Download Slides

International Networking for e-VLBI
What is JIVE?
Operate the EVN correlator and
support astronomers doing
VLBI.
A collaboration of the major radioastronomical research facilities in
Europe, China and South Africa
A 3 year program to create a
distributed astronomical instrument of
inter-continental dimensions using eVLBI, connecting up to 16 radio
telescopes
Radio Astronomy
Courtesy of NRAO
Radio vs. Optical astronomy
The imaging accuracy (resolution) of a telescope:
θ ≈ 1.2 λ/D (λ = wavelength, D = diameter)
Hubble Space Telescope:
λ ≈ 600nm (visible light)
D = 2.4m
θ = 0.06 arcsecond
Onsola Space Observatory:
λ = 6cm (5GHz)
D = 25m
θ = 600 arcseconds
Wanted: 240km dish
Very Long Baseline Interferometry
• Create a huge radio telescope by using telescopes in
different locations around the world at the same time
• Resolution depends on
distance between dishes,
milli-arc second level
• Sensitivity on dish area,
time and bandwidth
• Requires atomic clock
stability for timing
• Processed in a special
purpose super-computer:
Correlator, 16x 1024Mb/s
Very Long Baseline Interferometry
• Initially (1990) we used
large single-reel tapes
• Then came
harddisk-packs
tekst
tekst
• And now: e-VLBI
Very Long Baseline Interferometry
Latency: 2 weeks
Latency: 150ms
“Never underestimate the bandwidth of a station
wagon laden with computer tapes hurtling down the
highway”
(Andy Tanenbaum)
Why e-VLBI
• Quick turn-around
• Rapid response
• Check data as it comes in, not weeks later
(You can’t redo just 1 telescope)
• More bandwidth
• Logistics (disks delayed/deleted/damaged/destroyed)
Example: Cyg X-3
• Star + black hole
• Flares irregularly
• Timescale: days
• Left: 2 weeks late
• May: Observed
flare with e-VLBI
International Year of Astronomy
Observation of J0204+15 (IYA)
Observation of J0204+15 (IYA)
Observation of J0204+15 (IYA)
Networking challenges
e-VLBI is:
• High Bandwidth: > 1 Gb/s
• Long Distance: Worldwide
• Real-time
• Long duration: 12 hours
• Can accept a little
packet loss
TCP Research
• Mirror port (span)
• eVLBI: RTT up to 375ms
• Window Size (kernel vers.)
• SACK-bugs
• TCP Tuning defeats fairness
• Conclusion:
• UDP
• ‘private’ connections
(LP, VLAN, dark fiber)
Lightpaths
• Dedicated point-to-point circuit
• Based on SDH/Sonet timeslots (NOT a lambda)
• Stitched together at cross-connects
• Guaranteed bandwidth
e-VLBI
• But also: a string of SPFs
Lightpaths
• Especially the longer lightpaths have many outages
• NRENs usually very good about announcing maint.
• A -lot- of email.
• e-VLBI is becoming a ‘target of opportunity’
instrument, planned and unplanned observations
Network Overview
Telescope
CC
Bandwidth
RTT (ms)
Sheshan
CN
622M LP / 512M R
354 / 180
ATNF
AU
1G LP
343
Kashima
JP
512M R
288
Arecibo
PR
512M VLAN
154
TIGO
CL
95M R
150
Westford
US
512M R
92
Yebes
ES
512M R
42,1
Torun
PL
1G LP / 10G R
34,9
Onsala
SE
1.5G VLAN
34,2
Metsahovi
FI
10G R
32,7
Medicina
IT
1G LP
29,7
Jodrell Bank
UK
2x 1G LP
18,6
Effelsberg
DE
10G VLAN
13,5
WSRT
NL
2x 1G CWDM
0,57
Network overview
JIVE Network Setup
The 1Gb/s speedbump
• VLBI (tape based) comes in fixed speeds, power of 2:
128Mb/s, 256Mb/s, 512Mb/s - and 1024Mb/s
• 1024Mb/s > 1Gb/s! (with headers it’s more like 1030)
• Dropping packets works but is sub-optimal
• Dropping ‘tracks’ to <1Gb/s:
Takes a LOT of CPU work
• Lightpaths come in ‘quanta’ of
150Mb/s, but Ethernet doesn’t
The Trouble with Trunking
• Standard trunking: LACP (802.3ad)
• Uses a hash of source/destination MAC, IP and/or Port to
choose outgoing port
• This is to prevent re-ordering
• A single TCP/UDP stream will use only 1 link member!
• Recent Linux kernels come with bonding, ‘ifenslave’
• Round Robin traffic distribution
• Keep both halves in separate
VLANS/Lightpaths as switches
in between only speak LACP
“Do NOT cross the streams!”
CWDM from WSRT to JIVE
Much cheaper than upgrading to 10Gb/s
All the colours of the rainbow...
SX: 850nm
1470nm
1610nm
1550nm
ZX: 1550nm
1510nm
LX: 1310nm
1570nm
... and then some.
Timing is everything
• Originating network interface often has more
bandwidth than the end-to-end network path
• Example: 1Gb/s interface, 622 Mb/s lightpath
• Trying to send 520 Mb/s for e-VLBI - should fit...
• Ethernet either sends at 1Gb/s, or 0 Gb/s
• Linux timer interrupt: 250Hz (2.6) or100Hz (2.4)
• Data will be sent as bursts: 4ms at 1Gb/s is 512kB
• Buffer space at choke-point is limited, so packets get
dropped when queue fills up
Timing is everything
• Linux timer interrupt, even at 1000Hz, is too slow
• Use a CPU to run a calibrated delay at full speed
• And mount a big cooling fan
• Good packet spacing - problem solved?
• A real-time problem requires a real-time kernel
• High-resolution nanosleep( ) in 2.6.17 and beyond
• Available by default in e.g. Debian Lenny (2.6.26)
• Sleeping instead of spinning
• CPU-core load from 99% to 5% - much greener!
Multicast - it’s not just for video
• MERLIN - Multi Element Radio Linked Interferometer
• 6 UK telescopes, connected at 128Mb/s
• Up to 4 recorded on one server
• Disk: copy the disks upon arrival
• e-VLBI: network to copy data
• Using a span/mirror port (ugly hack)
• Multicast 512Mb/s UDP stream
• Needs hardware multicast support in switch/router
• Now in regular production use: ‘Merlincast’
The Elliptical Robin
• Jodrell Bank ‘home’ telescope:
full 1024 Mb/s
• 2 lightpaths to JIVE, 1Gb/s each
• Use ethernet bonding (‘Round Robin’) - distribute
the packets over 2 links
• No room left over for 512 Mb/s Merlincast?
• 8 line changes to .../drivers/net/bonding/bond_main.c
• # modprobe bonding skew=4
• 5 packets on first interface, then 1 on the other
e-VLBI today
• ToO observation of Cygnus X-3 outburst
• 3 observation epochs (May 23rd , June 10th, July 4th)
• Observations at K-band (1cm, 22GHz)
• Telescopes from Australia, Japan and Europe
e-VLBI today
• ToO observation of Cygnus X-3 outburst
• 3 observation epochs (May 23rd , June 10th, July 4th)
• Observations at K-band (1cm, 22GHz)
• Telescopes from Australia, Japan and Europe
Future e-VLBI
• More bandwidth increases sensitivity (by √B)
• Current correlator limited to 1024Mb/s (per station)
• Researching software correlators, GPUs
• Researching FPGA based correlators (64 Gb/s /st)
• New telescope backends
• More telescopes increases image accuracy
• Current correlator limited to 16 stations
• N * (N - 1) / 2 baselines
• FPGA based design targetting 32 stations
Questions?