Download InGrid 2008 presentation - Joint Institute for VLBI in Europe

iBOB Data Acquisition
and Network Streaming
Jan Wagner et al.
TKK / Metsähovi Radio Observatory
7th International eVLBI Workshop
Shanghai, 2008
Introduction
Current e-VLBI Data Acquisition
•
•
Telescope data generated by Data Acquisition (DAQ) hardware
e-VLBI : telescope data is sent via networks to correlator
a) capture DAQ data locally to RAID disk array, send afterwards
b) send data in real-time, capture and store or real-time process remotely
•
Current DAQ situation in the EVN:
•
•
•
•
•
Receiver and RF front-end
VLBA rack downconverters
Mark4-style A/D digitizers
max theoretical rate
max rate in practice
: »512 MHz RF bandwidth
: VCs & BBCs, ≤8 MHz base-band / ch
: max 32 Ms/s/channel 2-bit → 64 Mb/s / ch
: 16 x 2-ch BBCs → 2 Gbit/s (512 MHz RF)
: formatting, Mark5 disk&NIC → ≤ 1 Gbit/s
=> Stuck at 256/512 MHz, 1 Gbit/s?
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai
Slide #2
Introduction
Towards More DAQ Bandwidth
• Today’s networks and PC systems easily support over 1G rates
=> just a matter of providing antenna data samples faster
=> need a DAQ with more bandwidth
• More RF bandwidth is beneficial for astro VLBI
• Correlation/synthesis yields better
sensitivity, noise: improved images
• Observation time per object could be
reduced, more objects per session
• More bandwidth from bit reso allows
AGC, band equalizer, RFI mitigation, …
• To improve on the current DAQ:
C. Phillips, e-VLBI Scientific Benefits,
ATNF CSIRO memo, 2004
• Input bandwidth: off-the shelf gigasample A/D converter(s)
• Processing: keep as wideband for astro, fine channelization for geo(?)
• Output interface: copper Ethernet, UDP packet streams
• For Metsähovi DAQ development and tests, used iBOB platform
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai
Slide #3
iBOB DAQ Description
iBOB DAQ Hardware Platform
• UC Berkeley’s iBOB FPGA board
• Virtex II Pro 2vp50 with PPC CPUs
• two copper 10GbE ports (CX4)
• two expansion ports (Z-DOK)
• Berkeley iADC board
• Atmel AT84AD001
• dual-channel, 8-bit, 1 Gs/s/ch
=> 1 GHz MHz BW, 16 Gbit/s / board
• Nat LMX2531 or ADI ADF4360
• ~950..1080 MHz PLL for ADC fs
• Similar board ROACH (meerKAT)
available “soon”, has 4 x 10GbE, iBOB designs should drop in
• For iBOB, created several DAQ “firmware” test designs
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai
Slide #4
iBOB DAQ Description
iBOB IF-to-Ethernet Design Example
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai
Slide #5
UDP Packet Format
iBOB DAQ and Protocol for 10GbE
• Real-time e-VLBI: data integrity less important than timely send
• data loss factors f≤10% tolerable, image degrades ~ 1  f
<=> packet loss and reorder ok, payload integrity non-critical
<=> don’t bother to retransmit lost packets
<=> UDP checksums unnecessary: frame CRC32->2·10-9*BER
• congestion control: none - or client-specified sample mode changes
• The only essential points – implemented in Test-DAQ UDP
• must know which RF band/channel a sample comes from:
• sample series from same channel placed into same UDP packet
• for <60k channels can just send channel X to IPv4/6 port# Y
• samples must be time stamped:
• trigger acquisition at 1PPS, give packets a 64-bit packet sequence#
• assume 4kB UDP => ambiguity after 4.5e15 samples : 245 years @ 10Ts/s
• rest is “constant” a priori info, doesn’t belong into the stream (YMMV)
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai
Slide #6
iBOB Streaming Tests
iBOB DAQ Tests
• Local 10G LAN tests
•
•
•
•
12 June 2008
Analog signal sampled with iBOB, streamed as UDP to 10G LAN
UDP stream recording with ‘udp2raid’, ‘raw2raid’ (~tcpdump, gulp)
udp2raid’ed to RAID-0 at up to 4 Gbps (current RAID ~4.3G, XFS)
Data, spectrum checks with Matlab and swspectrometer
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai
Slide #7
iBOB Streaming Tests
Mh-On Streaming/Network Tests
•
First tests on 14/15th May
•
•
•
•
•
•
•
Supplied Onsala with test firmware
100 MHz clock, 1024 Ms/s sampling
I/Q: VLBI IF 90..540 MHz, -10dBm
On streamed to Mh RAID-PC
Mh PC busy with other computations…
Recorded 1.6, rest with slow-PC loss
Checked for and found PCAL
Second test 11th June
•
•
used ”adc2tsunami”, proto/test design
sent 2 channels (I/Q) to On
IBOB % regwrite dest_ip_ich 0x82F20A05
IBOB % regwrite dest_port_ich 46330
IBOB % regwrite dest_ip_qch 0x82F20A05
IBOB % regwrite dest_port_qch 46331
IBOB % regwrite teng_wordsperframe 512
IBOB % regwrite teng_decfactor 0
12 June 2008
3.2G, (6.4G), 1.6 Gbps; 2 channels
© NORDUnet (http://www.nordu.net/stats/)
# 130.242.10.5, Onsala
# I-channel to port 46330
#…
#…
# 512x8 : 4kB UDP
# sample rate reduction 1/(1+n)
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai
Slide #8
iBOB Streaming Tests
Mh-On Streaming Test - Route
© NORDUnet (http://www.nordu.net/stats/)
Funet→Nordunet→Sunet
Mh
On
12 June 2008
© OptoSUNET (http://stats.sunet.se/)
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai
Slide #9
iBOB Streaming Tests
Mh-On Streaming Test - Traffic
2G, 4G, 7.8G
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai
Slide #10
iBOB Streaming Tests
Mh-On Streaming Test - Errors
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai
Slide #11
iBOB Streaming Tests
Mh-On Streaming Test - Comments
•
Borje Josefsson (Sunet chief tech):
“AFAIK the tests started at 10AM Swedish time, why can't I ever go and
have a beer at that time :-)”
•
•
•
BJ made Internet Land Speed record in 2004,
4.3 Gbps over 42 hops of shared networks from commercial ISP
http://proj.sunet.se/LSR3-s/
Will upgrade Sunet-Nordunet to 40G this week
Roy Molini (Funet/CSC):
“It is definitely exciting to see data streaming from Finland at such
incredible rates across infrastructures built specifically for meeting just
those demands. The idea of Research Networking is clearly shown.”
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai
Slide #12
Conclusion
Conclusions
•
Basic Mh iBOB designs allow to
• digitize antenna signal, different rates
• stream as UDP to local or remote IP
•
On->Mh RAID recording was lossless “only” at 1.6 Gbps
• try again when PC really free - local tests give 4G to disk
Mh->On network streaming
• No problems with 0 – 7.8 Gbps traffic
• Credit goes to Nordunet, Sunet, Funet: it is straightforward
and welcome to stream multi-Gbps over Internet/RN
•
•
Future test or demo: stream and record a maser source
with 2-bit / 8-bit, 2 x 512 MHz BW, analyze in swspectrometer
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai
Slide #13
Conclusion
Questions/Answers
•
Contact information
Jan Wagner
M.Sc./Researcher
EXPReS (Metsähovi)
[email protected]
•
Additional Information
http://expres-eu.org/
[note: only one “s”]
http://www.metsahovi.fi/en/vlbi/
•
EXPReS is made possible through the support of the European
Commission (DG-INFSO), Sixth Framework Programme,
Contract #026642
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai
Slide #14
Conclusion
Refs
•
Comparison of two frequency synthesizers: LMX2531 and ADF4360
http://www.metsahovi.fi/en/vlbi/ibob/Comparing-ADI-NAT.pdf
•
Multiplier-Free Band-Selectable Digital Filters
Arnaud Santraine, Sébastien Leprince, Fred Taylor,
ICASSP 2004
•
Programmable, multiplier free channelizer
Arnaud Santraine, Sébastien Leprince
12 June 2008
iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, Shanghai
Slide #15