Download pdf

SoNIC: Precise Real1me So3ware Access and Control of Wired Networks Ki Suh Lee, Han Wang, Hakim Weatherspoon Cornell University 4/11/13 SoNIC NSDI 2013 1 Interpacket Delay and Network Research Applica1on •  Interpacket gap, spacing, arrival 1me, … IPG Transport Packet i Network Data Link Physical Packet i+1 IPD •  Important metric for network research –  Can be improved with access to the PHY Packet Genera1on Increasing Throughput Detec1ng 1ming channel Characteriza1on 4/11/13 SoNIC NSDI 2013 Packet Capture Es1ma1ng bandwidth 2 Network Research enlightened via the PHY Applica1on •  Valuable informa1on: Idle characters IPG Transport Network Data Link Physical 4/11/13 Packet i Packet i+1 IPD –  Can provide precise 1ming base for control •  Each bit is ~97 ps wide SoNIC NSDI 2013 3 Network Research enlightened via the PHY Applica1on 12 /I/s I=dle 100bits = 9.7ns •  Valuable informa1on: characters IPG Transport Network Data Link Physical Packet i – 
One Idle character (/I/) Can provide pbrecise 1ming base for control = 7~8 its •  Each bit is ~97 ps wide Packet Genera1on 4/11/13 Packet i+1 Detec1ng 1ming channel SoNIC NSDI 2013 Packet Capture 4 Principle #1: Precision Precise network measurements is enabled via access to the physical layer (and the idle characters and bits within interpacket gap) 4/11/13 SoNIC NSDI 2013 5 How to control the idle characters (bits)? Applica1on •  Access to the en1re stream is required IPG Transport Network Data Link Physical Packet i Packet i+1 •  Issue1: The PHY is simply a black box –  No interface from NIC or OS –  Valuable informa1on is invisible (discarded) Packet i Packet i Packet i+1 Packet i+1 •  Issue2: Limited access to hardware Packet i+2 Packet i+2 –  We are network systems researchers a.k.a. we like so3ware 4/11/13 SoNIC NSDI 2013 6 Principle #2: So3ware Network Systems researchers need so3ware access to the physical layer 4/11/13 SoNIC NSDI 2013 7 Precision + So3ware = Physics equipment??? •  BiFocals [IMC’10 Freedman, Marian, Lee, Birman, Weatherspoon, Xu] –  Enabled novel network research –  Precision + So3ware = Laser + Oscilloscope + Offline analysis –  Allowed precise control in so3ware •  Limita1ons –  Offline (not real'me) –  Limited Buffering –  Expensive 4/11/13 SoNIC NSDI 2013 8 Principle #3: Real1me Network systems researchers need access and control of the physical layer (interpacket gap) con1nuously in real1me 4/11/13 SoNIC NSDI 2013 9 Challenge Applica1on •  Goal: Control every bit in so-ware in real'me IPG Transport Network Data Link Physical Packet i Packet i+1 IPD –  Enable novel network research •  Challenge –  Requires unprecedented so3ware access to the PHY 4/11/13 SoNIC NSDI 2013 10 Outline •  Introduc1on •  SoNIC: So3ware-­‐defined Network Interface Card –  Background: 10GbE Network Stack –  Design •  Network Research Applica1ons •  Conclusion 4/11/13 SoNIC NSDI 2013 11 SoNIC: So3ware-­‐defined Network Interface Card Applica1on •  Implements the PHY in so3ware IPG Transport Network Data Link Physical Packet i Packet i+1 IPD –  Enabling control and access to every bit in real1me –  With commodity components –  Thus, enabling novel network research •  How? –  Backgrounds: 10 GbE Network stack –  Design and implementa1on 4/11/13 •  Hardware & So3ware •  Op1miza1ons SoNIC NSDI 2013 12 10GbE Network Stack Applica1on Data Transport Network Data Link Physical 64/66b PCS Encode Decode Scrambler Descrambler Gearbox Blocksync PMA Preamble Eth Hdr L3 Hdr Data L2 Hdr L3 Hdr Data L2 Hdr L3 Hdr Data characters (/I/) 64 bit Idle 2 b
it syncheader /S/ /D/ /D/ /D/ /D/ CRC Gap 10.3125 Gigabits /T/ /E/ 16 bit 011010010110100101101001011010010110100101101001011010010110100101101 PMD 4/11/13 SoNIC NSDI 2013 13 10GbE Network Stack Applica1on Data Transport Network SW Data Link Physical 64/66b PCS Encode Decode Scrambler Descrambler Gearbox Blocksync PMA PMD 4/11/13 L3 Hdr Data L2 Hdr L3 Hdr Data L2 Hdr L3 Hdr Data Packet i Preamble Eth Hdr Packet i+1 CRC Gap HW /S/ /D/ /D/ /D/ Packet i /D/ /T/ /E/ Packet i+1 011010010110100101101001011010010110100101101001011010010110100101101 Commodity NIC SoNIC NSDI 2013 14 10GbE Network Stack Applica1on Data Applica1on L3 Hdr Data Transport L2 Hdr L3 Hdr Data Network L2 Hdr L3 Hdr Data Data Link CRC SW Transport Network Data Link Preamble Physical 64/66b PCS Encode Decode SW Eth Hdr /S/ Packet i /D/ HW /D/ Packet /D/ i+1 Gap Physical 64/66b PCS Encode /T/ Decode /E/ /D/ Scrambler Descrambler Scrambler Descrambler Gearbox Blocksync Gearbox Blocksync PMA PMD 4/11/13 HW 011010010110100101101001011010010110100101101001011010010110100101101 PMA SoNIC NetFPGA SoNIC NSDI 2013 PMD 15 SoNIC Design Applica1on Data Transport Network Data Link Preamble Physical 64/66b PCS Encode Decode SW Scrambler Descrambler Gearbox Blocksync PMA PMD 4/11/13 /S/ Eth Hdr /D/ L3 Hdr Data L2 Hdr L3 Hdr Data L2 Hdr L3 Hdr Data /D/ /D/ /D/ CRC /T/ Gap /E/ HW 011010010110100101101001011010010110100101101001011010010110100101101 SoNIC SoNIC NSDI 2013 16 SoNIC Design and Architecture Applica1on Data Transport Network L2 Hdr Data Link Preamble Physical 64/66b PCS Encode Decode SW Scrambler Descrambler Gearbox Blocksync PMA PMD 4/11/13 /S/ Eth Hdr /D/ L3 Hdr APP Data Userspace L3 Hdr APP Data Kernel L2 Hdr TX M
L3 AC Hdr /D/ HW Data RX MAC CRC /D/ /D/ /T/ TX PCS RX PCS Gearbox Blocksync Gap /E/ Hardware 011010010110100101101001011010010110100101101001011010010110100101101 Transceiver Transceiver SoNIC SoNIC NSDI 2013 SFP+ 17 SoNIC Design: Hardware •  To deliver every bit from/to so3ware Applica1on Network –  High-­‐speed transceivers –  PCIe Gen2 (=32Gbps) Data Link •  Op1mized DMA engine Transport Physical 64/66b PCS Encode Decode SW Scrambler Descrambler Gearbox Blocksync PMA PMD 4/11/13 SFP+ SFP+ FPGA HW PCIeGen2 SoNIC NSDI 2013 18 SoNIC Design: So3ware Applica1on Port 0 Port 1 Transport APP APP Network TX MAC RX MAC TX MAC RX MAC TX PCS RX PCS TX PCS RX PCS Data Link Physical 64/66b PCS Encode Decode SW Scrambler Descrambler Gearbox Blocksync PMA PMD 4/11/13 •  Dedicated Kernel Threads HW –  TX / RX PCS, TX / RX MAC threads –  APP thread: Interface to userspace Packet i SoNIC NSDI 2013 Packet i+1 19 SoNIC Design: Synchroniza1on Applica1on Port 0 Transport APP Network Low-­‐latency Port 1 FIFOs APP TX MAC RX MAC TX MAC RX MAC TX PCS RX PCS TX PCS RX PCS Data Link Physical 64/66b PCS Encode Decode SW Scrambler Descrambler Gearbox Blocksync PMA PMD 4/11/13 SFP+ SFP+ FPGA Pointer-­‐polling No Interrupts HW PCIeGen2 SoNIC NSDI 2013 20 SoNIC Design: Op1miza1ons G( x) = x58 + x39 +1
Applica1on •  Scrambler Transport Naïve Implementa1on Op1mized Implementa1on Network s ç state d ç data for i = 0 è63 do in ç (d >> i) & 1 out ç (in ⊕
(s >> 38) ⊕
(s >> 57))&1 s ç (s << 1) | out r ç r | (out << i) state ç s end for s ç state d ç data r ç (s >> 6) ⊕
(s >> 25) ⊕
d r ç r ⊕
(r << 39) ⊕
(r << 58) state ç r 0.436 Gbps 21 Gbps Data Link Physical 64/66b PCS Encode Decode Scrambler Descrambler Gearbox Blocksync PMA PMD 4/11/13 •  CRC computa1on •  DMA engine SoNIC NSDI 2013 21 SoNIC Design: Interface and Control •  Hardware control: ioctl syscall •  I/O : character device interface •  Sample C code for packet genera1on and capture 1: #include "sonic.h" 2: 3: struct sonic_pkt_gen_info info = { 4: .mode = 0, 5: .pkt_num = 1000000000UL, 6: .pkt_len = 1518, 7: .mac_src = "00:11:22:33:44:55", 8: .mac_dst = "aa:bb:cc:dd:ee:ff", 9: .ip_src = "192.168.0.1", 10: .ip_dst = "192.168.0.2", 11: .port_src = 5000, 12: .port_dst = 5000, 13: .idle = 12, 14: }; 15: 16: /* OPEN DEVICE*/ 17: fd1 = open(SONIC_CONTROL_PATH, O_RDWR); 18: fd2 = open(SONIC_PORT1_PATH, O_RDONLY); 4/11/13 19: /* CONFIG SONIC CARD FOR PACKET GEN*/ 20: ioctl(fd1, SONIC_IOC_RESET) 21: ioctl(fd1, SONIC_IOC_SET_MODE, PKT_GEN_CAP) 22: ioctl(fd1, SONIC_IOC_PORT0_INFO_SET, &info) 23 24: /* START EXPERIMENT*/ 25: ioctl(fd1, SONIC_IOC_START) 26: // wait 1ll experiment finishes 27: ioctl(fd1, SONIC_IOC_STOP) 28: 29: /* CAPTURE PACKET */ 30: while ((ret = read(fd2, buf, 65536)) > 0) { 31: // process data 32: } 33: 34: close(fd1); 35: close(fd2); SoNIC 22 Outline •  Introduc1on •  SoNIC: So3ware-­‐defined Network Interface Card •  Network Research Applica1ons –  Packet Genera1on –  Packet Capture –  Covert 1ming channel •  Conclusion 4/11/13 SoNIC NSDI 2013 23 Network Research Applica1ons Applica1on •  Interpacket delays and gaps IPG Transport Network Data Link Packet i Packet i+1 IPD Physical Packet Genera1on 4/11/13 Detec1ng 1ming channel SoNIC NSDI 2013 Packet Capture 24 Packet Genera1on and Capture •  Basic func1ons for network research –  Genera1on: SoNIC allows control of IPGs in # of /I/s –  Capture: SoNIC captures what was sent with IPGs in bits APP APP TX MAC RX MAC TX MAC RX MAC TX PCS RX PCS TX PCS RX PCS 1518B 1518B 1518B 1518B 1518B 9Gbps, IPD =13992 bits (1357ns) 4/11/13 SoNIC NSDI 2013 25 Packet Genera1on •  SoNIC allows precise of IPGs CDF ocf ontrol generated IPDs 1"
SoNIC"
CDF 0.8"
0.6"
APP TX MAC RX MAC TX PCS RX PCS 0.4"
0.2"
0"
1000"
1518B Specialized NIC Higher variance Sniffer"10G"
APP SoNIC Zero variance!!! 1500"
2000"
2500"
Interpacket delays (ns) 1518B 1518B 1518B TX MAC RX MAC TX PCS RX PCS 3000"
1518B 9Gbps, IPD =13992 bits (1357ns) 4/11/13 SoNIC NSDI 2013 26 Packet Capture •  SoNIC captures wCDF hat ois f scaptured ent IPDs 1"
CDF 0.8"
SoNIC"
Kernel"
Userspace"
Sniffer"10G"
0.6"
APP TX MAC RX MAC TX PCS RX PCS 0.4"
0.2"
APP TX MAC RX MAC TX PCS RX PCS 0"
0"
1518B 1000"
2000"
3000"
4000"
Interpacket delays (ns) 1518B 1518B 1518B 5000"
1518B 9Gbps, IPD =13992 bits (1357ns) 4/11/13 SoNIC NSDI 2013 27 Covert Timing Channel •  Embedding signals into interpacket gaps. –  Large gap: ‘1’ –  Small gap: ‘0’ Packet i Packet i Packet i+1 Packet i+1 •  Covert 1ming channel by modula1ng IPGs at 100ns APP TX MAC RX MAC TX PCS RX PCS 4/11/13 •  Overt channel at 3 Gbps •  Covert channel at 250 kbps •  Over 4-­‐hops with < 1% BER SoNIC NSDI 2013 APP TX MAC RX MAC TX PCS RX PCS 28 Covert Timing Channel •  Modula'ng IPGS at 100ns scale (=128 /I/s) 1"
0.6"
APP 3562 /I/s 3562 -­‐ 128 /I/s APP 0.4"
TX MAC RX MAC TX PCS RX PCS 0.2"
0"
500"
‘1’ ‘0’ 1500"
2500"
3500"
Interpacket delays (ns) ‘1’: 3562 + 128 /I/s ‘0’: 3562 – 128 /I/s 4/11/13 3562 + 128 /I/s BER = 0.37% CDF 0.8"
SoNIC"
Kernel"
TX MAC RX MAC TX PCS RX PCS 4500"
‘1’: 3562 + a /I/s ‘0’: 3562 – a /I/s SoNIC NSDI 2013 29 Contribu1ons •  Network Research –  Unprecedented access to the PHY with commodity hardware –  A pla•orm for cross-­‐network-­‐layer research –  Can improve network research applica1ons •  Engineering –  Precise control of interpacket gaps (delays) –  Design and implementa1on of the PHY in so3ware –  Novel scalable hardware design –  Op1miza1ons / Parallelism •  Status –  Measurements in large scale: DCN, GENI, 40 GbE 4/11/13 SoNIC NSDI 2013 30 Conclusion •  Precise Real1me So3ware Access to the PHY •  Commodity components –  An FPGA development board, Intel architecture •  Network applica1ons –  Network measurements –  Network characteriza1on –  Network steganography •  Webpage: h‚p://sonic.cs.cornell.edu –  SoNIC is available Open Source. 4/11/13 SoNIC NSDI 2013 31 Thank you Demo tonight! h‚p://sonic.cs.cornell.edu