Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Stream Processing in PNEs George Porter Edge Services Session Winter Retreat - 2004 Overview Streams prevalent in the edge network PNEs—general platform for packet processing (roughly layers 2-4) in the edge We need a general mechanism for stream processing (roughly layer 7) that is integrated with the current PNE design and architecture I am proposing an extension to PNEs to specify streams and a mechanism for executing that specification at high speeds in a general way from the PNE interface This mechanism provides a clear separation between the transport of packets and the structure of the protocol Packet-vs.-stream processing: what’s the difference? Packet Processing Ethernet, IP, and TCP headers in every packet Decisions are stateless between packets Resilient to loss, reorder, duplication Stream Processing Assumes reliable transport Sequence of variable length data units over TCP Simple for endhosts, since O/S delivers in order Loss, reorder, dups a problem for in-network elements PNEs must track data unit boundaries Ethernet IP TCP Ethernet … Data Ethernet IP TCP iSCSI1 Data Data1 IP TCP Data Ethernet IP TCP Data1 iSCSI2 Data2 Examples of observed phenomenon (observed on OASIS testbed) header1 data1 header2 data2 Original Stream: 1) 2) 3) (overlaps with first And last part of header) Need for stream processing Necessary to track any layer-7 protocol that exists on top of TCP Storage Web/P2P iSCSI: storage virtualization, load-balancing, security, caching, in-network optimizations HTTP pipelining: load-balancing P2P: HTTP often used as a transport protocol Measurement/Monitoring Tracking state of observed protocols over time PNE mechanism for tracking ADUs H H StreamTracker Packet flow (fast path) Packet(s) containing ADU boundary Priority Queue Computed ADU size FixedLogic (iscsi) Paramaterized by specification PNE mechanism for tracking ADUs StreamTracker Packet flow (fast path) Packet(s) containing ADU boundary Priority Queue Computed ADU size VariableLogic (http) Paramaterized by specification Evaluation Platform Deployed onto testbed consisting of 20 Pentium-500 Linux machines 100 Mbit ethernet-based network iSCSI source and target through Intel drivers 100MB ramdisk on target / 30gig IDE Seagate drive Code deployment Baseline: Click modular router running in Kernel “Tracking”: Click router in kernel with custom elements Results: #pkts examined Only a small number of packets are examined in detail: Workload Packets examined Total Packets Percent Unpack and compile Apache 2272 42850 5.30% Resize two ~80Kb images 15 266 5.64% Store/Read 20Mb MPEG file 595 37207 1.60% FS caching does not effect above measurements Results: overhead of packet examination Overhead of examining packets in depth is large, but infrequent: Workload Stock Click CPU cycles/pkt Tracking CPU Cycles/pkt Percent Increase iSCSI regression Test 1430 1871 30.8% iSCSI regression Test (1% loss) 1328 1937 45.9% Unpack and compile Apache 1591 2104 32.2% Next Steps Implement in PNE software artifact Write compiler for specification language Deployment (see poster for deployment overview) “VideoCollective”, “VideoCollective with sharing”, “VideoCollective with collaborative sharing” Measurement of storage statistics in above deployment