* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download IP Traffic Measurement
Survey
Document related concepts
Multiprotocol Label Switching wikipedia , lookup
Internet protocol suite wikipedia , lookup
Piggybacking (Internet access) wikipedia , lookup
Zero-configuration networking wikipedia , lookup
Distributed firewall wikipedia , lookup
Asynchronous Transfer Mode wikipedia , lookup
Computer network wikipedia , lookup
Recursive InterNetwork Architecture (RINA) wikipedia , lookup
List of wireless community networks by region wikipedia , lookup
Wake-on-LAN wikipedia , lookup
Cracking of wireless networks wikipedia , lookup
Airborne Networking wikipedia , lookup
Deep packet inspection wikipedia , lookup
Network tap wikipedia , lookup
Transcript
IP Traffic Measurement: Technologies, Tools, and Protocols Jürgen Quittek NEC Europe Ltd., Network Laboratories, Heidelberg, Germany [email protected] Outline • Applications requiring traffic measurement • General traffic measurement process • Tools • Protocols and Standards © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 2 Applications (1) Requiring Traffic Flow Measurement • Usage-based accounting – input to charging and billing – various business model • time-based, volume-based, QoS class-based • per application, per user, per user group • Traffic engineering – optimizing network usage – traffic analysis on congested links • origin of traffic • type of traffic • dynamic behavior (bursty, adaptive, …) • Traffic profiling © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 3 Applications (2) Requiring Traffic Flow Measurement • QoS monitoring – (passive) measurement of QoS properties – validating Service Level Agreements • Attack detection and analysis – detecting (high volume) traffic patterns – investigation of origin of attacks • Intrusion detection – detecting unexpected or illegal packets • … © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 4 The Traffic Measurement Process Optional: traffic generation Conversion Integrate Classification & Flow Recording into TE, attack detect., QoS monitoring, accounting, ... Transport Sampling Packet Capturing Filtering (FlowScan) Store (TCPdump) Display PAYLOAD HEAD PAYLOAD HEAD PAYLOAD HEAD PAYLOAD HEAD (Ethereal) … other … Observation Point © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg Visualize 5 IP Flow Definition • “A flow is a set of packets with a set of common packet properties.” • Application level flow versus flow monitored at a single observation point – between endpoints <--> at one or more obs. points – using same path <--> using different paths – end-to-end packets only <--> also dropped packets • Uni-directional <--> bi-directional • typical case: separation by 5-tuple – IP addresses, transport type, port numbers © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 6 Observation Points sender receiver probe • Shared Media – shared wire Ethernet/Token Ring: OK – Ethernet with HUB: OK – Ethernet with switch: only broadcasts – Radio networks: not reliable • Point-to-point sender – Capturing only on end points or with splitter © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg receiver probe 7 Packet Capturing at Routers • Capturing on central CPU line card – observation point is complete router – typically SW solution – not scalable line card line card … • Capturing on line card – restricted observation point – typically hardware support – scalable © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 8 CPU Packet Capturing and Filtering Technology: PCAP • Library libpcap available on almost all Unix systems – creates copies of packets (up to a specified offset) in kernel spaces – delivers copies to user space by callback functions – includes kernel space packet filter BPF (Berkeley Packet Filter) – filter specified by user, compiled by libpcap, transferred into kernel – commonly used: TCPdump, NeTraMet, … – native in BSD systems – Linux, AIX, Solaris, HP-UX have compatible kernel-level and/or user-level implementations • sometimes with restricted functionality • For probe: network interface card in promiscuous mode © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 9 Packet Capturing, Flow Recording and Transport Technology: NetFlow • • • • • • Developed by Cisco De-facto standard Available for (almost) all Cisco & Juniper router products Dedicated probes available Implementations on central CPU or line card Packet capturing and flow recording with hardware support on line cards • Measures all 5-tuple flows at a line card or at the entire router • Exports flow records using NetFlow protocol: simple records sent over UDP • Supported by a huge variety of tools receiving NetFlow records © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 10 CAIDA Tools • Developed and supported by CAIDA at University of California at San Diego: http://www.caida.org/tools/ – – – – – – cflowd RTG skitter NeTraMet CoralReef Beluga © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 11 CAIDA Tools (2) • cflowd – flow analysis tool currently used for analyzing NetFlow records – collections, storage, and basic analysis modules – data collection and analysis for capacity planning, trends analysis, and characterization of workloads • CoralReef – software suite collecting and analyzing data from passive Internet traffic monitors – in real time or from trace files – Realtime monitoring via • libpcap • high-speed fiber network interface cards © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 12 CAIDA Tools (3) • NeTraMet – open-source implementation of the IETF RTFM architecture for Network Traffic Flow Measurement • RTG – flexible, scalable, high-performance SNMP statistics monitoring system. – collects time-series SNMP data from a large number of targets quickly. – uses data base – includes utilities that generate configuration and target files, traffic reports, 95th percentile reports and graphical data plots (supporting web-based interfaces). © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 13 CAIDA Tools (4) • skitter – actively probing the Internet in order to analyze topology and performance. • measures forward IP paths hop by hop • measures round trip time (RTT) • visualizes network connectivity • Beluga – provides a real-time graph of RTTs and packet loss to an end host – total round trip time and per-hop round trip time © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 14 More Tools • See a long list of (NetFlow-related) tools at – http://www.switch.ch/tf-tant/floma/software.html • FlowScan – analysis and nice graphical reporting of NetFlow input – http://net.doit.wisc.edu/~plonka/FlowScan/ • National Internet Measurement Infrastructure (NIMI) – http://ncne.nlanr.net/nimi/ • ntop – shows current network usage (like Unix ‘top’ program – http://www.ntop.org/ntop.html © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 15 Transport of Flow Records • Requires inter-operation between sender and receiver • Standardization desirable – de-facto standard NetFlow has some problems • IETF Standards – RTFM (Meter MIB) • Real-Time Flow Measurement – IPFIX (in progress) • IP Flow Information eXport – PSAMP (in progress) • Packet Sampling © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 16 IPFIX Scope and General Requirements • Goal: Find or develop a basic common IP Traffic Flow measurement technology to be available on (almost) all future routers • Fulfilling requirements of many applications • Low hardware/software costs • Simple and scalable • Metering to be integrated in general purpose IP routers and other devices (probes, middleboxes) • Data processing to be integrated into various applications • Interoperability by openness or standardization © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 17 IPFIX Requirements (1) • Distinguishing flows by 5-tuple – – – – IP addresses, transport type, port numbers Supporting MPLS, DiffServ Going on to more flexible flow definitions Flexible aggregation of flows • Metering Process – – – – Reliability Timestamps, time synchronization Flow timeouts Overload behavior • sampling, simplifying, stopping © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 18 IPFIX Requirements (2) • Data Export – Information model • many header fields and statistics required • anonymization? – Data model • flexible, extensible – Data Transfer • • • • • • reliability security congestion awareness push and pull model reporting? regular reporting interval notification on specific events • Configuration © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 19 IPFIX Architecture Overview Flow Information Export Exporter Probe Collector Flow Record (meter) PAYLOAD HEAD PAYLOAD HEAD PAYLOAD HEAD PAYLOAD HEAD PAYLOAD HEAD PAYLOAD HEAD PAYLOAD HEAD PAYLOAD HEAD Observation Point © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 20 Application IPFIX Scenarios Probe Simple Router Complex Router Multiple Exporters E M O E M OOO E E E M M OOO OOO Protocol Converter E (Meter MIB) M O © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg E M EO M E O M O M M OOO OOO Concentrator Proxy C M E C E 21 … Current State of IPFIX Standardization • Requirement specification complete • Protocol Selection in progress – no new protocol development – selection of an already existing protocol or of a protocol contributed externally • Elaboration / improvement of selected protocol will be last step before standardizing it © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 22 Existing Technologies • IETF standards – RTFM – RMON, RMON2 • Proprietary technologies – NetFlow (Cisco) – sFlow (InMon) – LFAP (Riverstone) – Crane (XACCT) –… © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 23 Real-Time Flow Measurement (RTFM) • Very flexible and powerful meter Application – programmable rule sets Manager – can serve several readers – programmable overload behavior • Reader polls meter • Realization by SNMP Meter MIB • Free software implementation NeTraMet • No acceptance at manufacturers • Complicated to use (too powerful) • Specified by RFCs 2720 - 2724 © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 24 Meter Reader Remote Network Monitoring MIB • Very flexible and powerful • Serves more general goals (analysis on layers 2-4) • Just a monitoring tool, no measurement architecture defined • Suited for very specific analysis tasks • High (hardware) performance requirements • Too complicated and too expensive for massive usage in routers • Specified by RFCs 2021(RMON2), 2613, 2819(RMON), 2895, 2896, 3144 © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 25 NetFlow Application • • • • • • Proprietary by Cisco, but de-facto standard Fast and efficient, implemented for IOS Data Configurable measurement per 5-tuple collector Unreliable (measurement & data transport) Hardware-supported on some models Meter Not well documented – re-engineered by Juniper • Versions 1-7 – fixed data model • Version 9 (under development) – data model templates – optional reliable transport © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 26 Router sFlow Application • • • • • • • By InMon Corporation Includes metering and data transmission Data Probabilistic sampling at meter collector Packet sampling and counter sampling Timestamping by data collector sMon Meter Configuration by sFlow MIB Poorly documented by informational RFC 3176 • Not adapted yet by other vendors © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 27 LFAP • • • • • • • • • • Application Light-weight Flow Accounting Protocol Proprietary by Riverstone (Cabletron) Just data transfer protocol FAS Meter at Connection Control Entity (CCE) communicates to Flow Accounting Server (FAS) Tight and reliable interaction CCE between CCE and FAS Reliable data transport Flexible TLV coding of transferred data Larger overhead than NetFlow More cost-intensive at meter/CCE and at data collector/FAS See <draft-riverstone-lfap-00.txt> © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 28 CRANE • Common Reliable Accounting for Network Element (CRANE) Protocol • Proprietary by XACCT • Just data transfer protocol • Template-based data model • Focus on reliability • Not yet in extensive commercial use • See <draft-kzhang-crane-protocol-02.txt> © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 29 IETF PSAMP Working Group • Established in Summer 2002 • Focus on sampling and capturing packets and on transferring them to data collectors • Target applications – traffic profiling – monitoring network behavior • Closely related to IPFIX • Defines packet sampling with much more detail – developing packet filtering and sampling information model – includes standardization of meter configuration • Hot Issue: (partial) export of payload © NEC Europe Ltd., 2002 Network Laboratories, Heidelberg 30