Download Gigascope A stream database for network monitoring

Gigascope
A stream database for network
monitoring
Chuck Cranor
Yuan Gao
Theodore Johnson
Vladislav Shkapenyuk
Oliver Spatscheck
AT&T Labs - Research
Fast and Flexible Network Monitoring
• High speed
– Monitor Gigabit Ethernet with a low loss rate
• Flexible
– SQL-like language to express monitoring queries
– Simple C-language interface for applications
– Perl interface
• Applications
–
–
–
–
Network debugging
Protocol debugging
Network security
Ad-hoc applications
Architecture
• HFTA : high level query node
• LFTA : low level query node
• Clearing house : data stream manager
HFTAs and applications subscribe
to data streams at the clearing house.
HFTAs also register data streams.
C/C++
Appl’n
HFTA
C/C++ and Perl
host libraries
HFTA
Clearing house
The clearing house manages data
streams and registers the queries
and schemas of data stream producers
LFTAs
FTA
registry
LFTAs can run either in the
clearing house, or in the
Network Interface Card (NIC)
supported by a NIC RTS
Perl
Appl’n
PCAP
library
Standard device driver
NIC
Firmware
interface
G’scope device driver
LFTAs
NIC
NIC RTS
Query Language
• Gigascope queries are written in GSQL
– Similar to SQL
– Support for stream database queries
• Stream fields can have ordering properties
• Deduce when aggregates are closed and can be flushed to the output
stream
– Currently limited to selection and aggregation
• Stream merge and stream join in the works.
Query traffic_count.gsql:
Select timebucket, sourceIP, destIP,
source_port,dest_port,
SUM(length)
From TCP
Where protocol=6 and
(source_port=80 or dest_port=80)
Group by time/5 as timebucket,
sourceIP, destIP,
source_port,dest_port
Schema of output stream:
STREAM traffic_count {
UINT timebucket ( INCREASING ) ;
UINT sourceIP
UINT destIP ;
UINT source_port ;
UINT dest_port ;
UINT SUM_length ;
}
Query Architecture
• GSQL queries are translated into C or C++ code
– LFTAs : translated into C code, interface with a Run Time System
(RTS)
– HFTAs : translated into C++ code, using templatized push-based
operators
• Self-documenting executables
– Generated code contains the defining query and the schema of the
output stream
• GSQL queries can read from a network packet stream, or
from the output of a GSQL query
– Queries that read packets from the network become LFTAs
• LFTA queries are tightly resource constrained
– Intended for execution in the kernel or the NIC
– Gigascope automatically splits queries into an LFTA and an HFTA
Query Splitting
Select timebucket, sourceIP, destIP,
source_port,dest_port, SUM(length)
From TCP
Where protocol=6 and
(source_port=80 or dest_port=80)
Group by time/5 as timebucket,
sourceIP, destIP, source_port,dest_port
LFTA query:
DEFINE{
query_name _fta_trafficcnt
}
Select timebucket, sourceIP, destIP,
source_port,dest_port,
SUM(length)
From TCP
Where protocol=6 and
(source_port=80 or dest_port=80)
Group by time/5 as timebucket,
sourceIP, destIP,
source_port,dest_port
HFTA query:
Select timebucket, sourceIP, destIP,
source_port,dest_port,
SUM(SUM_length)
From _fta_trafficcnt
Group by timebucket,
sourceIP, destIP,
source_port,dest_port
Performance
• Goal : Simple and rapid application development while
increasing performance.
• Experiment : measure packet loss rate with different levels
of traffic
– Gigabit Ethernet network
– 2% loss rate is acceptable
– Application : measure the volume of HTTP1.0 and HTTP1.1
traffic using port 80
• Four approaches
–
–
–
–
Dump all data to disk
Monitor network using libpcap, but do no processing
Gigascope using libpcap
Gigascope running queries on the Gigabit Ethernet NIC
Performance of Monitoring Alternatives
12
Dump to disk (libpcap)
G'scope on libpcap
Do nothing (libpcap)
G'scope on NIC
Loss rate (Percent)
10
8
6
4
2
0
0
100
200
300
400
Offered data rate (Mbit/sec)
500
600
700