Download ppt

Traffic Instrumentation and
Management
CSG, January 2002
Traffic Instrumentation
✦
What are you looking for?
✧
✧
How’s the bandwidth being spent?
Locate anomalies
✧
✧
✦
Intrusions
Outgoing Denial of Service (DoS) attacks
Where should you look?
✧
Gateway routers get us most of what we want
Solution: Network Logs
✦
Network logs let you analyze past events
✧
✧
Log specific information: source, dest, time,
amount of traffic, etc.
Packet contents are overkill
✧
✧
✦
Privacy issues
Disk space
Do you need to log all connections?
✧
✧
Doing so allows forensics
Our disk space usage: 100GB gives 3 months
Network Forensics
✦
What happened on the network?
✦ Three Examples
✧
✧
✧
Who’s launching a DoS?
Where’s the network bandwidth gone?
A compromised machine
✧
How was it compromised? When? Where?
Outgoing DoS
✦
DoSes are generally spoofed
✧
✧
✧
Network logs aren’t too useful
Egress filtering helps, but the DoS tools figure
out how much spoofing they can safely do
(spoof from the same class C)
Blocked spoof attacks can flood network logs
Graphs for identifying DoS
✦
✦
Graphs are useful as DoS attacks stand out
How much can you graph?
✧
✧
Graphing each network port may be impractical
Other traffic may interfere
DoS Identification
✦
If you catch the DoS while it’s occurring,
you can check the current bandwidth
usage on the switches
✧
✦
“show top pkts”
If it’s spoofed, and you don’t catch it while
it’s happening… now what?
Where’s the Bandwidth Gone?
The “Napster” question
✦ Use statistical analysis
✦
✧
Which udp/tcp ports and/or ip addresses are
using a lot of bandwidth at times of high
bandwidth?
Is it an abuser?
✦
Is one machine using more than their fair share
of bandwidth?
✧
✧
Look at the top ten bandwidth users
Maybe… most of the IPs are of known high
bandwidth services (usenet, ftp, backup)
% flow-stat -f11 < ft-v06.2002-01-06.140000 | sort -nr +2 -3 | head -10
# IPaddr
flows
octets
packets
128.135.137.92 9543
2477490152
2897200
224.2.177.155
160
2337752653
4446397
128.135.136.147 1258
1947979335
2123565
128.135.108.92 4775
1676599523
2105520
128.135.12.170 1391
1510492347
2570530
128.135.147.43 6765
1079172157
1396155
198.49.215.223 16
868834755
979761
128.135.221.135 1610
848575034
866508
128.135.112.72 3855
829361150
940891
66.27.181.42
43
807246316
876126
Is it a specific program
✦
File sharing is high
✧
✦
✦
KaZaA (port 1214) and eDonkey (4662)
http is high (no surprise)
Port 55524 only has a few flows.
✧
✧
Probably a few large file transfers
Flow-extract shows us that it is multicast traffic
% flow-stat -f7 < ft-v06.2002-01-06.140000 | sort -nr +2 -3 | head -10
# port
flows
octets
packets
1214
232837
16971643884
20705354
80
1696461
10397156269
21971307
4662
14292
2652388190
3526641
55524
83
2313245819
4410164
119
1503
1571612208
2510833
6346
86042
1067187821
3034293
1737
787
809319373
882134
6348
1695
799340259
1403576
1156
1592
715081006
754911
47087
10
678618965
691006
The Compromise
✦
willard.uchicago.edu compromsed
✦ We know the approximate time of the
compromise: the morning of December
18th.
✦ We want to know what else they got into
and how they got in.
Logs, Part 1
✦
✦
✦
✦
Look for connections to machine at right time
Compromise was via ssh
ftp’d to a home.com address
Weird connections to port 40911
% flow-extract -d willard.uchicago.edu.ft-v06 -e ' since 2001-12-18 00:00 { print }'
12/18/2001 04:59:42 -> 12/18/2001 04:59:42
12/18/2001 04:59:42 -> 12/18/2001 04:59:42
12/18/2001 04:59:42 -> 12/18/2001 04:59:43
12/18/2001 04:59:42 -> 12/18/2001 04:59:43
12/18/2001 04:59:42 -> 12/18/2001 04:59:42
12/18/2001 04:59:42 -> 12/18/2001 04:59:43
12/18/2001 04:59:42 -> 12/18/2001 04:59:42
6
6
6
6
6
6
6
12/18/2001 05:03:57 -> 12/18/2001 05:03:58
12/18/2001 05:03:58 -> 12/18/2001 05:03:58
12/18/2001 05:03:57 -> 12/18/2001 05:03:58
12/18/2001 05:05:46 -> 12/18/2001 05:05:46
12/18/2001 05:05:49 -> 12/18/2001 05:06:11
12/18/2001 05:05:46 -> 12/18/2001 05:06:12
6
6
6
6
6
6
02 willard.uchicago.edu 22 <-> 101 host230.avlogic.com 4658 2
100
00 -SR-A11 willard.uchicago.edu 22 <-> 01 host230.avlogic.com 4658 2
100
00 -SR-A11 willard.uchicago.edu 22 <-> 01 host230.avlogic.com 4658 6
543
10 F-RPA02 willard.uchicago.edu 22 <-> 101 host230.avlogic.com 4658 6
543
10 F-RPA103 host230.avlogic.com 4658 <-> 02 willard.uchicago.edu 22 5
296
00 FS-PA103 host230.avlogic.com 4658 <-> 02 willard.uchicago.edu 22 3
120
10 --R--11 willard.uchicago.edu 22 <-> 01 host230.avlogic.com 4157 1
60
00 -S--A[clip]
11 willard.uchicago.edu 2170 <-> 01 cc17926-a.wlgrv1.pa.home.com ftp 2 112 00 -S--A103 cc17926-a.wlgrv1.pa.home.com ftp <-> 02 willard.uchicago.edu 2170 1 60 00 -S--A02 willard.uchicago.edu 2170 <-> 101 cc17926-a.wlgrv1.pa.home.com ftp 2 112 00 -S--A02 willard.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1705 1 48
00 -S--A02 willard.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1705 48 3107
10 ---PA103 55.icafe.euroweb.ro 1705 <-> 02 willard.uchicago.edu 40911 43 2602
00 -S-PA-
More forensics
✦
✦
What’s on port 40911
Looks like a back door
% telnet willard.uchicago.edu 40911
Trying 128.135.149.73...
Connected to willard.uchicago.edu (128.135.149.73).
Escape character is '^]'.
SSH-1.5-1.2.27
Investigating 40911
✦
✦
Did they connect to other machines on port
40911? (Yes, ultraviolet)
Could also scan the whole network for port
40911
% flow-cat * | flow-extract -e 'port = 40911 { print }'
12/18/2001 04:52:01 -> 12/18/2001 04:52:01 6 02 ultraviolet.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1662 1
12/18/2001 04:52:01 -> 12/18/2001 04:52:29 6 103 55.icafe.euroweb.ro 1662 <-> 02 ultraviolet.uchicago.edu 40911 36
12/18/2001 04:52:01 -> 12/18/2001 04:52:29 6 02 ultraviolet.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1662 37
12/18/2001 05:01:50 -> 12/18/2001 05:02:10 6 02 ultraviolet.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1662 39
12/18/2001 05:01:50 -> 12/18/2001 05:02:11 6 103 55.icafe.euroweb.ro 1662 <-> 02 ultraviolet.uchicago.edu 40911 48
12/18/2001 05:03:03 -> 12/18/2001 05:03:05 6 103 55.icafe.euroweb.ro 1662 <-> 02 ultraviolet.uchicago.edu 40911 6
12/18/2001 05:03:03 -> 12/18/2001 05:03:04 6 02 ultraviolet.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1662 4
12/18/2001 05:05:46 -> 12/18/2001 05:05:46 6 02 willard.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1705 1
12/18/2001 05:05:49 -> 12/18/2001 05:06:11 6 02 willard.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1705 48
12/18/2001 05:05:46 -> 12/18/2001 05:06:12 6 103 55.icafe.euroweb.ro 1705 <-> 02 willard.uchicago.edu 40911 43
48
2294
3131
2512
2800
340
296
48
3107
2602
00 -S--A00 -S-PA10 ---PA10 ---PA00 ---PA00 ---PA10 ---PA00 -S--A10 ---PA00 -S-PA-
Reading the logs through
✦
What happened later on in the logs?
✧
✦
This can give us more information on what else was
compromised
Connections in from avanti0.hab.de
12/17/2001 21:50:40 -> 12/17/2001 21:50:41
12/17/2001 21:50:40 -> 12/17/2001 21:50:41
12/17/2001 21:50:41 -> 12/17/2001 21:50:42
12/17/2001 21:50:41 -> 12/17/2001 21:50:42
12/18/2001 06:12:05 -> 12/18/2001 06:12:05
12/18/2001 06:12:05 -> 12/18/2001 06:12:05
12/18/2001 06:12:05 -> 12/18/2001 06:12:10
6
6
6
6
6
6
6
90 avanti0.hab.de
4222 <->
02 willard.uchicago.edu 22 <->
02 willard.uchicago.edu 22 <->
90 avanti0.hab.de
4222 <->
02 willard.uchicago.edu 22 <->
90 avanti0.hab.de
1023 <->
90 avanti0.hab.de
1023 <->
02 willard.uchicago.edu 22 5
90 avanti0.hab.de
4222 2
90 avanti0.hab.de
4222 6
02 willard.uchicago.edu 22 3
90 avanti0.hab.de
1023 1
02 willard.uchicago.edu 22 2
02 willard.uchicago.edu 22 20
296
100
543
120
60
112
1647
00 FS-PA00 -SR-A10 F-RPA10 --R--00 -S--A00 -S--A10 ---PA-
Repeat the process
✦
✦
Looking at who avanti0.hab.de connected to can
reveal more compromised machines
We find one more… aupc1.uchicago.edu
% flow-cat * |flow-extract -e 'host = avanti0.hab.de && host != willard { print }'
12/18/2001 00:40:14 -> 12/18/2001 00:40:14 6 90 avanti0.hab.de
4284 <->
12/18/2001 00:40:14 -> 12/18/2001 00:40:14 6 90 avanti0.hab.de
4284 <->
12/18/2001 00:40:14 -> 12/18/2001 00:40:14 6 02 aupc1.uchicago.edu 22 <->
12/18/2001 00:40:14 -> 12/18/2001 00:40:14 6 02 aupc1.uchicago.edu 22 <->
12/18/2001 08:00:52 -> 12/18/2001 08:00:52 6 02 aupc1.uchicago.edu 22 <->
02 aupc1.uchicago.edu 22 5
02 aupc1.uchicago.edu 22 2
90 avanti0.hab.de
4284 4
90 avanti0.hab.de
4284 1
90 avanti0.hab.de
1023 1
248
80
283
44
44
00 FS-PA10 --R--10 F--PA00 -S--A00 -S--A-
Logging Methods
✦
We use flow logs from routers (and some
switched).
✧
Mark Fulmer’s flow-tools
✧
✧
<http://www.splintered.net/sw/flow-tools>
Flow-extract
✧
<http://security.uchicago.edu/tools/net-forensics
Flow Logs
✦
Advantages
✧
✧
✧
Straight from router
No sense of state
Authoritative
✦
Disadvantages
✧
✧
✧
Need to have a router
that supports flows
where you want to log
Missing useful
information (e.g.
sequence number)
No sense of state
Logging Methods
✦
Argus – QoSient, LLC – Carter Bullard
✧
✧
<http://www.qosient.com/argus>
OpenSource effort and proprietary version
✧
✧
Same flow model, performance and scaling
Origin/History:
✧
✧
✧
Early 1990’s Work at CERT
Guerilla work until startup in 1999
Continued analysis/experimentation at CMU
✧
Validation, IDS, web logging (FlowScan-style)
Argus
✦
Applications – audit
✧
✧
✧
✧
✧
Edge Traffic Characterization
Security
Anonymized research data (use analysis)
Traffic accounting
Service/Policy Discovery
✧
✧
✧
✧
who/how/how much
Unexpected service delivery?
QoS validation
Internet Call records
✧
✧
Who talks to whom – not what’s said
Contrast to Carnivore
Argus Flow Logs
✦
Advantages
✧
✧
✧
Authoritative
Transaction flow aggregation
Strong flow model/semantic
✧
✧
✧
✧
✧
✧
TCPdump selection syntax
Scalable – multiple probes
Flexible – put probe anywhere
✧
✧
✧
TCP window delta/retrans
ICMP aggregation
Accurate timestamps
Subnet/switch/host
Limited access to user data
Higher level tools for
analysis/indexing
✦
Disadvantages
✧
✧
✧
Technology, no sexy apps
Limited documentation
Probe Architecture
✧
✧
✧
Vs switches, IPSEC, etc
Scaling factors
DoS vulnerability
Argus
✦
Quick Demo
Interesting Questions
✦
Aggregate transaction analysis
✧
✧
✦
Application fingerprinting
✧
✦
Web trans frames smtp spam
Probes followed by specific connections
Regardless of port
Network service Provision
✧
✧
End2End or Edge2Ether
Ask for a service, not a connection
Problems in identifying traffic
✦
What if the port number jumps around
✧
✧
✧
Many file sharing programs are beginning to
do this to evade firewalls
If it’s used by a lot of people it will look like
random traffic from a statistical view point and
will just appear as noise
Application layer analysis can help
✧
✧
What if the traffic is encrypted?
Need lots of storage and a fast machine to keep up
Network Graphs
✦
Allows quick visualization of network use
✦ MRTG
✧
✦
Cricket
✧
✦
<http://people.ee.ethz.ch/~oetiker/webtools/m
rtg.html>
<http://cricket.sourceforge.net>
FlowScan
✧
<http://net.dois.wisc.edu/~plonka/FlowScan>
Traffic Management
✦
Traditional Rate Limiting
✧
Who to rate limit?
✧
✧
✧
✧
✧
Just the dorms?
Everyone?
Known abusers?
How much to Rate Limit?
Can’t do application layer limiting, so it may
be ineffective to programs that jump ports
Traffic Management
✦
Packeteer, etc.
✧
Can do application level
✧
✧
Can’t do high bandwidth
✧
✦
What if the traffic is encrypted?
~100Mb/sec okay, ~1Gb/sec not
Other options…?