Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Monitoring and Early Warning for Internet Worms Cliff C. Zou, Lixin Gao, Weibo Gong, Don Towsley Univ. Massachusetts, Amherst 1 How to detect an unknown worm at its early stage? Monitoring: Monitor worm scan traffic (non-legitimate traffic). Observation data is very noisy. Connections to nonexistent IP addresses. Connections to unused ports. Old worms’ scans. Port scans by hacking toolkits. Detecting: Anomaly detection for unknown worms Traditional anomaly detection: threshold-based Check traffic burst (short-term or long-term). Difficulties: False alarms; threshold tuning. 2 “Trend Detection” Detect traffic trend, not burst Trend: worm exponential growth trend at the beginning Detection: the exponential rate should be a positive, constant value Monitored illegitimate traffic rate 60 60 60 50 50 50 40 40 40 30 30 20 20 10 10 0 0 10 20 30 40 50 30 20 10 0 10 20 30 40 50 Exponential rate a on-line estimation 0.2 0.2 0.2 0.15 0.15 0.1 0.1 0.1 0.05 0.05 0.05 0 0 0 -0.05 -0.05 -0.1 -0.1 0.15 10 20 30 40 50 10 20 10 20 30 40 50 -0.05 -0.1 10 Non-worm traffic burst 20 30 40 50 30 Worm traffic 40 50 3 Why exponential growth at the beginning? The law of natural growth reproduction Exponential growth — fastest growth pattern when: Negligible interference (beginning phase). All objects have similar reproductive capability. Large-scale system — law of large number. Fast worm has exponential growth pattern Attacker’s incentive: infect as many as possible before counteractions. If not, a worm does not reach its spreading speed limit. Slow spreading worms can be detected by other ways. 4 Worm modeling — simple epidemic model # of contacts I S Simple epidemic model: It # of infected hosts : # of susceptible : # of infectious : Total # of hosts : Infectious ability 100 200 300 400 500 600 Time 10 # of infected hosts Discrete model: 10 with exponential rate At very early stage: 10 4 2% 1% 3 2 50 100 150 Time t 200 250 5 Why use simple epidemic model? Can model most scan-based worms. We can use other worm models as well with minor modifications (such as exponential model). Code Red SQL Slammer Figures from: D. Moore, V. Paxson, S. Savage, C. Shannon, S. Staniford, N. Weaver, “Inside the Slammer Worm”, IEEE Security & Privacy, July 2003. 6 Kalman Filter Estimation Equivalent to Recursive Least Square Estimator: Give estimation at each discrete time. Robust to noise. System: Discrete-time simple epidemic model System state: Worm infection rate a. (a= bN, exponential growth rate at beginning) Epidemic parameter b. (worm infectious ability) Measurement from monitors: Ci : cumulative # of observed infected, Zi :# of scans at time i. 7 Kalman Filter Estimation System: where Kalman Filter for estimation of Xt : 8 Code Red simulation experiments Population: N=360,000, Infection rate: a = 1.8/hour, Scan rate h = N(358/min, 1002), Initially infected: I0=10 Monitored IP space 220, Monitoring interval: D = 1 minute Consider background noise x 10 5 3.5 Real value of a Estimated value of a 0.25 Estimated infection rate # of infected hosts 3 Infected I t Observed Infected Ct 2.5 2 0.2 0.15 1.5 1 0.05 0.5 0 0 0.1 100 200 300 400 Time t (minute) 500 600 0 150 200 250 Time t (minute) 300 350 Before 2% (223 min): estimate is already stabilized and oscillating a little around a positive constant value 9 SQL Slammer simulation experiments Population: N=100,000, Monitored IP space 220, Scan rate h = N(4000/sec, 20002), Initially infected: I0=10 Monitoring interval: D = 1 second, Consider background noise 10 x 10 4 0.5 Estimated infection rate # of infected hosts # of infected hosts It 8 6 4 2 0 0 50 100 Time t (second) 150 200 Real value of a Estimated value of a 0.4 0.3 0.2 0.1 0 20 40 60 Time t (second) 80 100 Before 1% (45 sec): estimate is already stabilized and oscillating around a positive constant value 10 Early detection of Blaster 3.5 Blaster: sequentially scans from a starting IP address: 3 1.5 1 0.5 0 0 5 100 200 300 400 500 Time t (minute) 600 After using low-pass filter 3.5 4 3 # of infected hosts 1024-block monitoring 16-block monitoring 2 It follows simple epidemic model. x 10 4 2.5 40% from local Class C address. 60% from a random IP address. x 10 x 10 3.5 2.5 4 1024-blocks monitoring 16-block monitoring 3 2 2.5 2 1.5 1.5 1 0.5 0 100 200 95 percentile (Blaster) 5 percentile (Blaster) 95 percentile (Code Red) 5 percentile (Code Red) 300 400 500 600 700 Time t 1 0.5 0 0 100 200 300 400 Time t (minute) 500 600 700 11 700 Bias correction for uniform-scan worms Bernoulli trial for a worm to hit monitors (hitting prob. = p ). Bias correction: : Average scan rate 3.5 # of infected hosts 3 2.5 5 4 Infected hosts It Observed infected Ct Estimated It after bias correction 3.5 # of infected hosts x 10 2 1.5 1 2.5 5 Infected hosts It Observed infected Ct Estimated It after bias correction 2 1.5 1 0.5 0 3 x 10 0.5 100 200 300 400 500 Time t (minute) 600 Monitoring 217 IP space 700 0 100 200 300 400 500 Time t (minute) 600 Monitoring 214 IP space Bias correction can provide unbiased estimate of It 12 Prediction of Vulnerable population size N x 10 5 3.5 Direct from Kalman filter: 3 2.5 2 1.5 1 0.5 0 h : A worm sends out h scans per D time (derived from egress scan monitor) Estimated population N Alternative method: 6 x 10 5 100 200 300 400 500 600 5 4 3 2 1 0 50 Real population size N From estimated b From h and estimated a 100 150 200 250 300 Time t (minute) 350 400 Estimation of population N 13 Use exponential growth model 4 2% 1% # of infected hosts # of infected hosts 10 At the early stage: 10 10 3 2 100 50 : observation data Model #2: Autoregressive (AR) model 200100 300 500 150400 200 Time Timet 600 250 Early stage of worm propagation Model #3: Transformed linear model 14 Comparison between three estimation models 0.2 0.2 0.2 0.15 0.15 0.15 0.1 0.1 0.1 0.05 0.05 0.05 0 0 140 160 180 200 Time t (minute) 220 240 Epidemic model 0 140 160 180 200 Time t (minute) 220 240 AR exponential model 140 160 180 200 Time t (minute) 220 240 Transformed linear model Observations AR exponential model is smoother than epidemic model Transformed linear model gives best results Detect a worm when it infects about 0.5% population 15 Simple analysis of three estimation models Why AR exponential model is smoother than epidemic model? Introduced errors from measurement data: Epidemic model AR exponential model Why transformed linear model is better than AR model? assume AR exponential model: Transformed linear model: where 16 Summary Trend detection: non-threshold-based methodology Principle: detect traffic trend, not burst Pros : Robust to background noise low false alarm rate Cons: Rely on worm model, representation of measurement data Epidemic model, exponential model Using low-pass filter on noisy observation data For uniform-scan worms Bias correction: Forecasting N: ( IPv4 ) : scanning IP space : scan hitting prob. Routing worm : Average scan rate : Infection rate : cumulative # of observed infectious 17