Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Intrusion Detection Alert Correlation Mark Shaneck 2/11/2005 1 Outline Problem Statement Different Correlation Approaches A Comprehensive Approach Good News and Bad News A Better Approach? 2 What’s The Problem? Large organizations get tons of alerts Possibly up to 20,000 per day! Many false alarms 3 Also… Alerts can come from many different sources – Signature based IDS (Snort) – File System Integrity Checkers – System Call Traces Alerts may represent multiple stages in one attack Hard to make sense out of a large pile of alerts! 4 So What Is Alert Correlation? 3 general categories – Alert Clustering – Matching Predefined Attack Scenarios – Prerequisites/Consequences 5 Alert Clustering Main Sources: – A. Valdes, K. Skinner, “Probabilistic Alert Correlation”, RAID 2001 – O. Dain, R. Cunningham, “Building Scenarios from a Heterogeneous Alert Stream”, IEEE Workshop on Information Assurance and Security, 2001 6 General Idea Join alerts together in some meaningful groups Group alerts into attack threads - one thread contains all alerts related to one attack For a new alert, compare to all alert threads – Join to the closest match – Or start new thread if none match 7 Similarity Measure Feature Overlap - only consider features present in both (source, target, ports, attack class, timestamps, etc.) Each feature has a similarity measure – How much do port lists overlap? – Is one port contained within another’s list? (target port was previously scanned) – Are the IPs from the same subnet? – Attack classes have a similarity matrix 8 Similarity Expectation Different levels of similarity are expected for different features in different situations – SYN FLOOD with source spoofed • Expectation of similarity for source IP is 0 – Scanning port(s) • Expectation of target IP is low (but not 0 - since it usually scans the subnet) 9 Minimum Similarity Threshold for similarity measure Similarity is 0 if not above the minimum Adjusting thresholds – Synthetic Threads • high for sensor id, IPs – Security Incidents • low for sensor id, high for attack class • fuse alerts from multiple sources – Multistep attack detection • low for attack class 10 So What Is Alert Correlation? 3 general categories – Alert Clustering – Matching Predefined Attack Scenarios – Prerequisites/Consequences 11 Matching Predefined Attack Scenarios Main sources – H. Debar, A. Wespi, “Aggregation and Correlation of Intrusion-Detection Alerts”, RAID 2001 – B. Morin, H. Debar, “Correlation of Intrusion Symptoms : an Application of Chronicles”, RAID 2003 12 Aggregation and Correlation Correlation – Group alerts that are part of the same attack trend – Duplicates – Consequences (chain of related alerts) Aggregation – Group alerts based on certain criteria to aggregate severity level, reveal trends, clarify attacker’s intentions – Situations 13 Duplicates Duplicates Definition – Initial Alert Class – Duplicate Alert Class – List of Attributes (that must be equal) – Severity Level (new severity level for new merged alert) Specified by analyst 14 Consequences Consequences Definition – Initial Alert Class – Initial Probe Token – Consequence Alert Class – Consequence Probe Token – Severity Level – Wait Period Links together alerts that are sequential in nature 15 Aggregation Aggregate based on three axes – Alert Class – Source – Target Putting wildcards for different cases gives different views Aggregate into scenarios 16 Scenarios Same source/target/attack class – A single attacker launching attacks against a single victim Same source/destination – Single attacker running many attacks on a single victim Same target/attack class – Distributed attack against a single victim Same source/attack class – A single attacker running the same attack against multiple victims 17 Chronicles “Set of events, linked together by time constraints, whose occurrence may depend on the context” Similar to plan recognition Used to model known attack “chunks” – Long attack scenarios may have many paths – Certain small sequences of events almost certainly occur together 18 So What Is Alert Correlation? 3 general categories – Alert Clustering – Matching Predefined Attack Scenarios – Prerequisites/Consequences 19 Prerequisites/Consequences F. Cuppens, A. Miège, “Alert Correlation in a Cooperative Intrusion Detection Framework”, In IEEE Symposium on Security and Privacy, 2002 P. Ning, D. Reeves, et al. (many papers) – Check my website for the list – Or the very last slide….. 20 Prerequisites/Consequences Prerequisite: the necessary condition for the attack to be successful Consequence: the possible outcome of the attack Represented as a logical formula – Using only AND and OR connectives 21 Hyper Alert Type (fact, prerequisite, consequence) SadmindBufferOverflow = ({VictimIP, VictimPort}, ExistHost(VictimIP) AND VulnerableSadmind(VictimIP) {GainRootAccess(VictimIP)}) 22 Prepare-For Relationships An alert “prepares for” another alert if it contributes to the second alert’s prerequisite set Also must occur earlier in time 23 Correlation Graph Directed acyclic graph, with the nodes being alerts and the edges being the prepares-for relations Could be huge! 24 Adjustable Reduction Aggregation of alerts of the same type Can result in overly simple graphs Adjustable – Analyst can specify a time interval – Only alerts with time gap less than the interval are merged 25 Adjustable Reduction 26 Focused Analysis Logical combination of comparisons between attribute names and constants SrcIP = 129.174.142.2 OR DestIP = 129.174.142.2 Useful for focusing on a critical server 27 Graph Decomposition Cluster alerts based on “common” features Use clusters to separate large graph into smaller ones (A1.SrcIP = A2.SrcIP) AND (A1.DestIP = A2.DestIP) Clustering constraints are specified by the analyst 28 Reduced and Decomposed Graph Example 29 Matching Attack Strategies Attack Strategy Graph – Set of events linked together by certain constraints • Time Order • IP Addresses Events can be generalized to deal with variations SadmindBufferOverflow TooltalkBufferOverflow RPCBufferOverflow 30 Measuring Similarity Between Attack Strategies Error Tolerant Graph Isomorphism Use edit distance to derive a similarity measure Can be used to find similar attacks or to match against predefined strategies 31 Hypothesizing About Missed Attacks Missed attacks can break up the graphs – One attack graph becomes two disconnected, seemingly unrelated, attack graphs Indirect Prepares-for Similarity based merging of attack graphs Prune hypotheses with network traffic – E.g. one hypothesized attack is ICMP ping, but no ICMP traffic occurred during that time 32 Outline Problem Statement Different Correlation Approaches A Comprehensive Approach Good News and Bad News 33 A Comprehensive Approach F. Valeur, G. Vigna, C. Kruegel, R. Kemmerer, "A Comprehensive Approach to Intrusion Detection Alert Correlation", In IEEE Transactions on Dependable and Secure Computing, 2004 34 Alert Fusion Combine alerts that are independent detection of the same attack instance – Must be temporally close – From different sensors – Identical overlapping attributes 35 Alert Verification Idea: False positives can negatively impact alert correlation Filter out false positives and irrelevant positives (alerts that correspond to failed attacks) 36 Alert Verification Passive: use network knowledge to see if attack could succeed (low overhead, low confidence) – Listing of existence of/services running on IPs – Firewall configurations Active: check for evidence (high overhead, high confidence) – – – – See if service is still running and available See if extra ports are open Use vulnerability scanner to test target machine Remote login and run scripts 37 Thread Reconstruction Group alerts that refer to attacks launched by one attacker against a single target Merge alerts with same source and destination and within a time interval 38 Attack Session Reconstruction Link network based alerts to host based alerts Manually specify links between network events and process events – Alert on web server process (or one of its children) can be correlated to a (temporally) nearby network alert targeted to that machine on port 80 39 Focus Recognition Identify hosts that are the source or target of lots of attacks Merge these alerts together into one Source: Scanning Target: DDoS 40 Multi-Step Correlation Identify attack patterns that are made up of multiple individual attacks Create attack patterns by means of expert knowledge Simply match the merged alerts to the attack strategies 41 Experimental Results Defcon9 – Input: 6,378,096 alerts – Output: 203,303 alerts – Reduction: 96.81% TreasureHunt – Input: 2,811,169 alerts – Output: 1,080 alerts – Reduction: 99.96% MIT/LL 2000 – Input: 36,635 alerts – Output: 17,220 – Reduction: 53.00% 42 Benefits of Alert Correlation Higher level representation of alerts reduces clutter and can show attack structure Reduce false positives – False positives are unlikely to correlate with other alerts May find many attacks and respective scenarios 43 Limitations of Correlation Relies on IDS to alarm each step of the attack – Exploit mutations – Novel attacks – Bad sensor placement – Sensor overload - packet loss – Restricted ruleset for better performance Relies heavily on a priori expert knowledge 44 Limitations of Correlation (cont) Cannot provide a comprehensive view on network attacks 45 MINDS Level 2 Level 1 IDS alerts Anchor Point Identification Context Extraction Attack Characterization Behavior/Host Profiling 46 Questions? Paper links located at: http://www.cs.umn.edu/~shaneck/wormlist.html – At the bottom of the page Slides available: http://www.cs.umn.edu/~shaneck/Correlation.ppt 47 A Budding Hacker 48 Peng Ning Reference List 1. 2. 3. 4. 5. 6. 7. 8. P. Ning, D. Reeves, Y. Cui, "Correlating Alerts Using Prerequisites of Intrusions", Technical Report, TR-2001-13, North Carolina State University, Department of Computer Science, December 2001 P. Ning, Y. Cui, D. Reeves, "Analyzing Intensive Intrusion Alerts via Correlation", In Recent Advances in Intrusion Detection, 2002 P. Ning, Y. Cui, D. Reeves, "Constructing Attack Scenarios through Correlation of Intrusion Alerts", In CCS 2002 P. Ning, D. Xu, "Learning Attack Strategies from Intrusion Alerts", In CCS 2003 P. Ning, D. Xu, C. Healey, R. St. Amant, "Building Attack Scenarios through Integration of Complementary Alert Correlation Methods", NDSS, February 2004 Y. Zhai, P. Ning, P. Iyer, D. Reeves, "Reasoning about Complementary Intrusion Evidence", 20th Annual Computer Security Applications Conference, December 2004 D. Xu, P. Ning, "Alert Correlation Through Triggering Events and Common Resources", 20th Annual Computer Security Applications Conference, December 2004 P. Ning, D. Xu, "Hypothesizing and Reasoning about Attacks Missed by Intrusion Detection Systems", ACM Transactions on Information and System Security, 2004 49