INFORMATION FUSION FOR CYBER SITUATION AWARENESS

1. INFORMATION FUSION FOR CYBER SITUATION AWARENESS George Tadda Fusion Technology Branch Information Directorate Air Force Research Laboratory E-mail: george.tadda@rl.af.mil Phone: 315-330-3957

2. Outline • Introduction • Motivation • Situation Awareness Reference Model • Metrics • Application of Lessons Learned

3. Work in Situation Awareness (SA) • Used reference models to demonstrate/build prototype systems for: • Cyber (Defense & Security (D&S) ’05, SIMA ‘05) • Tactical (ISIF ’02) • Global (ISIF ’04) • Maritime • and Many Others • Developed Metrics (D&S ’04) to Evaluate Level 2 Systems and applied them to Cyber (D&S ’05) • After much discussion we questioned the difference between tracking objects and situations and whether the majority of the metrics are just another way to measure integrity of tracks • Additional Activities: • Jean Roy, under The Technical Cooperative Program, presented a definition of situational analysis and included in "Concepts, Models, and Tools for Information Fusion“ • Snidaro, M. Belluz, G. Foresti, “Domain knowledge for security applications”, ISIF’07 defined types of events (simple, spatial, and transitive) • Dale Lambert, formalizing situation awareness through mathematics 07-210

4. Motivation(Reality of Today’s Environment) Moving Objects 80/sec 1000’s of Objects Data • The Analyst/Operator • Drowning in data and Inundated with “dots” on map or messages. INFORMATION STARVED • INCOMPLETE, CONFLICTING DATA • SA is Highly Operator Dependent and 100% Mental Process • - Stress • - Fatigue • - Experience • LIMITED BY INDIVIDUAL’S ABILITIES Class B Address Space 26,000 Alerts/day Data 3 – 4 Petabytes/day (E-mail, Published Pages, etc) Data Today WE Have… Objects Tactical Alerts Cyber Events Global …and MORE 07-291

5. Motivation Knowledge Of Units Today WE Have… Anticipation Most Likely/ Worst Case eCOA Sensemaking What is… A Measure of Success SPATIAL (Obj Types/No.) Data/Information Ratio (DIR) (Examples) Plausible Futures (Intent, Opportunity, Capability) Units Objects Tactical Pre Iran/Kuwait Conflict Data Information Objects*: 16,203 No. Units: 42 DIR: 386 Knowledge Of Atcks TRENDS (Network, Host) Plausible Futures (Intent, Opportunity, Capability) *No noise/clutter ALERTS Attacks Alerts Cyber SKAION Datasets (3s8, 3s26, 3s28, 3s29) Data Information Events Attacks DIR 20,131 107 188 19.531 66 296 8,681 62 140 31,513 155 203 Knowledge Of Sits TRENDS (Economic, Military) Plausible Futures (Intent, Opportunity, Capability) Situations Events Global EVENTS Unit A xx xx xx xx xx Data Information Information Information Information Information Assessment Assessment Assessment Assessment …and MORE (STEP 1: From Data -> Complex Relations/Situation(s)) (STEP 2: From Complex Relations/Situation(s) -> Anticipation) 07-291

6. Goal Focused Data Focused BALANCE Sharing the Stage (From A Model Perspective) SITUATION AWARENESS FUSION - TACTICAL • Most popular is the Joint Director’s of Laboratory (JDL) Model (Sensor-based) • Functional Model • 5 Levels (Level 0, 1, 2, 3, 4) • Published By Llinas, Hall, White (1992) • Most work concentrated on Level 0/1/4 (Dots on Map) • Little definition of Level 2/3 (What do they mean?) • Bottom-up, Data Driven • Receiving Much Attention Today from the Cognitive Community • Mental Model • 3 Levels: Perception, Comprehension, Projection • Developed by: M. Endsley (1995) • Extended by McGuinness and Foy for Resolution • Top Down, Goal Driven 07-291

7. Assessment Situation Situation Awareness Reference Model (Combining The “Best” Of Both Worlds) • Based on JDL & Endsley’s Models • - Plus Initial Data Requirement • - Textual Inputs (Info Exploit) • Define Problem/Goal – Top Down • - What/Where/Who… • Processing Flow ( ) • - Projection – The Alert(s) • - Comprehension • -- Model Analysis • - Perception • -- Data Collection • -- Parsing/Extraction • -- Data Cleansing • - JDL: Level 0/1 • Process Refinement ( ) • - Missing Data • - Additional Data • - Input for Sensor Mgmt • Off-Line Processing ( ) • - Knowledge Discovery Level 0/1 Data Parsing Sources Sources Collection Extraction Perception Data Cleansing Data Requirements Evidence Model Knowledge Knowledge Additional Analysis Discovery Discovery Info Tools Tools Tools Comprehension Matches/ Partial Matches Target *Missed Questions Models ANTICIPATION Potential New Relationships Tools The “Problem” *Based on Model Unfolding 07-291

8. Client/Host Configurations Mission Model Matching Algorithm Equip Fail Attack A Attack N Potential Attacks Assessment Intrusion Attempt Recon Situation Privilege Escalation A Priori Knowledge Situation Awareness Reference Model (Applied to Cyber SA) Open Source Host IDS Snort Dragon Network Stats Web Logs Sys Logs Level 0/1 Post Proc Data Parsing Data Parsing Collection Perception Extraction Sources Sources Collection Extraction Perception Data Cleansing Data Cleansing Data Requirements Evidence (Alerts) Evidence Model Knowledge Knowledge Additional Analysis Discovery Discovery Info Comprehension Tools Tools Tools Comprehension Matches/ Partial Matches Target *Missed Questions Models ANTICIPATION Potential New Relationships Tools The “Goal” Anticipation 07-291

9. Situation Awareness Reference Model(Applied to Cyber Domain) Model Template Y A Priori Knowledge Client/Host Configurations Snort Model Template X Recon Intrusion Attempt Dragon Privilege Escalation Multi-Stage Attack Goal Potential Attacks Host IDS Attack A The Network Potential to Advance to Next Stage Evidence Model Matching Algorithm Attack B Sys Logs Equip Failure Post Proc List Based On Risk Anticipation Web Logs Impact Assessment Network Stats Business Model Perception Comprehension Open Source TBD 07-291

10. Lexicon(Background) • Evidence • IDS Alerts (i.e., Snort, Dragon) • System Logs • Service Logs (i.e., Apache, IIS) • Network Flow Data • Track – collection of all evidence available against one or more targets originating from one or more attackers • Situation – set of tracks at a snapshot in time • Situation Awareness of a Network – analyst’s mental model of the situation • True Positive* – successful attack • False Positive* – incorrectly identified attack • Non-relevant Positive* – correctly identified attack that fails or is incomplete (i.e., try to exploit a ‘blocked’ vulnerability) *Valeur et al, “A Comprehensive Approach to Intrusion Detection Alert Correlation, IEEE Transactions on Dependable and Secure Computing, Jul-Sep 04 06-081

11. Metrics Overview • Confidence – measures the ability of the system to correctly identify the track(s) • Recall: Percentage of tracks detected in relation to the “total known” • Precision: Percentage of correct tracks detected in relation to number of detections • Fragmentation: Percentage of tracks reported as multiple tracks that should have been reported as a single track • Mis-Association: Percentage of tracks that are neither correct nor a fragment in relation to the number of detections • Purity – characterizes the quality of the detections • Mis-Assignment Rate: Percent of evidence incorrectly assigned to a given track • Evidence Recall: Percentage of evidence detected in relation to the “total known” evidence • Cost Utility – a single weighted measure of the system in identifying “important or key” tracks with respect to a concept of cost • Timeliness – measures the ability of the system to respond within time requirements of a particular domain 06-081

12. Cost Utility(Weighted Cost and Attack Score) ∑Weighted Values for Results Weighted Cost = ∑Weighted Values for Ground Truth [No. Attacks in Results][No. Results] – [[Sum of Positions of Attacks in Results] – [Geometric Sum ([No Attacks in Results] -1)]] Attack Score = [No. Attacks in GT][No. Results] Given: 100 pts ATTACK 5 pts Background Scan 5 pts Background Attack -50 pts False Positive Proposed Attacks Ground Truth R0 Background Scan R1 UNASSIGNED R2 Attack R3 Background Scan R4 Background Scan R5 Background Attack 5 - 50 100 5 5 5 70 GT0 Background Scan GT1 Background Attack GT2 Background Scan GT3 Attack GT4 Background Scan 5 5 5 100 5 120 NOTE: Sorted Based on Score Weighted Cost = 70/120 = .5833 Attack Score = [(1)(6) – (2-(1-1))]/(1)(6) = .6667 06-081

13. The Infrastructure Skaion Dataset Processing Results Results UsingAFRL Schema Viewing Ground Truth Cyber Fusion System AFRL Results Analyzer Tools AFRL Ground Truth Correlation Ground Truth Assignment Matrix .csv .html List of Potential Attacks REPORTS Alerts correlated to selected Attack Track Filter by score Play Buttons Metric Report (Confidence, Purity, Cost) 06-081

14. Work has Raised Many Questions … Resulting in Few Answers • Where do groups, events, activities fit in? • Can we not track a group, an activity (Why only Objects?) • Is a group or activity only a complex object? • What is a Situation? Is there more than one? Is it Context-based? • Where does Knowledge Discovery exist? Forensics? • What is Situation Assessment? • Is Threat Assessment only of the future – what about current threat? • What about forecasting or projecting the “future” state? No one model answers ALL of these questions and even addresses them! 07-210

15. …so Then What • Treating Situation as a composite of activities and tracking activities as complex objects allows for a “cleaner” distinction between fusion levels • Situation(s)-> Activity(s) -> Group(s)/Entity(s) -> Event(s): These are ALL OBJECTS THAT CAN BE TRACKED • Object Assessment has really been performing Tracking & Identification – LET’S TRACK ALL TYPES OF OBJECTS • Knowledge Discovery and a priori knowledge necessary and integral to building “complex” objects (e.g., Groups, Activities) • Updating knowledge/relationships (models) is continuous and part of refinement process • Define Situation Assessment based on Jean Roy’s Definition for Situational Analysis: • Behavior Analysis – Activity Level Analysis • Intent Analysis – Salience Analysis • Capacity/Capability Analysis – Impact Analysis • Threat Analysis 07-210

16. …and • Use Time to distinguish between JDL Level 2 and 3 as does Endsley’s comprehension and projection • Same analysis is done for both levels only difference is time • Thus JDL Level 2 is assessment of “current situation and JDL Level 3 is the assessment of the current situation projected forward in time. • Process Refinement involves not only sensor movement/collection (sensor management) BUT fusion algorithm management (which algorithms and which parameters to use) and model management from ALL processes. Possible sources to refinement include: L1: Prediction where object is moving/next event L2: Missing data, increase certainty of current assessments L3: Forecasted actions/placement to pre-position sensors 07-210

17. Revised Situational Awareness Reference Model (Based on Previous Suggestions) Level 1: Object Tracking and Identification Level 3: Assessing the Forecasted Situation(s) Level 2: Assessing the Current Situation(s) *Based on JDL, Endsley’s, and Jean Roy’s work 07-210

18. Wrap Up • We proposed a revised Reference Model that includes many of the lessons learned to date • Plans are to continue to apply this revised model to Air, Cyber and Space Situation Awareness – UNIVERSAL SITUATION AWARENESS • …with emphasis on current and forecasted situation assessment 07-210