Meeting the Challenges of Unmanned and Autonomous System Test and Evaluation

1. Meeting the Challenges of Unmanned and Autonomous System Test and Evaluation Thomas Tenorio, Subject Matter Expert for UAST Executing Agent 10 March 2010, USC

2. Activities • Working Group • Roadmapping, Surveys, Networking, Tech Eval • BAA Supply Space Surveys • UAST Roadmap • FY2009 Unmanned Integrated Roadmap Findings • FY2010 UAST Roadmap • Community / Professional Organization Engagement • Test Communities • Standards Communities • ITEA: tutorials and workshops • AUVSI • INCOSE (International Council of System Engineers)

3. The Test Resource Management Center • TRMC under AT&L • UAST part of T&E S&T JMETC 6.5 CTEIP 6.4 T&E S&T 6.3 UAST DET MST NII SET NST

4. TRMC Mission and Vision • Mission • "Plan for and assess the adequacy of the…MRTFB…[and] to provide adequate testing in support of development, acquisition, fielding, and sustainment of defense systems; and, maintain awareness of other T&E facilities and resources, within and outside the Department, and their impacts on DOD requirements." • Vision • The DoD T&E ranges and facilities will be fully capable of supporting the Department with quality products and services in a responsive and affordable manner. 22 ranges Tagged as National Asset Major Range and Test Facility Base

5. UAS Test Bed and Environment S&T for physical test capabilities associated with Test Bed and Environment • Stimulus selection or generation • Sensors • Data Acquisition and Management • Data compression and characterization (signatures) • Data reduction/analysis/interpretation • Power Technologies: providing increased mission time and capability without increasing the logistics footprint. • Test conducting including situation awareness • Test operations safety • FAA and other civil coordination • C4ISR interoperability

6. TRMC Research and Interaction • Test and Evaluation, Science and Technology • Three groups organized by TRL level • T&E S&T • TRL 3-6 or 6.3 funding • $95M / year • CTEIP • TRL 6-9 or 6.4 funding • $140M / year • JMETC • 6.5 funding • $10M / year

7. TRL Technology Readiness Levels • Criteria used to assess project eligibility and status • Measures system maturity • Non-linear in time, money and effort • Not well understood by community

8. T&E S&T • Test and Evaluation, Science and Technology • 7 Focus Areas • UAST • SET • NST • NII • MST • HSHT • DET

9. Unmanned Systems Integrated Roadmap • The UAS Challenge • Capabilities (311 JCA named targets) • Systems (138 systems) • Performance Envelope Aspects (41) • Technologies (17)

10. UAST Augmented Study Methodology Documented (PoR) UAS Missions and Technologies per USIR Draft Drivers, Use Cases, Test Concept ,Test Plans Draft Test Requirements Refined Drivers, Use Cases, Test Concept ,Test Plans Refined and Extended (beyond PoR) UAS Missions and Technologies with Working Group Validation T&E Needed to Test Specific UAS Technologies Facility Specific Descriptions & Requirements (Hard Numbers) Test Resource Requirements Baseline T&E Capabilities (we do not want to conduct an exhaustive survey of all the T&E capabilities that exist and could support UAS T&E) Tri-Service Baseline Capability Test Resource Survey Baseline Gaps sensitive to New Approaches and New Paradigms Draft Test Resource Need Statements Needs Analysis (Gap) Refined Test Resource Need Statements Refined Needs Statements Validation BAA, RFI, White Papers & Proposals Develop Solutions BAAs, etc Roadmap Prioritize Solutions Road-Map

11. Relevant Commentary on Test

12. Urgent Needs for UAS

13. Outcome Timelines

14. Means, Ways, and Ends to UAST as a Value Proposition UAS Safety, Suitability, Survivability, Effectiveness Test and Evaluation of UAS as Highly Complex Systems V&V What if --? Integrity Assessment Surrogate & Simulate Live Contrived Live in situ UAST Ends Emulate Cross-Domain Commonality ……......... Specificity Space Air Ground Maritime (surface/under) Predicting Behavior UAST Ways Emulating Mission & Environmental Complexity with Assured Safety Assessing Effects and Capabilities UAST Tools & Techniques Reference Data Sets Ground Truth Decision & Behavior Protocols & Design Test Bed and Environment UAST Means UAST Technologies Risks Constraints Near Mid Far

15. Use Case Driven Investment

16. The Target for UAST

17. BAA Technology Investment Categories

18. Integrating Findings of the FY2009 Unmanned Integrated Roadmap

19. UAS for Operational Necessity (rapid tempo) Evolutionary acquisition with JUTLS & JUONS Booming capability development Capability challenge of 311 named systems Majority system non-PORs Fielding tech in months: 4-6 months for joint operational necessity Ensure test capabilities support the fielding of unmanned systems that are effective, suitable, and survivable • Test of UAS (increasing tempo for Integrated T&E) • Accelerate incrementally improving T&E • Augment legacy and improvised capability • Emerging arguments for UAST value proposition • AS-IS: OT&E emphasis with emergent Joint T&E • Fielding UAST in years: 3 year in S&T and 4 years in T&E UAST: Knowledge Risk Reduction UAS: Warfighter Capability

20. The Interacting Communities of UAS Advantage UAS: Warfighter Capability UAST: Knowledge Risk Reduction UAST S&T: Next Gen Tech For UAST

21. Value Proposition of UAST • UAST to further ensure Safe, Suitable, Effective, and Survivable UAS • Pace and tempo to secure advantage for UAS • Advancing Knowledge Generation capabilities for risk reduction in the production of UAS

22. UAST Driver Overview • UAST has primary drivers • Autonomous Behavior of unmanned and autonomous systems that sense, understand and act upon the environment in which they operate • Safety of Autonomous systems in Mission and Environment • Secondary drivers • Sensory Capacity & Perception Loops (Observe) • Knowledge Models of Ground Truth & Behavior (Orientation) • Decision Making (Decide) • Supervised Autonomy Behavior (Action) • Context • Systems Testing of Human Independent Behavior • Emulating UAS in Mission and Environmental Complexity with Assured Safety • Assessing UAS Effects and Capabilities in Joint Capability Areas

23. UAST Exemplar Overview

24. Leveraging S&T for T&E Capability Development Standard Systems T&E • System Level T&E • Mission & Environment T&E • Joint Capability Areas T&E Autonomous System T&E • Predicting Autonomous Behavior • Emulating Mission and Environmental Complexity with Assured Safety • Assessing UAS Effects and Capabilities Unmanned & AutonomousSystems Test UAS Test & Evaluation Focus Areas Non-IntrusiveInstrumentation Spectrum Efficiencies Multi-Spectral Sensors Netcentric Systems

25. UAST Systems EngineeringCapabilities Reference Framework • Predicting Unmanned and Autonomous System Behaviors (T&E/E&A) • Emulating Mission and Environmental Complexity with Assured Safety (T&E/E&A) • Assessing UAS Effects and Capabilities (E&A) • Autonomous Test Protocols and Design (T&E) • Test Bed and Environments for UAST (T&E) • UAST World Models (Ground Truth, Decision, & Behavior) (T&E) • Tools and Techniques for Systemic UAST (T&E/E&A) 1 7 3

26. T&E Capabilities Reference Framework 1 7 3 1 2 7 4 5 6 * * Safety Considerations in all 7 Areas Predicting Unmanned and Autonomous System Behaviors Emulating Mission and Environmental Complexity for Assured Safety Assessing UAS Effects and Capabilities Autonomous Test Protocols and Design Test Bed and Environments for UAST UAST Reference Data Sets (Ground Truth, Decision, & Behavior) UAST Tools and Techniques

27. Essence of UAST Challenge • At >10-fold reduction in • Cycle time • Cost • And exemplary ROI of S&T • Determine • False Positives and False Negatives • Dynamic Limits of behavior • Integrity Limits of intended functions Across UAS(s) OODA • Regarding Mission • Effectiveness • Suitability • Survivability • Safety

28. S&T Opportunities for T&Eof Autonomy with Assured Safety Tools for Design of Experiments in system, multisystem, and system of systems scenarios considering also implications of mission scenarios, opposition capability, and physical context. Ability to incorporate UAS design models into warfighter-scope models/simulations in order to anticipate mission suitability, safety, effectiveness and survivability (including countermeasures).  Determining how to manipulate live physical scenarios including Red Forces. Acquiring ground truth data during actual test operations. Bayesian Belief Networks and similar tools for conflating test data to mission-level expectations. Ensuring that Net Centric Systems and relevant test assets are sufficiently agile to enable span and dynamics of UAS test scenarios. Systems architecting and engineering of a family of composable UAST’s.

29. UAST Systems EngineeringCapabilities Reference Framework • Predicting Unmanned and Autonomous System Behaviors (T&E/E&A) • Emulating Mission and Environmental Complexity with Assured Safety (T&E/E&A) • Assessing UAS Effects and Capabilities (E&A) • Autonomous Test Protocols and Design (T&E) • Test Bed and Environments for UAST (T&E) • UAST World Models (Ground Truth, Decision, & Behavior) (T&E) • Tools and Techniques for Systemic UAST (T&E/E&A) 1 7 3 Readiness Assessment

30. Readiness Assessment Capability What: Discover internal bugs and vulnerabilities and external incompatibilities in UAS’s, across UAS’s and in T&E systems. Where: In executable code, source code, data bases, system models, mission simulations, and SoS configurations. At development, integration, warfighter and depot locations. Why: Generate warfighter-confident knowledge. Cut test cycle time and cost in half. User controllable degree of False Positives and False Negatives. How: Code inspection. Test beds not required. Mathematically rigorous assessment method and tools. Enabled by next generation pattern recognition semiconductor chips with throughput ≈ 1 Gb/sec When: TRL3@2010, TRL5 @2011, TRL6@2012, TRL9@2014

31. UAST Systems EngineeringCapabilities Reference Framework • Predicting Unmanned and Autonomous System Behaviors (T&E/E&A) • Emulating Mission and Environmental Complexity with Assured Safety (T&E/E&A) • Assessing UAS Effects and Capabilities (E&A) • Autonomous Test Protocols and Design (T&E) • Test Bed and Environments for UAST (T&E) • UAST World Models (Ground Truth, Decision, & Behavior) (T&E) • Tools and Techniques for Systemic UAST (T&E/E&A) 1 7 3 Safety Assurance *

32. Safety Assurance What: Discern and Referee the contest between autonomy and safety, both a) test safety, including, e.g., FAA, and b) Operational safety, e.g., fratricide and innocent civilians. Enable both static and evolving limits. Assess efficacy of UAS Self-test capability, resilience to cyber threats, probable error in M&S evaluations of UAS(s). Where: Throughout 5000.02 phases and Warfighter stages. Across UAS, UAS’s, SoS. Spans both on-board and administrator functions. Why: Avoid unintended consequences of UAS operations. Generate warfighter-confident knowledge. Cut test cycle time and cost in half. User controllable degree of False Positives and False Negatives. How: A ‘Do No Harm’ OODA loop inside the autonomy loop of both the UAS and the UAST. Method for Preempting UAS behaviors, separate from Planner capability, preferably non-destructive. When: for autonomy Level 1@2010, 2@2011, 3@2012, 4@2013, 5@2014

33. Contact Information Thomas.tenorio@us.army.mil (575) 678-4671