CREATE: The Opportunities and Challenges

1. CREATE: The Opportunities and Challenges Douglass Post for The CREATE Team Chief Scientist & CREATE Program Manager, DoD HPCMP HPC Users’ Forum, Norfolk, VA April 14, 2008 CREATE will develop and deploy three computational engineering tool sets for acquisition program engineers to harness the exponential growth in supercomputer power to rapidly develop and analyze engineering design options. These tool sets include Aircraft tools (Aerodynamics & Structures), Ship tools (Hydrodynamics and Structures) and Antenna Integration tools (Electromagnetics). Distribution Statement A: Approved for public release; distribution is unlimited.

2. The future is bright for HPC but challenges loom as we go from petaflops/s to exa-flops/s • Extrapolation to 2020 • (1-10 GFlops/core) • 2000: 7.2 TFLOPs/s • ~5000 cores • 2010: 2x103 TFLOPs/s • 105-6 cores • 2020: 106 TFLOPs/s • 108-10 cores • How do we program for 108-10 cores? Especially if the cores are different?

3. Next Generation Computers Offer Society Unparalleled Power to Address Important Problems Next Generation Computers Will Provide Exciting Opportunities To Develop And Deploy Very Powerful Application Codes, Much More Powerful Than Present Tools: Utilize Accurate Solution Methods Include All The Effects We Know To Be Important Model A Complete System Complete Parameter Surveys In Hours Rather Than Days To Weeks To Months Greatest Opportunities Include Large-scale Codes That Integrate Many Multi-scale Effects To Model A Complete System Science And Engineering Design

4. How do we get the codes that can exploit the next generation of computers? • Developing such codes is the major bottleneck! • Codes that can exploit the next generation of computers incorporate multi-scale, multi-disciplinary effects and scale to many, many thousands of processors • Developing such codes Requires large (10 to 30 professionals), multi-disciplinary, multi-institutional teams 5 to 10 years • Who will pay ~ 100 $M for each code? • Who will wait ~ 10 years for the development of these codes? • What’s the path forward?

5. Developing a Large, Multi-scale, Multi-effect Code Takes a Large Team a Long Time 2003 ~20

6. How do we realize the potential? • We need to pay attention to the business model • Simple codes are cheaper than computers (2-3 developers for a few years ~ 1 – 4 $M) • Agencies fund Scientific research, not code development • Large, Complex codes are more expensive than computers (~ 100 $M), especially large scale engineering codes! • Cheaper, simpler codes will run on tomorrow’s cheap workstations and small clusters that will be as powerful as today’s supercomputers • To justify supercomputers in 2020, we must explicitly fund code development because it’s no longer cheap

7. What Is Needed For Large-scale Computational Science And Engineering? We Need A Complete Problem Solving Capability: Computers Codes V&V Users Sponsors Sponsors

8. Getting Big Funding is About Establishing Value Added! • Value for sponsor: • Accurate and timely information that can be used to make decisions • Value for the customers (users) • Tools (software and hardware) that are easy to use and can yield reliable answers in a relevant timeframe. • Value for V&V • Tools whose accuracy has been established • Code developers • Computing environment that facilitates code development • Computers • Computers that balance performance and utilization with ease of use for users, V&Vers and code developers

9. How Do We “Market” CSE? • Must Establish That We Can Solve A Problem That People With Funding Need Solved! • Approach: • Problem • Benefit Of Solving Problem • Our Credibility To Solve The Problem • Vision For Solving The Problem • DoD Problem ― Major Acquisition Programs • Over Budget, Behind Schedule, Not Agile And Flexible, Performance Shortfalls • Return On Investment > 1 For Other Systems

10. Computational Research and Engineering Acquisition Tools and Environments (CREATE) CREATE Goal Is To Enable Major Improvements In The DoD Acquisition Process Detect And Fix Design Flaws Early In The Design Process Before Major Schedule And Budget Commitments Are Made Begin System Integration Earlier In Acquisition Process Increase Acquisition Program Flexibility And Agility To Respond To Rapidly Changing Requirements Improve The Ability Of DoD Institutions To Develop And Exploit Large-scale Computational Science And Engineering Tools STRENGTHEN ENGINEERING & TEST EFFORTS BY INJECTING COMPUTATIONALRESEARCH & ENGINEERINGFOR ACQUISITIONTOOLS & ENVIRONMENTS (CREATE) A B C IOC FOC CONCEPT REFINEMENT TECHNOLOGY DEVELOPMENT SYSTEM DEVELOPMENT &DEMONSTRATION PRODUCTION & DEVELOPMENT OPERATIONS & SUPPORT Concept Decision Design Readiness Review FRP Decision Review

11. F-18E/F Separated Flow Loss of Control • $360M 12-year Program to develop & deploy 3 computational engineering tool sets for acquisition engineers • Aircraft design tools: Aerodynamics, air frame, propulsion, control, early rapid design • Ship design tools: Early stage design, ship survivability, and hydrodynamics performance • RF Antenna design tools: RF Antenna performance and integration with platforms • Computational Infrastructure Explosion Survivability C4ISR Antennas in Network-Centric Battlespace

12. Immature Engineering Designs Are Major Contributors to DoD Acquisition Program Cost & Schedule Growth C R E A T E CAN HELP Impact on DOD is $10s of Billions

13. CREATE Is A Cross-service Program With Funding That Ramps Up Gradually POM08 POM08 13

14. How Did We “Market” CREATE? • Problem • Budget Overruns, Schedule slippage, performance shortfalls, rigid and slow process • Benefit Of Solving Problem • Reduced cost; faster delivery; good performance; rapid, flexible and agile process • Our Credibility To Solve The Problem • Track record, lessons learned, solid credentials • Vision For Solving The Problem • Develop and deploy computational engineering tools to optimize designs, detect design defects and fix them, respond rapidly to requirements changes, reduce testing by getting the design right the first time.

15. CREATE Is A Multi-institutional Program & Team • National Search  Key Positions • Acquisition Programs  Gaps • Gaps  Products • Products  Institutions and Staff OSD, Andre van Tilborg, DUSD(S&T) HPCMP Executing Agent, Cray Henry, Director Official HPCMP Advisory Panel CREATE Program, Doug Post (PM), Sara Arevalo, Analyst Air Vehicles Robert Meakin, (PM), Chris Atwood (Dep) RF Antennas Kueichien Hill (PM), David Van Veldhuizen (Dep) Project Boards Computational Infrastructure,David Fisher* (PM) Project Boards Ships Myles Hurwitz (PM), Vacant (Dep) Project Boards Project Boards Requirements Team (Govt Only) Code Development Tools, Environments, and Services KESTREL Eglin AFB, Scott Morton Hydrodynamics, NSWC-Carderock, Joseph Gorski Test Team (Govt, Industry) Education and Training Shadow-Ops, NAVAIR, Pax River Joe Laiosa (act) Product Team 1 (Govt, Industry, Academia) Product Team 2 (Govt, Industry, Academia) T. Blacker, Geometry and Meshing, SNL Survivability NSWC-Carderock, Tom Moyer Rotorcraft, Ames, R. Strawn (collab.) Application Advisory Panel (Govt, Industry, Academia) Collaboration Tools and Infrastructure Automated Structural Layout Technical Advisory Panel (Govt, Industry, Academia) Rapid Design, NAVSEA 05D, Steve Wynn Software Engineering Propulsion Integration *Govt includes DoD, DoE, NASA *CREATE Chief Engineer

16. Coordinating collocated code development by one institution has proven very challenging Coordinating non-collocated code development by multiple institutions will be even more challenging Establish the right culture, behavior and control Form a team whose members have trust and respect for each other and a strong commitment to the success of the project Provide support for collaboration tools (hardware, software, and user help). Effective desktop video communication Effective daily communication Establishing a Multi-Institutional, Multi-Disciplinary Collaboration Is a Daunting Challenge • Propose an aggressive program to develop and deploy collaboration tools and methods, budgeting up to $750k/year • Firewall issues have made this more difficult than we planned

17. CREATE is Designed for Engineers Results & Analysis Develop Code Code Use CSE Developers CSE Developers Journals CSE Developers Operations Design Engineers Test & Evaluation Test Fails Develop Code Code Use Test Model Build System Build Model CREATE Developers Model Builders Manufacturers • Computational Science is challenging: • Develop a complex code, apply it to study a scientific research problem, and publish the findings. • CREATE is attempting something much more challenging: • And the development will done by non-collocated, multi-institutional teams.

18. Identified Customers and Their Requirements and Needs • Working with the acquisition communities to identify the capability gaps • Identifying the gaps that computational engineering can fill • Developing concepts for computational engineering tools that could fill those gaps • Selecting tools that can be developed by the CREATE program given the available resources and schedule • Documenting these requirements in Initial Capability Documents (ICD) • Begun the process of validating the requirements in the ICDs with the customers

19. CREATE-AV 0110 01010 110010001011100 A B C IOC FOC CONCEPT REFINEMENT TECHNOLOGY DEVELOPMENT SYSTEM DEVELOPMENT &DEMONSTRATION PRODUCTION & DEVELOPMENT OPERATIONS & SUPPORT Concept Decision Design Readiness Review FRP Decision Review Example: CREATE Air Vehicles We’ve turned the crank once on this process – and are implementing Step-8 now

20. CREATE-AV 0110 01010 110010001011100 Acquisition Processes That HPC Can Improve Targeted Acquisition Processes

21. CREATE-AV 0110 01010 110010001011100 Separation Turbulence Aero-structure interaction Jets Wakes Shocks Vortices Many Processes Need The Same Physics • Numerous acquisition processes . . . A commonality of governing physics makes it possible for a relatively small set of CSE software tools to impact a large number of important acquisition processes.

22. CREATE-AV 0110 01010 110010001011100 Identified Acquisition Program Gaps To Select CREATE Products • Proposed Computational Engineering Software Products and Activity • ASL: AUTOMATED STRUCTURAL LAYOUT. Next generation software to enable CSE insertion into early phase acquisition, advanced conceptual design, and virtual prototyping • KESTREL: Next generation high-fidelity multi-physics simulation for FIXED-WING air vehicles • HELIOS: Next generation high-fidelity multi-physics simulation for ROTARY-WING air vehicles • PAI: Next generation software to enable high-fidelity analysis of AIRFRAME/PROPULSION INTEGRATION • SHADOW-OPS: Primary mechanism to validate AV CSE software products and process changes to targeted acquisition workflows; transition CREATE-AV technology into acquisition workforce; and to build bridges between AV CSE software development teams and targeted acquisition organizations. Technical & Development Transition & V&V

23. FY03 OPNAV Sponsored Cruiser Concept CREATE Ships has three subprojects • Ship Hydrodynamics • Accelerate and improve all stages of ship hydrodynamic design by providing the next generation of computational modeling and simulation tools. • Ship Shock, Damage, and Survivability • Supplement full ship shock and live fire effect tests with computational analysis • Rapid Design Capability and Design Synthesis • Develop and deploy computational tools for rapid design and assessment of candidate ship designs to avoid cost versus capability mismatches

24. RF Antenna Project projects focus on meeting near and long term needs • Requirements Team--Identify evolving needs • Test Team--Test and transition computational tools • Customer Interface Team--Lower thresholds for users • Near Term Capability Team--Improve legacy tools • Long Term Capability Team--Develop tools to utilize next-generation computers • Technical Advisory Panel--Ensures technical soundness of tools • Application Advisory Panel--Ensure that the tools meet acquisition program needs and support Board of Directors

25. Strategy for CREATE-Computational Infrastructure VIPR • Near-term • Provide state-of-practice development environment • Collaborative intranet for CREATE developers & managers • Best available software development tools • Shared repository for CREATE artifacts • Code repository & backup mechanism • Configuration, version, and release control [SVN] • Issue tracking [JIRA] • Interactive database [Deki Wiki] • Collaboration tools (multiple candidates available for test) • Common run-time architecture • Shared agreement on software development process • Training & support for CREATE projects • Begin development of shared acquisition tool components • Long-term • Transition environment to one specialized to development of production quality physics-based CSE tools • Address broader set of shared acquisition tool components • Strive for SE methods & tools better suited to CSE development 25 11/17/2014

26. Requirements & Goals Mesh Generation Production Runs Analysis of Results Decisions & Final Design Concept & Design Including Geometry Manual or automated Design Iterations Problem Generation takes up to 90% of the calendar time • Must reduce problem generation time (geometry and mesh) from weeks—months to minutes • Brought in T. Blacker from SNL to develop a program and begin to execute it

27. Air-Vehicle Problem Set-Up: Complex geometry and flow physics modeling is a key barrier to application of Computation-Based Engineering in all phases of acquisition. Unsteady time dependent body motion with complex flow interaction problems P-3C Aircraft, weapon and interior slice of volume grid for simulation of unsteady 6DOF weapon bay release Close up of weapon in bay surface and flowfield discretization

28. Two Major Challenges for Engineering Codes • Engage The Acquisition Community From The Beginning And Get The Requirements Right • Create Shadow Operations Group To Use Tools To Support Acquisition Programs And Determine User Needs • Adds Value, And Tracks User Requirements As They Evolve • Develop The Right Codes For The Short And Long Term • Provide Value And Impact In The Short Term (2008-2010) • Next Generation Codes For The Long Term (2010-2019), designed for Supercomputers in 2020 (109 GFLOPS, 109 processors) • Solution • Develop Light-weight Integrated Design Environment (IDE) Around Legacy Codes—Interface and linkage packages • Transition IDE Capability To Acquisition Programs • Apply Lessons Learned To Design And Develop Next Generation Codes for Supercomputers in 2020

29. Prototype Codes Will Be Gradually Replaced With Next Generation Codes Short Term Deliverables Long Term Deliverables Next Generation Codes For Computers in 2020 Existing Legacy Codes Relative Code Development Effort Delivered Code Capability Knowledge transfer Users 2008 2010 2013 2016 2019 Time ShadowOps Engages Customers, Tracks Customer Needs And Requirements

30. Problem Generation Interface link Interface link Output Analysis Incremental Early Delivery With Long Term Goals And Deliverables — Kestrel Example Fly the Airplane on the computer 1 to 3 years Time step / iteration loop Air Frame Structural Mechanics Aircraft Control Aerodynamics (Air Flow) • 2008-2009 • Build light-weight software infrastructure to link legacy codes airframe response airframe loads Legacy CSM codes NASTRAN LS-Dyna ANSYS DyTran ABAQUS IDEAS … Legacy CFD codes Cobalt Overflow AVUS Fluent ANSYS CFX… 101-3 GFLOPS 102-3 Cores • 2010-2019 • Replace legacy codes with next generation codes (109 GFLOPs) 5 – 10 years Next Generation CFD Codes Next Generation CSM Codes 109 GFLOPS And 109 Cores Build And Test New Codes To Replace Legacy Codes Separated Flow Loss of control

31. Kestrel Single Executable Single Executable Additional Additional Of Modules Of Modules Executables Executables Fluid Fluid - - Structure Structure Rigid Rigid - - Grid Grid Structural Structural CREATE-AV CFD CFD 0110 01010 110010001011100 Interface Interface Move Move Solver Solver Solver Solver Integrated Force & Integrated Force & CFD CFD Mesh Mesh Mesh Mesh Moment Calculator Moment Calculator Solver Solver Adaptation Adaptation Deformer Deformer Infrastructure Infrastructure Aircraft Aircraft On On - - the the - - Fly Fly Engine Thrust Engine Thrust 6DOF 6DOF Autopilot Autopilot Autopilot Autopilot Trim Trim Visualizer Visualizer Model Model Store Store - - Release Release Control Surface Control Surface Prescribed Prescribed Constraints Constraints Deflection Deflection Motion Motion Fixed Wing Virtual Aircraft

32. Batch Interactive What is the right trade-off? Batch provides greatest computer utilization User waits for the computer Interactive maximizes human interaction Computer waits for the user Code development is mostly done interactively Science runs are mostly done with batch systems, and have been for last 60 years What’s the best model for design calculations?

33. Code Development is interactive Computational Science Workflow Test Component Optimize Component Validation Expts. Debug Component Validation Tests Set global Requirements Write Component Verification Tests Select Programming Model Analyze Results Identify algorithms Develop Approach Develop Approach Customer input Detailed Design Detailed Goals Recruit Team Decide; Hypothesize Formulate questions Define Goals V&V Formulate questions Identify Customers Develop Code Production Runs Complete Run Schedule Runs Optimize runs Execute Runs Initial Analysis Analyze Run Store Results Define tests Document Decisions Make Decisions Identify Next Run Setup Problems Identify Uncertainties Identify Next Step Define General Approach Regression Tests Identify Models Computing environment Upgrade existing code or develop new code Not the WaterFall Model! • Requirements • Design • Code • Test • Run ―D. E. Post, R. P. Kendall, Large-Scale Computational Scientific and Engineering Project Development and Production Workflows, CTWatch (2006), vol.2-4B,pp68-76. 33 11/17/2014

34. Requirements & Goals Mesh Generation Production Runs Analysis of Results Decisions & Final Design Concept & Design Including Geometry Manual or automated Design Iterations Computer requirements for acquisition computing depend on designer workflows • Designer Workflow is iterative • Mesh generation is a major bottleneck • Geometry generation is a key (sometimes neglected) requirement

35. Operating System in Control Generate input deck Waits in job queue Job runs Analysis of Results Decisions & Final Design Submit Job CDC 7600 Standard scientific computing workflow is linear • Batch operating system without a human in the loop once job submission is complete • Just like the good old days.

36. Design Automated Automated Design Different Services may require a mixture of computers linked in a local network License Server Workstation Mesh Generation Cluster • Service Oriented Architecture (SOA) Approach • Loosely coupled codes passing data through XML files, and other approaches Geometry (CAD) Solver 1 (CFD) Workstation Data Store Designs, Results.. Big Cluster Requirements Driver Solver 2 (FE CSM) Cluster Decision Results Analysis (Viz) Big Cluster Workstation Cluster Boeing, industry…―R. Haimes, MIT

37. The Utility Of High Performance Computing To The DoD Is Increasing • DoD Values HPC Solely For Its Benefits To The DoD • We Have To Continually Establish The Value of HPC To The DoD • As Computing Capability Evolves Our Business Model And Our Service Model Must Evolve To Meet Changing Environments and Requirements • There Is Tremendous Potential For Making A Very Positive Impact, But The Challenges Are Immense • It’s Going To Be A Lot Of Fun