300 likes | 314 Views
Explore the evolution in scientific software development and the Snark Project’s philosophy, goals, and benefits. Discover how this framework streamlines model integration and enhances productivity for geoscience.
E N D
Towards Rapid GeoScience Model Development Bill AppelbeVictorian Partnership for Advanced ComputingDirector, Software Development - ACcESSbill@vpac.orgMay 5th , 2002
Outline • Scientific Software – • Why is the software not keeping pace with the models? • How can we change the way we build scientific software? • Goals for scientific software development • The Snark Project • Philosophy, toolkits, off-the-shelf stuff • Snark-itechture • Snark Finite Element Particle in Cell (PIC) models • Snark Particle Models (SPH) • Model Integration • Other Collaborations • Timelines, deliverables, availability
Scientific Software – Requirements for New Models • Increasing demands on scientific software • Must be 3D • 2D just not realistic • 3D requires massive increase in computational requirements • 10,000x10,000 2D grid (108) is 1012 in 3D! • Therefore must be scaleable-parallel • Parallel programming is hard! • Only MPI-parallel programs scale to hundreds of processors • But MPI programs must be crafted by hand
Scientific Software – Requirements for Geoscience • Increasing demands to be able to rapidly • Quickly test new models • Incorporate different rheological models • Requires changing the solver, not just the data • Incorporate new observational data • Models must be linked and interoperate
Scientific Software – What's Wrong With This Picture • Scientific Software • Largely been built "from scratch" in Fortran and C • Each team or project builds their own specialized programs (codes) • To meet their own specific models • Codes start small, but soon grow by "feature-creep" • Evolve to "hero-codes" – maintenance and extension require heroic efforts, eventually all thrown away • "Cut and paste" programming • Little or no reuse or sharing of modules • Limited to simple, low-level libraries (e.g. BLAS) • Little or no change in productivity in decades
What Can We Learn from Commercial Software? • Commercial software development • An order-of-magnitude increase in productivity since 1990, by • Using Generators, Frameworks, extensible libraries • Component-based technology • For example, JBuilder • Why has scientific software development not kept pace?
What Inhibits Productivityin Scientific Software? • Focus on publications and grants • No incentive to build reusable software • No publication or grant credit for doing it • Lack of expertise in modern software engineering among computational scientists • Until recently, no effective large-scale reusable components
What Are We Doing About It in Australia? • Form a major national earth sciences consortium – ACcESS • All the major computational earth science research groups involved • ANU, UQ, Monash U, U of Melbourne, CSIRO, VPAC, ... • Get long-term Federal funding • Mostly for software • A fair amount of compute grunt
Goals for Geoscience Software • Reduce custom code by an order of magnitude • Thus reduce the time to build and test new models from months to days • Scaleable parallel programs • Create a framework for combined particle and grid codes • Model interoperability
Goals for Geoscience Software - Can it be done? YES! • At VPAC, we have prototypes demonstrating all the above • First versions should be released by mid-year
The Snark Project - Goals • Create a framework for geoscience software • Building on top of "state of the art" tools, components and libraries • Rebuild and re-engineer existing geoscience models into this framework • Integrate particle and mesh based models
Snark – Philosophy, toolkits, off-the-shelf stuff • "Productivity is measured by the number of lines of code NOT written" • Careful survey and testing of existing scientific software tools & components • See vpac.org/snark • The core component technology that Snark uses is PETSc • A powerful extensible library of solvers and support software from Argonne
Using PETSc • Strong support group and documentation • Large library of linear and non-linear solvers, grid support functions, etc. • Scaleable parallelism built-in! • Continuously being extended • We are adding particle libraries • The programmer can focus on "setting up the problem", not the algorithm details • The only drawback is the "learning curve" • See ... http://www-fp.mcs.anl.gov/petsc/
Snark – Other tools • Off the shelf tools are available for many ancillary functions • Graphics, viewing data sets • Vis5D, VU, .. • Grid refinement/triangulation • LaGRIT, Triangle • Domain decomposition • Parmetis • All these tools just hook in to PETSc!
Snark –Top-Level Architecture • Almost all grid and particle models fit into one top-level architecture! • There is an isomorphism between irregular grids and particle ensembles Grid Point Particle • Grid tools that can • partition and refine irregular grids • generate grid communication, etc work equally well for particle ensembles
Snark –Top-Level Architecture Initialize boundary conditions, grids/particle ensembles Partition across processors Derived from constitutive laws Update matrix Update particle properties PETSc Library Solve matrix Solve particle properties Timestepper & IO
The Snark Engine –Detailed Architecture MainRoutine Appl. Init. LaGriT/GMV ParMETIS FEM & Shape Fun. Ax = b setup & loading PIC Prometheus or FEAP PETSc Unstr. Mesh VPAC code Vis5D or VU Other code Visualization Post-Processing Evaluations TAO
Snark – Particle in Cell Models • Based on the Ellipsis PIC code • A "state of the art" code for creeping viscoelastic fluids • Uses a novel Particle in Cell model • Coupled Eulerian multigrid solver to Lagrangian particles • Used to model problems in earth sciences, e.g. mantle convection and crustal deformation • Limitations of Ellipsis • A classic "hero-code" • 30,000 lines of code, several variants • Not parallel, very difficult to adapt and maintain for new constitutive laws • 2D and 3D versions, but only for regular rectangular grids • 3D version has taken person-years to adapt
Ellipsis • Very efficient for rheological simulations on large timescales
Snark – Particle in Cell Models • The Snark version of ellipsis • 3,000 lines of code • An order of magnitude reduction • Does both viscoelastic and thermally driven convection • Fully parallel and scaleable • A quick Snark video
Snark – Particle in Cell Models (cont) • Current limitations of the Snark version of Ellipsis • Particle tracking implemented • But not integration/update of matrix from particle states • Coming soon! • Only 2D • But 3D version for regular rectangular grids fairly trivial • 3D and 2D supported by PETSc • Estimate delivery June/July of prototype • 3D irregular grid with arbitrary topology by end of 2002 • Programmer must hand code matrix setup/update for constitutive laws and boundary conditions • Medium-term solution will evolve to setup/update generators from operator specifications
Snark –The Mesh/Particle Continuum • There is a clear continuum between mesh and particle based codes • PETSc’s support of particle codes is in its infancy • To move towards unified particle/mesh codes, we are putting a “pure particle” code into Snark • SPH, Smoothed Particle Hydrodynamics • Used extensively in astrophysics • Ideally suited to: rock fracture, multi-phase mixed fluids • The code framework is readily extensible to incorporate other particle models such as DEM or LMSheath
Snark –The Mesh/Particle Continuum • 3D parallel version of SPH built, being extended to • Allow specification of arbitrary boundary conditions • Simulation of Tsunamis and dam fracture • Handle multi-fluids • Student working on simulation of pyroclastic flows • Integrate with PETSc framework of PIC codes • Extend to DEM codes • A quick video or two
Collaboration • Surface process modeling (ANU) • Reconstruction software (Monash) • LSM (UQ) • Griddles • Linking applications across the grid • With NO source language changes
Couple to Other Models • The Surface models (ANU, UoM);
Couple to Other Models • PlatyPlusPlus (ACRC, Monash U)
Couple to Other Models • LSM (QUAKES, UQ)
Timelines, Deliverables, Availability • Snark V1 will release midyear • There is a website with copious documentation –vpac.org/snark • Programmer must hand code matrix setup/update for constitutive laws and boundary conditions • Medium-term goal is to enable constitutive laws to be input as operator specifications • Key project • Be able to model the evolution of the Tibetan plateau
Thank You! VPAC's HPC Facility – 128 processor Compaq AlphaServer SC