A dynamic parallel coupler E-Mail : palm@cerfacs.fr URL: http://www.cerfacs.fr/~palm

1. A dynamic parallel coupler E-Mail : palm@cerfacs.fr URL: http://www.cerfacs.fr/~palm Andrea Piacentini Rennes - November 2007

2. Overview • The genesis of the PALM project • Code coupling issues • PALM main features • How to set up a PALM coupled application • Some PALM applications

3. The genesis of the PALM project Origin of the PALM project: in 1996 with the operational ocean forecasting project MERCATOR. A data assimilation suite can be designed and implemented as a coupling. The different tasks - running a forecast, apply the observation operator, computing the misfit, approximate the error statistics, invert matrices, minimize a cost function and so on - are thought of as independent pieces of code to be assembled within a portable, flexible and efficient framework.

4. The genesis of the PALM project • The PALM Team at CERFACS was in charge of the development of the software backbone of the MERCATOR project. • First a fully featured prototype was implemented using the available technology at the end of the 90's. At the issue of this phase the first version of the coupler was released with the name PALM_RESEARCH. • Based on the expertise developed on the prototype, the final MPI2 based version of PALM was designed and implemented. It is currently released with the name PALM MP. • Thanks to the flexibility of the approach, the non-specific formalism used to describe the coupling interfaces and the portability of the software, PALM established itself as a natural choice for all sorts of dynamic parallel coupling and code assembling projects.

5. Overview • The genesis of the PALM project • Code coupling issues • PALM main features • How to set up a PALM coupled application • Some PALM applications

6. Code coupling issues • Why coupling scientific computing codes? • To model a system as a whole Ocean - atmosphere Fluid - structure Atmosphere Ocean + NEMO ARPEGE Flux exchanges Climate coupled modelling • To set up new applications from existing codes Parallel post-processing Optimisation: algorithm on top of a model Research of the optimal position of the intakes in a combustion chamber • To build modular applications by assembling their elementary components (data assimilation…) Split the problem in order to reach a better task scheduling, taking advantage of the intrinsic parallelism of the application

7. Code coupling issues Code 1 Code 2 data • What does it imply? • Manage executions (serial or parallel) • Drive the information exchanges between codes Commitments: -Easy set-up: non intrusive interfaces -Efficiency: no loss in performances -Portability: standardtechnical solutions -Flexibility: test different configurations with a few changes -Genericity: reuse components in other couplings

8. Code coupling issuesImplementation • First solution: code merging prog1 & prog2 Program prog1 … Call sub_prog2(in, out) … end Program prog2 Subroutine sub_prog2(in,out) … end • Very efficient in computing time (data exchange by reference, …) • BUT: • Integration problems (the code independency is not respected): common blocks, logical units, compilation options, languages, … • No flexibility (problems in maintenance, evolution, …) • Potential memory wastes • Task parallelism cannot be implemented • N.B. This is merging and not coupling (parallelism not taken into account)

9. Code coupling issuesImplementation • Second solution: use a communication mechanism (PVM, MPI, CORBA, pipe UNIX, files, …) Prog1 Prog2 Program prog2 … Call XXX_recv(prog1, data) end Program prog1 … Call XXX_send(prog2, data) end • More or less efficient depending on the mechanism • BUT: • Coupling is not generic • Good experience in parallel computing required • Lack of flexibility (parallelism, interpolation, …) • Not always portable depending on the chosen mechanism • Too complex with more than two codes and many exchanges

10. Code coupling issuesImplementation OASIS Third solution: choose a coupler PALM Prog1 Prog2 • The coupler starts the executables in a sequence or in parallel • Data exchanges are invoked by general primitives PUT/GET • The coupler uses a portable and efficient communication protocol (MPI) • The coupler provides interpolation tools, data redistributions, … • Hugegainin integration ease, performances, maintenance and evolution of the coupled application.

11. Overview • The genesis of the PALM project • Code coupling issues • PALM main features • How to set up a PALM coupled application • Some PALM applications

12. PALM main features First function of the coupler: Managing the componentsexecution

13. Code1 Code2 Spawning and execution Classical couplers PALM coupler Static coupling Dynamic coupling condition Code1 Loop time Code4 Code3 Code2 Both codes are started at the beginning of the simulation and exit together at the end of the application Example: Ocean/Atmosphere coupling in climate modelling Parallel or serial executions Loops and conditional executions Complex algorithms can be easily end efficiently implemented Dynamic resources management (processors and memory)

14. Spawning and execution How to describe complex algorithms? A Graphic User Interface for: • Representing the components • Handling the parallelism • Describing loops and conditions • Describing the communications (exchanges) With all the benefits of a G.U.I. as, for instance: • Coherency checks on the exchanged data • Pictorial flow chart representation of the algorithm • Performance analysis, debugging and run-time monitoring

15. Snapshot of the G.U.I.

16. PALM glossary • Unit: computational component • - Code • - Module of a code • - Data loaders • - Pre-defined units (F77/F90 subroutines, C or C++ functions, Java, Python, Matlab and so on through F90 or C/C++ interfaces) 32 • Parallel unit

17. PALM glossary • Branch: sequence of units and instructions => algorithm

18. PALM main featuresTwo levels of parallelism Task parallelism: branches Parallel computing “simply drawing” Distributed components: units Components assembling: blocks

19. PALM main features Second function of the coupler: Driving the data exchanges

20. PALM main features “End point” communication paradigm •  NO explicit indication of the receiving units on the producer side, NOR of the producer on the receivers side In the sources of the units: potentialsending or reception Announce data production: PROVIDER CALL PALM_Put (…) Register a data request: CLIENT CALL PALM_Get (…)

21. Communication: exchange of an object between two units PALM glossary • Object: data chunk produced or received by a unit N.B.: “Packed data” for coherent data batches can simplify the data flow representation

22. PALM main featuresCommunication mechanism Actual communications are described by connecting the plugs in the graphic interface Handshaking between clients and providers

23. PALM main featuresCommunication mechanism A PALM_Put has been issued on the provider side A PALM_Get has been issued on the client side A “tube” has been drawn between the two plugs A communication takes place only if

24. PALM main featuresIndependent units (0) (1) (2) (3) 00 10 20 30 21 31 01 11 (0) (0) (1) (1) Parallel data exchanges • Automatic remapping of the objects if the distribution on the provider side is different from the distribution on the client side

25. PALM main featuresCommunication mechanism Direct Send (MPI) Storage in a local mailbox (memory to memory copies) Storage on a mailbox on the driver (MPI) Optional use of files for too large objects

26. Overview • The genesis of the PALM project • Code coupling issues • PALM main features • How to set up a PALM coupled application • Some PALM applications

27. How to set up a PALM coupled application Preliminary step: How a code becomes a PALM unit ?

28. How a code becomes a PALM unit • Pre-requisites: • Code sources should be available • (it is possible to couple black boxes, but efficiency is much lower) • Codes have to be written in a compiled language (C, C++, FORTRAN 77 ou 90) or interfaced with such languages (Python, Java) • Codes must run under Linux/Unix

29. How a code becomes a PALM unit For all kinds of units • Replace PROGRAM (FORTRAN) by subroutine or main (C) by a function name For a parallel unit • Skip the calls to MPI_INIT and MPI_FINALIZE • Replace MPI_COMM_WORLD by PL_COMM_EXEC • Replace the calls to STOP, EXIT by PALM_Abort

30. How a code becomes a PALM unit unit space object distributor Create an identity card: !PALM_UNIT \ ! -name unit name\ ! -functions { function to be invoked } ! -comment { optional comment } ! !PALM_SPACE \ ! -name space name \ ! -shape (.,., … ,.) \ ! -element_size PL_[INTEGER, REAL, DOUBLE_PRECISION, …] \ ! -comment { optional comment } ! !PALM_OBJECT \ ! -name object name \ ! -space name of its space \ ! -intent [IN, OUT, INOUT]\ ! -time [ON, NO] \ ! -tag [ON, NO] \ ! -comment {optional comment } ! !PALM_DISTRIBUTOR ! -name distributor name \ ! -type [regular, custom] \ ! -nbproc number of processes in the distribution \ ! -function name of the distribution function \ ! -comment {optional comment }

31. How a code becomes a PALM unit Error code Variable pointing to the data associated to the object Integer further tag for the current instance of the object Integer time stamps for the current instance of the object String with the name of the object as it is declared in the i.d. card String with the name of the object space Add the PALM primitives: CALL PALM_Put (‘space1’, ‘obj1’, time, tag , array, error) CALL PALM_Get (‘space2’, ‘obj2’, time, tag , array, error)

32. Pre-defined PALM algebra units Load the unit from the Algebra Toolbox Execution and communications are managed as if it were a user unit Definition: Pre-defined units interfacing common mathematical libraries (BLAS, LAPACK, …) applied on the fly to objects exchanged between units. (Interesting for units conversion, grid to grid interpolation, …) Users can compose (and share) their specific algebra units.

33. How to set up a PALM coupled application Main step: Once the units have been defined, the coupled application (algorithm and communications) is described in the graphic user interface PrePALM

34. Description of the coupling algorithm In the graphic interface the user define the execution sequences (branches), with control structures Observations Branch Model Branch

35. Description of the communication fluxes The actual communications are described in the graphic interface by linking the plugs corresponding to PALM_Put and PALM_Get KEY POINT: Analysis of the algorithm data flow and coherent definition of the interfaces between sub-models N.B.: PALM handles time interpolations for objects produced by units with a different time step.

36. Compile and run a PALM_MP application User defined units IHM PrePALM .f .c .f .f .ppl .pil … PALM library .f .c .f PALM library .a .a .a .a .a .a Palm_main Palm_main Entity_exe Entité_exe .a .a mpirun –np 1 palm_main MPI and algebra libraries MPI and algebra libraries

37. Debugging, analysis and monitoring tools • A PALM output file per branch with different verbosity levels for different classes of messages • A user defined debugging function, called on the PALM_Put and PALM_Get sides of an active communication to check the coherency of the objects • A performance analyser accounting for the elapsed and the CPU time • A graphic post-mortem replay or run-time monitoring of the application

38. Performance analyser in PrePALM

39. Post-mortem replay and run-time monitoring

40. To sum up PALM is a software dealing with couplings or with complex applications built upon sub-modules with no loss of performances. A friendly and simple G.U.I. to describe, implement and supervise applications. Many pros: • Evolution and maintenance of a complex application and its components are easier • A framework for the integration of existing independent codes in an application (sub-models assembling, multi physics coupling, collaborative development, …) • Take the best advantage from the intrinsic parallelism of an application

41. Overview • The genesis of the PALM project • Code coupling issues • PALM main features • How to set up a PALM coupled application • Some PALM applications

42. PALM in the SEVE project Sol Eau Végétation Energie: integrated modelling of land surface processes at different space and time scales (water and carbon cycles) Pool of actors:INRA CSE (Avignon) , INRA EPHYSE (Bordeaux) , LISAH (Montpellier) , CESBIO (Toulouse) , CNRM/GMME (Toulouse), LTHE, CEMAGREF, CERMICS Ongoing actions:Complex landscapes description and integrated interactions modelling : Segmentation of landscapes Hetherogeneity, different scales, discontinuities and linear structuresMulti-physics and multi-scale coupling, dynamic time stepping3D transferts modelling in different compartments at a landscape scale Models enhancement: data assimilation from teledetectionData bases

43. PALM for NitroScape ??

44. PALM in the ANR corayl projectCoupling combustion and radiatif chambre de combustion très hautes températures. phénomènes radiatifs prépondérants. calcul massivement parallèle. PhD Jorge Amaya CERFACS EM2C (Ecole Centrale Paris) LGPSD (Ecole des mines d’Albi) DOMASIUM AVBP Radiation equations in complex geometry Reactive Filtered Navier-Stokes equations. Two levels of parallelism. Independent coupled models Energy sources. Thermal fluxes on the walls. TemperatureSpecies Coupling scheme: n CFD time steps for each radiative step Parallel model Parallel model CFD - LES Radiative model

45. Model nesting:theCOMODO-PALM project IMAG/CNRS - Laurence Viry OPA 1/3°NATL3 OPA NorthAtlantic 1/15°BABY15 GascogneBay With PALM Before PALM • The components communicate by message passing (MPI) • Very aggressive method, with deep specific changes in the source coedes: rewriting at every model or algorithm update • Implementation of the two levels of parallelism rather cumbersome • Independent coupled components (MPI2) • Flexibility, modularity, performances • Run-time components spawning • Loops around parts of the application (iterative algorithms of domain decomposition) • Coupled units does not depend on the nesting algorithm • Automatic parallel communications • Code reuse in new developments

46. Temperature forecast in combustion chambers Eric Mercier (SNECMA Moteur), Florent Duchaine Itération 1 Itération 3

47. Shape optimisation MIPTO (INTELLECT D.M.) Florent Duchaine CERFACS - TURBOMECA

48. RANS – LES coupling - Florent Duchaine Code reuse

49. PALM in MERCATOR From regional to global, from OIto variational, from R&D to operational 1/15° 1/15° 1/4° 2° 1/3° Operational oceanography Allthe data assimilation suites have been developed withPALMThe same methods are used for the different resolutions. The upgrades of the direct model do not impact on assimilationThe PALM_MP coupled implementation is lessmemoryconsuming than a standard implementation

50. PALM in the ADOMOCA project 4D-VAR Assimilation de DOnnéesdans les MOdèlesde Chimie Atmosphérique Pool of the main actors in the atmospheric chemistry french researchSA, LA, LSCE, LPCE, LMD , IPSL, CERFACS, L3AB, CNRM, ACRI, CNES Choice of a commontool for sharing: components: models and operators methods: algorithms and diagnostics 3D-FGAT The assimilation algorithm is edited in the graphic interface PrePALM: no changes in the source codes. In this case, new units are plugged in: e.g. the tangent linear and adjoint models CERFACS CNRM