1 / 31

Présentation des grilles de calcul et initiation à la grille de calcul EGEE Lyon, 17 mars 2010

Présentation des grilles de calcul et initiation à la grille de calcul EGEE Lyon, 17 mars 2010 Sylvain Reynaud. GOS. NAREGI. La grille dans le monde. middleware. infrastructures. La grille en Europe (bientôt). middleware. GOS. NAREGI. infrastructures.

khotchkiss
Download Presentation

Présentation des grilles de calcul et initiation à la grille de calcul EGEE Lyon, 17 mars 2010

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Présentation des grilles de calcul et initiation à la grille de calcul EGEE Lyon, 17 mars 2010 Sylvain Reynaud

  2. GOS NAREGI La grille dans le monde middleware infrastructures JSAGA

  3. La grille en Europe (bientôt) middleware GOS NAREGI infrastructures JSAGA

  4. La grille dans les NGI (exemples) middleware GOS NAREGI infrastructures JSAGA

  5. Standardisation • Besoins et concepts communs • Une communauté d'industriels et de chercheurs définit des standards • pour l'architecture globale d'une grille (Open Grid Service Architecture) • pour chaque composant d'un middleware de grille JSAGA

  6. Sécurité grid-proxy-init VOMS CE, SE… Authentification Autorisation (décision partagée) Interopérer avec les infrastructures de sécurité locales (sans les compromettre) Délégation Négociation Isolation des utilisateurs Traçabilité, détection d’intrusion LCAS / LCMAPS JSAGA Source: The Open Grid Services Architecture, Version 1.5

  7. Servicediscovery Loadbalancing Problemdetermination Executionmanagement Accounting Resourcereservation Applicationmonitoring Systèmes d'information Architectures et interfaces dépendent de… type d'information (statique/dynamique, fréquence de rafraîchissement) type d'utilisation (critères de QoS différents) BDII InformationServices Registry Asynchronous notification Consumers Producers Retrieval • Reliable • Secure • Efficient Logger R-GMA LB JSAGA Source: Hiro Kishimoto

  8. SRM Data Resources Managed Storage Serviceinterface Resourceinterface Gestion des données LFC, AMGA FTS Non-OGSA client APIs & other services Gestion de stockage Accès aux données Gestion de réplicats Catalogues de méta- -données Transfert de données Gestion de cache Transfer Lookup Transfer Registries Sink/ Source Description Data Management Other Data Services Storage Storage Management Access Access Description Sink/ Source Data Resources Transfer Protocols GridFTP JSAGA Source: The Open Grid Services Architecture, Version 1.5

  9. ? ? ? Information Service WMS Gestion de l'exécution Description du job Sélection de la ressource Soumission, suivi et contrôle du job Déploiement JDL Application Contents Service Config Desc, Deployment & Lifecycle Mgmt JSDL LCG-CE, CREAM-CE JSDL Job Manager Basic Execution Service resource user BDII Resource Selection Service JSAGA Source: Keisuke Fukui (ACS-WG)

  10. Interface utilisateur gLite-UI VOMS user WMS, LCG-CE, CREAM-CE SRM, gsiftp LFC, AMGA BDII JSAGA Source: S Jha, T Kielmann, A Merzky

  11. SAGA est une interface standard spécification de l' orientée objet simple pour 80% des besoins, 20% des fonctionnalités suffisent intégrée "look & feel" commun à tous les packages de l'API high-level les détails du fonctionnement du middleware sont cachés uniforme indépendante du middleware sous-jacent multi-langages implémentations Java et C++ wrappers pour python stable spécification SAGA 1.0 (15 janvier 2008) bientôt un 1er update mineur (ne casse pas la compatibilité ascendante) Interface utilisateur JSAGA

  12. Interface utilisateur • Exemple de code en Java (data management) Session session = SessionFactory.createSession(true); URL url = URLFactory.createURL("gsiftp://cclcgseli01.in2p3.fr/tmp/"); // instanciation de l'objet SAGA à l'aide d'une fabrique NSDirectory dir = NSFactory.createNSDirectory(session, url); // utilisation de l'objet SAGA List<URL> result = dir.list(); for (URL r : result) System.out.println(r); liste le contenu du répertoire gsiftp://… JSAGA

  13. Interface utilisateur • Exemple de code en Java (execution management) Session session = SessionFactory.createSession(true); URL url = URLFactory.createURL("wms://lapp-wms02.in2p3.fr/…"); // instanciation de l'objet SAGA à l'aide d'une fabrique JobService svc = JobFactory.createJobService(session, url); // utilisation de l'objet SAGA Job job = svc.runJob("/bin/date --utc"); job.waitFor(); System.out.println(job.getState()); soumet un job et attend la fin de son exécution JSAGA

  14. accès grille de super-calculateurs augmenter le nombre de ressources accessibles cluster JSAGA disponibilité de logiciels payants JSAGA ( ) débuguage, mise au point du job faible latence (e.g. job de consolidation), interactivité confidentialité pour certains traitements JSAGA

  15. EGEE D-Grid JSAGA jsaga-context-init JSAGA SAGA job WMS gsiftp Gatekeep. Globus VOMS VOMS WMS GridFTP Gatekeeper Unicore LCG-CE CREAM-CE JSAGA

  16. out.txt EGEE D-Grid JSAGA jsaga-job-run –Executable /bin/hostname –Output out.txt -FileTransfer gsiftp://…/out.txt<out.txt -r wms://lapp-wms02.in2p3.fr:7443/glite_wms_wmproxy_server JSAGA SAGA job WMS gsiftp Gatekeep. Globus VOMS JDL VOMS WMS GridFTP Gatekeeper Unicore LCG-CE CREAM-CE JSAGA

  17. out.txt EGEE D-Grid JSAGA jsaga-job-run –Executable /bin/hostname –Output out.txt -FileTransfer gsiftp://…/out.txt<out.txt -r gatekeeper://gt4-fzk.gridka.de:2119/jobmanager-pbs JSAGA SAGA job modified job wrapper script WMS gsiftp Gatekeep. Globus VOMS RSL VOMS WMS GridFTP Gatekeeper Unicore LCG-CE CREAM-CE JSAGA

  18. JSAGA interface Architecture logicielle implementation JSAGA

  19. Ready to use software, adapted to targeted scientific field A single interface for using all middlewares As many interfaces as ways to implement each functionality easy to develop new plug-in efficient usage of legacy APIs As many interfaces as used technologies end user application developer SAGA Java binding core engine JSAGA adaptors interfaces JSAGA adaptor developer adaptors JSAGA interface Architecture logicielle user applications legacy APIs implementation JSAGA

  20. Minimal dependencies on external libraries external tools e.g. no gLite-UI operating system tested on done construction planned gatekeeper gLite-WMS wsgram unicore6 cream naregi ssh local gatekeeper gLite-LB wsgram unicore6 cream naregi ssh local InMemCred Globus G. Legacy G. RFC820 MyProxy VOMS JKS SSH Login / pwd X509 Provided adaptors Grid techno. Others Execution Job control Job monitoring Grid techno. Others Data Logical Physical rns lfn srb irods catalog rbyteio gsiftp srm srb irods http https sftp ftp file zip cache Grid techno. Others Security JSAGA

  21. Adaptors validated by a middleware-independent SAGA test suite maven 'archetype' to generate skeleton of new adaptor project Automated build process download and install build tools external libraries generate source code execute test-suites unitary tests integration tests generate project web site documentation reports generate installer GUI extract library dependencies from maven project description Software quality # SAGA protocols test-suite configuration gsiftp.base=gsiftp://ccrugceli01.in2p3.fr/tmp/ gsiftp.base2=gsiftp://agena.c-s.fr/grid/tmp/ gsiftp.context=OpenPlast_proxy https.base=http://grid.in2p3.fr/html/Private/ https.context=Web_X509 file.base=file:///c:/tmp/ file.base2=file:///c:/ JSAGA

  22. Installer GUI JSAGA

  23. LGPL license for the core engine and for most adaptors Optional licenses for adaptors having external dependencies, which license is not compatible with LGPL then, end-user must… either accept the terms of the license agreement, or go back to previous screen and uncheck the adaptor(s) Licenses JSAGA

  24. data management jsaga-cat jsaga-cp jsaga-ls jsaga-mkdir jsaga-mv jsaga-rm jsaga-rmdir jsaga-stat jsaga-test jsaga-logical Applications – Commandes en ligne • JSAGA provides command line interfaces for… • security • jsaga-context-init • jsaga-context-info • jsaga-context-destroy • execution management • jsaga-job-run • jsaga-job-status • jsaga-job-cancel • jsaga-job-output • Also useful as code examples to start with SAGA JSAGA

  25. Applications – JUX JSAGA

  26. Elis@ a web portal for submitting jobs to industrial and research grid infrastructures JJS (Java Job Submission) a tool for running efficiently short-life jobs on EGEE SimExplorer a set of tools, including a workflow engine providing distributed computing facilities, for managing simulation experiments jCAE(Java CAE) Computer Aided Engineering on distributed computing environment / Applications – autres exemples JSAGA

  27. Thank you ! JSAGA

  28. Close to application developer needs object-oriented high-level uniform interface for all the supported technologies Design objectives easy to use … but << certainly not simple to implement >> (T. Kielmann) Close to existing middleware APIs service-oriented low-level as many interfaces as ways to implement each functionality + optional interfaces Design objectives easy to extend wrap m/w functionalities only code size: engine= 2x adaptors enable efficient usage of middleware APIs engine choose best interface SAGA adaptors interfaces JSAGA Design of adaptors interfaces JSAGA

  29. As many interfaces as ways to implement each functionality Example: ways to monitor jobs API mode poll job status listen to notifications about job status changes API granularity individual jobs list of jobs jobs filter (e.g. by user, by date, by tag…) done construction planned SAGA adaptors interfaces JSAGA Design of adaptors interfaces getState waitFor query status for individual job listen status for individual job query status for filtered jobs Job monitoring gatekeeper gLite-WMS wsgram unicore6 cream naregi ssh local JSAGA

  30. Optional interfaces for optimization Example: ways to copy a file data read/write methods either stream methods or get/put methods data copy e.g. enable third-party transfer delegated transfer done construction planned SAGA adaptors interfaces JSAGA Design of adaptors interfaces copy data reader data writer data copy Physical rbyteio gsiftp srm srb irods http https sftp ftp file zip cache JSAGA

  31. SAGA hides most of the differences between the technologies Information on remaining differences is available on JSAGA web site, for each adaptor module Frequently Asked Questions Configuration example with command line jsaga-help supported protocols associated security contexts context attr. usage/defaults usage: jsaga-help -a <ctxId>.<attr> | --config | -d <mode> | -h | -j <mode> | -s <mode> | -v [-D <ctxId>.<attr>=<value> where: -a,--attribute <ctxId>.<attr> Output the value of security context attribute --config Output the effective configuration -d,--data <mode> Information about data protocols. <mode> = service | context -D <ctxId>.<attr>=<value> Set context instance attribute (e.g. -DVOMS[0].UserVO=dteam) -h,--help Display this help and exit -j,--job <mode> Information about job services. <mode> = service | context -s,--security <mode> Information about security context instances. <mode> = usage | default | missing -v,--version Output version information and exit Provided adaptors JSAGA

More Related