Grid Tools

1. Grid Tools Jai Dayal & Casey Rathbone

2. Outline • Web Services • Unicore • BOINC • Globus • Condor

3. Web Services “A Web service is an abstract notion that must be implemented by a concrete agent. … The agent is the concrete piece of software or hardware that sends and receives messages, while the service is the resource characterized by the abstract set of functionality that is provided.”

4. Web Services

5. Web Services • Remote Procedure Calls (RPC) - distributed function call interface • Service Oriented Architecture (SOA) - utilizes message passing to communicate to services also known as "message-oriented" services • Representational State Transfer (REST) - emulates HTTP (or similar protocols) to communicate stateful resources

6. Unicore - Overview • Single point of entry – security • Uses SSL • Uses web services • SOAP Messages • WSDL • Reputation for simple deployment

7. Unicore - Architecture • Most basic resource • A cluster, a PC, web service, etc Vsite

8. Unicore - Architecture • The gateway is the point of entry to the Unicore grid Gateway Vsites

9. Unicore - Architecture • Gateway + Vsites = Usite Usite Gateway Vsites

10. Unicore - Architecture UNICORE - Architecture Usite Gateway Client Vsites

11. Unicore - Architecture • Client communicates using SOAP • SOAP/HTTP binding Usite Gateway Client SOAP Message Vsites • SOAP contains: • Address of gateway • Port number • Vsite name • task descriptions

12. Unicore - Security • Gateway handles authentication • SSL used to authenticate the client • SSL also used by client to authenticate gateway • Both present digital certificates • Vsite handles authorization • Gateway passes along certificate in SOAP header • If client has privileges, Vsite authorizes • This model allows end to end security • Removes some strain from gateway • Trust between gateway and Vsite

13. BOINC - Overview Grid Public Resource Computing Owned by institutions/orgs Individuals own pc's Managed/used by pro's Used by commoners High level of trust Low (no?) level of trust • Grid computing vs. public resource computing

14. BOINC - Overview Grid Public Resource Computing Owned by institutions/orgs Individuals own pc's Managed/used by pro's Used by commoners High level of trust Low (no?) level of trust • Grid computing vs. public resource computing • BOINC Goals • Cheap and easy • Project variety • Public participation → reward points

15. BOINC - Overview Grid Public Resource Computing Owned by institutions/orgs Individuals own pc's Managed/used by pro's Used by commoners High level of trust Low (no?) level of trust • Grid computing vs. public resource computing • BOINC Goals • Cheap and easy • Project variety • Public participation → reward points • BOINC projects • SETI at home • Folding at home • Predictor at home

16. BOINC - Architecture BOINC

17. BOINC - Architecture BOINC Projects

18. BOINC - Architecture BOINC Projects Servers Scheduling Data

19. BOINC - Architecture BOINC Projects Tools Servers Scheduling Data

20. BOINC - Architecture BOINC Projects Tools Servers Scheduling Data Client

21. BOINC - Workflow • Client contacts Scheduling Server • Server returns XML Document • Workunit: inputs for a computation/job • Applications • Input files • Environment variables • Result: • Reference to workunit • Reference to output files • Client runs job and places results in output files

22. BOINC – Fault tolerance • Public Resource Computing is volatile • Users leave before job completing • Over-clocking can lead to bad results • Malicious users • Homogenous Redundancy • Computer architecture specific jobs • Transitioner • Identifies bad results • Validator • Determines canonical result • Assimilator • Handles canonical result • These daemons can run on different hosts

23. BOINC – Client settings • Client implements “local scheduling policy” • Utilizes maximum resources • Meet job deadlines • Respect clients resource allocation wishes • Offers some variety to the client • Anonymous platform • Used for open source projects • Clients can modify apps to fit their resources • Communicate with scheduling server using XML config file.

24. Globus Toolkit 4 • GT4 is a open-source toolkit developed to build grids • It provides full capabilities for sharing computing power and databases • Usage of Globus is extensive throughout the scientific community within NSF, DOE, DARPA, IBM, NASA, and Microsoft projects • The Large Hadron Collider project utilizes a Globus based grid for the European Data Grid.

25. Globus Toolkit 4 Globus Toolkit is a suite of Web-Services and Unix that controls provides an interface to submit, monitor, and cancel jobs on the Grid resources. Globus Toolkit is an interface that allows for jobs to be submitted and executed at the appropriate resources within the grid

26. Globus Toolkit 4

27. Globus Toolkit 4 Security • GT4 uses (Grid Security Infrastructure) GSI with an public key cryptography • GSI Functional Layers - Authorization, Authentications, Delegation, Message Protection, and Message Format

28. Globus Toolkit 4 Data Management • Data Management tools within the toolkit fall into either of two categories data replication and data movement • Data Replication consists of Replica Location Service (RLS)‏ • Data Movement consists of GridFTP and Reliable File Transfer (RFT)‏

29. GT4: Data Replication Replica Location Service provides the capability to track and maintain multiple locations of data across the grid.  It is a distributed registry system that allows users and applications to register the locations of data.

30. GT4: Data Movement • GridFTP - GT4 provides a server implementation, scriptable command-line interface and several GUI client interfaces • Reliable File Transfer (RFT) - A Web Services Resource Framework that schedules file transfers based on a set of criteria of when specific resources and bandwidth is available

31. Globus Toolkit 4 Execution Management • Community Scheduler Framework - Grid level meta-scheduling framework • Grid Teleoperation Control Protocol - A service interface that has been used to • Dynamic Accounts - This service provides functionality to create groups and accounts for specific jobs that need to be submitted • Grid Resource Allocation & Management - Used to locate, submit, monitor and cancel jobs on Grid computing resources.  It is the fabric that ties all the schedulers and job queues together to utilize the resources.

32. GT4: Community Scheduler Framework Open-Source implementation of OSGA-based meta scheduler Provides an interface to existing grid platforms making integration more efficient

33. GT4: Community Scheduler Framework The diagram below shows the basic layout of the scheduler framework http://www.globus.org/grid_software/computation/csf.php

34. GT4: Grid Teleoperation Control Protocol The GTCP is used to control heterogenous physical and computational simulations. It exposes the WSRF-compliant service interface used to control remote instruments and simulations and a interface to the backend GTCP to allow for additional control abstraction.

35. Globus Toolkit 4 Information Services Monitoring and Discovery Service is a suite of web services used to discover resources and services on a grid This service is dedicated to handling the complex task of organizing resources into Virtual Organizations (VO) http://www.globus.org/toolkit/docs/4.0/info/key-index.html

36. Globus Toolkit 4 Common Runtime • GT4 contains a multitude of implementations of libraries allowing the suite to be platform independent. • C Common Libraries • C WebService Libraries • Java WebService Libraries • Python WebService Libraries • XIO

37. Condor - Overview • University of Wisconsin – Madison • Open source • Linux, Solaris, Mac OS, Windows • No program modification • Jobs tend to be architecture specific • Designed on x86 → job run on x86 • Java applications possible exception • Overall concept is simple • Install it on each desired machine • Designate one as central manager • Practice is much harder • Installation is tricky • Lots of commands to learn

38. Condor • Central Manager • Detects idle machines • Matches job with available resources • Considers each processor as separate resource • Jobs • Submitted via “submit file” • Universe, executable name, inputs, outputs, error, log file • Sequential Jobs • Standard universe • Used for long jobs (weeks, months)‏ • Allows partial execution, pausing • Can relocate job to another machine • Vanilla universe • For short term jobs • Local universe • Submitter executes job • Parallel Jobs • Supports MPI and PVM

39. Condor - variety • Flocking • Connects multiple pools • If current pool has no resources, job “flocks” • Set “flock_to” variable in config file • Condor-G • Distributed flocking • Utilizes Globus • Universe = Globus • Also specify “globusscheduler” • Points to the globus job manager

40. References [1] D. P. Anderson, BOINC: A System for Public-Resource Computing and Storage.  In Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing. November, 2004.[2] S. Bagchi, E. Hung, A. Iyengar, N. Vogl, and N. Wadia, Capacity Planning Tools for Web and Grid Environments.  In Proceedings of the 1st international conference on Performance evaluation methodolgies and tools. October, 2006.[3] F. Hernández, P. Bangalore,  K. Reilly, End-User Tools for Grid Computing. In Proceedings of the first workshop on End-user software engineering. July, 2005.[4]I. Habib, Getting Started with Condor.  Linux Journal, Volume 2006, Issue 149, September 2006. [5]Menday, R., "The Web Services Architecture and the UNICORE Gateway," Telecommunications, 2006. AICT-ICIW '06. International Conference on Internet and Web Applications and Services/Advanced International Conference on , vol., no., pp. 134-134, 19-25 Feb. 2006 [6]Hellinckx, P.; Stuer, G.; Hendrickx, W.; Arickx, F.; Broeckhove, J., "Grid-user driven grid research, the CoBRA grid," Cluster Computing and the Grid Workshops, 2006. Sixth IEEE International Symposium on , vol.2, no., pp. 8 pp.-, 16-19 May 2006