Introduction to Grid Computing:

1. Introduction to Grid Computing: Ding Qing SSE USTC

2. Overview 1. Background 2. Globus Toolkit 3. Future directions 4. Related tools

3. 1. Background • Introduction • Towards global (Grid) computing • Grid Challenges and Technologies • Grid Architectures • Grid Applications

4. Introduction

5. Computing and Communication Technologies Evolution * HTC * P2P * PDAs Minicomputers * * PCs * Workstations * Mainframes * Grids COMPUTING * Computing Utility * PC Clusters * Crays * MPPs * WS Clusters * XEROX PARC worm * e-Science * e-Business * IETF * W3C * TCP/IP Communication * Ethernet * HTML * Mosaic * Web Services * Email * Sputnik * SocialNet * Internet Era * WWW Era * XML * ARPANET 2010 1960 1970 1975 1980 1985 1990 1995 2000

6. 2100 2100 2100 2100 2100 2100 2100 2100 2100 Scalable Computing PERFORMANCE + Q o S Administrative Barriers • Individual • Group • Department • Campus • State • National • Globe • Inter Planet • Universe Personal Device SMPs or SuperComputers Global Grid Inter Planet Grid Local Cluster Enterprise Cluster/Grid

7. Cluster 1 Scheduler Master Daemon LAN/WAN Submit Graphical Control Cluster 3 Execution Daemon Scheduler Clients Master Daemon Cluster 2 Scheduler Submit Graphical Control Execution Daemon Master Daemon Clients Submit Graphical Control Execution Daemon Clients Cluster of Clusters

8. http://www.sun.com/hpc/ Towards global (Grid) computing Grid: An Internet Computing model for coordinated resource sharing Grid enables: • Resource Sharing • Selection • Aggreation Metaphor: Applications draw computing power from a Computational Gridin the same way electrical devices draw power from an electrical grid.

9. database A Typical Grid Computing Environment Grid Information Service Grid Resource Broker Application R2 R3 R4 R5 RN Grid Resource Broker R6 R1 Resource Broker Grid Information Service

10. What is Grid ?(there are several definitions) • A type of parallel and distributed system that enables the sharing, selection, & aggregationof geographically distributed “autonomous” resources: • Computers – PCs, workstations, clusters, supercomputers, laptops, notebooks, mobile devices, PDA, etc; • Software – e.g., ASPs renting expensive special purpose applications on demand; • Catalogued data and databases – e.g. transparent access to human genome database; • Special devices/instruments – e.g., radio telescope – SETI@Home searching for life in galaxy. • People/collaborators. depending on their availability, capability, cost, and user QoS requirements. Widearea

11. Various Types of Grid Services • Computational Services – CPU cycles • SETI@Home, NASA IPG, TeraGrid, I-Grid,… • Data Services • Data replication, management, secure access--LHC Grid/Napster • Application Services • Access to remote software/libraries and license management—NetSolve • Interaction Services • eLearning, Virtual Tables, Group Communication (Access Grid), Gaming • Knowledge Services • The way knowledge is acquired and managed—data mining. • Utility Computing Services • Towards a market-based Grid computing: Leasing and delivering Grid services as ICT utilities. Utility Grid Knowledge Grid Interaction Grid ASP Grid Data Grid Computational Grid

12. Prominent Grid Drivers: Emerging e-Science and e-Business Apps • Next generation experiments, simulations, sensors, satellites, even people and businesses are creating a flood of data. They all involve numerous experts/resources from multiple organization in synthesis, modeling, simulation,analysis, and interpretation. ~PBytes/sec High Energy Physics Brain Activity Analysis Newswire & data mining: Natural language engineering Digital Biology Life Sciences Astronomy Quantum Chemistry Finance: Portfolio analysis Internet & Ecommerce

13. 2100 2100 2100 2100 2100 2100 2100 2100 Distributed instruments Distributed data E-Science Elements Peers sharing ideas and collaborative interpretation of data/results E-Scientist Distributed computation Remote Visualization Data & Compute Service

14. Molecules Protein Molecular Docking for Drug Design • It involves screening millions of chemical compounds (molecules) in the Chemical Databases to identify those having potential to serve as drug candidates. Chemical Databases (legacy, in .MOL2 format) [Collaboration with WEHI for Medical Science, Melbourne]

15. LHC – High Energy Physics Collaboration(fundamental investigation on the origin of mass)

16. Online System Tier2 Centre ~1 TIPS Australian Centre ~1 TIPS Tier2 Centre ~1 TIPS Tier2 Centre ~1 TIPS Tier2 Centre ~1 TIPS Storage Storage Storage Storage Storage 1 TIPS is approximately 25,000 SpecInt95 equivalents LHC Grid Computing Model ~PBytes/sec ~100 MBytes/sec Offline Processor Farm ~20 TIPS There is a “bunch crossing” every 25 nsecs. There are 100 “triggers” per second Each triggered event is ~1 MByte in size ~100 MBytes/sec Tier 0 CERN Computer Centre ~622 Mbits/sec France Regional Centre Italy Regional Centre US Regional Centre Asia Pacific Centre ~4 TIPS Tier 1 ~622 Mbits/sec Tier 2 ~622 Mbits/sec Melbourne~0.25TIPS Tier 3 Institute Institute Institute Physics data cache ~10 to 100 Mbits/sec Tier 4 Physicist desktop computers

17. Traditional Model Grid Based Model Enterprise Computing Applications Service Virtualization Layer & Load Balancing Email server Web server Database server Apps server Upgrade to a new server to handle more users Horizontal integration of Email, Web, Data, and Apps servers

18. Traditional (e.g., Oracle 9i) Tight/Vertical Integration of Storage, Database, Application Hosting Server, and Application Elements They reside on a single computing resource. Enhancing capability means a new investment: Replace a machine by new one or upgrade it. Can’t leverage existing resources. Expensive approach. Grid Based (e.g., Oracle 10g) Disintegration of Storage, Database, Application Hosting Server, and Application Elements They reside on a different resources in a Grid environment. Enhancing capability means: Leveraging existing resources Dynamic provisioning Cost-effective approach Oracle 10g: Towards Enterprise Grid Model

19. Security Data locality Resource Allocation & Scheduling Uniform Access System Management Resource Discovery Application Construction Network Management Grid Challenges and Technologies

20. Realizing the Grid The Grid Architecture

21. Virtual Organizations • Distributed resources and people • Linked by networks, crossing admin domains • Sharing resources, common goals • Dynamic R R R R R R R R R R R R R R VO-A VO-B

22. Virtual Organizations • Distributed resources and people • Linked by networks, crossing admin domains • Sharing resources, common goals • Dynamic • Fault tolerant R R R R R R R R R R R R VO-A VO-B

23. Grid Realization Steps • The integration of individual s/w & h/w components into a combined networked resource (single system image cluster). • Low-level middleware to provide a secure and uniform access to services provided by different resources. • User-level middleware to support application development and aggregation of distributed resources. • The construction of distributed applications.

25. APPLICATIONS Applications and Portals … Prob. Solving Env. Collaboration Engineering Web enabled Apps Scientific USER LEVEL MIDDLEWARE Development Environments and Tools Languages/Compilers Libraries Debuggers Monitors … Web tools Resource Management, Selection, and Aggregation (BROKERS) CORE MIDDLEWARE Distributed Resources Coupling Services Security Information Data Process Trading … QoS SECURITY LAYER Local Resource Managers FABRIC … Internet Protocols Libraries & App Kernels Queuing Systems Operating Systems Networked Resources across Organizations … Computers Networks Storage Systems Data Sources Scientific Instruments Layered Grid Architecture

26. Major Grid Projects and Initiatives

27. Australia Nimrod-G Gridbus GridSim Virtual Lab DISCWorld GrangeNet. ..etc Europe UK eScience EU Data Grid Cactus XtremeWeb ..etc. India I-Grid Japan Ninf DataFarm Korea... N*Grid Singapore NGP USA AppLeS Globus Legion Sun Grid Engine NASA IPG Condor-G Jxta NetSolve AccessGrid and many more... Cycle Stealing & .com Initiatives Distributed.net SETI@Home, …. Entropia, UD, SCS,…. Public Forums Global Grid Forum Australian Grid Forum IEEE TFCC CCGrid conference P2P conference Some Grid Projects & Initiatives http://www.gridcomputing.com

28. NetSolve mix-and-match Object-oriented Internet/partial-P2P Grid Computing Approaches Network enabled Solvers Economic-based Utility / Service-Oriented Computing Nimrod-G

29. Overview 1. Background 2. Globus Toolkit 3. Future directions 4. Related tools

30. The Role of the Globus Toolkit • A collection of solutions to problems that come up frequently when building collaborative distributed applications • Heterogeneity • A focus, in particular, on overcoming heterogeneity for application developers • Standards • We capitalize on and encourage use of existing standards (IETF, W3C, OASIS, GGF) • GT also includes reference implementations of new/proposed standards in these organizations

31. Layers in the Grid

32. A Typical eScience Use of Globus:Network for Earthquake Eng. Simulation Links instruments, data, computers, people

33. Without the Globus Toolkit A ComputeServer SimulationTool B ComputeServer WebBrowser WebPortal RegistrationService Camera TelepresenceMonitor DataViewerTool Camera C Database service ChatTool DataCatalog D Database service CredentialRepository E Database service Certificate authority Users work with client applications Application services organize VOs & enable access to other services Collective services aggregate &/or virtualize resources Resources implement standard access & management interfaces

34. With the Globus Toolkit GlobusGRAM ComputeServer SimulationTool GlobusGRAM ComputeServer WebBrowser CHEF Globus IndexService Camera TelepresenceMonitor DataViewerTool Camera GlobusDAI Database service CHEF ChatTeamlet GlobusMCS/RLS GlobusDAI Database service MyProxy GlobusDAI Database service CertificateAuthority Users work with client applications Application services organize VOs & enable access to other services Collective services aggregate &/or virtualize resources Resources implement standard access & management interfaces

35. The Globus Toolkit:“Standard Plumbing” for the Grid • Not turnkey solutions, but building blocks & tools for application developers & system integrators • Some components (e.g., file transfer) go farther than others (e.g., remote job submission) toward end-user relevance • Easier to reuse than to reinvent • Compatibility with other Grid systems comes for free • Today the majority of the GT public interfaces are usable by application developers and system integrators • Relatively few end-user interfaces • In general, not intended for direct use by end users (scientists, engineers, marketing specialists)

36. A B 1 1 9 9 Shared Distributed Infrastructure The Application-Infrastructure Gap Dynamicand/orDistributedApplications

37. Provisioning Bridging the Gap:Grid Infrastructure Users • Service-oriented Gridinfrastructure • Provision physicalresources to support application workloads • Service-oriented applications • Wrap applications as services • Compose applicationsinto workflows Composition Workflows Invocation ApplnService ApplnService

38. Grid Infrastructure • Distributed management • Of physical resources • Of software services • Of communities and their policies • Unified treatment • Build on Web services framework • Use WS-RF, WS-Notification (or WS-Transfer/Man) to represent/access state • Common management abstractions & interfaces

39. Globus is Open Source Grid Infrastructure • Implement key Web services standards • State, notification, security, … • Software for Grid infrastructure • Service-enable new & existing resources • E.g., GRAM on computer, GridFTP on storage system, custom application services • Uniform abstractions & mechanisms • Tools to build applications that exploit Grid infrastructure • Registries, security, data management, … • Enabler of a rich tool & service ecosystem

40. Web Browsers / Batch Processes (typically several thousand requests) IPC Server Request:Price Query 2 1 IPC Dispatcher Delegation of Request 2 IPC Server Response: PricelistDepending on: - Time - Discount - Number of Items - … 3 An eBusiness Use of Globus:SAP Demonstration @ GlobusWorld • 3 Globus-enabled applns: • CRM: Internet Pricing Configurator (IPC) • CRM: Workforce Management (WFM) • SCM: Advanced Planner & Optimizer (APO) • Applications modified to: • Adjust to varying demand & resources • Use Globus to discover & provision resources SAP AG R/3 Internet Pricing & Configurator (IPC)

41. Overview • Background and Globus approach • Globus Toolkit • Future directions • Related tools

42. The Globus Toolkit is a Collection of Components • A set of loosely-coupled components, with: • Services and clients • Libraries • Development tools • GT components are used to build Grid-based applications and services • GT can be viewed as a Grid SDK • GT components can be categorized across two different dimensions • By broad domain area • By protocol support

43. GT Domain Areas • Core runtime • Infrastructure for building new services • Security • Apply uniform policy across distinct systems • Execution management • Provision, deploy, & manage services • Data management • Discover, transfer, & access large data • Monitoring • Discover & monitor dynamic services

44. GT Protocols • Web service protocols • WSDL, SOAP • WS Addressing, WSRF, WSN • WS Security, SAML, XACML • WS-Interoperability profile • Non Web service protocols • Standards-based, such as GridFTP • Custom

45. “Stateless” vs. “Stateful” Services FileTransferService Client move (A to B) move • Without state, how does client: • Determine what happened (success/failure)? • Find out how many files completed? • Receive updates when interesting events arise? • Terminate a request? • Few useful services are truly “stateless”, but WS interfaces alone do not provide built-in support for state

46. FileTransferService (without WSRF) FileTransferService Client move (A to B) : transferID move whatHappen state tellMeWhen cancel • Developer reinvents wheel for each new service • Custom management and identification of state: transferID • Custom operations to inspect state synchronously (whatHappen) and asynchronously (tellMeWhen) • Custom lifetime operation (cancel)

47. Service State representation Resource Resource Property State identification Endpoint Reference State Interfaces GetRP, QueryRPs, GetMultipleRPs, SetRP Lifetime Interfaces SetTerminationTime ImmediateDestruction Notification Interfaces Subscribe Notify ServiceGroups Resource RPs WSRF in a Nutshell Service GetRP GetMultRPs EPR EPR EPR SetRP QueryRPs Subscribe SetTermTime Destroy

48. Transfer RPs FileTransferService (w/ WSRF) FileTransferService Client createResource createResource (A to B) : EPR getRP queryRPs destroy • Developer specifies custom method to createResource and leaves the rest to WSRF standards: • State exposed as Resource + Resource Properties and identified by Endpoint Reference (EPR) • State inspected by standard interfaces (GetRP, QueryRPs) • Lifetime management by standard interfaces (Destroy)

49. Core Globus Toolkit version 4 (GT4) Contrib/Preview Grid Telecontrol Protocol Depre-cated Delegation Data Replication Community Scheduling Framework WebMDS Python WS Core CommunityAuthorization Data Access & Integration Workspace Management Trigger C WS Core Web ServicesComponents Authentication Authorization Reliable File Transfer Grid Resource Allocation & Management Index Java WS Core Pre-WS Authentication Authorization GridFTP Pre-WS Grid Resource Alloc. & Mgmt Pre-WSMonitoring & Discovery C Common Libraries Non-WS Components Credential Mgmt Replica Location eXtensible IO (XIO) www.globus.org Security Data Mgmt Execution Mgmt Info Services CommonRuntime

50. Globus Toolkit: Open Source Grid Infrastructure Globus Toolkit v4 www.globus.org Data Replication CredentialMgmt Replica Location Grid Telecontrol Protocol Delegation Data Access & Integration Community Scheduling Framework WebMDS Python Runtime Reliable File Transfer CommunityAuthorization Workspace Management Trigger C Runtime Authentication Authorization GridFTP Grid Resource Allocation & Management Index Java Runtime Security Data Mgmt Execution Mgmt Info Services CommonRuntime