The Gridbus Toolkit for Service-Oriented Grid and Utility Computing

1. WW Grid The Gridbus Toolkit for Service-Oriented Grid and Utility Computing Grid Computing and Distributed Systems (GRIDS) Lab. Dept. of Computer Science and Software EngineeringThe University of MelbourneMelbourne, Australiawww.gridbus.org Rajkumar Buyya Collaboration: Nimrod-G Project

2. Pointers and Acknowledgements Download PPT Slides from: http://www.buyya.com/tut/gridbus.zip

3. Acknowledgements: Co-authors, Collaborators, and Funding Sources • Gridbus Project Members • Rajkumar Buyya (Gridbus PI) • Srikumar Venugopal (Ph.D. student) • Jia Yu (Ph.D. student) • Elan Kovan (Ph.D. student) • Anthony Sulistio (Ph.D. student) • Chee Shin Yeo (Ph.D. student) • Manjuka Soysa (Ph.D. student) • Shoaib Burq (Research Assistant) • Martin Placek (B.E student) • Rajiv Ranjan (Masters by Research) • Alex Barmouta (from UWA probing GridBank ) • Ding Choon-Hoong • Akshay Luther (Alchemi .NET Grid framework) • Virtual Lab - Docking • Kim Branson, WEHI for Structural Biology • NeuroGrid • Susumu Date, Osaka University • HEPGrid • Lyle Winton, School of Physics (HEP data catalogue and BASF application) • Natural Language Engineering: • Baden Hughes badenh@cs.mu.oz.au • Finance/Portfolio/investment risk analysis • Rafael Moreno-Vozmediano, Complutense University of Madrid • Industry Collaborator • Wolfgang Gentzsch, Sun Microsystems • Benjamin Khoo, IBM Global Services, Singapore • Collaborators • David Abramson & Jon Giddy, Monash – Nimrod-G • Manzur Murshed, Monash (GridSim core 1.0) • Heinz Stokinger, CERN (grid economy models) • Jahanzeb Sherwani, Nosheen Ali, Nausheen Lotia, Zahra Hayat (LUMS) • Hussein Gibbins, X – RA, GRIDS lab, Melborune (Gridscape)

4. Agenda • Technology Evolution and Introduction to Grid Computing • Grid Challenges, Approaches, and Architecture • Grid Architecture for Computational Economy • Grid Economy driven Technologies [+demo] • Grid Market Directory • Grid Bank • Nimrod-G and Gridbus Data Broker • Visual Parametric Modeler • Grid Simulation (GridSim) Toolkit • Scheduling Applications on Global Grids [+demo] • Summary and Conclusions

5. Technology Evolution

6. * HTC * P2P * PDAs Minicomputers * * PCs * Workstations * Mainframes * Grids COMPUTING * PC Clusters * Crays * MPPs * WS Clusters * XEROX PARC worm * IETF * W3C * TCP/IP Communication * Ethernet * HTML * Mosaic * Web Services * Email * Sputnik * Internet Era * WWW Era * XML * ARPANET 1960 1970 1975 1980 1985 1990 1995 2000 Computing and Communication Technologies Evolution

7. 2100 2100 2100 2100 2100 2100 2100 2100 2100 Scalable Computing PERFORMANCE + Q o S Administrative Barriers • Individual • Group • Department • Campus • State • National • Globe • Inter Planet • Universe Personal Device SMPs or SuperComputers Global Grid Inter Planet Grid Local Cluster Enterprise Cluster/Grid

8. Application Drivers for Tech. Evolution • Solving grand challenge applications using modeling, simulation and analysis Aerospace Internet & Ecommerce Life Sciences Military Applications CAD/CAM Digital Biology

9. Internet, Web, and Grid Effect 140 ‘Web Services and Grid Effect’ 120 100 The 'Network Effect’ kicks in, and the web goes critical' Number of hosts (millions) Business $$$ 80 60 40 20 0 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010 TCP/IP HTML Mosaic XML 4. with XML PHASE 2. The Internet is Born 3. The World Wide Web 5. The Grid 1. Packet Switching Networks HTML hypertext system created 1969: 4 US Universities linked to form ARPANET TCP/IP becomes core protocol CERN launch World Wide Web 1972: First e-mail program created Domain Name System created IETF created (1986) NCSA launch Mosaic interface 1976: Robert Metcalfe develops Ethernet

10. Clusters: Emerged as Mainstream, Low Cost and High Performance Computers

11. Adoption of Cluster Approach

12. Cluster 1 Scheduler Master Daemon LAN/WAN Submit Graphical Control Cluster 3 Execution Daemon Scheduler Clients Master Daemon Cluster 2 Scheduler Submit Graphical Control Execution Daemon Master Daemon Clients Submit Graphical Control Execution Daemon Clients Multi-Clustering: Coupling Multiple Clusters for Solving Lager Problems

13. http://www.sun.com/hpc/ Grid Emergence: Towards Global Computing Grid enables: • Resource Sharing • Selection • Aggreation - Unification of geographically distributed resources

14. What is Grid ? • A type of parallel and distributed system that enables the sharing, selection, & aggregationof geographically distributed resources: • Computers– PCs, workstations, clusters, supercomputers, laptops, notebooks, mobile devices, PDA, etc; • Software– e.g., ASPs renting expensive special purpose applications on demand; • Catalogued data and databases– e.g. transparent access to human genome database; • Special devices/instruments – e.g., radio telescope – SETI@Home searching for life in galaxy. • People/collaborators. depending on their availability, capability, cost, and user QoS requirements for solving large-scale problems/applications. Widearea

15. Formal Definition! • Grid is a type of parallel and distributed system that enables the sharing, selection, and aggregation of geographically distributed "autonomous" resources dynamically at runtime depending on their availability, capability, performance, cost, and users' quality-of-service requirements.

16. database A Bird Eye View of World-Wide Grid Environment Grid Information Service Grid Resource Broker Application R2 R3 R4 R5 RN Grid Resource Broker R6 R1 Resource Broker Grid Information Service

17. What do Grids aim for and how to support them. • Grids aim at exploiting synergies that result from cooperation of autonomous distributed entities. Synergies include: • Resource sharing • “On-demand” Virtual Enterprises creation • Aggregation of resources on demand. • For this cooperation to be sustainable, participants needs to have (economic) incentive. • Therefore, “incentive” mechanisms should be considered as one of key design parameters of Grid computing.

18. Synergies that Result from Cooperation:Classes of Grid Application Drivers • Distributed HPC (Supercomputing): • Computational science. • High-Capacity/Throughput Computing: • Large scale simulation/chip design & parameter studies. • Content Sharing (free or paid) • Sharing digital contents among peers (e.g., Napster) • Remote software access/renting services: • Application service provides (ASPs) & Web services. • Data-intensive computing: • Drug Design, Particle Physics, Stock Prediction... • On-demand, realtime computing: • Medical instrumentation & Mission Critical. • Collaborative Computing: • Collaborative design, Data exploration, education. • Service Oriented Computing (SOC): • Towards economic-based Utility Computing: New paradigm, new applications, new industries, and new business.

19. Drivers: Next Generation Applications (NGA) • Next generation experiments, simulations, sensors, satellites, even people and businesses are creating a flood of data. They all involve numerous experts/resources from multiple organization in synthesis, modeling, simulation,analysis, and interpretation. ~PBytes/sec High Energy Physics Brain Activity Analysis Newswire & data mining: Natural language engineering Digital Biology Life Sciences Astronomy Quantum Chemistry Finance: Portfolio analysis Internet & Ecommerce

20. Common Attributes/Needs/Challenges of NGA • They involve Distributed Entities: • Participants/Organizations • Resources • Computers • Instruments • Datasets/Databases • Source (e.g., CDB/PDBs) • Replication (e.g, HEP Data) • Application Components • Heterogeneous in nature • Participants require share analysis results of analysis with other collaborators (e.g., HEP) • Grids offer the most promising solution & enable global collaborations. • The beauty of the grid is that it provides a secure access to a wide range of heterogeneous resources. • But what does it take to integrate and manage applications across all these resources?

21. Grid Challenges

22. Computational Economy Security Data locality Resource Allocation & Scheduling Uniform Access System Management Resource Discovery Application Construction Network Management Grid Challenges

23. What Grids need to provide • Services that enable the execution of a job on a resource in different admistrative domain. • Security mechanisms that permit resources to be accessed only by authorized users. • App/Data Security (?) – A must for commercial users (protecting from GSPs/other users). • (New) programming tools that make our applications Grid Ready!. • Tools that can translate the requirements of an application/user into the requirements of computers, networks, and storage. • Tools that perform resource discovery, trading, selection/allocation, scheduling and distribution of jobs and collects results.

24. database Grid Operations Management Challenges Grid Information Service Grid Resource Broker Application R2 2 R3 R4 R5 RN Grid Resource Broker R6 R1 Resource Broker Grid Information Service challenges * Resource Management * Application Construction

25. Sources of Complexity in Grid Resource Management • Size (large number of nodes, providers, consumers) • Heterogeneity of resources (PCs, Workstatations, clusters, and supercomputers) • Heterogeneity of fabric management systems (single system image OS, queuing systems, etc.) • Heterogeneity of fabric management polices • Heterogeneity of applications (scientific, engineering, and commerce) • Heterogeneity of application requirements (CPU, I/O, memory, and/or network intensive) • Heterogeneity in demand patterns • Geographic distribution and different time zones • Differing goals (producers and consumers have different objectives and strategies) • Unsecure and Unreliable environment

26. Centralized Vs Distributed Management • They use centralised policy that need • complete state-information and • common fabric management policy or decentralised consensus-based policy. • Due to too many heterogenous parameters in the Grid it is impossible to define: • system-wide performance matrix and • common fabric management policy that is acceptable to all. • Therefore, the Grid resource management policy need to be distributed in natures so that autonomous entities organise themselves and yet benefit from cooperation.

27. Realizing the Grid Grid Architecture, Approaches, and Efforts

28. Grid Realization Steps • The integration of individual s/w & h/w components into a combined networked resource (single system image cluster). • Low-level middleware to provide a secure and uniform access to services provided by different resources. • User-level middleware to support application development and aggregation of distributed resources. • The construction of distributed applications.

30. Layered Grid Architecture APPLICATIONS Applications and Portals … Prob. Solving Env. Collaboration Engineering Web enabled Apps Scientific USER LEVEL MIDDLEWARE Development Environments and Tools Languages/Compilers Libraries Debuggers Monitors … Web tools Resource Management, Selection, and Aggregation (BROKERS) CORE MIDDLEWARE Distributed Resources Coupling Services Process Data Information Accounting … QoS Trading SECURITY LAYER Local Resource Managers FABRIC … Internet Protocols Libraries & App Kernels Queuing Systems Operating Systems Networked Resources across Organizations … Computers Networks Storage Systems Data Sources Scientific Instruments

31. Australia Nimrod-G Gridbus GridSim Virtual Lab DISCWorld GrangeNet ..new coming up Europe UNICORE Cactus UK eScience EU Data Grid EuroGrid MetaMPI XtremeWeb and many more. India I-Grid Japan Ninf DataFarm Korea... N*Grid USA Globus Legion OGSA Sun Grid Engine AppLeS NASA IPG Condor-G Jxta NetSolve AccessGrid and many more... Cycle Stealing & .com Initiatives Distributed.net SETI@Home, …. Entropia, UD, Parabon,…. Public Forums Global Grid Forum Australian Grid Forum IEEE TFCC CCGrid conference P2P conference Some Grid Projects & Technologies http://www.gridcomputing.com

32. NetSolve mix-and-match Object-oriented Internet/partial-P2P Grid Computing Approaches Network enabled Solvers Economic-based Service-Oriented Computing Nimrod-G

33. WW Grid Many Testbeds ? & who pays ?, who regulates demand and supply ? GUSTO (decommissioned) World Wide Grid Legion Testbed NASA IPG

34. Testbeds so far -- observations • Who contributed resources & why ? • Volunteers: for fun, challenge, fame, charismatic apps, public good like distributed.net & SETI@Home projects. • Collaborators: sharing resources while developing new technologies of common interest – Globus, Legion, Ninf, Ninf,,... Unless you know lab. leaders, it is impossible to get access! • How long ? • Short term: excitement is lost, too much of admin. Overhead (Globus inst+), no incentive, policy change,… • What we need ? Grid Marketplace! • Regulates supply-and-demand, offers incentive for being players, simple, scalable solution, quasi-deterministic – proven model in real-world.

35. Gridbus: Towards Building Grid Economy and Business Toolkit Offers Incentive and Enable the Creation and Promotion of: Grid Marketplace (competitive) ASP Service Oriented Computing . . . And let users focus on their own work (science, engineering, or commerce)!

36. The Gridbus Vision: To Enable Service Oriented Grid Computing & Bus iness! WW Grid Gridbus World Wide Grid! + marketplace for Service-Oriented Computing

37. GRIDS Lab @ the U. of Melbourne, The Gridbus Project: www.gridbus.org • R& D in Distributed Computational Economy and end-to-end infrastructure for Service-Oriented/Utility Computing: : • Architecture, Specification, and “Open Source” Reference Implementation in Collaboration with interested Global Peers. • Platform: Cluster, Grid, P2P for diverse applications. • Visual Tools for Creation of Distributed Applications • Grid Economy & Scheduling (via Nimrod-G Broker) • Grid Modeling and Simulation (GridSim) • Economic Cluster Scheduler (Libra) • Grid Accounting Services Architecture (GridBank) • Grid Service Publication (GMD) • World Wide Grid (WWG) – A Global Grid Testbed • Jxta-based P2P Compute Power Market (CPM) • Application Enabler Projects: • Virtual Laboratory Toolset for Drug Design • High-Energy Physics and the Grid Network (HEPGrid) • Brain Activity Analysis on the Grid (NeuroGrid) • GridEmail for Spam Management

38. Grid Economy and its Challenges

39. What do users want ? • Grid Consumers • Execute jobs for solving varying problem size and complexity • Benefit by selecting and aggregating resources wisely • Tradeoff timeframe and cost • Strategy: minimise expenses • Grid Providers • Contribute (“idle”) resource for executing consumer jobs • Benefit by maximizing resource utilisation • Tradeoff local requirements & market opportunity • Strategy: maximise return on investment

40. Grid Characteristics & Operations Management Challenges • Grid has Large Heterogeneity • Resource Types: PC, WS, Clusters • Resource Architecture: CPU Arch, OS • Applications: CPU/IO/message intensive • Access Price: different for different users, resources and time. • Availability: varies with time. • Grid has Distributed • Resources, Ownership, Users • Grid has self-interested entities: • Users/Grid Service Consumers: Minimise expenses • Resource Owners/Grid Service Providers: Maximise profit • The heterogeneity and decentralization that is present in Grid is very similar to one present in “human economies” where market-based mechanisms have been used successfully to manage them. • Therefore, we advocate the use of economic/market-models for managing Grid resources by treating their services as ICT commodities/ utilities.

41. Grid Economy: Incentive (sustained sharing), Regulation (supply & demand)

42. Service Pricing and Economic Models • Price-based: Supply, demand, value, and wealth of economic system • Commodity Market Model • Posted Price Model • Bargaining Model • Tendering (Contract Net) Model • Auction Model • English, first-price sealed-bid, second-price sealed-bid (Vickrey), and Dutch (consumer:low,high,rate; producer:high, low, rate) • Proportional Resource Sharing Model • Monopoly (one provider) and Oligopoly (few players) • consumers may not have any influence on prices. • Bartering • Shareholder Model • Partnership Model

43. Benefits of Computational Economy • It provides an effective paradigm for managing self interested and self-regulating entities (resource owners and consumers) • Helps in regulating supply-and-demand of resources. • Services can be priced in such a way that equilibrium is maintained. • User-centric / Utility driven • Scalable: • No need of central coordinator (during negotiation) • Resources(sellers) and also Users(buyers) can make their own decisions and try to maximize utility and profit. • Adaptable, • It allows to offer different QoS (quality of services) to different applications depending the value users place on them. • It offers incentive for resource owners for being part of the grid! • It offers incentive for resource consumers for being good citizens. • It improves the utilisation of resources.

44. GRACE: A Reference Grid Economy Services Architecture GRid Architecture for Computational Economy (GRACE)

45. Grid Economy & Users’ Challenges • Grid Service Providers (GSPs) • How do I decide service pricing models ? • How do I specify them ? • How do I translate them into resource allocations ? • How do I enforce them ? • How do I advertise & attract consumers ? • How do I do accounting and handle payments? • ….. • Grid Service Consumers (GSCs) • How do I decide expenses ? • How do I express QoS requirements ? • How do I trade between timeframe & cost ? • How do I map jobs to resources to meet my QoS needs? • ….. • They need mechanisms and technologies for value expression, value translation, and value enforcement.

46. Market-based Computing Systems Requirements • To enable users (GSPs and GSCs) to realise economic value, market-based systems need to provide mechanisms for: • Value Expression • a means to express their requirements, valuations, and objectives • Value Translation • scheduling policies to translate them to resource allocations • Value Enforcement • mechanisms to enforce the selection and allocation of differential services, and dynamic adaptation to changes in their availability at runtime • Market mechanisms, accounting and payment, Reservation of resources.

47. GRACE: A ReferenceService-Oriented Grid Architecture for Computational Economies Data Catalogue Grid Bank Information Service Grid Market Services Sign-on HealthMonitor Info ? Grid Node N … Grid Explorer … Secure ProgrammingEnvironments Job Control Agent Grid Node1 Applications Schedule Advisor QoS Pricing Algorithms Trade Server Trading Trade Manager Accounting Resource Reservation Misc. services … Deployment Agent JobExec Resource Allocation Storage Grid Resource Broker … R1 R2 Rm Grid Middleware Services Grid Consumer Grid Service Providers

48. Realising Market-based Grid: Minimal New Components • Grid Market Directory Services • Grid Trading Services – • for different economic models • Grid Metering Services • Grid Accounting and Payment Services • Grid Service Broker

49. Globus Grid Technologies for Realising the Vision of Grid Economy Framework High Energy Physics Brain Activity Analysis Grid Apps. Natural Language Engineering Molecular Docking Portfolio Analysis Grid Email High-level Services and Tools … User-LevelMiddleware (Grid Tools) Gridscape G-Monitor Programming Framework Grid Brokers & Schedulers Nimrod-G Gridbus Data Broker Alchemi: .NET Grid Services +Clustering of desktop PCs Data Management Services GridBank GMD Core Grid Middleware MDS GRAM GASS PKI-basedGrid Security Interface (GSI) .NET JVM Condor PBS SGE Libra Tomcat Grid Fabric Windows Linux AIX IRIX OSF1 HP UX Solaris

50. Gridbus Technologies • Application Construction Tools • Visual Parametric Modeller (VPM) • Grid Economy Services • Grid Market Directory • A Registry for publication of GSPs and their Services – VO/VE • Grid Bank • A Grid Accounting Services • Grid Trading Services • Data Grid Service Broker • QoS based Scheduling of Distributed Data Oriented Apps on global Grids • Builds on the Nimrod-G work • Gridscape • Interactive Grid Testbed Portal Generator • G-monitor • Grid Application Execution Management Portal • GridSim • A Grid Simulation Toolkit • Libra • Economy based Cluster Scheduling