Lightweight grid computing workshop, 3rd May 2006

1. Lightweight grid computing workshop, 3rd May 2006 WEDS: Web services Environment for Distributed SimulationJames SuterCentre for Computational ScienceUniversity College London

2. Contents • Problems with existing grid middleware • Lightweight middleware? • WEDS • Enabling grid-based computational science • Examples

3. Grid Computing • We define grid computing as distributed computing performed transparently across multiple administrative domains. • By computing, we refer to any activity involving digital information. Transparency, implying minimal complexity for the users of the technology, has been missing from all existing distributed computing infrastructures. • A general computational grid should provide easy access to many different types of resources to enable users to pick and choose those required to achieve their intended scientific objectives. See: Phil Trans R Soc London A (2005)

4. Storage devices UK and US Grid Technologies Instruments: XMT devices, LUSI,… Grid Middleware HPC resources Scalable MD, MC, mesoscale modelling User with laptop Steering Performance control/monitoring Visualization engines VR and/or AG nodes

5. Grid Computing • Such demands have so far proved hard to meet and as a result very few scientists have wanted to get near so-called grid technology. • Important for the development of the grid as a whole that as many diverse groups of scientists begin to use the grid; it is critical that the uptake of grid technologies is outside of specialized grid-centric projects. • Existing grid middleware obstructs rather than facilitates access to grid resources. This has hindered scientific progress with the grid.

6. Problems for users • The existing toolkits have an excessively heavy set of software and administrative requirements, even for relatively simple demands from applications. • Existing toolkits are painful and difficult to install and maintain. • Existing standards bodies and the task forces within the UK e-Science programme are not engaging sufficiently with the applications community. • The effort and time required to overcome these problems are such that many scientists are reticent to deal with grid resources. • Application scientists generally use legacy applications, often written without knowledge of grid computing. Porting of these codes to the grid environment with ease of installation and facility is crucial to user uptake.

7. Our solution • We have therefore developed simple, lightweight Grid middleware that is "good enough" for rapid adoption, rather than waiting for a solution which will, supposedly, suit all needs. • Such a toolkit must be • substantially more portable, lightweight, and modular in design than current middleware. • By being produced in very close collaboration with application scientists and having full documentation, this will allow end-users to port existing codes and use Grid techniques with the minimum of hassle.

8. Lightweight middleware • What do we mean by lightweight? • Minimal dependencies on third-party software • Small learning-curve for new users – removing the need to learn new programming methods • Interoperable with other WSRG implementations • Easy to write, and so to adapt to new specs, etc. • Uses WSRF::Lite (interoperable with other WSRG implementations)

9. Lightweight middleware • OGSI::Lite/WSRF::Lite • by Mark McKeown of Manchester University • Lightweight OGSI/WSRF implementation, written in Perl • uses existing software where possible; simple installation • We have developed WEDS--a Web service hosting Environment for Distributed Simulation

10. About WEDS • Developed to make life easier for application scientists • Easy to deploy – sits inside a WSRF::Lite container, has no additional software requirements • For use within an administrative domain • Provides all the tools and glue required to: • expose an unaltered binary application as a service • create and interact with service instances • Broker service manages creation of services, to load balance across a pool of machines See Coveney et al., 2004, NeSC Tech Rpt

11. WEDS Architecture • Each resource runs a WSRF::Lite container containing a WEDS machine service and factory services for each hosted application. • Each machine that a user wishes to use is registered with a broker service • The user contacts the broker with the details of the job to run • The broker match-makes the job details with the capabilities advertised by each machine service and decides where to invoke the service • The broker passes back the contact details of the service instance to the client Client Broker Machine Service Service Factory Wrapper Service Invoked Application Managed resource

12. More complex uses of WEDS • Under this framework, an executable can be deployed remotely using a 'standard' script as if it were being run locally. • An object-oriented representation of a simulation wrapper can be written in Perl, which allows scripts (or users) to start, visualize and control simulations with commands as simple as • $g=$sim->getValue('gravity'); or • $sim->setValue('pressure', 200); • The term 'loosely coupled' is often used and well describes the way a library of Web services could be easily connected to run simulations or analysers, which automatically exchange their results between each other

13. The Polysteer Application Developed by Daniel Mason (Imperial; ReG partner) • New Monte Carlo polymer simulation code • Create as many chain conformations as possible Task farming of configuration generation • Equilibration is difficult from arbitrary start point • Need to watch chains relax Attach visualisation client • Monte Carlo moves are complex • Modify parameters on-the-fly to optimise efficiency Attach steering client

14. Lightweight hosting of Polysteer application

15. Visualisation client attaches to a running simulation • Data transfer via files using ReG steering library • Fortran main code to Java visualiser Visualisation • Showing each atom is unreadable • Potentials treat CHx, Bz as single entities • We visualise ellipsoids rather than spheres

16. Work flow example: Fourier transform of molecular dynamics output • A user script coordinates a workflow between molecular dynamics, Fourier transform and visualization services. The binary data moves directly between elements, not via the user's machine, halving bandwidth demands. • Example: MD program LAMMPS interfacing with a Fourier Transform analysis program

17. Steering example: LB2D • LB2D is a workstation class lattice-Boltzmann code written in C simulating multiphase fluid flow in two dimensions. Preparing the code for use with WEDS involved linking to a steering library • A single broker and machine service allows a single simulation process to be executed. The simulation involved stepping the value of the parameter g_cc from 2, for the first 200 time-steps, to 0, for the second 200 time-steps, and back to 2 for the final 200 time-steps, giving a simulation run 600 time-steps long in total.

18. Summary • Lightweight middleware such as WEDS greatly facilitates deployment of applications on grids • We’re now working with several “computational user communities” from physics through to biology • All our middleware will be in the public domain: Download from www.realitygrid.org/WEDS

19. Acknowledgements • Prof. Peter Coveney, Radhika Saskena, Matt Harvey, Laurent Pedesseau, Mark Mc Keown, Stephen Pickles, Daniel Mason, Jonathan Chin • EPSRC • OMII