The Inferno Grid (and the Reading Campus Grid)

1. The Inferno Grid(and the Reading Campus Grid) Jon Blower Reading e-Science Centre Many others, School of Systems Engineering, IT Services http://www.resc.rdg.ac.uk resc@rdg.ac.uk

2. Introduction • Reading are in early stages of Campus Grid construction • Currently consists of two flocked Condor pools • More of which later • Also experimenting with the Inferno Grid • Condor-like system for pooling ordinary desktops • Although (like Condor) it could be used for more than this • The Inferno Grid is commercial software but free to UK e-Science community • Secure, low maintenance, firewall-friendly • Perhaps not (yet) as feature-rich as Condor

3. The Inferno operating system • The Inferno Grid is based upon the Inferno OS • Inferno OS is built from the ground up for distributed computing • Mature technology, good pedigree (Bell Labs, Pike & Ritchie) • Extremely lightweight (~ 1MB RAM) so can run as emulated application identically on multiple platforms (Linux, Windows, etc) • Hence it is a powerful base for Grid middleware • Everything in Inferno is represented as a file or set of files • cf. /dev/mouse in Unix • So to create a distributed system, just have to know how to share “files” – uses a protocol called Styx for this • Inferno OS is released under Liberal Licence (free and open source) for non-commercial use • Can run applications in the host OS (Linux, Windows etc) • Secure: certificate-based authentication, plus strong encryption built-in at OS level

4. The Inferno Grid • Built as an application in the Inferno OS • Hence uses OS’s built-in security and ease of distribution • Can run under all platforms that Inferno OS runs on • Essentially high-throughput computing cf. Condor • Created by Vita Nuova (http://www.vitanuova.com) • Free academic licence, but also used “for real”: • Evotec OAI (speeds up drug discovery) 90% utilisation of machines • “Major government department” modelling disease spread in mammals • Other major company (can’t say more!) • University installations at Reading and York • (AHM2004 – created Inferno Grid from scratch easily)

5. (Can also run Inferno native on bare hardware) Could write all applications in Limbo (Inferno’s own language) and run on all platforms, guaranteed! Inferno Grid software (a Limbo program) Inferno OS (Virtual OS) Host OS (Windows, Linux, MacOSX, Solaris, FreeBSD)

6. Inferno Grid system overview • Matches jobs submitted to abilities of “worker nodes” • The whole show is run by a scheduler machine • Jobs are ordinary Windows/Linux/Mac executables • Process is different from that of Condor • Unless Condor has changed/is changing… • In Condor, workers run daemon processes that wait for jobs to be sent to them • i.e. “scheduler-push” • Requires incoming ports to be open on each worker node • In the Inferno Grid, workers “dial into” the scheduler and ask “have you got any work for me?” • i.e. “worker-pull” or “labour exchange” • No incoming ports need to be open • Doesn’t poll – uses persistent connections • Studies have shown this to be more efficient (not sure which ones… ;-)

7. Worker firewalls: No incoming ports open. Single outgoing port open (to fixed, known server) Architecture Workers can be in different admin. domains Workers can connect and disconnect at will. Job submission is via supplied GUI. Could create other apps (command-line, Web interface) Firewall: single Incoming port open Scheduler – listens for job submissions and workers “reporting for duty”

8. Display All scheduled jobs with current status and priority Control Create, start, stop, delete and change job priority with immediate effect Information Job description and parameters Status Detailed progress report for currently selected job Job Administration

9. Control Include, exclude and delete nodes Display All known nodes and current status Job Group Assign nodes to individual job groups Connected Not Connected Dead Blacklisted Information See node operating system & list of installed packages At a Glance Viewing Quickly see the current state of the grid with colour coded job ids Power Bar See how much of the grid is being utilised % available Job id Task id % in use Node Administration

10. Pros and Cons • Pros: • Easy to install and maintain • Good security: • See next slide • “Industry quality” • Cons: • Small user base and not-great documentation • Hence learning curve • Doesn’t have all Condor’s features • E.g. migration, MPI universe, reducing impact on primary users • No Globus integration yet • But probably not hard to do – JobManager for Inferno? • Security mechanism is Inferno’s own • But might see other mechanisms in Inferno in future • Question over scalability (100s of machines, fine: 1000s… not sure) • Inferno Grids don’t “flock” yet

11. Security and impact on primary users • Only one incoming port on the scheduler needs to be open through the firewall • Nothing runs as root • All connections in the Inferno Grid can be authenticated and encrypted • Public-key certificates for auth, variety of encryption algs • Cert. usage is transparent, user is not aware it’s there • Similar to SSL in principle • Can setup worker nodes to only run certain jobs • So can prevent arbitrary code from being run • Doesn’t have all of Condor’s options for pausing jobs on keyboard press, etc • Runs jobs under low priority • But could set up so that workers don’t ask for work if they are loaded • But what happens to a job that has already started?

12. Other points • Slow-running tasks are reallocated until whole job is finished. • Could fairly easily write different front-ends for Inferno Grid for job submission and monitoring • Don’t have to use supplied GUI • ReSC’s JStyx library could be used to write Java GUI or JSP • In fact, code base is small enough to make significant customisation realistic • Customise worker node behaviour • “Flocking” probably not hard to do • Schedulers could exchange jobs • Or workers could know about more than one scheduler • Inferno OS can be used to very easily create a distributed data store • This data store can link directly with the Inferno Grid • Caveat: We haven’t really used this in anger yet!

13. Building an Inferno Grid in this room… • These are conservative estimates (I think) • Install scheduler (Linux machine) – 10 minutes • Install worker node software (Windows) – 2 minutes each • Run toy job and monitor it within 15 minutes of start • Set up Inferno Certificate Authority – 1 minute • Provide Inferno certificates to all worker nodes – 2 minutes per node • Provide Inferno cert to users + admins – 2 minutes each • Fully-secured (small) Inferno Grid up and ready in an hour or two. • If you know what you’re doing!! (remember that docs aren’t so good…  )

14. Reading Campus Grid so far • Collaboration between School of Systems Engineering, IT Services and e-Science Centre • Haven’t had as much time as we’d like to investigate Inferno Grid • But have an embryonic “Campus Grid” of two flocked Condor pools • Although both at Reading, come under different admin domains • Getting them to share data space was challenging, and firewalls caused initial problem • (Incidentally, the Inferno Grid had no problems at all crossing the domains) • Small number of users running MPI and batch jobs • Animal and Microbial Sciences, Environmental Systems Science Centre • Ran demo project for University • An “heroic effort” at the moment, but we are trying to secure funding

15. Novel features of RCG • Problem: Most machines are Windows but most people want *nix environment for scientific programs • “Diskless Condor”: • Windows machines reboot into Linux overnight • Loads Linux from a network-shared disk image • Uses networked resources only (zero impact on hard drive) • In morning, reboots back into Windows • Looking into CoLinux (www.colinux.org): • Free VM technology for running Linux under Windows • Early days, but initial look is promising.

16. Future work • Try to get funding! • Intention is to make CG key part of campus infrastructure • IT Services are supportive • Installation of SRB for distributed data store • Add clusters/HPC resources to Campus Grid • Working towards NGS compatibility

17. Conclusions • Inferno Grid has lots of good points, especially in terms of security, ease of installation and maintenance • Should be attractive to IT Services… • We haven’t used it “in anger” yet but it is used successfully by others (in academia, industry and govt) • Caveat: these people tend to run a single app (or small number of apps) rather than general code • Doesn’t have all of Condor’s features • We don’t want to fragment effort or become marginalised • Would be great to see good features of Inferno appear in Condor, esp. “worker pull” mechanism