1 / 11

The UCL Condor Pool Experience

Environment from the Molecular Level A NERC eScience testbed project. The UCL Condor Pool Experience. John Brodholt 1 , Paul Wilson 3 , Wolfgang Emmerich 2 and Clovis Chapman 2 . 1. Department of Earth Sciences, University College London, Gower Street, London WC1E 6BT, UK

abel
Download Presentation

The UCL Condor Pool Experience

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Environment from the Molecular Level A NERC eScience testbed project The UCL Condor Pool Experience John Brodholt1, Paul Wilson3, Wolfgang Emmerich2and Clovis Chapman2. 1. Department of Earth Sciences, University College London, Gower Street, London WC1E 6BT, UK 2. Department of Computer Science, University College London, Gower Street, London WC1E 6BT, UK 3. Anvil Software, London, UK.

  2. Environment from the Molecular Level A NERC eScience testbed project The UCL Condor Pool Approximately 946 Windows machines (yesterday) 1 to 2.4 GHtz Intel processors 256 to 512 MBytes memory (a few are more) They are in “open access” student cluster rooms PCs are all thin client “WTS” machines with network bootable operating systems. (Citrix/Bpbatch - hit spacebar to upload new operating systems image) The pool is very simple – one manager, one submit machine (via ssh).

  3. Environment from the Molecular Level A NERC eScience testbed project Q. Why was it someone from an Earth Science Dept. who got it going? Because three years ago, the eScience grants call made me look up “the Grid” on the web and by chance I came across the Condor web site. I also happened to know how Information Systems at UCL managed their student PCs. Persuaded the Director of UCL’s Education and Information Systems Division that I could put it in our eMinerals grant (I think he assumed it wouldn’t get funded).

  4. Environment from the Molecular Level A NERC eScience testbed project Key Political Issues Even though the Director of EISD had agreed for us to put it in the grant, we had to convince Information Systems themselves. Numerous meetings …. IS produced a five page document outlining what they thought their policy on a large Condor cluster would be – i.e. the primary purpose of the student cluster rooms must not be compromised. Nor should IS staff use their time on the project … etc. Needed testing (one cluster, then one image type). Perhaps the key moment was when the UCL presented its eScience projects to Tony Hey and the UCL Provost.

  5. Environment from the Molecular Level A NERC eScience testbed project Timescale Desktop - June 2002 (2 nodes) Earth Science Student Cluster Room - Oct 2002 (18 nodes) Physics Department (one WTS image) – Jan 2003 (150 nodes) Campus – October 16th 2003 (930 nodes) 1 millionth hour of CPU – April 2nd, 2004 This matched exactly the timescale we outlined in the eMinerals grant

  6. Environment from the Molecular Level A NERC eScience testbed project Other Issues Difficult to persuade the scientists to get involved for just a few machines. Some needed to compile their codes for Windows machines – “It’s simple, just convert them to Java ..” Wolfgang Emmerich, 2002! Our central manager died a few times when a user submitted a few thousand jobs all at the same time (took 24 hours to repair disk with fsck). Now have a manager and a submit machine. Students will do anything to reserve a machine – steal the mouse, put out of order signs on them, and UNPLUG them. Also, IS themselves briefly turn machines in some clusters off in order to clear the room. This restricts the length of job.

  7. Environment from the Molecular Level A NERC eScience testbed project

  8. Environment from the Molecular Level A NERC eScience testbed project Spikes in user demand: Not many users Most are using simple schemes to produce lots of initial input files and send off to pool. Get results back and spend a long time processing them/extracting data/planning next set of inputs.

  9. Jobs Database Steering Database Analysis Database Existing e-science technology User Input: Structural model Si/Al, cation types, [H2O] etc. Distributed resources (Condor pools etc.) Distributed Computing Portal Model/Configuration Generator Analysis(geometry, energy, fit) Improve generation / modelstrategy User Input: Diffraction data, chemical analysis,building units Si/Al, cation types, [H2O] etc. Drip feeding and interactive steering of a Condor pool using relational databases Dewi Lewis, Rosie Coates and Sam French UCL Chemistry / RI

  10. Environment from the Molecular Level A NERC eScience testbed project THE Science.1. Simulation of pollutants in the environment Binding of heavy metals and organic molecules in soils. 2. Studies of materials for long-term nuclear waste encapsulationRadiocactive waste leaching through ceramic storage media. 3. Studies of weathering and scalingMineral/water interface simulations, e.g oil well scaling.also4. The Earth’s core and mantleMany codes:DL-POLY, GULP, METADISE, CRYSTAL, CASTEP, SIESTA, …

  11. Environment from the Molecular Level A NERC eScience testbed project Now what? Expand pool to include staff WTS machines ~ 1500 machines (received 3 page email from IS - who owns them?). UCL Staff machines at hospitals ~ ???? machines. Federate with other pools: hopefully make it more flexible smooth spikes in demand.

More Related