1 / 25

GridPP2 Data Management Part 2: Mass Storage & Local Storage Management

This work area focuses on metadata management, local storage management, cleanup of temporary space, and enabling Grid access to storage at UK Tier 1 and 2. The goal is to integrate with deployment software and work with storage staff at various sites.

robertsonk
Download Presentation

GridPP2 Data Management Part 2: Mass Storage & Local Storage Management

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. GridPP2 Data Management Work Area – Part 2 Mass storage & local storage mgmt J Jensen j.jensen@rl.ac.uk

  2. Data Management Work Area • Metadata management • See Gavin’s talk • Local Storage Management • Cleanup temporary space • Reservations? • Mass Storage Management • Enable Grid access to storage at UK Tier 1 and 2

  3. Overview Local Storage Metadata Deployment Support ARDA SRM2 EGEE WSRF? dCache Storage management

  4. Common • Links to external middleware groups • Common support & deployment requirements • Integrate with deployment software • LCFG-ng / Quattor / Manual install • Not contractually obliged to deliver • Must use tools from EGEE JRA1/SA1 (tar/make) • Can we use EGEE CVS ? • Savannah • Work with OMII

  5. Storage Group • Form a Storage Group within GridPP2 • Already set up! • Requirements, evaluation, development, deployment, support, evaluation • Work with storage staff at sites • RAL, Bristol, Edinburgh (, Glasgow) • Will set up web site, mailing list soon • Regular phone conferences

  6. Storage Management – Pf2 • Need to support MSS at UK Tier 2 sites • Evaluate existing solutions in the UK • Support Tier 1 first, then Tier 2 • Prioritize Tier 2 in order of storage committed… • …or ease of implementation • SRM 1 required • Move to SRM 2 later?

  7. Storage Management – Pf3 • SRM version 1 implementations • Enstore, dCache, CASTOR, EDG SE,… • Pick one to deploy initially • Initial focus on Tier 1 • Pick one to develop and support • Which is best for supporting storage systems at Tier-2 sites in the UK?

  8. Local Storage Management – Pf2 • Evaluations • Clean up temporary files • Reservations – difficult to support • May be necessary to integrate with WLMS • May be necessary to integrate with batch systems

  9. Local Storage Management – Pf3 • Various ways to provide partial functionality • EDG Zambo • dCache • Condor • Integrate something with batch systems • Initial deployment • dCache • For batch systems, focus on PBS initially, then LSF • Evaluate which is best for long term • Additional developments may be required

  10. Requirements input • Other GridPP areas • EGEE Architecture Group • HEPCAL, in particular HEPCAL I, ARDA,… • EGEE NA4? • Security - JRA3 • SRM group

  11. Before GridPP2 • EDG SE developments • More work on SRM 1 • Currently testing with dCache client and GFAL • Improve metadata implementation • Integrate VOMS, deploy ACL stuff • Set up support structures (started already!) • Set up EGEE tracking

  12. Support • Work with Tier-1 and Tier-2 support staff • Use Bugzilla? Or Savannah? • Bugzilla to be set up by Edinburgh if req’d • Provide mailing lists for • Announcements • Help – integration • Perhaps internally, for developments/discussions

  13. DeliverablesDeployment and Support • Releases to match EGEE release m and LCGn, mє{1,2}, n є{3,…}. • PM03: Initial deployment • dCache for LSM • EDG SE (and SRM?) • PM05: Release for LCG3 • PM07: Release for EGEE release 1 • PM16: Release for EGEE release 2 • PM26: Final post-EGEE release • For EGEE follow-on project

  14. DeliverablesEvaluation and Development • Evaluations, developments design • PM08: [Storage only] Report on mass storage used in UK • PM10: Report on integration with EGEE release 1 • PM14: [Local Storage only] Report on local storage management • PM20: Report on integration with EGEE release 2

  15. DeliverablesManagement and timeline • Plans, report on interactions with EGEE, e-Science, OMII • PM01: Initial project management plan • Risk analysis, plan for evaluating and supporting storage systems, etc • PM12: Annual report • PM24: Annual report • PM36: Final project report • Describing delivered and usable software, EGEE and LCG deployment and evaluation, dissemination, contributions to int’l standards, deviations from plans,

  16. DeliverablesIntegration with EGEE • “Higher level” EGEE tracking and integration (architecture, security, …) • PM01: Report describing requirements arising from EGEE Architecture • PM11: Report describing requirements arising from EGEE Architecture • PM18: Integration plan for EGEE follow-on

  17. SRM 1 • Mass storage control protocol • GridFTP for data transfer • SRM 1 req’d for interoperability with DoE Labs • Some interoperability problems between various implementations • Parts of protocol open to interpretation

  18. EGEE • Integrate directly with • JRA1 – middleware • Data Management “cluster” • Architecture Group • SA1 – support and deployment • Also watch • JRA2 – QA • JRA3 – security • And…? • NA2, 3, 4: dissemination, training, use cases

  19. LCG • LCG testbed – match LCG3 • Future LCG milestones less well defined (or secret)? • ARDA • Can provide early prototypes • What is “early” – pre-GridPP2? • For Mass Storage, SE to interface with RM • For Local Storage Management, dCache • GFAL

  20. LCG: GFAL • LCG (will) decide to use GFAL – the “Grid File Access Library” • It was decided to interface to EDG SE using SRM 1 interface • For now using EDG RM POSIX interface SRM 1 client EDG SRM EDG 2.1 Storage Element Mass Storage

  21. Web services • Already using web services for mass storage control interfaces • Need HTTPS-compatible delegation • Need to evaluate WSRF

  22. SRM version 2 • Protocol (very nearly) defined • More advanced optional functionality • Volatile, Durable, and Permanent files • Volatile, Durable, and Permanent space • Evaluate need/requirement for this as we go along • SRM Group to be in GGF as “GSM”

  23. Dissemination • Work with NA2 • Work with e-Science programme • We used AHM 2002, 2003 to promote GridPP, EDG SE (posters, papers) • We will be present at AHM 2004 (September)

  24. Non-LCG liaison • e-Science Core programme • Liaise with e-Science GSC, ETF, STF,…? • ADS – used by diverse scientific community • EGEE – WP formerly known as 10 • WP formerly known as 9 is not in EGEE • Babar • No Grid plans yet…

  25. EGEE: DICOM server support The Grid Metadata Metadata Storage Element WP10 DM2 Store patient metadata Store key Encrypt, anonymise DICOM Server Access control on metadata required; different ACLs for different types of metadata

More Related