1 / 27

An Introduction to the

An Introduction to the. By: Roman Olschanowsky roman2u@sdsc.edu. Outline. SDSC and History of SRB Example Project Introduction to SRB Discussion on SRB basics SRB Clients Overview of a Data Grid Infrastructure Topology. TeraGrid Linux Cluster (IA64). 4.4 TF.

nuala
Download Presentation

An Introduction to the

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An Introduction to the By: Roman Olschanowsky roman2u@sdsc.edu

  2. Outline • SDSC and History of SRB • Example Project • Introduction to SRB • Discussion on SRB basics • SRB Clients • Overview of a Data Grid • Infrastructure • Topology

  3. TeraGrid Linux Cluster (IA64) 4.4 TF Sun F15K Disk Server Storage Area Network Disk 600 TB www.sdsc.edu Human infrastructure:Experienced multi-disciplinary staff support a broad spectrum of national science, engineeringand technology projects Archival Systems Blue Gene/L (Due 12/04) DataStar IBM Power4 6 PB Networking Visualization Storage and Compute Resources 2.8/5.7 TF 10.4 TF

  4. Sites Using the SRB

  5. SDSC SRB Projects (60 million, .5 PB ) • Digital Libraries • UCB, Umich, UCSB, Stanford,CDL • NSF NSDL - UCAR / DLESE • NASA Information Power Grid • Astronomy • National Virtual Observatory • 2MASS Project (2 Micron All Sky Survey) • Particle Physics • Particle Physics Data Grid (DOE) • GriPhyN • SLAC Synchrotron Data Repository • Medicine • Digital Embryo (NLM) • Earth Systems Sciences • ESIPS • LTER • Persistent Archives • NARA • LOC • Neuro Science & Molecular Science • TeleScience/NCMIR, BIRN • SLAC, AfCS, …

  6. The SCEC Project • Southern California Earthquake Center • 400 people, the best earthquake seismologists in the country (33 states) and several from abroad (9 countries). (Sep. 2004 SCEC AHM attendees) • Simulating a 7.7 earthquakein the L.A. basin • 10 year effort • 100+ TB of input data ( soil conditions, topography, grid coordinates, etc… ) • 240 procs on SDSC Datastar cluster, 5 days, 1 TB RAM, 2GB/sec IO Thanks! • SDSC, scientific applications group, with porting the code; parallelizing the calculation and the IO; and generalizing the code for scaling up to a large run. Offered invaluable insights regarding IO management.  • SRB, took care of draining the GPFS cache regularly,  moving 43 TB of data safely to archive storage.   That task was completed a mere 36 hours after the end of the calculation. The SRB was critical in this achievement.

  7. SDSC & SRB Example

  8. Storage Resource Broker (SRB) • A distributed file system (Data Grid) • Client-Server, Server-Server architecture. • Abstracts physical • SRB provides the ability to transparently share data across remote sites. • Heterogeneous Resources • Single sign on • Single logical file hierarchy

  9. What we are familiar with

  10. What we are not familiar with, yet

  11. How do the file systems differ? • Logical Abstraction • Folders are NOT physical • Files do NOT inherit physical location • Everything is potentially distributed • Access Control • Permissions are NOT rwxrwxrwx • Permissions ARE on a object by object basis • Groups and permissions ARE more similar to NTFS • Domains • Geographical / logical grouping of users • Namespace scalability: john@harvard john@mit • Also doubles as groups

  12. Interfaces to theStorage Resource Broker • inQ – Windows Client • Scommands – UNIX, DOS Command line Client • Jargon – Java API and GUI components • mySRB – Web Client • Matrix – WSDL, Data Grid Workflows • C, C++ – C and C++ API • Python – Python API • Perl – Perl API

  13. Sinit Senv Spwd Sls Scd Sget Sput Ssh Scp Smv (logical) Sphymove (physical) Srm Smkdir Srmdir Serror Schmod Sexit Common Scommands (69 total)

  14. mySRB

  15. BIRN Portal (perl based)

  16. NEEScentral Portal (php based)

  17. Biomedical Informatics Research Network (BIRN) • Major collaboration with SDSC, several of the projects’ Co-Investigators and Co-PIs are at SDSC. • BIRN’s purpose is to provide it’s consortium of neuroscience laboratories the ability to share, compute, and collaborate. • The Storage Resource Broker provides the ability to transparently share data across remote sites.

  18. The BIRN SRB Data Grid

  19. Doing this “Manually”

  20. The BIRN Data Grid

  21. The grid is in the details

  22. File Replication Sls /home/Demo/SRB-Tutorial/files-2: Doc.txt Sls -l /home/Demo/SRB-Tutorial/files-2: romanoly 0 z-ucsd-ncmir-nas1 15 2003-07-09-05.15 Doc.txt romanoly 1 z-jhu-cis-nas0 15 2003-07-09-05.16 Doc.txt romanoly 2 z-stanford-lucas-nas 15 2003-07-09-05.16 Doc.txt romanoly 3 z-umn-cmrr-nas0 15 2003-07-09-05.16 Doc.txt romanoly 4 z-uci-bic-nas0 15 2003-07-09-05.17 Doc.txt

  23. DR SRB “Location” or “Slave Server” SRB “Physical Resources” z-jhu-cis-nas0 z-jhu-cis-nas1 “Location” z-jhu-cis-nas2 SRB “Logical Resource” “jhu-cis-nas”

  24. 0.7 TB 5.2 TB 0 TB 1.6 TB 0.8 TB 0.8 TB 3.2 TB 0.8 TB 2.4 TB 0.8 TB 0.8 TB 2.4 TB 1.6 TB 0.8 TB 5.0 TB 0.78 TB 0.08 TB Pooling physical resources

  25. Logical / Compound Resources SRB “instant replication” “fast archival” “resource pooling” “My-Resource” SRB

  26. Logical Resources

  27. Thanks! SRB handles large data and provides the ability to share and collaborate on distributed heterogeneous resources. Questions? www.sdsc.edu/srb srb@sdsc.edu

More Related