1 / 27

Introduction to iRODS

Introduction to iRODS. 10 th Oct 2014. Christine Staiger SURFsara. Contents. iRODS Concepts The Basics Installation Tools for administration Adding data and metadata iRODS rules. iRODS Concepts. What is iRODS ?. What does iRODS provide?.

leslie-rogers
Download Presentation

Introduction to iRODS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. IntroductiontoiRODS 10th Oct 2014 Christine Staiger SURFsara

  2. Contents iRODS Concepts The Basics Installation Tools for administration Adding data and metadata iRODS rules

  3. iRODS Concepts

  4. What is iRODS?

  5. What does iRODS provide? Storage virtualization of different disk and tape storage systems A logical namespace across storage locations A rule engine to automate data management according to defined policies Various client interfaces

  6. iRODS Components

  7. In the background:iRODSResources Cache Resource Archive Resource Compound Resource High Latency Low Latency POSIX like IO calls (Storage) Resource is a Software or Hardware system that stores data 3 Resource classes:

  8. Resource Groups: Distributing files Resource Group Cache Resource Cache Resource Cache Resource Low Latency Low Latency Low Latency POSIX like IO calls @Server A @Server B @Server C

  9. Resource Groups: Data replication Resource Group Replicate Cache Resource Cache Resource Cache Resource Low Latency Low Latency Low Latency POSIX like IO calls @Server A @Server B @Server C

  10. The users view:iRODSData Collections Data Object Collection0 Logical Name Space (iCAT)independent from physical resources Data Object Data Object • Similar to Unix Directory and Files • ichmodto set user/group ACLs • Core.re: msiSetDefaultResc(Resource) • .irodsEnv: irodsDefResource=Resource Data Object A Data Object Collection1 Data Object Data Object X …/Collection0/ …/Collection0/DataObject A-B …/Collection0/Collection1/ …/Collection0/Collection1/DataObject X-Z

  11. Hands on The Basics

  12. Demo Data Grids Virtual machines:sshuser@IP

  13. Installation and configuration • iRODS3: Install script ./irodssetup • iRODS4: (rpm, dmg) package and source code • ~/iRODS/config/irods.config # Database configuration $DATABASE_TYPE = 'postgres'; … $DATABASE_HOST = 'localhost'; $DATABASE_PORT = '5432'; $DATABASE_ADMIN_PASSWORD = 'irods2'; $DATABASE_ADMIN_NAME = ’alice'; # iRODS configuration IRODS_HOME = '/home/alice/iRODS'; $IRODS_PORT = '1247’; … $IRODS_ADMIN_PASSWORD = ’alice'; $IRODS_ICAT_HOST = '';

  14. .irodsEnv file # iRODS personal configuration file. # # This file wasautomaticallycreatedduringiRODSinstallation. # CreatedMon Jun 18 13:13:27 2012 # # iRODS server host name: irodsHost’surfsara-cloud' # iRODS server portnumber: irodsPort1247 # Default storageresourcename: irodsDefResource'demoResc' # Home directory in iRODS: irodsHome'/alicesZone/home/alice' # Currentdirectory in iRODS: irodsCwd'/alicesZone/home/alice' # Accountname: irodsUserName’alice' # Zone: irodsZone‘alicesZone' The .irodsEnv determines the zone and the user for the icommands-client. You can only use one such file at a time.

  15. iRODS i-commands Documentation: https://wiki.irods.org/index.php/icommands (wedon’tprovide a exhaustivelist of i-commands)

  16. iRODS control • Execute ./irodsctl <arg> in the folder iRODS • Restarting, starting and stopping the iRODS and iCATserver with • <arg> = restart, start, stop • Restarting, starting and stopping only the iRODSserver with • <arg> = irestart, istart, istop • Initialise an account and log in with iinit

  17. ilsresc • ilsresclistsiRODSresources and resource-groups • Usage: ilsresc [-lvVhA] [Name] alice@sara-cloud:~$ ilsresc demoResc alice@sara-cloud:~$ ilsresc –l demoResc resource name: demoResc resc id: 10010 zone: alicesZone type: unix file system class: archive location: surfsara-cloud vault: /home/alice/iRODS/Vault …

  18. User and Data administration • Change to admin mode: iadmin • Quit admin mode:q • iadminmkresc, iadminrmresc • Create new resource: • mkdir Data (in your home, not in the iRODS environment) • iadminmkresc<RescName>“unix file system” archive • you.rIP.add.ress“/home/<un>/Data” • Make/delete a user: • mkuser, rmuser, moduser • (iadmin) mkuser user1 rodsuser • (iadmin) moduser user1 password ***** • Parameters and their values: iadminlt, iadminltuser_type

  19. Put and replicate data • Add the file /home/login/put1.txt to resource • iput -v put1.txt • ils -l or ils -L (gives also physical location) • iput-h • Create a collection and add data • imkdir /<yourZone>/archive • iput put2.txt /<yourZone>/archive/put2.txt • Check physical location of the data ils -L /<yourZone>/archive • Replicate data across resources: irepl -R<YourResc> put1.txt • Check with ils -L • Trim number of copies: itrim -S<YourResc> -N 1 put1.txt

  20. iRODS Metadata • Commands: imeta, iquest, idbo • Addmetadata: • imetaadd -d put1.txt “Date” “08.10.2013” • imetaadd-dput1.txt “AnotherDate” “04.04.2014” • imetals -d put1.txt • Removemetadata: • imetarm -d put1.txt “Date” “08.10.2013” • QueryingtheiCAT: • iquest"SELECT DATA_NAME,DATA_CHECKSUM WHERE DATA_RESC_NAME like ’%Resc%’” • DATA_NAME = put1.txt • DATA_CHECKSUM = c16d7d3488677a3348b12eb82795d28c

  21. iRODSMicroservices and rules • Micro-services are small, well-defined procedures/functions that perform a simple task • Examples: “msiCollCreate”, “msiDataObjRepl”, “msiDataObjChksum”, … • Convert any C procedure (createCollection) to a microserviceby creating an interface routine (msiCreateCollection) • Micro-services can be combined in rules to enforce policies • Rules can be executed on action, with time-delay or periodically

  22. iRODS Rule Engine

  23. Replicatetoanotherresource • replicate{ • msiWriteRodsLog("startingreplication", *status); • msiWriteRodsLog("source = *source", *status); • msiDataObjRepl(*source, *destRescName, *CC); • } • INPUT *source=”/<yourZone>/home/<UN>/put1.txt”, *destRescName=“testResc” • OUTPUT ruleExecOut • Save as replicate.r • Execute irule-vFreplicate.r • Check with ils -L

  24. Replicatetoanothercollection • Usefulforbackup of a data collection • Changingthestoragemedium • iput put2.txt • replicate{ • msiWriteRodsLog("startingreplication", *status); • msiWriteRodsLog("source = *source", *status); • msiDataObjRsync(*source, "IRODS_TO_COLLECTION", *destRescName, *destination, *rsyncStatus); • } • INPUT *source="/<yourZone>/home/<UN>/put2.txt", *destination=”/<yourZone>/coll1", *destRescName=“<RescName>" • OUTPUT ruleExecOut • ils-L /irodsXZone/coll1/home/irodsX

  25. Thank you !

  26. Appendix

  27. Troubleshooting ERROR: putUtil: put error for /alicesZone/home/bob#bobsZone/put1.txt, status = -9000 status = -9000 SYS EXCEED CONNECT CNT Host configuration problem on the server. Solution: Edit iRODS/server/config/irodsHost localhostyou.rIP.XX.XXsurfsara-cloud localhost 127.0.0.1 ./irodsctl restart

More Related