1 / 17

A possible SRB to iRODS Migration (a path from here to there) ‏ Adil Hasan RAL

A possible SRB to iRODS Migration (a path from here to there) ‏ Adil Hasan RAL. Why iRODS?. SRB used by large number of projects. Provides data management system virtualizes data. X509 authentication. Misses functionality that's commonly required like: checksum data regularly,

morrisprice
Download Presentation

A possible SRB to iRODS Migration (a path from here to there) ‏ Adil Hasan RAL

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A possible SRB to iRODS Migration (a path from here to there)‏ Adil Hasan RAL

  2. Why iRODS? • SRB used by large number of projects. • Provides data management system virtualizes data. • X509 authentication. • Misses functionality that's commonly required like: • checksum data regularly, • extract metadata store in different database etc.

  3. Why iRODS? • Missing functionality has to be scripted outside of SRB. • Difficulty in interacting with SRB. • SRB has a lot of functionality, even more than most people use, not easy to exclude unwanted functionality. • SRB starting to become difficult to maintain.

  4. What else is there? • Domain specific Data Management Systems (like ESA DMS). Difficult to generalize, require extra s/w. • Distributed file-system (nfs, webdav). Data not easily queriable, requires interface to MSS. • Build your own, problem is you have to maintain it. • Better to be part of a bigger group contributing to an DMS.

  5. iRODS • Rule-oriented system. • Each data management task regarded as a rule. • Each rule is executed by one or more micro-services. • Rules can be queued, run asynchronously. • Configurable system. • Can compile-in only micro-services required. • Ability to enhance storage resources.

  6. Migration Issues • But we have data in SRB how do we present data in iRODS? • Migration plan important. • Also helps to define what we want from the system. • Criteria system should meet, services it should provide etc.

  7. Migration Issues • Important to have group in place that can: • Pool together requirements and functionality the system must provide. • Be used to organize effort on producing a production-quality iRODS system. • Share knowledge on iRODS (a self-help group). • Group should primarily be EU-focussed (time-zones). But, have interest from Australians (maybe EU++).

  8. Migration Issues • Group called DataGridsEXchange (dgex): dgex@googlegroups.com (coords J-Y Nief, a.hasan).

  9. iRODS Requirements • Based on our systems want: • Ability to handle small files. • Interface to our MSS. • Single-Sign-On capabilities (X509, Kerberos). • Fine-grained access-control (to be refined). • Auditing (who accessed, when etc). • Replication (automatic replication to resources subject to certain criteria).

  10. iRODS Requirements • Synchronize data (ie update iRODS copy or replicas). • Checksum data (md5, system V etc). • Bulk operations. • Robust file-transfer (handle flaky networks) maybe with checkpointing. • GUI front-ends and web-based front end (drag and drop on all platforms desirable).

  11. iRODS Requirements • Ability to queue and run rules at regular intervals (can pause queue etc). • APIs important in Java, Python, Perl, C. • Audit info of rules (which version when run etc, incl versioning of micro-services). • Easy to install, update. • Monitoring of system health. • Load balancing.

  12. iRODS Requirements • File-system-like interface (webdav, nfs) for legacy apps. • Understandable uniform logging. • Interoperation with SRB and other systems.

  13. Migration Plan • Get people to join dgex (the more the merrier). • Identify critical (ie must have) common components of the system. • Test iRODS: • See how iRODS scales with number of files. • See how iRODS scales with concurrent access. • See how iRODS handles error situations. • Compare with SRB.

  14. Migration Plan • Based on tests either: • provide feedback to SDSC as to what needs to be improved (with suggestions). • provide SDSC with modified code. • Requires SDSC (or someone) hosts Gforge system to manage projects.

  15. Migration Plan • Work on missing, critical components. • Test, test, test. • Work on interoperation of SRB with iRODS. • Able to access SRB data through iRODS. • Work on migration tools extracting data from SRB inserting into iRODS.

  16. Migration Plan • Creating a production iRODS system will require close collaboration from all interested parties. • Communication through dgex will be important.

  17. Take Care of Your Data I'm here to make protect all the data in this building by making sure all the rules are obeyed and incinerating those who don't obey! Aiee! Save me! All my data is being eaten by these bug-eyed monsters! And now they want to Sashimi my brains! We Do Want This We Don't Want This

More Related