Storage Resource Management: Version 3.0 Fundamentals Storage Resource Management

1. Storage Resource Management Storage Resource Management: Version 3.0 Fundamentals Storage Resource Management Working Group Meeting Jefferson Lab September 14-15, 2005 http://sdm.lbl.gov/srm-wg

2. Finalizing SRM V3.0 • Extending functions for advance features • Grouping functions into core and advanced features • File and Space Types in Features • File Sharing • Releasing and removing files • Reserving and Releasing Spaces • Directory Management

3. FEATURE 6 f1 (F3) FEATURE 1 FEATURE 5 f2 (F3) f1 (F1) f2 (F1) CORE f3 (F1) f1 (C) f3 (C) f2 (C) f4 (C) FEATURE 2 FEATURE 4 f7 (C) f5 (C) f1 (F3) f2 (F3) f6 (C) f8(C) f3 (F3) FEATURE 3 f4(F3) f1 (F3) f2 (F3) Extending functions for advance features • Issue: what to do in the case of functions that are in core, but are also used in advanced features with additional functionality • E.g. Specifying a space-token for srmCopy when “space reservation” feature is supported. • Solution: Extending parameters according to features

4. Extending parameters according to features:An example • 2.1. srmAbortRequest • srmAbortRequest terminates the previously submitted request. • InOut • String userID • String authorizationID • String storageSystemInfo • String requestToken • Boolean isRemoval • 2.1.1. Core Behavior • a) Terminate all files in the request regardless of the file state. Remove files from the queue, and release cached files if applicable. Expired files are released, too. • b) isRemoval flag is false by default. When isRemoval flag is true, all cached files for the request associated with the request token must be removed. • c) Abort must be allowed to all requests with requestToken, including requests such as srmLs or srmCopy. • d) If any file transfers are active, SRM should attempt to stop the transfer. If not possible, abort takes place after the pending transfers are completed. • Behavior With Advanced Features • 2.1.2. Copy Feature • a) srmAbortRequest with the isRemoval flag on must be propagated for any calls to remote SRMs. ReturnStatusForRequest { EnumStatusCode statusCode, string explanation }

5. Grouping functions into core and advanced features (1) BASIC ADVANCED • File movement • PrepareToGet • PrepareToPut • Copy • Request capabilities • Multi-file Streaming • Trans. Prot. Negotiation • File lifetime negotiation • File types • Volatile • Permanent • durable yes yes yes yes no yes yes yes yes yes no yes yes yes yes (for MSS) yes no yes

6. Grouping functions into core and advanced features (2) BASIC ADVANCED • Space reservations • Space-time negotiation • Space types • Remote access • gridFTP • Other SRMs • User-specified Directory • Volatile • Permanent • Durable • Terminate/suspend • Abort file • Abort request • Suspend/resume request no yes yes no no yes no yes yes yes yes yes no yes yes yes yes yes no yes

7. File and Space Types in Features 2. Core Functions 2.1. srmAbortRequest 2.2. srmAbortRequestedFiles 2.3. srmChangeFileStorageType 2.4. srmChangeFileStorageTypeStatus 2.5. srmExtendFileLifetime 2.6. srmGetFeatures 2.7. srmGetRequestSummary 2.8. srmGetRequestTokens 2.9. srmGetSRMStorageInfo 2.10. srmGetTransferProtocols 2.11. srmLs 2.12. srmLsStatus 2.13. srmPrepareToGet 2.14. srmPrepareToPut 2.15. srmPutFileDone 2.16. srmPutRequestDone 2.17. srmReleaseFiles 2.18. srmRm 2.19. srmRmStatus 2.20. srmStatusOfGetRequest 2.21. srmStatusOfPutRequest 4. Advanced feature set 2 : Space Management 4.1. srmCleanupFilesFromSpace 4.2. srmCompactSpace 4.3. srmGetSpaceMetaData 4.4. srmGetSpaceTokens 4.5. srmReleaseSpace 4.6. srmReserveSpace 4.7. srmUpdateSpace 5. Advanced feature set 3 : Directory Management 5.1. srmCp 5.2. srmCpStatus 5.3. srmMkdir 5.4. srmMkdirStatus 5.5. srmMv 5.6. srmMvStatus 5.7. srmRmdir 6. Advanced feature set 4 : Authorization 6.1. srmCheckPermission 6.2. srmReassignToUser 6.3. srmReassignToUserStatus 6.4. srmSetPermission 7. Advanced feature set 5 : Request Administration 7.1. srmResumeRequest 7.2. srmSuspendRequest 3. Advanced feature set 1 : Remote Access 3.1. srmRemoteCopy 3.2. srmStatusOfCopyRequest

8. File and Space Types in Features • File types MUST be the same as the space typethey are in • Are space types global to all features of the SRM? • e.g. Core functions support Volatile and Permanent, but want space reservations for Volatile only • e.g. Core functions support Volatile and Permanent, butdirectory functions for Permanent only • Should all functions in a feature support the same space type? • Recommendation • allow space types by feature • all functions in a feature support same space type • Extend discovery function to include space types supported • Example • {core [volatile, permanent], space_reservation [volatile]}

9. File Sharing • From the user’s point of view – there is NO file sharing • If files are physically shared, the SRM needs to keep • Lifetime per user • Space where file is per user • State of file per user – released / removed • May need to copy files between spaces depending on space types • Simplest implementation • Copy multiple uses of files to users’ spaces • Copy can be made locally – avoid getting from remote site • Space efficient implementation • Spaces are virtual • Keep track of file status virtually

10. Releasing and removing files (1) • Files can be released (or removed) from spaces that are: • Implicitly assigned (Core or default) • Only one space of each type can be implicitly assigned • Need to specify “space type” in call • Explicitly assigned (Space Reservation feature) • Can have many spaces of same type • If more than one space of a certain type exists, need to provide “space token” • srmReleaseFiles • Requires: Request_Token, SURLs • To release all file, need: releaseAllCurrentlyPinnedFiles flag • srmRm • Does not require: Request_Token, only SURLs • Removed only for requesting user for all of his/her active requests

11. Releasing and removing files (2) • srmCleanupFilesFromSpace (Space_Token) • Releases all files in space regardless of request • If “remove” flag is on files will be removed • If files are shared, this has no effect on other users • srmAbortRequestedFiles • Releases files specified for Request_Token • If “remove” flag is on files will be removed • Files that are still in queue (not brought to cache) – are removed from queue • File in-transfer – stop transfer if possible, and remove partial file; otherwise complete transfer and release (or remove) • srmAbortRequest • Same as above, but applies to all files • Aborting an srmRemoteCopy • Need to notify remote site, of course • Push case – needs to propagate “release / remove” to remote site

12. Reserving and Releasing Spaces (1) • Reserve space • Can provide: “guarantee” + “total” (difference is best-effort) • No confirmation expected, refusal is done by srmReleaseSpace • Can reserve multiple spaces of the same type, subject to total allowed • Lifetime of space is also “negotiated” • Files in spaces should have shorter lifetime than space lifetime • All files in spaces have the same type as space type • Space assigned canbe discovered with srmGetSpaceMetadata • Updating space • Can request new sizes and lifetime (from current time) • No consolidation of spaces • Can compact space • Compact space • Space size is reduced by amount of unused space • Released file are removed • doDynamicCompactFromNowOn flag – for dynamic compacting

13. Reserving and Releasing Spaces (2) • Release space • Only owner can release a space • Once released – cannot be used or updated • If space not empty when release requested • Volatile • Released files will be removed and space compacted • Files not released – space release is postponed till they are released or lifetime expires • Advisable to “cleanup” space before releasing space • Durable • Released files will be removed and space compacted • Files not released – space release is postponed till they are released • Files not released and lifetime expires– space release is postponed till durable-action takes place (e.g. archived and owner notified) • Permanent • Released files will be removed and space compacted • Files not released – stay in space, and space they occupy not released

14. Directory Management • A single directory for all files types • Files can belong to any spaces D1 D1 D2 D3 D2 D3 D4 D4 F1 (D) F2 (P) F3 (V) F1 (V) F2 (V) F3 (V) F4 (D) F5 (D) F6 (D) F7 (P) F8 (P) F4 (P) F5 (D) (1) Mixed file types (2) By file type • Supported functions: srmLs, srmMkdir, srmRmdir, srmRm, srmMv, and srmCp (srmLs and srmRm are Core functions) • Advantage of (1): no need to move files when file types are changed

15. V3.0 Client – to Access Existing Servers

16. Using v3.0 Client to access existing SRM implementations SRM V3.0 Client translator translator translator SRM V1.1 SRM V1.1 +Space R. SRM V2.1 SRM V3.0 Core SRM V3.0 Core+{AF} • Need to add discovery function to translators • Advantage: can continue to use existing implementations