1 / 17

Efficient Space Management for Disk-Intensive Applications in Grid Environments

This paper discusses the challenges in storage system management, space reservation, and direct access for disk-intensive Grid applications. The solution, StoRM, addresses efficient I/O, space reservation, and management policies using Parallel File Systems and GPFS, with advanced functionalities for space management. The Space Management Interface enables direct access to reserved space, following SRM definitions for compatibility. StoRM aims to enforce management policies efficiently and enable direct access to reserved space for improved I/O performance.

jacquelinei
Download Presentation

Efficient Space Management for Disk-Intensive Applications in Grid Environments

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. StoRMdisk management middleware CHEP 2004, Interlaken 27th September 2004 Flavia Donno, INFN Antonia Ghiselli, CNAF-INFN Luca Magnoni, CNAF-INFNRiccardo Zappi, CNAF-INFN

  2. Problem statement (1/3) Storage System • For applications that are disk-intensive, particular attention should be paid to storage resources, such as those providing higher capacity, redundancy, scalability, etc.. • More than decent performance are required to access your data.

  3. Problem statement (2/3) Space Reservation • In a storage system, we can identify two kinds of resources : file and space. • It often authorized users on storage acts as concurrent consumer of storage resources. • Concurrent consuming of storage resources may cause jobs failure, specially if there is no certainty of space availability.

  4. Problem statement (3/3) Direct access • Data-intensive jobs want direct access on reserved space with good I/O performance. • Some data-intensive job might not be able to be adapted to use reserved space through an “external” space management application (backward compatibility).

  5. Requirements (1/2) • Storage • Efficient I/O • Capacity, Fault-tolerance • Space Management • Space reservation • Direct access to reserved space • Management policies

  6. Storage : Efficient I/OSolution: Parallel File System • An example of Parallel FS. POSIX I/O read() write() open() close() … Logical File

  7. Storage : Scalability, Fault recovery, ..Solution: GPFS • The following requirements: • Scalability • Large files/file-systems • Security • Failure Recovery are satisfied by modern parallel file system as GPFS.

  8. Requirements (2/2) • Storage • Efficient I/O • Capacity, Fault-tolerance • Space Management • Space reservation • Direct access to reserved space • Management policies

  9. Space Management : Interface • In SRM v2.1 advanced specification there are some new functionalities regarding space management : srmReserveSpace() srmReleaseSpace() srmUpdateSpace() srmCompactSpace() ... we have adopted SRM definitions to preserve compatibility with any Space Management Client that is SRM-compliant.

  10. Space Management : Direct access In order to enable applications for posix access to the reserved space in a storage, we consider a <path>/<file-name> as a part of spaceToken. path ::= [root] [relative-path] root ::= [root-directory] root-directory ::= "/" relative-path ::= path-element { "/" path-element } ["/"] path-element ::= name | parent-directory | directory-placeholder name ::= char { char } directory-placeholder ::= "." parent-directory ::= "..“

  11. Assembling all the pieces: StoRM • Storage Resource Manager • StoRM goal is to create a container for storage management functionalities as: • Space reservation (SRM-compliant) • Other SRM functionalities. • Management policyenforcement on storage (G-Pbox provided by INFN).

  12. Architecture : access to StoRM.

  13. Architecture : StoRM Server Management Request

  14. Use scenario : Use of reserved space

  15. Prototype status and future plan • Prototype of StoRM implementing reserve-space functionality exists. • Next steps : • Testing of all functionalities defined in SRM advanced involving Space Management. • Integration with other Storage Management Client SRM-Compliant (SRM File Transfer functionalities).

  16. Conclusion • We have presented StoRM, a solution that fulfils the requirements about space management and I/O performance in disk-intensive Grid applications. • StoRM prototype have shown how to create and use reserved space in efficient mode. • There is a lot of work to do, but we are confident in a success.

  17. End • Project references can be found at INFNForge : http://infnforge.cnaf.infn.it

More Related