1 / 13

Towards scalable, semantic-based virtualized storage resources provisioning

Towards scalable, semantic-based virtualized storage resources provisioning. Kornel Skałkowski, Renata Słota, Dariusz Król , Michał Orzechowski, Bartosz Kryza, Jacek Kitowski ACC Cyfronet AGH, Krakow, Poland.

palila
Download Presentation

Towards scalable, semantic-based virtualized storage resources provisioning

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Towards scalable, semantic-based virtualized storage resources provisioning Kornel Skałkowski, Renata Słota, Dariusz Król, Michał Orzechowski, Bartosz Kryza, Jacek Kitowski ACC Cyfronet AGH, Krakow, Poland KU KDM 2012 : fifth ACC Cyfronet AGH users' conference : Zakopane, March 07–09, 2012

  2. Outline • Introduction • The QStorMan toolkit overview • The QStorMan toolkit architecture • QStorMan usage • Recent improvements • Current status of QStorMan • Test results • Future Work

  3. Introduction • Data intensive applications and the 4th science paradigm • Resources virtualization becomes ubiquitous • Storage resources virtualization is often provided by cluster file systems like Lustre • IT infrastructure users expect more and more computing and storage power as well as an appropriate QoS level

  4. TheQStorMantoolkit • Main goal is to provide virtualized storage resources with QoS warrianties for data intensive applications • Users can define QoS requirements concerning storage resources on three levels: application, user, virtual organization • Currently we support the following non-functional requirements: • Average Read/Write transfer rate, • Current Read/Write transfer rate, • Free capacity, • Result cachability – dedicated for application, which generates a large number of small files. • The toolkit consists of three components: • Knowledge base (GOM) which stores semantic descriptions concerning storage resources and synchronizes the descriptions with a grid middleware • Dedicated monitoring service (SMED) which performs continuous, real-time monitoring of virtualized storage resources with semantic support • Intelligent resources matching service (SES) which combines information obtained from the GOM and SMED services as well as advanced semantic support in order to perfectly match a virtualized resource from the resources mesh

  5. TheQStorMantoolkitarchitecture

  6. QStorManusage • Using system C library (libses-wrapper): • declare your non-functional requirements in the GOM knowledge base • export LD_PRELOAD=<path_to_libses_wrapper_librart> 2. Using C++ programming library (libses): #include <LustreManager.h> #include <StoragePolicyFactory.h> using namespace lustre_api_library; LustreManager manager; StoragePolicy policy; policy.setAverageReadTransferRate(50); policy.setCapacity(100); int descriptor = manager.createFile(„nazwa_pliku.dat”, &policy);

  7. Recentimprovements • General purpose of the improvements is to provide a scalable, fully semantic-based solution for efficient provisioning of virtualized storage resources • SMED improvements: • Utilization of the enhanced C2MS storage resources semantic model for description of high-level QoS parameters • Application of semanatic reasoners on the monitoring level • SES improvements • Cache mechanism on demand – supporting large number of files generation • Automatic registration of users in knowledge base – decrease required administration effort • GOM improvements • Security enhancements • Scripts for administration

  8. The QStorMantoolkitcurrent status • Test installation is running at ACC Cyfronet AGH from over 1 year now • A lot of tests were performed and no major bugs were found • We have passed operational and security audits in PL-Grid succesfully • We now waiting for official deployment in ACC Cyfronet, PCSS Poznan, TASK Gdansk, and ICM Warsaw • Official tutorials, workshops and other material are on the way • Integrated with QoSCosGrid middleware from PCSS • We are willing to cooperate with anyone, who would like to test QStorMan in practice with an exisiting data intensive application

  9. Test description Synthetic test • The toolkit evaluation was performed by simulation of 8 users which were executing their applications on the Grid infrastructure • 3 users used the QStorMan toolkit during the applications execution, the others used plain Lustre file system • Every user periodically saved and read a 60 GB file with random sleep periods between the succeeding operations (10 reads and 10 writes) • Users started their applications with random delays in order to simulate real conditions in a Grid environment Test with real user’s application • Simulation of sound wave propagation inside human head • Out-of-core computations • No source code modifications • 5 instances of application running in parallel in order to generate enough load for storage system

  10. Synthethic test results • 12% speedup between two fastest applications • 26% speedup on average (~7:20 h vs ~10 h) • No source code modification

  11. Real user’s application test result • 15% speedup on average • Running on production infrastructure • No source code modification

  12. Futurework • Support for domain-oriented virtualized computing environments • Implementation of new storage resources selection strategies • Orientation toward Cloud computing environments • Dissemination and exploitation among possible users

  13. Questions?dkrol@agh.edu.plrena@agh.edu.pl

More Related