1 / 14

LCG Middleware Testing Report (CERN-INTAS 03-52-4297)

LCG Middleware Testing Report (CERN-INTAS 03-52-4297). E.Slabospitskaya, IHEP, Russia CERN-Russia Joint Working Group on LHC Computing June, 25, 2006. Outline. 1.Report for 2004 Testing against SRM ( E. Slabospitskaya,IHEP ) and GFAL ( M. Sapunov,IHEP ) II. Report for 2005

nodin
Download Presentation

LCG Middleware Testing Report (CERN-INTAS 03-52-4297)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. LCG Middleware Testing Report(CERN-INTAS 03-52-4297) E.Slabospitskaya, IHEP, Russia CERN-Russia Joint Working Group on LHC Computing June, 25, 2006

  2. Outline 1.Report for 2004 Testing against SRM (E. Slabospitskaya,IHEP) and GFAL (M. Sapunov,IHEP) II. Report for 2005 The 4 kind of tests have been developed by Russian developers: • 1. Testing of some functionality in LCG-2 Releases (D.Zaborov, ITEP,L.Stepanova, INR, S.Laptev, INR) • 2. Port SFT tests to gLite (M.Sapunov, IHEP) • 3. Port new gLite test suites to EDG/LCG Framework (E.Slabospitskaya, IHEP) • 4. New functionality in gLite test suites • (N.Klopov, PNPI) II Report for the first part of 2006 Using the EDG/LCG Framework via NMI (L.Stepanova,INR, M.Sapunov, IHEP)

  3. 2004Testing against SRM and GFAL • Storage Resource Manager (SRM) and Grid File Access Library (GFAL) are GRID middleware components used for transparent access to mass storage elements • These two Test Suites have been developed by IHEP team: • for testing of SRM interface v.1.1 • for testing against the GFAL file system. • The programs have been written in C and Perl languages

  4. 2005Testing of new functionality in LCG-2 Releases (1/3) • File Transfer Service (FTS) CLI based test suite 1. A basic test of the ability of the FTS to perform successful file transfer. 2. High level cancellation test implies: - submission of an FTS job with several files to transfer in parallel; - waiting until at least 1 file transfer is done; - canceling the job; - check that files with status "Done" are kept and not deleted; - check that files pending are canceled and not transferred.  3. FTS stress test allows to run many copies of the basic transfer test..  4. Tests of the reaction of FTS CLI commands to various "mistakes":  5. FTS management tests.

  5. Testing of some LCG-2 functionality (2/3) • Parsers for Gridftp and Gatekeeper logs Data processing is selecting from logs of gridftp and gatekeeper. Scripts in Perl have developed for these purposes. Results store to MySQL database for future processing. The Perl DBI module provides a generic interface for database access.

  6. Testing of some LCG-2 functionality (3/3) Parser for SFT logs Extractold statistics from early SFT history (in PHP4) about  good/bad sites and visualization of results ( jpgraph-1.19). - search name of region, and prepare list of regions in html - output image exposes the date statistic for sites into one region (via web browser)

  7. Port SFT tests to gLite • Site Functional Tests (SFT) Port the existence LGC-2 tests to gLite. gLite SFT consist of 2 main parts: Launcher and Tests. Launcher part has kind of GNU/Linux file structure and consist of 5 main modules. Each module consist of submodules. Main modules called "submit", "status", "publish", "cancel" and "clean" . Tests are using "sandbox" feature of gLite/LCG-2 middleware. It is easy to add more tests, rules have been defined..

  8. Port new gLite test suites to EDG/LCG Framework • I/O storm testing in EDG/LCG Framework This Framework is a set of Testing Suites produced by many of EDG and LCG developers. User launches multiple (parallel or one by one) jobs from User Interface to Worker Node. Each job on Worker Node launches a script which compiles and link C++ program and creates jdl file (this program is a part of job in input sandbox). This C++ program via API creates a file (size is optional) and transfers (writes/reads) these files between WN and I/O server (there and back). The result returns to user (to User Interface). Test creates a local file in temporary directory, copies it to I/O server, and writes it to tmp once again

  9. New gLite tests for the functionality different from LCG-2 • Direct Acyclic Graph (DAG) CLI based tests (requested by JRA1 team) - create the JDL files for DAG with the regular structure like tree.   DAG generation is realized based on the template that defines   the Executable and JDL-attributes for three parts of the DAG   (pre-jobs, main part and post-jobs) with given number of the   levels and nodes.- modify the created DAG structure: add new dependencies or delete   the existed ones, based on the appropriate config file. The nodes   without dependencies are removed (optional) at the end of this step.- add new JDL attributes (requirements) for the DAG at whole or for   one level or for the definite set of nodes according to config file.- run DAG job.- check status of the subjobs during DAG execution.- retrieve the nodes stdout and stderr files after DAG completion   and create the log file.

  10. 2006 • To be able to test daily grid specific software right after the build was chosen the NMI system. One of the goals was to submit and execute tests from edg-tests framework, used to test grid middleware in CERN IT\GD group.

  11. Next Generation Build System • Ease of use. • Ease of adding new components without modifying the build system code. • Ease of adding new platforms preferable without modifying the build system code. • Building multiple components at the same time if they are independent and follow the dependencies when required.. • Log the build/test related information in the database. • Should transfer the required externals for a particular component to the remote build machine. This can even include externals like Java. Flexibility on this feature is debatable. • Framework should be smart enough to handle the operating system signals and should report failure if any of the scripts is killed. • Web interface for component builders/testers.

  12. EDG/LCG tests were launched via NMI: • Tests are successfully done via NMI. • Test which is done partially via NMI. • Tests are not touched. M.Sapunov, IHEP and L.Stapanova ,INR

  13. Troubles with NMI system • One of many worry concerned with Grid middleware is using Grid proxy in self-acting software • Absence of the presentation layer is not a good point for NMI system. If the job produce the special results like html or xml files and you want to get it back, then you need to store thus files in result.tar.gz file, download and unpack it manually. • It's clear that NMI system is far away from production level. Summarizing all negative points CERN IT/GD group decide to wait 3-4 months until NMI becomes more mature.

  14. Main tasks in 2006-2007 • Task 1. "The development of full featured test suites that for the certification of the new grid MW components“ JINR (leading role), SINP-MSU, CERN-IT (consultations) • Task 2. "The development of testing procedures focused on special details of various software feature PNPI (leading role), JINR, IHEP, CERN-IT (consultations) • Task 3. ”Porting EDG-LCG framework to the gLite environment” IHEP (leading role), SINP-MSU , CERN-IT (consultations)

More Related