1 / 22

SA2 – Quality Assurance

SA2 – Quality Assurance. Alberto AIMAR (CERN) SA2 Leader. EMI Second EC Review 12 June 2012, Brussels. Outline. Context Objectives and Recommendations Impact on EMI Quality Build and Test (ETICS) Integration Infrastructure (EMI Testbed) Metrics Quality Control Conclusions.

alka
Download Presentation

SA2 – Quality Assurance

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SA2 – Quality Assurance Alberto AIMAR (CERN)SA2 Leader EMI Second EC Review 12 June 2012, Brussels

  2. Outline Context Objectives and Recommendations Impact on EMI Quality • Build and Test (ETICS) • Integration Infrastructure(EMI Testbed) • Metrics • Quality Control Conclusions https://bit.ly/emisa2 EMI 2nd EC Review

  3. Quality Assurance in EMI (SA2) EMI 2nd EC Review

  4. DoW Objectives DoW SO 1.4 • Common software quality assurance policies and metricsfor all engineering activities • Monitoring metrics and trends quality control activities value trends, reviews and related tests, providing support and consultancy in QA matters. • Allow EMI to pass customer acceptance criteriacontinually improve software quality and process itself DoW SO 1.5 • Continuous building, integration and testing process by selecting and maintainingtools and resources for building and testing either within the project or in collaboration with external resource providers EMI 2nd EC Review

  5. SA2 Quality Assurance SA2 Quality Assurance SA1 SA1 QA Policies QA Metrics Development Trackers Reports & Dashboards Quality Control Integration Infrastructure Build and Test Tools JRA1 JRA1 NA2 EMI 2nd EC Review

  6. SA2 QA Strategy EMI merged 4 established Middleware projects • Each had its own QA practices, tools, reports, etc. Working with a 3-years vision • Y1: explain, define and implement • Y2: review and automate • Y3: consolidate and optimize Benefitted from existing QA practices • Used existing QA tools, resources and expertise • Involved PTs, SA1, JRA1and also explained QA constraints • Associated policies with tools to support them for build, test, release and automated reports EMI 2nd EC Review

  7. Recommendations from 1st Review 1. Resubmit SQAP deliverable • Submitted in October 2011. Reorganized as a plan for the 3 years. 2. Reduce correlated deliverables • Merged QA and QC deliverable (DSA2.3.3) 3.Integrate Quality Control from SA1 and JRA1 into SA2 • Completed by June 2011 • Solid QC integration with policies, metrics and reports 4. Foster sustainable solutions after EMI • Added standard tools (Mock, Pbuilder) • Compel compliance to established distributions (EPEL, Debian) • Distribution of buildable sources in addition to binaries EMI 2nd EC Review

  8. SA2 Deliverables All SA2 deliverables submitted • DSA2.1 - Quality Assurance Planrewritten and resubmitted taking into account the review of Y1 • DSA2.2.3 - QA Tools Documentationupdate of previous deliverables DSA2.2.1 and DSA2.2.2 • DSA2.3.3 - Periodic QA and QC Reports includes DJRA1.7.3 - Software Development QC Report EMI 2nd EC Review

  9. EMI Tools and ETICS in Y2 DoW SO 1.5 : Continuous integration, building and testing process Full support Fedora (SL5, SL6), Debian 6 • Standard build tools (Mock and PBuilder) • Standard packaging/repositories (EPEL and APT) Two ways of building • Use ETICS for build and packaging • Give sources to build with Mock/Pbuilder All EMI software builds for EMI platforms • Single project configuration • Used by PTs and for global Nightly Builds • Better workload management • Maven mirror, VM images for developers • Elastic build infrastructure Build and Test Tools Montlhy ETICS Builds EMI 2nd EC Review

  10. Testing Infrastructure in Y2 DoW SO 1.5 : Continuous integration, building and testing process Fundamental for releases • 3 platforms x Releases x RCs x 32/64 bits • >200 installations of pre-EMI, EMI-1, EMI-2 products • CERN, CESNET, CNAF INFN, DESY, JUELICH, KOSICE, NIIF Everything has been deployed and tested on the QA Testbed • 15 EMI 1 Updates + EMI 2 Release • Provided feedback on documentation, and deployment issues to PTs Fundamental for testing, training, demos, monitoring • Adopted common EMI deployment tests, integration tests across products • Used training and demos, supported “EGI TF 2011 best demo” award Collaboration with EGI is operational and effective • Releases EMI project regularly pass the EGI validation  you will see later • Deployed SAM-NAGIOS for automated testing and monitoring Integration Infrastructure

  11. QA Metrics and Reports in Y2 DoW SO 1.4: Common software quality assurance policies and metrics Metrics on code, process, docs • Reaction to RfCs, delays in releases • Compliance to standards and templates Automated reports and live dashboards • Focus on metrics immediately useful • RfC management, certification and testing Total control of Development Process • Status of the products, tests, trackers • Used by QC and Product Teams Faster and compliant product releases • Submit all QA issues to PTs for prompt action • Products ready for passing the QC tests QA Metrics Reports & Dashboards EMI 2nd EC Review

  12. EMI RfC Dashboard http://emi-rfc.cern.ch Data from >30 trackers of all EMI Product Teams, status of EMI developmentAccess to all the RfC in each tracker, search and select, export to Excel EMI 2nd EC Review

  13. EMI Verification Dashboard http://emi-verification.cern.ch Access to all reports of each product, complete view of the products EMI 2nd EC Review

  14. QA Policies and Control in Y2 DoW SO 1.4: Common software quality assurance policies and metrics Software Quality Assurance Plan updated • Reorganised to match your recommendations • Added milestones and targets that were in other documents All Product comply with the platforms supported • Multiple platforms support (release, configuration, packaging, doc, etc.) Quality Control monitors compliance to policies • Uses the QA dashboards to view and extract the desired metrics • Metrics automatically extracted to tabular data (e.g. for SA2 and SA1) • Weekly reports and plots from the metrics and constant follow-up Quality Control has impact on work of Product Teams • Products passing are deployed, product not passing are discussed with the PTs • RfCs are issued and the execute the changes QA Policies Quality Control EMI 2nd EC Review

  15. QC of SA2 Services (KPIs) EMI 2nd EC Review

  16. QC – Testing Improvements Quality Control EMI 1 Updates 1-7 Updates 8-14 EMI 2nd EC Review

  17. QC – Regression Testing Report of Regression Tests for High/Immediate RfCs Quality Control Policy enforced by Release Manager after update 7 Data not collected before update 5 EMI 2nd EC Review

  18. QC – Compliance to Fedora/EPEL Quality Control Convergence to standards of EMI software 90 % 60 % EMI 2nd EC Review

  19. QC – Defect per Product Quality Control Number of Defects per Products per Quarter EMI 2nd EC Review

  20. From EGI TCB 11 – April 2012 DoW SO 1.4: Allow EMI to pass customer acceptance criteria 100% 90% 80% 70%

  21. Conclusions Main Achievements • New platforms successfully supported • Convergence to standards, QA has an impact on the actual quality • Faster testing and successful deployment of products • Very high success rate in the acceptance of our releases to EGI Excellent collaboration and results across all EMI activities • SA1 for releases, updates and deployment. NA2 for training, documentation. JRA1 quality control and dashboards Quality Control merge in SA2 a very good decision • Quicker feedback to EMT and within SA2 (metrics, reports, reviews) • Someone external to PTs checks products (deployment, compliance) Y3: Consolidate and Optimize • Continue to support/improve current tools, testbed, dashboard and reports • QC on sustainability of middleware products beyond the EMI Y3 EMI 2nd EC Review

  22. Thank You

More Related