Data Publication and Quality Control Procedure for CMIP5 / IPCC-AR5 Data

1. Scientific Q. Assurance Technical Q. Assurance DOI- Publication Permission: QC L2 QC Services QC Services Scientist QC Tool QC Tool Web User Environment Application Environment Scripts Metadata QC L2 passed? Users NERSC NCAR Community Tools DKRZWDCC Metadata Handling Data Products Publication Agency QC L1L2L3 Harvester Security LLNL PCMDI ESG Federated Gateways BADC Semantic Data Search & Discovery QC L1L2 QC L1L2 Metrics Aggregation To be replicated among ESGF? User Registration Publishing API NASA JPL Database Registration Agency ANU ORNL QC L1L2 PCMDI Data Node TDS Catalogs Node Database GFDL Data Node NASA/GISS Data Node Security Middleware … … Publisher THREDDS IPSL Data Node INM Data Node Disk Cache TIME Deep Archive PCMDI/BADC/WDCC IDF Data Published > 10 PBData and Metadata QC L1 -------------------------------- On globally distributed data nodes Replicated:Copied to PCMDI, BADC, WDCC & elsewhere ~ 1PB DN TDS QC L2 DN TDS QC L2 ESGGateways CIMQuestionnaire DOI Catalogue DN TDS QC L2 Store, Analyse,and Plot Results MD onModel /Simulation MD on Data QC Repository Data QC L2 passed? YES YES YES MD onQuality QC L3 passed? DOI Access CIMMD Repository Quality Results Harvest all MD QC L3 / STD-DOI NO NO NO YES Parts of Data /MD on Simulation TQA DOI Target Page Discard data MD on Data / Quality / Model / Simulation CERA2MD Harvest all MD oflong-term archive Register STD-DOI /URL of DOI Target Page DOI Assigned: Data formally citable Data can appear in IPCC-DDC ------------------------------------------ Automatic access granted after filling in ESGF registration page Data NOT formally citable Modelling group control access manually. Data NOT formally citable Automatic access granted after filling in ESGF registration page. Atarrabi DOI Data Long-term Archive SQA ofData Author WDCC: DOI Publication Agency (Informal citation still requested where formal citation not available) Multiple Sites performinglocal QC checks Central Repository DOI Publication Agency with long-term archive ProjectMetadata Repository QC Services QC Services DOI process QC Tool QC checks: • QC Run Service: QC tool run and Repository ingestsof configuration and results • QC Services for data analyses and exception statistics • QC Plotting Service and plot ingest in Repository • QC L2 assignment • Export QC results for DOI publication process DOI publication: Granularity of QC in CMIP5context Data Publication and Quality Control Procedure for CMIP5 / IPCC-AR5 Data Browsers & Scripts Martina Stockhause, Michael Lautenschlager, Heinke Hoeck, and Frank Toussaint EGU2011-2859 Future Perspective CMIP5 Organization & Infrastructure Components CMIP5 Quality Control (QC) For CMIP5 ca. 3 PB of officially requested data are expected to be archived. About 1 PB of thatdata will likely be of especially high interest and will be replicated by the three ESGF partners. The current DOI publication procedure is comparable to the publication of grey literature inscientific print media. For the integration of a peer review process quality procedures acceptedand agreed on by the earth system modelling community are necessary. For distribution of data connected to the next IPCC report, the Earth System Grid Federation(ESGF) was founded. Its members have different responsibilities within the data infrastructure: • PCMDI / LLNL: data and security infrastructure (ESG) • BADC (British Atmospheric Data Centre): metadata infrastructure (METAFOR / CIM) • WDCC (World Data Center Climate) / DKRZ: quality control, data publication (DataCite DOI) Because of the high data volume the QC checks up to level 2 are performed distributed amongthe ESGF. The final QC level 3 checks for the DOI assignment are carried out by WDCC.Afterwards CMIP5 data is formally citable and remains persistent. Browsing The distributed quality control approach could be reduced in complexity by the integration ofthe QC Repository into the CIM Metadata Repository. OPeNDAP GridFTP Product Manager CMIP5 Quality Control Workflow Distributed Quality Control Approach Data Publication Procedure .org Overall CMIP5 QC Workflow Quality Levels for CMIP5 Workflow of Distributed QC in CMIP5 Actors in DOI Publication Process DOI Publication Process Distributed QC Approach The final DOI data publicationprocedure is in agreement with theregulations of the DataCite consortium: For high volume data such as climatedata quality assurance has to becarried out at the data storage centrebefore opening the repository for dataaccess. Data distributed in a Data Gridwith its decentralized data repositorieshave to be checked at different siteswith comparable QC procedures. Three Quality Control (QC) Levelsare defined for CMIP5 data: • QC Level 1:Metadata: Technical checks on METAFOR questionnaire input dataData: CMOR2 and ESG publisher conformance checks • QC Level 2: Metadata: METAFOR questionnaire metadata checked by scientist Data: Technical checks e.g. on the reliability of variable ranges and the consistency checks between data and data requirements • QC Level 3 / DOI: Data approved by author and published as DOI • Scientific Quality Assurance: performed by the data author and documented via a publication service GUI (atarrabi) • Technical Quality Assurance: cross- and double checks of data and metadata integrity • DOI Publication: DataCite DOI metadata and DOI are separately send to the registration agency, a member of the DOI Foundation. Data and DOI remain unchanged and persistent. Thus a QC procedure/tools have to bedeveloped, maintained and distributedcentrally and agreed upon within thescientific community. Our distributed QC approach consistsof different software components: ___________________________________________________________________ESG: Earth System Grid, MD: Metadata, DN: Data Node, TDS: Thredds Data Server,TQA: Technical Quality Assurance, SQA: Scientific Quality Assurance. DOI Construction Rule for CMIP5: Quality Control Workflow CMIP5 doi: 10.1594/WDCC/CMIP5.<opaque bit> The different QC levels are connected with different access rights for registered users: Distributed QC Procedure in CMIP5 DOI Publication GUI atarrabi QC L3 / DOI Process in CMIP5 QC checks for data and metadataare performed, separately, for levels 1 and 2. During the cross-checksof QC L3 checks their results arereviewed. CMIP5 data is delivered to one of the three ESGFpartners, where it is ESG published and thus QC L1Data checked. Afterwards QC L2 Data consistencychecks are performed, before a data subset isreplicated among the ESGF. QC L2 results arestored in a central QC Repository. • Restricted Access (QC Level 1 Data): After ESG publication the access is restricted under control of the modelling centre. • Scientific Access (QC Level 2):The scientific community is granted access of data of QC L2. • QC Level 3 / DOI: With the DOI assignment the data archive is opened for access by non-scientific users. The cross- and double checks of theTechnical Quality Assurance make use of the QC result of the precedinglevels. Data as well as metadata isreviewed and data accessibility checked. Data assigned a DOI is formallycitable and is granted persistent access. Organisation of CMIP5 Data Federation During QC L3 / DOI checks the QC results areaccessed by the DOI Publication Agency WDCC.Other sources for cross- and double-checks arethe CIM Metadata Repository, the Thredds DataServer (TDS), and the metadata stored in the long-term archive at WDCC. The final author approval step issupported by the GUI atarrabi. Authors check basic metadata and add information about their own quality assurance (Scientific QA). Granularity of Quality Control QC is accomplished on DRS Atomic Dataset level. The QC results are aggregated on DRS experiment level. In the gateways data discovery is supported downto the level of Ensemble versions (ESG dataset). A DOI is assigned and registered atthe International DOI Foundation (IDF: dx.doi.org) via the Registration Agency DataCite. Thus, the effort of the QC L2 Data checks is sharedamong the ESGF. But the QC L3 / DOI checks areperformed at one site making use of the QC L2results stored in a central QC Repository. More Information: http://purl.org/org/cmip5/qc More Information: http://cera-www.dkrz.de/atarrabi More Information: http://purl.org/org/cmip5/qc martina.stockhause@zmaw.de WDC Climate / DKRZ: www.wdc-climate.de; www.dkrz.deCMIP5: cmip-pcmdi.llnl.gov/cmip5; CMIP5 Quality Control: purl.org/org/cmip5/qc