1 / 11

Data Handling

Data Handling. 2004. South Pole Test System (SPTS) operational South Pole System (SPS) assembled at PSL SPS shipped to the South Pole SPS installed at the South and operational with 1 string SPADE operational moving summer data and TestDAQ data now. 2005.

fuller
Download Presentation

Data Handling

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data Handling Berkeley IceCube Meeting 21-23 March

  2. 2004 • South Pole Test System (SPTS) operational • South Pole System (SPS) assembled at PSL • SPS shipped to the South Pole • SPS installed at the South and operational with 1 string • SPADE operational moving summer data and TestDAQ data now Berkeley IceCube Meeting 21-23 March

  3. 2005 • Data input methods for Data Warehouse • Web interface for Data Warehouse • Upgrade data storage facility • Implement IceCube Metadata standard and directory structure • Upgrade SPS and SPTS for 05/06 season Berkeley IceCube Meeting 21-23 March

  4. Data Warehouse • Well organized, accessible, catalogued, and controlled data repository • The basis for cataloging is metadata • Data input (ingestion) via application (Ingest) or direct write • Access via application (web) or direct read • Controls to ensure conformity to cataloging requirements and access control Berkeley IceCube Meeting 21-23 March

  5. Ingest Methods Solution is 4 models (Mackenzie) • Direct write, catalogued by directory and metadata supplied by user • Direct write, catalogued by directory and metadata supplied by application • Direct write, catalogued by file and metadata supplied by user • Via application, catalogued by file and metadata supplied by application Berkeley IceCube Meeting 21-23 March

  6. Ingest Methods Long term • Web application for metadata creation, modification, and catalog searching • Interfaces to standard simulation and analysis Short term • Manual creation of directory locations by Data Center via help@icecube.wisc.edu request • Metadata by filling out templates • SPADE adapted for application based ingestion Berkeley IceCube Meeting 21-23 March

  7. Directory Structure <DATAHOME>/(exp|sim|ana)/<experiment>/<yyyy>/<category>/[subcategory/] \[mmdd/]filename Examples /data/exp/IceCube/2005/TestDAQ/0314 /data/sim/AMANDA/2000/corama Berkeley IceCube Meeting 21-23 March

  8. Metadata Group: DIF Entry_ID: 12345 Entry_Title: AMANDA Data Group: Data_Center Data_Center_Name: UWI-MADISON > University of Wisconsin-Madison AMANDA/IceCube Group Data_Center_URL: http://icecube.wisc.edu/datacenter Group: Personnel Role: Data Center Contact End_Group End_Group Group: Parameters Category_Topic_Term_Variable: PHYSICS > ASTROPHYSICS > NEUTRINOS > ATMOSPHERIC End_Group Summary: Data from the South Pole Sensor_Name: AMANDA > Antarctic Muon and Neutrino Detector Array Group: Temporal_Coverage Start_Date: 2000-01-01 Stop_Date: 2001-12-31 End_GroupEnd_Group Berkeley IceCube Meeting 21-23 March

  9. Metadata <?xml version="1.0" encoding="UTF-8"?><DIF> <Entry_ID>12345</Entry_ID> <Entry_Title>AMANDA Data</Entry_Title> <Data_Center> <Data_Center_Name>UWI-MADISON > University of Wisconsin-Madison AMANDA/IceCube Group</Data_Center_Name> <Data_Center_URL>http://icecube.wisc.edu/datacenter</Data_Center_URL> <Personnel> <Role>Data Center Contact</Role> </Personnel> </Data_Center> <Parameters> <Category_Topic_Term_Variable>PHYSICS > ASTROPHYSICS > NEUTRINOS > ATMOSPHERIC</Category_Topic_Term_Variable> </Parameters> <Summary>Data from the South Pole</Summary> <Sensor_Name>AMANDA > Antarctic Muon and Neutrino Detector Array</Sensor_Name> <Temporal_Coverage> <Start_Date>2000-01-01</Start_Date> <Stop_Date>2001-12-31</Stop_Date> </Temporal_Coverage></DIF> Berkeley IceCube Meeting 21-23 March

  10. Data Handling Berkeley IceCube Meeting 21-23 March

More Related