A Brief History of the 50-Year-Old Space Physics Data Environment

1. A Brief History of the 50-Year-Old Space Physics Data Environment Joseph H. King Perot Systems at NASA/Goddard Space Flight Center, Greenbelt, MD

2. A Framework for the Talk How do selected, publicly accessible, space physics systems and interfaces, created at various times, satisfy various functional and search requirements? Some repository-created systems provide value-added access to data held at those repositories Other systems are specifically oriented towards a distributed data environment Focus is not on underlying technologies, although their evolution has enabled the great progress in satisfying functional requirements. We suppress programmatic history (e.g., NASA’s Rice U. and River Bend meetings). We underemphasize format and documentation issues. We do not address data preservation/stewardship issues nor creation of value-added data products like “OMNI”.

3. Functional Requirements Data Finding Data product level Data granule level Data Retrieval File level Subset by parameter Data Display Lists Plots Data Processing to higher levels Data Analysis tools Coordinate transformations Cross correlations Etc. Data Usability Supporting material, including error bars or equivalent, enables long-term correct and independent usability

4. Search Requirements Free text Physical Observables Time span Time resolution Spectral range Spectral resolution Region sensed In situ Remote sensing Observatory and Instrument Person Repository Rare event occurrence Parameter inside or outside specified range Concurrence of data in multiple favorable locations

5. Some repository-specific systems and interfaces (many dates are approximate) Pre-network 1978 CDAW at NSSDC, 10’s of data sources Pre-Web 1981 SDDAS @ SWRI for Cluster, DE, Tether, Astrid… 1983 OMNI captive account at NSSDC UCLA magnetic field data from Polar, ISEE 1,2 3, IMP8… 1992 SDAC @ GSFC for SMM, Yohkoh, SOHO, Web era 1994 OMNIWeb @ NSSDC 1994 COHOWeb @ NSSDC for Voyager,Ulysses, P10&11... 1994 CDAWeb @ SPDF for ISTP s/c and many more SPIDR @ NGDC for GOES, etc. DARTS @ JAXA, Japan, for Geotail, Reimei, Akebono CDPP (Toulouse), for Cluster, Wind, Ulysses, ARCAD3… ACE Science Center (Caltech) for ACE and SAMPEX Etc.

6. Coordinated Data Analysis Workshop Developed at NSSDC in 1977-8 to support International Magneto- spheric Study analysis Major effort to find good events (spacecraft well located, interesting geophysical events, good data coverage) Event selection supported by Satellite Situation Center with satellite-satellite and satellite-ground orbits and conjunctions Participants submitted space and ground data in advance, then came to NSSDC for workshop Eight CDAW’s hosted, 1978-1990. Clusters of JGR papers from CDAW’s 2, 6, 8. (10’s of data sets per CDAW, e.g., 65 data sets used for CDAW 6) Last 2 CDAW’s hosted on VAX’s, based on CDF and network-accessible Paper data catalogs used by participants, plots and lists made on the fly at CDAW. New parameters creatable on the fly. The term “CDAW” was taken over by the solar community in about 1999

7. Southwest (SWRI) Data Display and Analysis System (SDDAS) First created at SWRI in 1981 to support Dynamics Explorer data analysis. Using calibration and related data and information, worked with telemetry-level data and produced plots of geophysical parameters for analysis. Required Instrument Data File Set (IDFS), involving a header file, a data file, a “virtual instrument” data file (VIDF), and an optional “plot information and description file” (PIDF). Can output data files in several formats (CDF, netCDF, ASCII). Now operates in client-server mode, with IDFS-packaged data at sites in addition to SWRI (e.g., MSSL, IRF) – significant usage in Cluster community. Spacecraft supported: TSS, UARS, AMPTE, ARIA, DE, DMSP, ISEE, Spacerad, Interball, IMP, Astrid, Cluster, MARS, Prognoz, Viking, Freja, ISIS, NOAA-12, etc.

8. “NSSDC Account” for OMNI Data Access Captive account created by NSSDC in ~1983 to provide access to OMNI data and then some other data. Allowed users to select from list of several tens of physical parameters per OMNI record and to receive output listing or file of just the selected parameters for the time span specified. Retired when OMNIWeb became available in ~1994. _________ OMNI: A multi-spacecraft compilation of hourly resolution solar wind magnetic field and plasma data, energetic particle data and geomagnetic and solar activity indices spanning 1963 to near current. HRO (High Resolution OMNI: 1-min and 5-min) spanning 1995 to near current recently made available; data are as shifted to Earth’s bow shock nose.

9. COHOWeb System created at NSSDC to provide access to coordinate-system-uniformized and content-uniformized hourly solar wind magnetic field and plasma data from several deep-space spacecraft. Provides subset, plot and list capabilities. Provides additional analysis capabilities: Scatter plots and linear regression fits Cross correlation coefficients Distribution functions, means and medians Filtering for any of these by any physical parameters Spacecraft supported: Voyager 1 and 2 Pioneer 10, 11 and PVO Ulysses Helios 1 and 2

10. Main COHOWeb Web Page

11. COHOWEB scatter plot interface

12. Solar Wind flow speed vs. heliolatitude for first half orbit leg,1992.0-1995.2

13. Solar Wind flow speed vs. heliolatitude for second half orbit leg,1995.2-1998.0

14. Solar Wind flow speed vs. heliolatitude for third half orbit leg,1998.0-2001.4

15. Solar Wind flow speed vs. heliolatitude for forth half orbit leg,2001.4-2004.2

16. Solar Wind flow speed vs. heliolatitude for fifth half orbit leg,2004.4-2007.3

17. Systems for supporting distributed data NSSDC’s NASA Master Directory SPDS’s Subdiscipline-specific web pages Virtual Solar Observatory Virtual Space Physics Observatory Virtual Heliospheric Observatory Virtual Solar Terrestrial Observatory Others in development

18. NASA Master Directory Started by NSSDC in ~1985 to enable data product finding. Discipline horizon spanned space and Earth science data. Became space science specific after Global Change MD separated in early 1990’s. Query functionality: Free text Parameters measured Time span Region sensed, location Observatory and instrument Person Network connections via SPAN/DECnet to underlying systems holding data or further information (about 20 such systems in 1991). Abandoned in mid-1990’s due to limited usage and difficulties maintaining information current.

19. SPDS Information Systems A series of information systems about distributed data created in ~1996-7; now largely abandoned. Intended to assist in the data product finding effort. MOLD – Magnetospheric OnLine Data directory, Jeff Reeves, LANL. Hosted by GSFC/SPDF in later years. Cosmic & Heliospheric – NSSDC ITM – SWRI(?) MOLD and C&H reached a reasonable population level and had modest-level usage.

20. Virtual Space Physics Observatory Started at GSFC (A. Roberts) in about 2003 to aid in data finding, retrieval and display. Discipline horizon is space and solar physics. ~ 400 space- and ground-based product descriptions as of today Developed in parallel with SPASE, therefore highly SPASE compatible. Query functionality: Free text Parameters measured Time span and resolution, Spectral range Region or remote object sensed. Observatory and Instrument Person, Repository Provides parameter-subsetted data retrieval and display functionalities for some data by linking to underlying interface (e.g., CDAWeb) via Web services or by emulating underlying interface (e.g., SPDF’s FTPBrowser)

21. VirtualHeliospheric Observatory In development at GSFC (Szabo et al) with a distributed team. Provides current access to ACE, Wind, Genesis, Helios, IMP 8, SOHO and MGS field and plasma data. Search by time, location (ranges in X,Y,Z gse), physical observable (field or plasma), spacecraft, instrument Clickable file-level URL’s returned.

22. Virtual Solar Terrestrial Observatory In development at NCAR (P. Fox) Provides current access to data at CEDAR and MLSO (Coupling, Energetics and Dynamics of Atmospheric Regions) (Mauna Loa Solar Observatory) Significant emphasis on ontology (developed independently of SPASE) Search by time, instrument or physical parameter, with “filtering” by domain (CEDAR vs. MLSO) or by parameter type. Deals with registration requirement for downloads of CEDAR digital data. Get URL’s of files satisfying requirements, with option to choose format (FITS, ASCII, JPEG, etc.)

23. Thoughts for current VO activities Data modeling and ontology work is very valuable for data finding, but to ensure data understanding and correct and independent usability, more data-descriptive documentation is needed than that required to pass SPASE (or equivalent) validation tests. Whole data file delivery is desired even when a (desirable) subsetting option is available, especially if the files are immediately usable (I have a bias for ASCII files!). Data environment should facilitate users’ access to either repositories’ data access paths or to VOs’ access paths. (e.g., VSPO does this for VSO-accessible data) Data environment should support data access by either people or by computer programs. In multi-level multi-nodal data environment, new resource-acknowledgement practices will be important to each node’s desire for continuation.