1 / 37

NC Geospatial Data Archiving Project (NCGDAP)

Cooperative Project with Library of Congress on Preservation of Digital Geospatial Data Steve Morris Head of Digital Library Initiatives NCSU Libraries. NC Geospatial Data Archiving Project (NCGDAP). Partnership between NCSU Libraries and NC Center for Geographic Information & Analysis

chip
Download Presentation

NC Geospatial Data Archiving Project (NCGDAP)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Cooperative Project with Library of Congress on Preservation of Digital Geospatial DataSteve MorrisHead of Digital Library InitiativesNCSU Libraries

  2. NC Geospatial Data Archiving Project(NCGDAP) • Partnership between NCSU Libraries and NC Center for Geographic Information & Analysis • $520,000 funding – 3 years • Focus on state and local geospatial content in North Carolina (statedemonstration) • Address NC OneMap objective: “Historic and temporal data will be maintained and available.” • One of eight projects in the first NDIIPP funding round: “Building a Network of Partners” Note: Percentages based on the actual number of respondents to each question

  3. Note: Percentages based on the actual number of respondents to each question

  4. NDIIPP Overview • National Digital Information Infrastructure and Preservation Program • Congress appropriated $100 million for this effort, which instructs the Library to spend an initial $25 million to develop and execute a congressionally approved strategic plan • Eight initial projects, 2004-2007: • web pages, cultural heritage, numeric data, video, business records, mixed content, geospatial (2) • Developing partnerships and identifying issues • Extensive interaction among NDIIPP projects Note: Percentages based on the actual number of respondents to each question

  5. Targeted Content • Resource Types • GIS “vector” (point/line/polygon) data • Digital orthophotography • Digital maps • Tabular data (e.g. assessment data) • Content Producers • Mostly state, local, regional agencies • Some university, not-for-profit, commercial • Selected local federal projects Note: Percentages based on the actual number of respondents to each question

  6. Risks to Digital Geospatial Data .shp .mif .gml .e00 .dwg .dgn .bsb .bil .sid Note: Percentages based on the actual number of respondents to each question

  7. Risks to Digital Geospatial Data • Focus on current data • Archiving data does not guarantee “permanent access” • Future support of data formats in question • Need to migrate formats or allow for emulation • Data failure • “Bit rot”, media failure • Preservation metadata requirements • Descriptive, administrative, technical, DRM • Shift to “streaming data” for access Note: Percentages based on the actual number of respondents to each question

  8. Time series – vector data Parcel Boundary Changes 2001-2004, North Raleigh, NC Note: Percentages based on the actual number of respondents to each question

  9. Time series – Ortho imagery Vicinity of Raleigh-Durham International Airport 1993-2002 Note: Percentages based on the actual number of respondents to each question

  10. Today’s geospatial data as tomorrow’s cultural heritage Note: Percentages based on the actual number of respondents to each question

  11. Earlier NCSU Acquisition Efforts • NCSU University Extension project 2000-2001 • Target: County/city data in eastern NC • “Digital rescue” not “digital preservation” • Project learning outcomes • Confirmed concerns about long term access • Need for efficient inventory/acquisition • Wide range in rights/licensing • Need to work within statewide infrastructure • Acquired experience; unanticipated collaboration Note: Percentages based on the actual number of respondents to each question

  12. One Earlier Project Outcome: Directory of County and City Services Among top 15 most used resources on library web site 99.5% of directory users from outside ncsu.edu Note: Percentages based on the actual number of respondents to each question

  13. NDIIPP Project Phases • Content Identification and Selection • Content Acquisition • Partnership Building • Content Retention and Transfer All 8 NDIIPP cooperative projects adhere to this structure Note: Percentages based on the actual number of respondents to each question

  14. Content Identification and Selection • Work from NC OneMap Data Inventory • Combine with inventory information from various state agencies and from previous NCSU efforts • Develop methodology for selecting from among “early,” “middle,” and “late” stage products • Develop criteria for time series development • Investigate use of emerging Open Geospatial Consortium technologies in data identification Note: Percentages based on the actual number of respondents to each question

  15. Content Acquisition • Work from NC OneMap Data Sharing Agreements as a starting point (the “blanket”) • Secure individual agreements (the “quilt”) • Investigate use of OGC technologies in capture • Use METS (Metadata Encoding and Transfer Standard) as a metadata wrapper • Bundle data files, metadata, ancillary documentation • Supplement FGDC metadata with additional administrative, technical, and descriptive metadata • Encode rights (Digital Rights Management – DRM) • Links to services Note: Percentages based on the actual number of respondents to each question

  16. Partnership Building • Work within context of the NC OneMap initiative • Explore state, local, federal partnerships • Defined characteristic: “Historic and temporal data will be maintained and available” • Advisory Committee drawn from the NC Geographic Information Coordinating Council subcommittees • Seek external partners • National States Geographic Information Council • FGDC Historical Data Committee • … more Note: Percentages based on the actual number of respondents to each question

  17. Content Retention and Transfer • Ingest into Dspace open source digital repository software • Look more generically at the issue of putting geospatial content into digital repositories • Investigate re-ingest into a second platform • Start to define format migration paths • Special problem: geodatabases • Purse long term solution • Roles of data producing agencies, state agencies; NC OneMap; NCSU Note: Percentages based on the actual number of respondents to each question

  18. Big Geoarchiving Challenges • Format migration paths • Management of data versions over time • Preservation metadata • Preserving cartographic representation • Keeping content repository-agnostic • Preserving geodatabases • Harnessing geospatial web services • More … Note: Percentages based on the actual number of respondents to each question

  19. Vector Data Format Issues • Vector data much more complicated than image data • ‘Preservation’ vs. ‘Permanent access’ • An ‘open’ pile of XML might make an archive, but if using it requires a team of programmers to do digital archaeology then it does not provide permanent access • Piles of XML need to be widely understood piles • GML: need widely accepted application schemas (like OSMM?) • The Geodatabase conundrum • Export feature classes, and lose topology, annotation, relationships, etc. • … or use the Geodatabase as the primary archival platform (some are now thinking this way) Note: Percentages based on the actual number of respondents to each question

  20. Geography Markup Language Issues • GML still more useful as a transfer format than an archival format, support limited even for transfer • FGDC Historical Data Working Group investigations into GML for use in archiving • Plans for environmental scan of existing GML profiles and application schemas or profiles • schema name (e.g. OSMM, top10NL, ESRI GML, LandGML) • responsible agency; scheme has official government status? • GML version; known unsupported GML components • schema history; known interoperation with other schemas • vendor support; translator support Note: Percentages based on the actual number of respondents to each question

  21. Managing Time-versioned Content • Many local agency data layers continuously updated • Older versions not generally available • Individual versioned datasets will wander off from the archive • How do users “get current metadata/DRM/object” from a versioned dataset found “in the wild”? • How do we certify concurrency and agreement between the metadata and the data? Note: Percentages based on the actual number of respondents to each question

  22. Preservation Metadata Issues • FGDC Metadata • Many flavors, incoming metadata needs processing • Other standards: PREMIS, MODS • Metadata wrapper • METS (Metadata Encoding and Transmission Standard) vs. other industry solutions • Need a geospatial industry solution for the ‘METS-like problem’ • GeoDRM a likely trigger—wrapper to enforce licensing (MPEG 21 references in OGC Web Services 3) Note: Percentages based on the actual number of respondents to each question

  23. Preserving Cartographic Representation • The true counterpart of the old map is not the GIS dataset, but rather the cartographic representation that builds on that data: • Intellectual choices about symbolization, layer combinations • Data models, analysis, annotations • Cartographic representation typically encoded in proprietary files (.avl, .lyr, .apr, .mxd) that do not lend themselves well to migration • Symbologies have meaning to particular communities at particular points in time, preserving information about symbol sets and their meaning is a different problem Note: Percentages based on the actual number of respondents to each question

  24. Preserving Cartographic Representation Note: Percentages based on the actual number of respondents to each question

  25. Preserving Cartographic Representation • Image-based approaches (“dessicated data”) • Generate images using Map Book or similar tools • Harvest existing atlas images • Capture atlases from WMS servers • Export ‘layouts’ or ‘maps’ to image • Vector-based approaches • Store explicitly in the data format (e.g. Feature Class Representation in ArcGIS 9.2) • Archive and upward-migrate existing files .avl, .apr, .lyr, .mxd, etc. • SVG, VML or other XML approaches • Other? Note: Percentages based on the actual number of respondents to each question

  26. Preserving Cartographic Representation Note: Percentages based on the actual number of respondents to each question

  27. Preserving Cartographic Representation Note: Percentages based on the actual number of respondents to each question

  28. Preserving Geodatabases • Not just data layers and attributes—also topology, annotation, relationships, behaviors • ESRI Geodatabase archival issues • XML Export, Geodatabase History, File Geodatabase, Geodatabase Replication • Growing use of geodatabases by municipal, county agencies • Some looking to Geodatabase as archival platform (in addition to feature class export) Note: Percentages based on the actual number of respondents to each question

  29. Geodatabase Availability • According to the 2003 Local Government GIS Data Inventory, 10.0% of all county framework data and 32.7% of all municipal framework data were managed in that format. Note: Percentages based on the actual number of respondents to each question

  30. Evolving Geodatabase Handling Approaches Note: Percentages based on the actual number of respondents to each question

  31. Harnessing Geospatial Web Services • Automated content identification • ‘capabilities files,’ registries, catalog services • WMS (Web Map Service) for batch extraction of image atlases • last ditch capture option • preserve cartographic representation • retain records of decision-making process • … feature services (WFS) later. • Rights issues in the web services space are ambiguous Note: Percentages based on the actual number of respondents to each question

  32. Partnerships • ESRI • Discussing software requirements: meetings with development teams April 2005 • Open Geospatial Consortium (OGC) • Meet with Architecture Working Group Nov. 2005 • National Archives and Records Administration • Investigations into GML for archiving; planned presentation to NARA technology team • FGDC Historical Data Working Group • General geospatial data preservation issues Note: Percentages based on the actual number of respondents to each question

  33. Partnerships • EDINA (University of Edinburgh, UK) • NCSU is Associate Partner on UK project for geospatial institutional repositories • UC Santa Barbara & Stanford University • Other NDIIPP geospatial project • EROS Data Center • Planned site visit • Project visits to regional GIS groups • Albemarle Regional GIS meeting Nov. 3 • More planned … Note: Percentages based on the actual number of respondents to each question

  34. Progress to Date • Completion of project agreements • Hiring staff • Acquisition and deployment of storage system (12.4 TB capacity – two 16.8 TB systems) • Testing and deployment of repository software • Development of metadata workflow • Development of ingest workflow • Pilot project with NC Geologic Survey data … Initial focus on developing the “plumbing” Note: Percentages based on the actual number of respondents to each question

  35. Questions for You? • What are your current practices for: • Archiving data and managing time versions • Managing geodatabase versions • Transfer mechanisms for data • to regional entities? • to off-site storage for disaster recovery? • Archiving project files and finished products • What rights issues exist with regard to putting county and city data into an archive? • What would you like this project to do? Note: Percentages based on the actual number of respondents to each question

  36. Ways to Participate in NCGDAP • Identifying data for inclusion in the repository • Discussing data format strategies • Sharing ideas about archiving approaches and architectures • Sharing and identifying concerns about rights issues, liability, etc. • Host project visits to regional GIS groups • Use Local Government GIS listserv to discuss preservation issues? Note: Percentages based on the actual number of respondents to each question

  37. Questions? Contact: Steve Morris Head, Digital Library Initiatives NCSU Libraries Steven_Morris@ncsu.edu http://www.lib.ncsu.edu/ncgdap Note: Percentages based on the actual number of respondents to each question

More Related