1 / 14

Storage

Storage. Why is storage an issue? Space requirements Persistence Accessibility Needs depend on purpose of storage Capture/encoding Access/delivery Preservation. Storage: Working Space. Space for storage of digital files during capture/encoding/quality control process Possibilities

rangle
Download Presentation

Storage

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Storage • Why is storage an issue? • Space requirements • Persistence • Accessibility • Needs depend on purpose of storage • Capture/encoding • Access/delivery • Preservation

  2. Storage: Working Space • Space for storage of digital files during capture/encoding/quality control process • Possibilities • PC hard drive • File server, e.g. marengo (LIT) • DLP file server • Issues • Capacity, backup, speed, accessibility

  3. Storage: Access/Delivery • Storage for web delivery of images, audio, text, etc. • Possibilities • UITS web server, under library account • UITS streaming media server (audio/video) • DLP web server • Issues: capacity, backup, performance, software integration, maintenance/migration

  4. Storage: Preservation • Much harder problem • Longer term • Issues of longevity of media, hardware, file format • Where are the files? • Larger files • Hard disk storage, traditional backup methods not cost-effective • Infrequency of access • Problems do not become immediately evident

  5. Long-Term Storage Options • Removable media • e.g. CD-R, DVD-R • Pros: cheap, easy, produces tangible item • Cons: Low capacity, physical space requirements, unknown longevity, migration • Nearline storage • UITS Massive Data Storage Service

  6. UITS MDSS • Massive Data Storage Service • HPSS (High Performance Storage System) software • Developed as collaboration of IBM and US national labs • Four tape robots (two at IUB, two at IUPUI) • Data can be mirrored • 540 TB total storage • ~75 TB used as of April 2001

  7. MDSS – A Sense of Scale • 2 Kilobytes : A typewritten page • 5 Megabytes : Complete works of Shakespeare OR 30 seconds of TV quality video • 1 Gigabyte (1000MB) : 1 pickup truck filled with paper OR a symphony in hi-fi sound • 1 Terabyte (1000GB) : All the X-ray films in a large hospital OR paper from 50,000 trees • 10 Terabytes : The printed collection of the US Library of Congress • 50 Terabytes : The contents of a large mass store system • 8 Petabytes (8000TB) : All information available on the web • 200 Petabytes : All the printed material (in the world!)

  8. MDSS Storage Infrastructure

  9. MDSS • Access • FTP/PFTP: (Parallel) File Transfer Protocol • DFS: Distributed File System (being phased out) • HSI • Not practical for delivery • Hierarchical storage (metadata on disk, data on tape -> 30-90 second to start transfer.) • File size – chunks of 50 MB or greater work best • Small files aggregated into larger .tar or .zip files

  10. DL Objects • Digital library “objects” have many parts • Metadata • Preservation files • Delivery files • How do we keep them connected? • Now: Good practice in file naming, directory organization, project documentation -not scalable! • Future: Digital object repository

  11. Data Persistence • Key is migration • Keeping the bits alive - MDSS responsibility • Physical media • Logical media format • Keeping the bits understandable - MDSS user responsibility • File format • Metadata • Small “pockets” of digital content pose a problem for migration

  12. DL Object Repository Preservation version in MDSS Repository System Users and applications Delivery version on web server Metadata records

  13. DL Repository Models • OAIS: Open Archival Information System Reference model • Fedora: Flexible and Extensible Digital Object and Repository Architecture • Developed at Cornell and UVa • IU DLP in deployment group

  14. DLP Storage Services • Consulting • Server space for production and access • Persistent naming service (PURL server) • Facilitation of access to UITS services • Streaming media • MDSS • Developing repository service • Contact: diglib@indiana.edu

More Related