1 / 26

Archiving and Preserving the Web: Challenges and the Future

Learn about the importance of archiving the web and the efforts being made by organizations like RLG and Internet Archive to preserve valuable online content. Discover the technologies used and the next steps in web archiving.

lkirsten
Download Presentation

Archiving and Preserving the Web: Challenges and the Future

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Archiving and Preserving the Web Dan Avery Kristine Hanna Merrilee Proffitt Internet Archive RLG April 2006

  2. Agenda RLG Internet Archive Archive-It Challenges The Future Q&A

  3. The importance of archiving the web • The web contains much of what will be the basis of scholarship in the future • record of events • official publications • personal viewpoints • ephemeral material

  4. RLG’s interest • RLG mission includes working with its member organizations to enhance their ability to provide research resources • RLG members have long been participating in web archiving, but so far, this has been an activity restricted to large organizations

  5. Members active in web archiving • Bibliothèque Nationale de France • British National Library • California Digital Library • Library of Congress • National Library of Australia • National Library of New Zealand

  6. Archive-It pilot partners • Indiana University • International Institute of Social History • University of Toronto • Swarthmore/Haverford College

  7. About Internet Archive • Founded in 1996 • Largest public web archive • 60 billion pages, 55 million sites • Have expanded to include texts, audio, moving images, and software: 2.6 million downloads a day

  8. What do we collect?Web Archive • Take a broad snapshot of the web every 2 months • 2 billion pages a month • Websites from every domain (.org, .com, .edu etc) • Content in 21 languages

  9. Policy • We follow Oakland Archive Policy, 2002 • Founded by commercial and non commercial organizations • Opt-out policy • We collect it all, and make it inaccessible if requested by site owner • Site owner directly blocks harvester on website

  10. Access to Web Archive • Entire archive accessible for free to the public via the website at www.archive.org • Receive100 hits/second • 60k unique users per day • Evolving/Fluid: through public use we hope to find out what is important and to continuously improve

  11. Why try to collect and preserve it all? • Web has no boundaries, no limits • What will be important? • What is there today may be gone tomorrow • “Capture now, ask why later” • “Grab it while you can, work it out later” • “Lose as little as possible”

  12. Open Source Technology primarily developed by Internet Archive and IIPC How do we collect it? • Heritrix: web crawler • Wayback Machine: access tool for rendering and viewing files • Nutchwax: Search engine • Arc File: archival record format (ISO work item)

  13. Wayback Machine

  14. Preservation • Store multiple copies of each Archive • 1300 machines/servers • Multiple copies at different geographical locations (U.S. Alexandria, Amsterdam) • Standard storage boxes, open source design

  15. Next Steps Institutions: • need to create collections around primary source web material • want to do more than broad crawling with specific and complete web archives • want a technology partner that could harvest, index, access, store and preserve their collections for them.

  16. 1. PartnerContract Crawls • In 2002, began to form partnerships with Library of Congress, NARA and other National Libraries, including Australia and France. • Library of Congress collections: • Iraq War: 450,000,000 documents and growing • U.S. National Elections • 2000:131,331,973 documents • 2004: 87,481,265 documents • Supreme Court Nomination 2005: 100 Million documents

  17. 2. Archive-It • Last year, early 2005, we had requests from state archivists, university librarians and other memory institutions to expand our archiving services and develop an application that acknowledge resource constraints • Developed Archive-It, web based service that allows partners to create,manage, search and store their web archives through an easy to use web interface • Does not require technical expertise or infrastructure • Pilot launched in September 2005 • 1.0 Release in February • 1.5 Release in April • 2.0 Release in July

  18. Pilot Partners • Center for Research Libraries • Research Libraries Group ( U of Toronto, U of Indiana, Haverford and Swarthmore Colleges, IISH) • University of Texas • Library of Virginia • State Archives South Dakota • State Archives North Carolina • State Archives Alabama • Minnesota Historical Society • Institut d'Etude Politique de Grenoble

  19. Archive-It • 1.0 Release in February • 1.5 Release in April • 2.0 Release in July

  20. Archive-It Collections Some samples: Virginia’s political landscape, 2005 (Gov. Mark Warner) Hurricane Katrina Jamestown 2007 Commemoration

  21. Archive-It Access • All collections are accessible for free to the general public, with text search, at: • www.archiveit. org • Partners websites with links • Plus, member web application with login

  22. Demo

  23. Dan’s slides Tech

  24. Challenges we face • Making the collections useful for a variety of end users (i.e. general public, researchers) • Making sure we capture the best and most relevant content • Continuing to develop our tools for access and harvesting (crawler.archive.org)

  25. Internet Archive’s priorities • Collaboration and Partnerships • Continue to act as a technology partner in providing web archiving services to government and memory institutions • Continue to develop Open Source software • Develop common tools, storage formats and standards through the IIPC (International Internet Preservation Consortium) • Open Content Alliance (OCA) digital books project • Multiple copies across the world • Within IA’s own facilities and with partners such as LC, Bnf, Library of Alexandria

  26. RLG’s web archiving program • Collaborative collection development. • Descriptive metadata for web archives. • Usability/user studies • Intellectual property concerns • Web Archiving 101 • Web archiving services and software

More Related