Repository Index

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

• pocket
• wget
• browser-bookmarks
• pinboard

An Awesome List for getting started with web archiving

• webarchiving
• awesome-list
• awesome

A High-Fidelity Web Archiving Extension for Chrome and Chromium based browsers!

• chromium
• extension
• web-archiving
• webrecorder

InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS

• ipfs
• warc
• wayback
• web-archiving

Streaming WARC/ARC library for fast web archive IO

• web-archives
• web-archiving
• warc
• pywb

WarcDB: Web crawl data as SQLite databases.

• crawling
• sqlite
• warc
• cli

An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.

• archivespark
• spark-framework
• spark
• web-archiving

Parse And Create Web ARChive (WARC) files with node.js

• webarchive
• webarchiving
• web-archives
• warc-files

A Memento Aggregator CLI and Server in Go

• web-archiving
• memento
• timemap
• memento-rfc

Web archive index server based on RocksDB

• web-archiving
• wayback

Converts HTTrack crawls to WARC files

• web-archiving

CLI implementation of httpreserve that can test links and retrieve internet archive replacements

• code4lib
• web-archiving
• digipres
• digital-preservation

A Splitable Hadoop InputFormat for Concatenated GZIP Files and *.(w)arc.gz

• hadoop
• spark
• webarchive
• web-archiving

Intelligent web page comparison tool with Wayback Machine support and visual regression testing

• visual-comparison
• wayback-machine
• web-scraping
• diff

Partition (W)ARC Files by MIME Type and Year

• web-archiving
• webarchive
• hadoop
• warc