Repository Index

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

• pocket
• wget
• browser-bookmarks
• pinboard

InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS

• ipfs
• warc
• wayback
• web-archiving

Streaming WARC/ARC library for fast web archive IO

• web-archives
• web-archiving
• warc
• pywb

WarcDB: Web crawl data as SQLite databases.

• crawling
• sqlite
• warc
• cli

Command line tools and libraries for handling and manipulating WARC files (and HTTP contents)

Tool and library for handling Web ARChive (WARC) files.

• python

Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.

A tool for exploring, analyzing, transforming, recombining, and extracting data from WARC (Web ARChive) files.

A tool for detecting viruses and NSFW material in WARC files

• antivirus
• nsfw-classifier
• warc
• warc-safe

The repository host gives no description.

Convert a bag-nabit dataset stored in a ZIP into a full-content WARC.