Repository Index

A collection of tools for archiving and analysing the internet.

AI-powered platform for OSINT intelligence analysis. Features archive discovery with hypothesis-driven investigation, GLiNER entity extraction, Mapbox geospatial visualization, network analysis, and document processing. Built with FastAPI, Next.js, Weaviate, and DSPy.

• ai-agents
• document-processing
• dspy
• entity-extraction

Crystal language Tar Module implements access to tar archives.

• crystal
• crystal-language
• crystal-lang
• tar

Web archive index server based on RocksDB

• web-archiving
• wayback

Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.

Salvaged websites, articles, contributions, text and docs

Zip-Ada: a standalone, portable pure Ada library for .zip archives. Includes LZMA and BZip2 byte stream encoder & decoder pairs.

• ada
• ada-language
• zip
• zip-archive

[DEPRECATED | Use graphene-django package] Integrate GraphiQL easily into your Django project

Create self-extracting compressed or self-mounting archives for MicroPython

Command-line tool and Rust library for handling Web ARChive (WARC) files

A Common Lisp library for reading archive (tar, cpio, etc.) files

An assembly tutorial for Game Boy showing how the scroll registers can be exploited to create some nice and interesting effects.

Various examples of notebooks for working with web archives with the Archives Unleashed Toolkit, and derivatives generated by the Archives Unleashed Toolkit.

• notebooks
• pyspark-notebook
• juypter-notebook
• spark

A tool for exploring, analyzing, transforming, recombining, and extracting data from WARC (Web ARChive) files.

Infinity GBC

DuckDB extension to fetch pages from Wayback Machine & Common Crawl

• archived
• common-crawl
• duckdb
• duckdb-extension

HAR (HTTP Archive) parser in Crystal

• har
• http-archive
• http-archive-format
• json

Create Robust Links from within Zotero

• references
• reference-rot
• link-rot
• memento

MusicXML parsing and layout library

Web application for distributed compute analysis of Archive-It web archive collections.

A simple script to generate a single geoJSON shape for each transit route in a GTFS archive.

RAAR is a ruby application to manage and browse an audio archive.

• ruby
• rails
• archive
• radio

golang readers for ARC and WARC webarchive formats

A JMeter plugin that lets you select a HAR (HTTP Archive) file to import into JMeter

• http-archive
• jmeter
• jmeter-plugin