A collection of tools for archiving and analysing the internet.
Repository Index
47,133 repositories indexed
25–37 of 37 · Most stars
AI-powered platform for OSINT intelligence analysis. Features archive discovery with hypothesis-driven investigation, GLiNER entity extraction, Mapbox geospatial visualization, network analysis, and document processing. Built with FastAPI, Next.js, Weaviate, and DSPy.
- ai-agents
- document-processing
- dspy
- entity-extraction
Datalogger for MicroPython and CPython.
- micropython
- datalogger
- esp32
- mqtt
Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
Create self-extracting compressed or self-mounting archives for MicroPython
A tool for exploring, analyzing, transforming, recombining, and extracting data from WARC (Web ARChive) files.
A simple script to generate a single geoJSON shape for each transit route in a GTFS archive.
A client for the Archive-It And Webrecorder WASAPI Data Transfer API
A Tool to Summarize Web Archive Holdings
- memento
- web-archive
- profiling
- mementomap
The repository host gives no description.
Intelligent web page comparison tool with Wayback Machine support and visual regression testing
- visual-comparison
- wayback-machine
- web-scraping
- diff
Script to merge GTFS ZIP archives into one.
- gtfs
- python
- script
- public-transport
Convert a bag-nabit dataset stored in a ZIP into a full-content WARC.