Extract web archive data using Wayback Machine and Common Crawl
- golang
- crawler
- commoncrawl
- wayback-machine
Repository Index
47,133 repositories indexed
1–11 of 11 · Most stars
An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
- archivespark
- spark-framework
- spark
- web-archiving
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
A robust web archive analytics toolkit
- python
- web
- warc
- bigdata
Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in this repo is now only for reference. For support and issues of 'warc-indexer', please communicate with NetArchiveSuite.
Parse And Create Web ARChive (WARC) files with node.js
- webarchive
- webarchiving
- web-archives
- warc-files
Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
golang readers for ARC and WARC webarchive formats
A Splitable Hadoop InputFormat for Concatenated GZIP Files and *.(w)arc.gz
Parse CDXJ(https://github.com/oduwsdl/ORS/wiki/CDXJ) files with node.js
- webarchive
- webarchiving
- web-archives
- cdxj
Partition (W)ARC Files by MIME Type and Year
- web-archiving
- webarchive
- hadoop
- warc