Repository Index

📜 A CLI toolkit for extracting and working with your digital history

• chronicle
• chronicle-etl
• personal-data
• etl

An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.

• archivespark
• spark-framework
• spark
• web-archiving

RDF-Centric Map/Reduce Framework and Freebase data conversion tool

A Substrate indexing framework

• substrate

Algorand's data pipeline framework.

Track Go benchmark performance over time by storing results in InfluxDB

• benchmark
• influxdb
• golang

EtlKit is a lightweight ETL (extract · transform · load) library and data integration toolbox for .NET. Build your own data pipelines programmatically in C# — readable, testable, and tracked in source control like the rest of your code.

• dotnet
• dotnet-core
• etl
• etl-framework

Just swoof it in

• golang
• mysql

Java importer skeleton for complicated, business-logic-heavy high-performance Neo4j imports directly from SQL databases, CSV files, etc.

• neo4j
• java
• neo4j-graphaware-framework

Internet Archive's Sparkling Data Processing Library

High performant ETL engine written in TypeScript

• data-processing
• etl
• etl-framework
• streaming-data