📜 A CLI toolkit for extracting and working with your digital history
- chronicle
- chronicle-etl
- personal-data
- etl
Repository Index
89,073 visible repositories in the index
49–70 of 70 · Most stars
Blockchain Indexing Engine
go-etl is a toolset for data extraction, transformation and loading.
- etl
- mysql
- go
- postgresql
A plugin to push to Algolia based on graphQl queries
- gatsby
- algolia
- search
- static-site-generator
An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
- archivespark
- spark-framework
- spark
- web-archiving
RDF-Centric Map/Reduce Framework and Freebase data conversion tool
Scalable Full History API Solution for Antelope (former EOSIO) based blockchains
- antelope
- api
- big-data
- blockchain
Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in this repo is now only for reference. For support and issues of 'warc-indexer', please communicate with NetArchiveSuite.
Friendly utility to load your on-prem data, whether large or small, to Einstein Analytics Datasets, with useful features such as autoloading, dataflow control and dataset inspection.
A high performance data transfer tool that can be used to transfer data from Bitcoin Core blockchain files to a SQL Server database
🇪🇺 Warehouse-native Analytics compatible with Google Analytics and Matomo tracking protocols. Ingest into ClickHouse, BigQuery or CSV/Parquet on S3/GCS/local fs while maintaining complete control over your data.
- clickhouse
- ga4
- tracker
- bigquery
Go tool + library that publishes GIS vector data to GeoServer via PostGIS — walks a directory of shapefile / GeoJSON / GeoPackage / KML, loads each into PostGIS via GDAL/OGR, then registers the resulting tables as GeoServer feature types. Idempotent. Tested on GeoServer 2.27 LTS + 2.28 stable.
Enables search with Algolia in Hugo static sites
Track Go benchmark performance over time by storing results in InfluxDB
Insert incoming MQTT values into InfluxDB. Follows mqtt-smarthome architecture.
EtlKit is a lightweight ETL (extract · transform · load) library and data integration toolbox for .NET. Build your own data pipelines programmatically in C# — readable, testable, and tracked in source control like the rest of your code.
- dotnet
- dotnet-core
- etl
- etl-framework
Collecting, shape-enhancing, validating, fixing and (partially) merging GTFS feeds
Just swoof it in
- golang
- mysql
R COBOL DI (Data Integration) Package : Import COBOL CopyBook data files directly into R as properly structured data frames.
- r
- data-science
- dataframe
- data-frames
Real-time GTFS database loader and ORM library.
WFFM Conversion Tool is a tool to convert Sitecore WFFM forms items and data to Sitecore Forms.
A tool to grab AVL data from the database and push it to the ddot-updates service