Repository Index

A fast to develop, fast to run, Go based toolkit for ETL and feature extraction on Hadoop.

Blockchain Indexing Engine

• polkadot
• substrate
• postgres
• indexing

go-etl is a toolset for data extraction, transformation and loading.

• etl
• mysql
• go
• postgresql

Event-driven OneLake pipeline with automated Upsert merge and Python reporting

• activator
• data-engineering
• data-factory
• data-pipeline

RDF-Centric Map/Reduce Framework and Freebase data conversion tool

Analyse GTFS Schedule data by importing it into a PostgreSQL database.

• gtfs
• postgresql
• postgres
• sql

Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in this repo is now only for reference. For support and issues of 'warc-indexer', please communicate with NetArchiveSuite.

Friendly utility to load your on-prem data, whether large or small, to Einstein Analytics Datasets, with useful features such as autoloading, dataflow control and dataset inspection.

A high performance data transfer tool that can be used to transfer data from Bitcoin Core blockchain files to a SQL Server database

• bitcoin

Hadoop Data Integration with various databases, ftp servers, salesforce. Incremental update, dedup, append, merge your data on Hadoop.

Database synchronizer for Magento 2 (and Wordpress), based on Magerun2. Keeping development, staging and production in sync easily.

• magento2
• magento
• magerun2
• import

🇪🇺 Warehouse-native Analytics compatible with Google Analytics and Matomo tracking protocols. Ingest into ClickHouse, BigQuery or CSV/Parquet on S3/GCS/local fs while maintaining complete control over your data.

• clickhouse
• ga4
• tracker
• bigquery

A blazing fast fully-automated CSV to database importer

• csv2sql
• csvs
• database
• csv-files

Go tool + library that publishes GIS vector data to GeoServer via PostGIS — walks a directory of shapefile / GeoJSON / GeoPackage / KML, loads each into PostGIS via GDAL/OGR, then registers the resulting tables as GeoServer feature types. Idempotent. Tested on GeoServer 2.27 LTS + 2.28 stable.

• gis
• postgis
• geoserver
• golang

A Substrate indexing framework

• substrate

Enables search with Algolia in Hugo static sites

Export your database from Lovable Cloud to Supabase

Track Go benchmark performance over time by storing results in InfluxDB

• benchmark
• influxdb
• golang

Insert incoming MQTT values into InfluxDB. Follows mqtt-smarthome architecture.

• mqtt
• influxdb
• smarthome
• grafana

Reverse ETL for the code-first data stack

• bigquery
• cli
• data-engineering
• dbt

EtlKit is a lightweight ETL (extract · transform · load) library and data integration toolbox for .NET. Build your own data pipelines programmatically in C# — readable, testable, and tracked in source control like the rest of your code.

• dotnet
• dotnet-core
• etl
• etl-framework

Java importer skeleton for complicated, business-logic-heavy high-performance Neo4j imports directly from SQL databases, CSV files, etc.

• neo4j
• java
• neo4j-graphaware-framework

Data logistics system enabling real-time pathogen surveillance. Built for the Seattle Flu Study.

• postgres
• postgresql
• python
• infectious-disease

Live import all your Google Sheets to your data warehouse

• data-warehouse
• data-vault
• etl
• google-sheets