Repository Index

The Apache Spark - Apache HBase Connector is a library to support Spark accessing HBase table as external data source or sink.

An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration

• java
• smooks
• enterprise-integration
• etl

Download, unpack from a ZIP/TAR/GZ/BZ2 archive, parse, correct, convert units and import Google Spreadsheets, XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models. Uses RemoteTable gem internally.

Powerful CSV & Excel Import experience for SaaS 🚀 Save months building data import experience from scratch 💰

• csv-import
• data-import
• data-mapping
• data-review

High performance Kafka consumer for InfluxDB. Supports collectd message formats.

• kafka-influxdb
• kafka-consumer
• logstash
• influxdb

Universal BLE Smart Scale bridge. Captures body composition from Renpho, Xiaomi & 20+ others, syncs to Garmin Connect, Strava, MQTT (Home Assistant), InfluxDB, Webhooks, Ntfy & local files (CSV/JSONL). Headless CLI for Raspberry Pi, Linux, macOS & Windows.

• garmin
• typescript
• mqtt
• ble

A fast to develop, fast to run, Go based toolkit for ETL and feature extraction on Hadoop.

📜 A CLI toolkit for extracting and working with your digital history

• chronicle
• chronicle-etl
• personal-data
• etl

Blockchain Indexing Engine

• polkadot
• substrate
• postgres
• indexing

go-etl is a toolset for data extraction, transformation and loading.

• etl
• mysql
• go
• postgresql

An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.

• archivespark
• spark-framework
• spark
• web-archiving

Scalable Full History API Solution for Antelope (former EOSIO) based blockchains

• antelope
• api
• big-data
• blockchain

Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in this repo is now only for reference. For support and issues of 'warc-indexer', please communicate with NetArchiveSuite.

This is a library that provides generic functionalities for the implementation of imports. In addition to maximum performance and optimized memory consumption, Pacemaker can also be used to implement imports in distributed scenarios that place the highest demands on speed and stability.

• magento
• import
• products
• categories

Bash script to export android content providers data to csv

• android
• adb
• csv
• content-provider

A blazing fast fully-automated CSV to database importer

• csv2sql
• csvs
• database
• csv-files

A Substrate indexing framework

• substrate

Export your database from Lovable Cloud to Supabase

Track Go benchmark performance over time by storing results in InfluxDB

• benchmark
• influxdb
• golang

Massively parallel RDBMS to Neo4j Importer

Insert incoming MQTT values into InfluxDB. Follows mqtt-smarthome architecture.

• mqtt
• influxdb
• smarthome
• grafana

EtlKit is a lightweight ETL (extract · transform · load) library and data integration toolbox for .NET. Build your own data pipelines programmatically in C# — readable, testable, and tracked in source control like the rest of your code.

• dotnet
• dotnet-core
• etl
• etl-framework

Collecting, shape-enhancing, validating, fixing and (partially) merging GTFS feeds

Interfaces ham radio APRS-IS servers and saves packet data into an influxdb database