Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 437 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Hop Orchestration Platform
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Actively maintained successor to Alibaba DataX — a fast, versatile, open-source ETL tool for 30+ RDBMS and NoSQL data sources.
Support agile Ontology DataOps Based on Flink, DataX and Flink-CDC with Web-UI
Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
- apache-beam
- dataflow-templates
- google-cloud-dataflow
- google-cloud-storage
High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.
- flow
- rust
- rust-lang
- stream
Python Fast Dataflow programming framework for Data pipeline work( Web Crawler,Machine Learning,Quantitative Trading.etc)
Scratch is a swiss army knife for big data.
A collection of Kafka Connect sinks and sources maintained by Lenses.io.
🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊
Input pipeline framework
ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)
High performance Kafka consumer for InfluxDB. Supports collectd message formats.
- kafka-influxdb
- kafka-consumer
- logstash
- influxdb
Friendly utility to load your on-prem data, whether large or small, to Einstein Analytics Datasets, with useful features such as autoloading, dataflow control and dataset inspection.
Flume NG MongoDB source.
Track Go benchmark performance over time by storing results in InfluxDB
Insert incoming MQTT values into InfluxDB. Follows mqtt-smarthome architecture.
Interfaces ham radio APRS-IS servers and saves packet data into an influxdb database
Data logistics system enabling real-time pathogen surveillance. Built for the Seattle Flu Study.
Read JMeter results in a csv file and put results in Influxdb database and allows you to create beautiful graphs with Grafana
ETL scripts for EOS.
- eos
- eosio
- etl
- apache-beam