Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Repository Index
63,269 visible repositories in the index
1–24 of 32 · Most stars
Incremental engine for long horizon agents 🌟 Star if you like it!
- ai
- change-data-capture
- data-indexing
- etl
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
- data-integration
- high-performance
- offline
- real-time
Flink CDC is a streaming data integration tool
- change-data-capture
- cdc
- batch
- data-integration
The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL
Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.
- machine-learning
- convolutional-networks
- embedding-vectors
- embeddings
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
- etl
- sql
- bigquery
- cloud-native
Self-hosted platform to unify wearable health data through one AI-ready API.
- wearables
- apple-health
- apple-watch
- fitness
A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.
- data
- ingestion
- replication
- management
Open-source IoT Gateway - integrates devices connected to legacy and third-party systems with ThingsBoard IoT Platform using Modbus, CAN bus, BACnet, BLE, OPC-UA, MQTT, ODBC and REST protocols.
Connect your company's data, let an agent draft an ontology over it, and load it into HelixDB as one graph you can explore and ask questions of. Hosted at foundry.helix-db.com, or run it on your own machine.
- ai-agents
- data-integration
- golang
- graph-database
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
- etl
- data-integration
- python
- postgresql
:elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop
MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)
🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.
- data-engineering
- reverse-etl
- data-pipeline
- data-activation
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 426 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
- data-engineering
- data-integration
- data-pipeline
- data-quality
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
- transform
- etl
- xml
- json
ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
Amazon Redshift Database Loader implemented in AWS Lambda
Distributed Data Transfer Service for MySQL
- mysql
- data-transfer
- data-migration
- replication
The data platform for manufacturing
- oee
- iot
The data transformation tool the open-source community deserves.
- data-engineering
- duckdb
- data-lineage
- data-pipeline
A decision-model-based ontology & knowledge-graph builder — builds governed knowledge graphs from your documents, every fact gated and traceable to its evidence.
- graphrag
- information-extraction
- knowledge-graph
- llm
Event-driven OneLake pipeline with automated Upsert merge and Python reporting
- activator
- data-engineering
- data-factory
- data-pipeline