Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Repository Index
132,241 visible repositories in the index
1–12 of 12 · Most stars
Incremental engine for long horizon agents 🌟 Star if you like it!
- ai
- change-data-capture
- data-indexing
- etl
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
- data-integration
- high-performance
- offline
- real-time
Flink CDC is a streaming data integration tool
- change-data-capture
- cdc
- batch
- data-integration
The open source repository for Electricity Maps data parsers that powers the world's most comprehensive electricity data platform
- sustainability
- data-visualization
- climate-change
- hacktoberfest
OLake Go: open-source replication from databases, Kafka and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.
Postgres, MySQL, or MariaDB to Elasticsearch/OpenSearch sync
- elasticsearch
- elasticsearch-sync
- postgresql
- change-data-capture
a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.
- elasticsearch
- go
- golang
- mongodb
DataLink是一个满足各种异构数据源之间的实时增量同步、离线全量同步,分布式、可扩展的数据交换平台。
- data-replication
- datalink
- data-exchange
- realtime-replication
🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊
- change-data-capture
- data
- data-collection
- data-engineering
Distributed Data Transfer Service for MySQL
- mysql
- data-transfer
- data-migration
- replication
This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)