Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Repository Index
132,241 visible repositories in the index
1–24 of 36 · Most stars
Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.
- change-data-capture
- kafka-connect
- apache-kafka
- debezium
Incremental engine for long horizon agents 🌟 Star if you like it!
- ai
- change-data-capture
- data-indexing
- etl
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
- data-integration
- high-performance
- offline
- real-time
Broadcast, Presence, and Postgres Changes via WebSockets
- elixir
- postgres
- postgresql
- realtime
Flink CDC is a streaming data integration tool
- change-data-capture
- cdc
- batch
- data-integration
The open source repository for Electricity Maps data parsers that powers the world's most comprehensive electricity data platform
- sustainability
- data-visualization
- climate-change
- hacktoberfest
Source-agnostic distributed change data capture system
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
- workflow
- governance
- azkaban
- davinci
Fast, horizontally scalable, multiprocess pub/sub queuing server and proxy for HTTP, long-polling, Websockets and EventSource (SSE), powered by Nginx.
Postgres change data capture to streams, queues, and search indexes like Kafka, SQS, Elasticsearch, HTTP endpoints, and more
- cdc
- change-data-capture
- elixir
- erlang
Fancy stream processing made operationally mundane. This repository is a fork of the original project before the license was changed.
- benthos
- bento
- connect
- connectors
OLake Go: open-source replication from databases, Kafka and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.
JSON output plugin for changeset extraction
- postgres
- replication
- plugin
- json
Postgres, MySQL, or MariaDB to Elasticsearch/OpenSearch sync
- elasticsearch
- elasticsearch-sync
- postgresql
- change-data-capture
a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.
- elasticsearch
- go
- golang
- mongodb
The Data Change Processing platform
- cdc
- change-data-capture
- change-detection
PostgreSQL replication with DDL changes
- golang
- postgresql
- replication
- ddl
DataLink是一个满足各种异构数据源之间的实时增量同步、离线全量同步,分布式、可扩展的数据交换平台。
- data-replication
- datalink
- data-exchange
- realtime-replication
🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊
- change-data-capture
- data
- data-collection
- data-engineering
An extensible distributed system for reliable nearline data streaming at scale
- distributed-systems
- data-streaming
- scalability
- kafka-mirror-maker
C# Twitch Chat, Whisper, API and PubSub Library. Allows for chatting, whispering, stream event subscription and channel/account modification. Supports everything that supports .NETStandard 2.0
SymmetricDS is database replication and file synchronization software that is platform independent, web enabled, and database agnostic. It is designed to make bi-directional data replication fast, easy, and resilient. It scales to a large number of nodes and works in near real-time across WAN and LAN networks.
- synchronization
- database
- mysql
- postgresql
WeDataSphere is a financial grade, one-stop big data platform suite.