Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Repository Index
47,133 repositories indexed
1–24 of 43 · Most stars
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
Instant messaging platform. Backend in Go. Clients: Swift iOS, Java Android, JS webapp, scriptable command line; chatbots
- chat
- chat-application
- chatapp
- go
Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.
- change-data-capture
- kafka-connect
- apache-kafka
- debezium
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
- data-drift
- jupyter-notebook
- pandas-dataframe
- machine-learning
Migrate to PostgreSQL in a single command!
- postgresql
- migration
- mysql
- mssql
The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL
Apache Ignite
- distributed-sql-database
- iot
- osgi
- network-client
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
- bigquery
- copy-database
- data-ingestion
- data-integration
A better compressed bitset in Java: used by Apache Spark, Netflix Atlas, Apache Pinot, Tablesaw, and many others
- java
- bitset
- roaring-bitmaps
- roaringbitmap
Source-agnostic distributed change data capture system
Sync data from one Postgres database to another
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
- workflow
- governance
- azkaban
- davinci
Database Subsetting and Relational Data Browsing Tool.
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
- data-science
- python
- dag
- data-engineering
🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.
- data-engineering
- reverse-etl
- data-pipeline
- data-activation
Probabilistic data structures for processing continuous, unbounded streams.
- bloom-filter
- go
- stable-bloom-filters
- cuckoo-filter
Copy a Postgres database to a target Postgres server (pg_dump | pg_restore on steroids)
Polo travels through your database and creates sample snapshots so you can work with real world data in development.
- database
- snapshot
- productivity
- activerecord
HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.
:lipstick: Durable and asynchronous data imports for consuming data at scale and publishing testable SDKs.
- porter
- data-import
- framework
- data-transformation
Amazon Redshift Database Loader implemented in AWS Lambda