Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Repository Index
47,133 repositories indexed
1–24 of 40 · Most stars
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
Always know what to expect from your data.
- pipeline-tests
- dataquality
- datacleaning
- datacleaner
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.
- steampipe
- postgresql
- postgresql-fdw
- cloud
Migrate to PostgreSQL in a single command!
- postgresql
- migration
- mysql
- mssql
The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL
A lightweight message queue. Like AWS SQS and RSMQ but on Postgres.
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
- bigquery
- copy-database
- data-ingestion
- data-integration
Sync data from one Postgres database to another
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
- etl
- sql
- bigquery
- cloud-native
Database Subsetting and Relational Data Browsing Tool.
High-performance time-series aggregation for PostgreSQL
- realtime
- stream-processing
- analytics
- sql
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
- etl
- data-integration
- python
- postgresql
Microservice native message and event store for Postgres
- event-store
- event-sourcing
- event-driven
- pub-sub
Copy a Postgres database to a target Postgres server (pg_dump | pg_restore on steroids)
Import CSV and JSON into PostgreSQL the easy way
High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.
- flow
- rust
- rust-lang
- stream
The ultimate distributed MQTT broker. Handles 100M+ connections and 10M msg/sec with ease. Built on Kafka to provide industrial-grade persistence and eliminate data loss.
MySQL to PostgreSQL replica system
- python
- postgresql
- mysql
- mysql-replication
Automatic data change tracking for PostgreSQL
- audit-log
- audit-trail
- change-data-capture
- change-tracking
Enhanced PostgreSQL logical replication
- postgresql
- logical-replication
- golang
Postgres change events (CDC) in Elixir [WAL]
- cdc
- change-data-capture
- elixir
- postgres