Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Repository Index
47,133 repositories indexed
1–24 of 50 · Most stars
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.
- steampipe
- postgresql
- postgresql-fdw
- cloud
Migrate to PostgreSQL in a single command!
- postgresql
- migration
- mysql
- mssql
The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL
A lightweight message queue. Like AWS SQS and RSMQ but on Postgres.
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
- bigquery
- copy-database
- data-ingestion
- data-integration
Sync data from one Postgres database to another
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
- etl
- sql
- bigquery
- cloud-native
Database Subsetting and Relational Data Browsing Tool.
High-performance time-series aggregation for PostgreSQL
- realtime
- stream-processing
- analytics
- sql
Compositional, streaming I/O library for Scala
- scala
- stream
- streaming-computation
- fp
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
- etl
- data-integration
- python
- postgresql
MongooseIM is Erlang Solutions' robust, scalable and efficient XMPP server, aimed at large installations. Specifically designed for enterprise purposes, it is fault-tolerant and can utilise the resources of multiple clustered machines.
- mongooseim
- erlang
- xmpp
- jabber
🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.
- data-engineering
- reverse-etl
- data-pipeline
- data-activation
Microservice native message and event store for Postgres
- event-store
- event-sourcing
- event-driven
- pub-sub
Copy a Postgres database to a target Postgres server (pg_dump | pg_restore on steroids)
Import CSV and JSON into PostgreSQL the easy way
High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.
- flow
- rust
- rust-lang
- stream
The ultimate distributed MQTT broker. Handles 100M+ connections and 10M msg/sec with ease. Built on Kafka to provide industrial-grade persistence and eliminate data loss.
🏎️ Streams & Reactive Programming paradigm for Go: declarative and composable API for event-driven applications
- event-driven
- generics
- go
- golang
Data validation toolkit for assessing and monitoring data quality.
- data-validation
- tabular-data
- data-testing
- data-quality