Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Repository Index
47,133 repositories indexed
1–24 of 42 · Most stars
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
Python SQL Parser and Transpiler
- transpiler
- sql
- python
- parser
BullMQ - Message Queue and Batch processing for NodeJS, Python, .NET, Elixir, Rust and PHP based on Redis or PostgreSQL
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.
- steampipe
- postgresql
- postgresql-fdw
- cloud
the portable Python dataframe library
The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL
The polyglot queue: Fast and reliable background jobs in Go, Java, JS, Ruby, and Rust on Postgres/SQLite.
- background-jobs
- go
- golang
- postgres
A lightweight message queue. Like AWS SQS and RSMQ but on Postgres.
Where data access meets operational intelligence
- mysql
- postgresql
- ai-agent
- data-catalog
Queueing jobs in Postgres from Node.js like a boss
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
- bigquery
- copy-database
- data-ingestion
- data-integration
.NET Transactional Document DB and Event Store on PostgreSQL
- marten
- postgres
- documentdb
- eventsourcing
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
- etl
- sql
- bigquery
- cloud-native
KubeBlocks is a Kubernetes Operator designed to manage a variety of databases and streaming systems, including MySQL, PostgreSQL, MongoDB, Redis, RabbitMQ, RocketMQ, and more, within Kubernetes environments.
- mysql-operator
- mongodb-operator
- postgresql-operator
- rabbitmq-operator
High-performance time-series aggregation for PostgreSQL
- realtime
- stream-processing
- analytics
- sql
High performance Node.js/PostgreSQL job queue (also suitable for getting jobs generated by PostgreSQL triggers/functions out into a different work queue)
- postgresql
- node
- worker
- job-queue
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
- etl
- data-integration
- python
- postgresql
Microservice native message and event store for Postgres
- event-store
- event-sourcing
- event-driven
- pub-sub
Test your code without writing mocks with ephemeral Docker containers 📦 Setup popular services with just a couple lines of code ⏱️ No bash, no yaml, only code 💻
- go
- golang
- testing
- integration-testing
Complete working example of using Domain Driven Design (DDD), Hexagonal Architecture, CQRS, Event Sourcing (ES), Event Driven Architecture (EDA), Behaviour Driven Development (BDD) using TypeScript and NestJS. Like what you see? Don't forget to star! ⭐ ^^^
One advanced and mature open-source MPP (Massively Parallel Processing) database. Open source alternative to Greenplum Database.
- database
- data-analysis
- data-warehouse
- olap