Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Repository Index
47,133 repositories indexed
1–24 of 31 · Most stars
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.
- steampipe
- postgresql
- postgresql-fdw
- cloud
Migrate to PostgreSQL in a single command!
- postgresql
- migration
- mysql
- mssql
The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
- bigquery
- copy-database
- data-ingestion
- data-integration
Sync data from one Postgres database to another
.NET Transactional Document DB and Event Store on PostgreSQL
- marten
- postgres
- documentdb
- eventsourcing
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
- etl
- sql
- bigquery
- cloud-native
KubeBlocks is a Kubernetes Operator designed to manage a variety of databases and streaming systems, including MySQL, PostgreSQL, MongoDB, Redis, RabbitMQ, RocketMQ, and more, within Kubernetes environments.
- mysql-operator
- mongodb-operator
- postgresql-operator
- rabbitmq-operator
High-performance time-series aggregation for PostgreSQL
- realtime
- stream-processing
- analytics
- sql
High performance Node.js/PostgreSQL job queue (also suitable for getting jobs generated by PostgreSQL triggers/functions out into a different work queue)
- postgresql
- node
- worker
- job-queue
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
- etl
- data-integration
- python
- postgresql
One advanced and mature open-source MPP (Massively Parallel Processing) database. Open source alternative to Greenplum Database.
- database
- data-analysis
- data-warehouse
- olap
High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.
- flow
- rust
- rust-lang
- stream
pmacct is a small set of multi-purpose passive network monitoring tools [NetFlow IPFIX sFlow libpcap BGP BMP RPKI IGP Streaming Telemetry].
- netflow
- ipfix
- sflow
- bgp
Logical Replication extension for PostgreSQL 17, 16, 15, 14, 13, 12, 11, 10, 9.6, 9.5, 9.4 (Postgres), providing much faster replication than Slony, Bucardo or Londiste, as well as cross-version upgrades.
- postgresql
- replication
- logical-decoding
- database-replication
Event store using PostgreSQL for persistence
- cqrs
- eventstore
- elixir
- event-sourcing
Data API Framework for AI Agents and Data Apps
The ultimate distributed MQTT broker. Handles 100M+ connections and 10M msg/sec with ease. Built on Kafka to provide industrial-grade persistence and eliminate data loss.
Free and open source schema versioning and database migration made natively with .NET/6. NEW THIS MAY 2022! v1.3.15 released!
The repository host gives no description.
- hacktoberfest
- data
- datalineage
- dbt