Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
Repository Index
47,133 repositories indexed
1–24 of 86 · Most stars
TranslatedAlibaba's MySQL binlog incremental subscription & consumption component
Apache Flink
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
A curated list of awesome System Design (A.K.A. Distributed Systems) resources.
- relational-database
- message-broker
- hadoop-ecosystem
- nosql
The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.
PRQL is a modern language for transforming data — a simple, powerful, pipelined SQL replacement
Python SQL Parser and Transpiler
- transpiler
- sql
- python
- parser
BullMQ - Message Queue and Batch processing for NodeJS, Python, .NET, Elixir, Rust and PHP based on Redis or PostgreSQL
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Apache Beam is a unified programming model for Batch and Streaming data processing.
- python
- java
- big-data
- beam
Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.
- steampipe
- postgresql
- postgresql-fdw
- cloud
Business intelligence as code: build fast, interactive data visualizations in SQL and markdown
the portable Python dataframe library
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL
The polyglot queue: Fast and reliable background jobs in Go, Java, JS, Ruby, and Rust on Postgres/SQLite.
- background-jobs
- go
- golang
- postgres
lakeFS - Data version control for your data lake | Git for data
- data-engineering
- data-versioning
- go
- object-storage
A lightweight message queue. Like AWS SQS and RSMQ but on Postgres.
Apache Ignite
- distributed-sql-database
- iot
- osgi
- network-client
Distributed stream processing engine in Rust
Where data access meets operational intelligence
- mysql
- postgresql
- ai-agent
- data-catalog
Maxwell's daemon, a mysql-to-json kafka producer