Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Repository Index
47,133 repositories indexed
1–12 of 12 · Most stars
Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
- kafka
- cpp
- kubernetes
- go
Incremental engine for long horizon agents 🌟 Star if you like it!
- ai
- change-data-capture
- data-indexing
- etl
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
lakeFS - Data version control for your data lake | Git for data
- data-engineering
- data-versioning
- go
- object-storage
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
- etl
- sql
- bigquery
- cloud-native
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
- data-science
- python
- dag
- data-engineering
A distributed MySQL binlog storage system built on Raft
- mysql
- raft
- golang
- mysql-binlog
Time-series database and pub/sub messaging server for IoT and real-time devices, in Go: an embeddable storage engine with TTLs, wildcard topics and encryption, and a clustered server over gRPC, TCP and WebSocket with replication and rolling upgrades.
Fast implementation of queue in OCaml
- queue
- ocaml
- gadt
Pipeline de ingestão e busca semântica de documentos para RAG — com rate limiting, circuit breaker e fila resiliente.