Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
132,241 visible repositories in the index
1–20 of 20 · Most stars
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
Fast web applications through dynamic, partially-stateful dataflow
Distributed stream processing engine in Rust
A modular implementation of timely dataflow in Rust
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
An implementation of differential dataflow using timely dataflow on Rust.
Kubernetes-native platform to run massively parallel data/streaming jobs
YTsaurus is a scalable and fault-tolerant open-source big data platform.
The Feldera Incremental Computation Engine
Python Stream Processing
A Rust framework for correct and performant distributed systems
High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.
Zero-cost & Type-safe Reactive Extensions for Rust. Supports both high-performance single-threaded (Local) and thread-safe (Shared) contexts. WASM & Tokio ready.
🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊
Real-time decision features without streaming infra. Turn live events into product reflexes — no Kafka, no Flink, no feature store.
Small, Fast event processing and monitoring for Erlang/OTP applications.
Open-source streaming SQL engine written in Rust using Apache Arrow and DataFusion. Supports continuous queries, temporal stream joins, tumbling/session windows, and CDC/Kafka connectors. Lightweight, embeddable, and sub-microsecond latency