Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
107,098 visible repositories in the index
1–24 of 28 · Most stars
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Fancy stream processing made operationally mundane
Fast and Lightweight Logs, Metrics and Traces processor for Linux, BSD, OSX and Windows
The Reactive Extensions for .NET
Canonical actor model implementation for .NET with local + distributed actors in C# and F#.
A network event stream processing system, in Clojure.
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
Kubernetes-native platform to run massively parallel data/streaming jobs
Goka is a compact yet powerful distributed stream processing library for Apache Kafka written in Go.
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
A lightweight stream processing library for Go
Probabilistic data structures for processing continuous, unbounded streams.
A Python stream processing engine modeled after Yahoo! Pipes
A platform that makes it easy for developers to build realtime, cost-effective, operations-focused applications
High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.
The Data Change Processing platform
🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊
Build platforms that flexibly mix SQL, batch, and stream processing paradigms
Tenzir is the data pipeline engine for security teams.
.NET Stream Processing Library for Apache Kafka 🚀
Streaming reactive and dataflow graphs in Python
Real-time decision features without streaming infra. Turn live events into product reflexes — no Kafka, no Flink, no feature store.
Scramjet Platform is a distributed runtime supervisor that can run data processing programs called Sequences and manage local resources on any Linux server, Docker on small edge servers, and even large-scale Kubernetes clusters in the cloud or datacenters without exposing network ports to the public internet.