Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
132,241 visible repositories in the index
1–24 of 57 · Most stars
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Apache Spark - A unified analytics engine for large-scale data processing
Apache Flink
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Fancy stream processing made operationally mundane
Distributed and fault-tolerant realtime computation: stream processing, continuous computation, distributed RPC, and more
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
Fast web applications through dynamic, partially-stateful dataflow
Canonical actor model implementation for .NET with local + distributed actors in C# and F#.
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
An implementation of differential dataflow using timely dataflow on Rust.
StreamAlert is a serverless, realtime data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.
Kubernetes-native platform to run massively parallel data/streaming jobs
High-performance time-series aggregation for PostgreSQL
Compositional, streaming I/O library for Scala
Open source framework for processing, monitoring, and alerting on time series data
YTsaurus is a scalable and fault-tolerant open-source big data platform.
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
Fancy stream processing made operationally mundane. This repository is a fork of the original project before the license was changed.
The Feldera Incremental Computation Engine
Lightweight data stream processing engine for IoT edge
👜 Tiny and fast reactive/iterable programming library