Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Repository Index
47,133 repositories indexed
1–24 of 34 · Most stars
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Fancy stream processing made operationally mundane
- message-queue
- stream-processing
- streaming-data
- message-bus
Apache Beam is a unified programming model for Batch and Streaming data processing.
- python
- java
- big-data
- beam
Apache Storm
- storm
- apache
- distributed
- streaming
The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
- cloud-native
- streaming
- rust
- real-time
Ultra-high performance reactive programming
Kubernetes-native platform to run massively parallel data/streaming jobs
- kubernetes
- stream-processing
- data-processing
- k8s
Project Flogo is an open source ecosystem of opinionated event-driven capabilities to simplify building efficient & modern serverless functions, microservices & edge apps.
- flogo
- iot
- integration
- golang
Compositional, streaming I/O library for Scala
- scala
- stream
- streaming-computation
- fp
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
- clickhouse
- kakfa
- sql
- redpanda
A lightweight stream processing library for Go
Python Stream Processing
- python
- stream-processing
- rust
- data-engineering
Python Stream Processing. A Faust fork
Lightweight data stream processing engine for IoT edge
👜 Tiny and fast reactive/iterable programming library
- callbacks
- reactive
- iterables
- observables
Probabilistic data structures for processing continuous, unbounded streams.
- bloom-filter
- go
- stable-bloom-filters
- cuckoo-filter
Stream Processing and Complex Event Processing Engine
- streaming-data-processing
- cep
- stream-processing
- complex-event-processing
Python Streaming DataFrames for Kafka
Real-time stream processing for python
- streaming-data
- python
- async
- real-time
Trill is a single-node query processor for temporal or streaming data.
- streaming-data
- temporal-data
Esper Complex Event Processing, Streaming SQL and Event Series Analysis
- cep
- complex-event-processing
- open-source
- compiler
Build platforms that flexibly mix SQL, batch, and stream processing paradigms
- golang
- event-sourcing
- stream-processing
- brokers