Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Repository Index
67,262 visible repositories in the index
1–24 of 52 · Most stars
Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
Apache Flink
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Fancy stream processing made operationally mundane
- message-queue
- stream-processing
- streaming-data
- message-bus
Apache Beam is a unified programming model for Batch and Streaming data processing.
- python
- java
- big-data
- beam
Fast and Lightweight Logs, Metrics and Traces processor for Linux, BSD, OSX and Windows
- fluentd
- c
- logging
- data-collector
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
- java
- hazelcast
- in-memory
- big-data
The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
- cloud-native
- streaming
- rust
- real-time
Fast web applications through dynamic, partially-stateful dataflow
Distributed stream processing engine in Rust
- data
- dev-tools
- infrastructure
- kafka
A modular implementation of timely dataflow in Rust
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
An implementation of differential dataflow using timely dataflow on Rust.
StreamAlert is a serverless, realtime data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.
- security
- kinesis
- serverless
- terraform
Kubernetes-native platform to run massively parallel data/streaming jobs
- kubernetes
- stream-processing
- data-processing
- k8s
Concurrent and multi-stage data ingestion and data processing with Elixir
- elixir
- genstage
- data-ingestion
- data-processing
A lightweight stream processing library for Go
Python Stream Processing
- python
- stream-processing
- rust
- data-engineering
Lightweight data stream processing engine for IoT edge
Probabilistic data structures for processing continuous, unbounded streams.
- bloom-filter
- go
- stable-bloom-filters
- cuckoo-filter
Stream Processing and Complex Event Processing Engine
- streaming-data-processing
- cep
- stream-processing
- complex-event-processing
Python Streaming DataFrames for Kafka