Repository Index

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

• batch-processing
• kafka
• pathway
• python

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

• database
• stream-processing
• rust
• postgresql

Apache Beam is a unified programming model for Batch and Streaming data processing.

• python
• java
• big-data
• beam

Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.

• java
• hazelcast
• in-memory
• big-data

The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL

• rust
• database
• sql
• streaming

Apache KIE (Knowledge Is Everything) is the home of Drools, OptaPlanner, jBPM, and Kogito.

• rule-engine
• java
• dmn
• hacktoberfest

Asynchronous Python framework for event-driven services. A thin client for Kafka, RabbitMQ, NATS, Redis and MQTT with full access to native broker features, plus AsyncAPI docs, in-memory tests and observability out of the box.

• asyncio
• kafka
• python
• rabbitmq

A network event stream processing system, in Clojure.

• riemann
• events
• clojure
• monitoring

High-performance time-series aggregation for PostgreSQL

• realtime
• stream-processing
• analytics
• sql

Project Flogo is an open source ecosystem of opinionated event-driven capabilities to simplify building efficient & modern serverless functions, microservices & edge apps.

• flogo
• iot
• integration
• golang

The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.

• clickhouse
• kakfa
• sql
• redpanda

Ruby and Rails efficient Kafka processing framework

• karafka-framework
• kafka
• kafka-client
• kafka-clients

Python Stream Processing

• python
• stream-processing
• rust
• data-engineering

Python Streaming DataFrames for Kafka

• kafka
• python
• stream-processing
• data-engineering

High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.

• flow
• rust
• rust-lang
• stream

Esper Complex Event Processing, Streaming SQL and Event Series Analysis

• cep
• complex-event-processing
• open-source
• compiler

Tenzir is the data pipeline engine for security teams.

• incident-response
• threathunting
• siem
• soc

HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.

• streaming
• database
• iot
• stream-processing

Compositional pipelines

Series and Panels for Real-time and Exploratory Analysis of Data Streams

• time-series
• real-time
• data-stream
• series-manipulation

The database purpose-built for stream processing applications.

• stream-processing
• real-time
• sql
• interactive

Parallelize _anything_ //

• ocaml-library
• parallel-computing
• stream-processing
• multi-processing

Open-source streaming SQL engine written in Rust using Apache Arrow and DataFusion. Supports continuous queries, temporal stream joins, tumbling/session windows, and CDC/Kafka connectors. Lightweight, embeddable, and sub-microsecond latency