Repository Index

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

• batch-processing
• kafka
• pathway
• python

Building event-driven applications the easy way in Go.

• watermill
• event-driven
• event-sourcing
• sagas

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

• database
• stream-processing
• rust
• postgresql

Fancy stream processing made operationally mundane

• message-queue
• stream-processing
• streaming-data
• message-bus

The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL

• rust
• database
• sql
• streaming

Asynchronous Python framework for event-driven services. A thin client for Kafka, RabbitMQ, NATS, Redis and MQTT with full access to native broker features, plus AsyncAPI docs, in-memory tests and observability out of the box.

• asyncio
• kafka
• python
• rabbitmq

Concurrent and multi-stage data ingestion and data processing with Elixir

• elixir
• genstage
• data-ingestion
• data-processing

High-performance time-series aggregation for PostgreSQL

• realtime
• stream-processing
• analytics
• sql

The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.

• clickhouse
• kakfa
• sql
• redpanda

Ruby and Rails efficient Kafka processing framework

• karafka-framework
• kafka
• kafka-client
• kafka-clients

👜 Tiny and fast reactive/iterable programming library

• callbacks
• reactive
• iterables
• observables

Distributed Stream Processing

• stream-processing
• stream-processor
• stream-processing-engine
• pony-language

🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊

• change-data-capture
• data
• data-collection
• data-engineering

Machine is a workflow/pipeline library for processing data

• golang
• golang-library
• golang-package
• workflow

Amazon Kinesis Aggregators provides a simple way to create real time aggregations of data on Amazon Kinesis.

Real-time decision features without streaming infra. Turn live events into product reflexes — no Kafka, no Flink, no feature store.

• analytics
• feature-store
• fraud-detection
• python

Data management for the sensor-edge-cloud continuum

• bigdata
• streamprocessing

Multi-core Window-Based Stream Processing Engine

• llvm
• incremental-computation
• aggregation
• stream-processing

Scramjet Platform is a distributed runtime supervisor that can run data processing programs called Sequences and manage local resources on any Linux server, Docker on small edge servers, and even large-scale Kubernetes clusters in the cloud or datacenters without exposing network ports to the public internet.

• nodejs
• stream-processing
• transform-as-a-service
• real-time

Stream Processing Made Easy

• machine-learning
• reactive-programming
• kafka
• reactive-systems

A light-weight package for in-app micro batch processing

• batch
• batch-processing
• event-processing
• go