Repository Index

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

• batch-processing
• kafka
• pathway
• python

Asynchronous Python framework for event-driven services. A thin client for Kafka, RabbitMQ, NATS, Redis and MQTT with full access to native broker features, plus AsyncAPI docs, in-memory tests and observability out of the box.

• asyncio
• kafka
• python
• rabbitmq

🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.

• cloud-native
• streaming
• rust
• real-time

Distributed stream processing engine in Rust

• data
• dev-tools
• infrastructure
• kafka

Kubernetes-native platform to run massively parallel data/streaming jobs

• kubernetes
• stream-processing
• data-processing
• k8s

Project Flogo is an open source ecosystem of opinionated event-driven capabilities to simplify building efficient & modern serverless functions, microservices & edge apps.

• flogo
• iot
• integration
• golang

Compositional, streaming I/O library for Scala

• scala
• stream
• streaming-computation
• fp

The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.

• clickhouse
• kakfa
• sql
• redpanda

Ruby and Rails efficient Kafka processing framework

• karafka-framework
• kafka
• kafka-client
• kafka-clients

A lightweight stream processing library for Go

• stream-processing
• pipeline
• etl
• kafka

Python Stream Processing

• python
• stream-processing
• rust
• data-engineering

👜 Tiny and fast reactive/iterable programming library

• callbacks
• reactive
• iterables
• observables

Stream Processing and Complex Event Processing Engine

• streaming-data-processing
• cep
• stream-processing
• complex-event-processing

Python Streaming DataFrames for Kafka

• kafka
• python
• stream-processing
• data-engineering

Distributed Stream Processing

• stream-processing
• stream-processor
• stream-processing-engine
• pony-language

High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.

• flow
• rust
• rust-lang
• stream

🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊

• change-data-capture
• data
• data-collection
• data-engineering

Build platforms that flexibly mix SQL, batch, and stream processing paradigms

• golang
• event-sourcing
• stream-processing
• brokers

Lightweight real-time big data streaming engine over Apache Pekko

• akka
• stream-processing
• bigdata
• scala

Apache StreamPipes - A self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams.

• iot
• analytics
• edge
• stream-processing

🏎️ Streams & Reactive Programming paradigm for Go: declarative and composable API for event-driven applications

• event-driven
• generics
• go
• golang

.NET Stream Processing Library for Apache Kafka 🚀

• kafka-streams-dotnet
• kafka-streams
• kafka-streams-net
• kafka

Streaming reactive and dataflow graphs in Python

• python
• python3
• stream
• data-pipeline

Machine is a workflow/pipeline library for processing data

• golang
• golang-library
• golang-package
• workflow