Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Repository Index
47,133 repositories indexed
1–24 of 51 · Most stars
Cross-platform, customizable ML solutions for live and streaming media.
- mediapipe
- c-plus-plus
- computer-vision
- deep-learning
A high-performance observability data pipeline.
- logs
- metrics
- observability
- forwarder
QuestDB is a high performance, open-source, time-series database
- time-series
- low-latency
- database
- sql
Translatedflink learning blog. http://www.54tianzhisheng.cn/ Covers Flink introduction, concepts, principles, hands-on practice, performance tuning, source code analysis, and more. Includes learning examples involving Flink Connector, Metrics, Library, DataStream API, Table API & SQL, and also shares large-scale project cases of Flink in production (PV/UV, log storage, real-time deduplication of 10 billion records, monitoring and alerting). You are welcome to support my column 《大数据实时计算引擎 Flink 实战与性能优化》 ("Big Data Real-Time Computing Engine Flink: Hands-On Practice and Performance Optimization")
A curated list of awesome big data frameworks, ressources and other awesomeness.
- awesome
- bigdata
- data-warehouse
- data-analytics
Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
- kafka
- cpp
- kubernetes
- go
A curated list of awesome System Design (A.K.A. Distributed Systems) resources.
- relational-database
- message-broker
- hadoop-ecosystem
- nosql
Building event-driven applications the easy way in Go.
- watermill
- event-driven
- event-sourcing
- sagas
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Fancy stream processing made operationally mundane
- message-queue
- stream-processing
- streaming-data
- message-bus
Python Stream Processing
- kafka-streams
- kafka
- python
- asyncio
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
- java
- hazelcast
- in-memory
- big-data
The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL
Upserts, Deletes And Incremental Processing on Big Data.
- hudi
- apachehudi
- datalake
- bigdata
🌊 Online machine learning in Python
- incremental-learning
- machine-learning
- python
- online-learning
Asynchronous Python framework for event-driven services. A thin client for Kafka, RabbitMQ, NATS, Redis and MQTT with full access to native broker features, plus AsyncAPI docs, in-memory tests and observability out of the box.
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
- cloud-native
- streaming
- rust
- real-time
Distributed stream processing engine in Rust
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
- etl
- sql
- bigquery
- cloud-native
Kubernetes-native platform to run massively parallel data/streaming jobs
- kubernetes
- stream-processing
- data-processing
- k8s
High-performance time-series aggregation for PostgreSQL
- realtime
- stream-processing
- analytics
- sql
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
- clickhouse
- kakfa
- sql
- redpanda
A lightweight stream processing library for Go