Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Repository Index
47,133 repositories indexed
1–24 of 102 · Most stars
Apache Kafka - A distributed event streaming platform
Apache RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications.
- rocketmq
- cloud-native
- eventing
- java
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
- deep-learning
- document-parsing
- machine-learning
- nlp
Apache Pulsar - distributed pub-sub messaging system
Logstash - transport and process your logs, events, or other data
Open source RabbitMQ: core server and tier 1 (built-in) plugins
Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.
- change-data-capture
- kafka-connect
- apache-kafka
- debezium
Node-based Visual Programming Toolbox
- photogrammetry
- computer-vision
- 3d-reconstruction
- alicevision
Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
- kafka
- cpp
- kubernetes
- go
Open-Source Web UI for Apache Kafka Management
- kafka-ui
- kafka-brokers
- kafka
- kafka-streams
Scalable real-time messaging server in a language-agnostic way. Self-hosted alternative to Pubnub, Pusher, Ably, socket.io, Phoenix.PubSub, SignalR. Set up once and forever.
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
- data-integration
- high-performance
- offline
- real-time
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Fancy stream processing made operationally mundane
- message-queue
- stream-processing
- streaming-data
- message-bus
Apache Beam is a unified programming model for Batch and Streaming data processing.
- python
- java
- big-data
- beam
Real-time monitor and web admin for Celery distributed task queue
- celery
- task-queue
- monitoring
- administration
Apache Storm
- storm
- apache
- distributed
- streaming
The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL
🪽 An open, easy, fast, reliable and battery-efficient solution for real-time communications
- api
- web-api
- server-sent-events
- subscriptions
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
- cloud-native
- streaming
- rust
- real-time
Apache Iggy: Hyper-Efficient Message Streaming at Laser Speed
Privacy and Security focused Segment-alternative, in Golang and React
- privacy
- warehouse-management
- data-warehouse
- customer-data
Ultra-high performance reactive programming