Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Repository Index
47,133 repositories indexed
1–24 of 166 · Most stars
Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.
- orchestration
- data-orchestration
- high-availability
- infrastructure-as-code
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
High-Performance server for NATS.io, the cloud and edge native messaging system.
- go
- golang
- messaging
- message-bus
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
The most scalable and reliable MQTT broker for AI, IoT, IIoT and connected vehicles
The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.
- metadata
- datadiscovery
- dataquality
- data-profiling
dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
- dbt-viewpoint
- slack
- pypa
- data-modeling
Open-Source Web UI for Apache Kafka Management
- kafka-ui
- kafka-brokers
- kafka
- kafka-streams
CMAK is a tool for managing Apache Kafka clusters
🥊 Tiny 200 byte functional event emitter / pubsub.
- mitt
- event-handlers
- pubsub
- tiny
Incremental engine for long horizon agents 🌟 Star if you like it!
- ai
- change-data-capture
- data-indexing
- etl
ZeroMQ core engine in C++, implements ZMTP/3.1
- zeromq
- zmq
- messaging
- concurrency
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
- kedro
- hacktoberfest
- mlops
- python
Scalable real-time messaging server in a language-agnostic way. Self-hosted alternative to Pubnub, Pusher, Ably, socket.io, Phoenix.PubSub, SignalR. Set up once and forever.
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
Efficient reliable UDP unicast, UDP multicast, and IPC message transport
- messaging
- java
- c-plus-plus
- ipc
Fancy stream processing made operationally mundane
- message-queue
- stream-processing
- streaming-data
- message-bus
Distributed Application Framework for .NET
- masstransit
- rabbitmq
- c-sharp
- azure-service-bus
Real-time monitor and web admin for Celery distributed task queue
- celery
- task-queue
- monitoring
- administration
Distributed transaction solution in micro-service base on eventually consistency, also an eventbus with Outbox pattern
The leader in Customer Data Infrastructure
- snowplow
- analytics
- data
- data-pipeline
Python Stream Processing
- kafka-streams
- kafka
- python
- asyncio