Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
- python
- workflow
- data-engineering
- data-science
Repository Index
47,133 repositories indexed
1–24 of 45 · Most stars
Apache RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications.
- rocketmq
- cloud-native
- eventing
- java
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- python
- luigi
- orchestration-framework
- scheduling
The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.
- metadata
- datadiscovery
- dataquality
- data-profiling
dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
- dbt-viewpoint
- slack
- pypa
- data-modeling
Open source RabbitMQ: core server and tier 1 (built-in) plugins
Build, Manage and Deploy AI/ML Systems
- machine-learning
- model-management
- ai
- ml
Building event-driven applications the easy way in Go.
- watermill
- event-driven
- event-sourcing
- sagas
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
🔩 Like builtins, but boltons. 250+ constructs, recipes, and snippets which extend (and rely on nothing but) the Python standard library. Nothing like Michael Bolton.
- python
- utilities
- cache
- data-structures
🍴 Web applications made easy. Since 2011.
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
- dataquality
- spark
- unit-testing
- scala
A distributed MQTT message broker based on Erlang/OTP. Built for high quality & Industrial use cases. The VerneMQ mission is active & the project maintained. Thank you for your support!
- mqtt
- m2m
- scalable
- broker
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.
- datalake
- lakehouse
- metadata
- federated-query
Kafka-style message streaming in Go. Built on NATS. Single binary, no JVM, no ZooKeeper.
A framework for building messaging apps with .NET and C#.
- command-processor
- command-dispatcher
- brighter
- asynchronously
Project Flogo is an open source ecosystem of opinionated event-driven capabilities to simplify building efficient & modern serverless functions, microservices & edge apps.
- flogo
- iot
- integration
- golang
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
- analytics
- bigquery
- data-modeling
- data-pipelines
EventMesh is a new generation serverless event middleware for building distributed event-driven applications.
- esb
- event-driven
- event-mesh
- event-governance
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
- data-engineering
- data-integration
- data-pipeline
- data-quality
A platform that makes it easy for developers to build realtime, cost-effective, operations-focused applications
Framework for building Event-Driven Microservices
- microservices
- messaging
- event-driven-microservices
- stream-processing