81,588 visible repositories in the index
1–12 of 12 · Most stars
Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
Build, Manage and Deploy AI/ML Systems
🧙 Build, run, and manage data pipelines for integrating and transforming data.
Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 426 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
Build and deploy a serverless data pipeline on AWS with no effort.
Distributed batch data processing framework