Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
- airflow
- apache
- apache-airflow
- python
Repository Index
47,133 repositories indexed
1–19 of 19 · Most stars
Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.
- orchestration
- data-orchestration
- high-availability
- infrastructure-as-code
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- python
- luigi
- orchestration-framework
- scheduling
An orchestration platform for the development, production, and observation of data assets.
- data-pipelines
- dagster
- workflow
- data-science
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
- workflow-schedule
- azkaban
- airflow
- task-scheduler
AI Infra / AI Orchestration / AI Control Plane
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
- data-science
- python
- dag
- data-engineering
Distributed Big Data Orchestration Service
- big-data
- bigdata
- orchestration
- configuration
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
- data-engineering
- data-integration
- data-pipeline
- data-quality
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.
- hpc
- workflow
- workflow-automation
- workflow-management
Polyglot workflows without leaving the comfort of your technology stack.
- orchestration-framework
- scheduling
- workflow
- event-driven
iWF is a Workflow-As-Code microservice orchestration platform offering an orchestration coding framework and service for building resilient, fault-tolerant, scalable long-running processes
- golang
- workflow
- distributed-system
- orchestration
A command-line tool for launching Apache Spark clusters.
- apache-spark
- ec2
- apache-spark-cluster
- orchestration
LittleHorse Server: Business-as-Code Platform for Microservices, Workflows, and Agents
The type-safe, event-driven workflow orchestration library that scales with your business. Build robust, distributed workflows in Go with compile-time safety, automatic retries, and horizontal scaling out of the box. Integrate with your existing tech stack using adapters 🚀
- eventdrivenarchitecture
- golang
- kafka
- reflex
Distributed, Scalable, Fault-Tolerant, Minimalistic Workflow Engine - No DAGs, No YAML, No Cumbersome Diagrams, Just Code
- business-process
- kafka
- kafka-streams
- micorservices
🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines
- etl
- etl-framework
- etl-pipeline
- python