Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
- airflow
- apache
- apache-airflow
- python
Repository Index
47,133 repositories indexed
1–7 of 7 · Most stars
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- python
- luigi
- orchestration-framework
- scheduling
An orchestration platform for the development, production, and observation of data assets.
- data-pipelines
- dagster
- workflow
- data-science
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.
- hpc
- workflow
- workflow-automation
- workflow-management
A command-line tool for launching Apache Spark clusters.
- apache-spark
- ec2
- apache-spark-cluster
- orchestration
🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines
- etl
- etl-framework
- etl-pipeline
- python