Repository Index

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

• airflow
• apache
• apache-airflow
• python

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

• python
• workflow
• data-engineering
• data-science

Turns Data and AI algorithms into production-ready web applications in no time.

• automation
• data-engineering
• data-ops
• data-visualization

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

• python
• luigi
• orchestration-framework
• scheduling

An orchestration platform for the development, production, and observation of data assets.

• data-pipelines
• dagster
• workflow
• data-science

CLI task management & automation tool

• python
• build-tool
• build-automation
• task-runner

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

• spark
• pyspark
• data-wrangling
• bigdata

Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.

• hpc
• workflow
• workflow-automation
• workflow-management

BatchFlow helps you conveniently work with random or sequential batches of your data and define data processing and machine learning workflows even for datasets that do not fit into memory.

• python3
• data-science
• machine-learning
• python

🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines

• etl
• etl-framework
• etl-pipeline
• python