84,583 visible repositories in the index
1–16 of 16 · Most stars
Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
Node-based Visual Programming Toolbox
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
Data-Centric Pipelines and Data Versioning
Translatedcube studio is an open-source, cloud-native, one-stop machine learning / deep learning / large model AI platform. It covers the full mlops algorithm pipeline, a computing power rental platform, online notebook development, drag-and-drop task flow pipeline orchestration, multi-machine multi-GPU distributed training, hyperparameter search, inference services with VGPU virtualization, edge computing, a labeling platform with automated labeling, SFT fine-tuning / reward model / reinforcement learning training for large models such as deepseek, multi-machine large model inference with vllm/ollama/mindie, a private knowledge base, and an AI model marketplace. It supports domestic Chinese CPUs/GPUs/NPUs and the Ascend ecosystem, supports RDMA, and supports distributed frameworks such as pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/ray/volcano.
Maestro: Netflix’s Workflow Orchestrator
A workflow language for data-driven computational pipelines
Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
Easy Machine Learning is a general-purpose dataflow-based system for easing the process of applying machine learning algorithms to real world tasks.
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
Simple but powerful DAG scheduler and dashboard
Distributed batch data processing framework