Build Agentic workflows, RAG pipelines, with rich AI model and tool support on one collaborative workspace. Deploy on cloud, VPC, or self-hosted, so teams move from prototype to production without rebuilding the stack.
Repository Index
47,133 repositories indexed
1–24 of 28 · Most stars
World's first open-source, agentic video production system. 12 production pipelines, 100+ tools, 700+ agent skill and production-knowledge files. Turn your AI coding assistant into a full video production studio.
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- batch-processing
- kafka
- pathway
- python
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
- airflow
- apache
- apache-airflow
- python
Open-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search, and conversational systems.
- semantic-search
- information-retrieval
- ai
- python
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
- pytorch
- speech-recognition
- paraformer
- punctuation
Turns Data and AI algorithms into production-ready web applications in no time.
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- python
- luigi
- orchestration-framework
- scheduling
An orchestration platform for the development, production, and observation of data assets.
- data-pipelines
- dagster
- workflow
- data-science
Unified framework for building enterprise RAG pipelines with small, specialized models
- parsing
- retrieval-augmented-generation
- agents
- generative-ai-tools
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
- kedro
- hacktoberfest
- mlops
- python
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
- data
- data-cleaning
- data-pipelines
- data-processing
ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.
- mlops
- machine-learning
- data-science
- production-ready
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
- dataops
- dataops-platform
- elt
- open-source
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
- etl
- data-integration
- python
- postgresql
Elyra extends JupyterLab with an AI centric approach.
- notebooks
- notebook-jupyter
- jupyterlab
- jupyterlab-extensions
A deep learning toolkit for healthcare AI: EHR (MIMIC, eICU, OMOP), clinical text, signals and imaging. 30+ models, reproducible pipelines.
- healthcare
- data-mining
- deep-learning
- preprocessing
Python Streaming DataFrames for Kafka
Extra blocks for scikit-learn pipelines.
- scikit-learn
- machine-learning
- common-sense
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.
- hpc
- workflow
- workflow-automation
- workflow-management
The world's cleanest AutoML library ✨ - Do hyperparameter tuning with the right pipeline abstractions to write clean deep learning production pipelines. Let your pipeline steps have hyperparameter spaces. Design steps in your pipeline like components. Compatible with Scikit-Learn, TensorFlow, and most other libraries, frameworks and MLOps environments.
- pipeline
- pipeline-framework
- machine-learning
- deep-learning
ChEMBL database structure pipelines