Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
- airflow
- apache
- apache-airflow
- python
103,113 visible repositories in the index
1–24 of 68 · Most stars
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
An orchestration platform for the development, production, and observation of data assets.
Graph-Native Semantic Layer for Trustworthy Enterprise AI
dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
Always know what to expect from your data.
Incremental engine for long horizon agents 🌟 Star if you like it!
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
🧙 Build, run, and manage data pipelines for integrating and transforming data.
Fancy stream processing made operationally mundane
LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
The Open Source Feature Store for AI/ML
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
lakeFS - Data version control for your data lake | Git for data
TranslatedDeepAnalyze is the first agentic LLM for autonomous data science. 🎈Your AI data analyst, automatically analyzes large amounts of data and generates professional analysis reports with one click!
Privacy and Security focused Segment-alternative, in Golang and React
SQL Translator is a tool for converting natural language queries into SQL code using artificial intelligence. This project is 100% free and open source.
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
Deepchecks: Tests for Continuous Validation of ML Models & Data. Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling to thoroughly test your data and models from research to production.