Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Repository Index
47,133 repositories indexed
1–18 of 18 · Most stars
An orchestration platform for the development, production, and observation of data assets.
- data-pipelines
- dagster
- workflow
- data-science
Node-based Visual Programming Toolbox
- photogrammetry
- computer-vision
- 3d-reconstruction
- alicevision
Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
- weak-supervision
- data-cleaning
- data-quality
- noisy-labels
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
- kedro
- hacktoberfest
- mlops
- python
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
MLBox is a powerful Automated Machine Learning python library.
- machine-learning
- auto-ml
- kaggle
- deep-learning
pyspark methods to enhance developer productivity 📣 👯 🎉
The world's cleanest AutoML library ✨ - Do hyperparameter tuning with the right pipeline abstractions to write clean deep learning production pipelines. Let your pipeline steps have hyperparameter spaces. Design steps in your pipeline like components. Compatible with Scikit-Learn, TensorFlow, and most other libraries, frameworks and MLOps environments.
- pipeline
- pipeline-framework
- machine-learning
- deep-learning
Engine for AI/ML/Data tracking, visualization, explainability, drift detection, and dashboards for Polyaxon.
- pandas
- pandas-summary
- dataframes
- data-science
Streaming reactive and dataflow graphs in Python
- python
- python3
- stream
- data-pipeline
amqplib fork
- python
- amqp-client
- amqp
- amqp-connection
Build and deploy a serverless data pipeline on AWS with no effort.
- aws-cdk
- data-pipeline
- stepfunctions
- glue
Helpers & syntactic sugar for PySpark.
Stream Processing Made Easy
- machine-learning
- reactive-programming
- kafka
- reactive-systems
Reverse ETL for the code-first data stack
- bigquery
- cli
- data-engineering
- dbt
Declarative data quality engine. Define checks in YAML, run anywhere.
- airflow
- anomaly-detection
- data-contracts
- data-quality
A detective for your data. Zero-config data quality monitoring — works with dbt, Postgres, BigQuery, Snowflake. No YAML.
- anomaly-detection
- bigquery
- cli
- data-engineering