Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
- airflow
- apache
- apache-airflow
- python
Repository Index
47,133 repositories indexed
1–24 of 24 · Most stars
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
An orchestration platform for the development, production, and observation of data assets.
- data-pipelines
- dagster
- workflow
- data-science
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
- pandas-profiling
- pandas-dataframe
- statistics
- jupyter-notebook
Always know what to expect from your data.
- pipeline-tests
- dataquality
- datacleaning
- datacleaner
Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
- weak-supervision
- data-cleaning
- data-quality
- noisy-labels
Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
- dataframe
- python
- bigdata
- tabular-data
The Open Source Feature Store for AI/ML
- machine-learning
- features
- ml
- big-data
🔩 Like builtins, but boltons. 250+ constructs, recipes, and snippets which extend (and rely on nothing but) the Python standard library. Nothing like Michael Bolton.
- python
- utilities
- cache
- data-structures
🌊 Online machine learning in Python
- incremental-learning
- machine-learning
- python
- online-learning
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
Koalas: pandas API on Apache Spark
CLI task management & automation tool
- python
- build-tool
- build-automation
- task-runner
Python Stream Processing
- python
- stream-processing
- rust
- data-engineering
Python Streaming DataFrames for Kafka
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
MLBox is a powerful Automated Machine Learning python library.
- machine-learning
- auto-ml
- kaggle
- deep-learning
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.
- hpc
- workflow
- workflow-automation
- workflow-management
Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!
Engine for AI/ML/Data tracking, visualization, explainability, drift detection, and dashboards for Polyaxon.
- pandas
- pandas-summary
- dataframes
- data-science
The repository host gives no description.
- hacktoberfest
- data
- datalineage
- dbt
A pure Python implementation of Apache Spark's RDD and DStream interfaces.
- data-science
- data-processing
- python
- apache-spark
BatchFlow helps you conveniently work with random or sequential batches of your data and define data processing and machine learning workflows even for datasets that do not fit into memory.
- python3
- data-science
- machine-learning
- python