Repository Index

1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.

• pandas-profiling
• pandas-dataframe
• statistics
• jupyter-notebook

Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.

• weak-supervision
• data-cleaning
• data-quality
• noisy-labels

Evidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.

• data-drift
• jupyter-notebook
• pandas-dataframe
• machine-learning

A light-weight, flexible, and expressive statistical data testing library

• pandas
• validation
• schema
• dataframes

Deepchecks: Tests for Continuous Validation of ML Models & Data. Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling to thoroughly test your data and models from research to production.

• machine-learning
• ml
• model-validation
• data-validation

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

• dataquality
• spark
• unit-testing
• scala

Open-source low code data preparation library in python. Collect, clean and visualization your data in python with a few lines of code.

• dataprep
• data-science
• datapreparation
• dataconnector

Library for exploring and validating machine learning data

TranslatedOntiCards is an enterprise-oriented AI Data Hub that offers one-stop capabilities around business databases: data source connection, metadata understanding, natural language query, cross-source fusion, data quality inspection, query auditing, data asset management, and more

pyspark methods to enhance developer productivity 📣 👯 🎉

• pyspark
• apache-spark

Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!

• python
• pandas
• spark
• data

Utility to compare data between homogeneous or heterogeneous environments to ensure source and target tables match

Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning scenarios using these algorithms. Desbordante has a console version and an easy-to-use web application.

• data-analytics
• data-cleaning
• data-cleansing
• data-engineering

Data validation toolkit for assessing and monitoring data quality.

• data-validation
• tabular-data
• data-testing
• data-quality

Canonical GTFS Validator project for schedule (static) files.

• gtfs-dataset
• gtfs-validator
• gtfs-zip
• gtfs-feed

The repository host gives no description.

• hacktoberfest
• data
• datalineage
• dbt

Lightweight DataFrame validation decorators for Pandas, Polars, Modin, and PyArrow. No custom types required.

• pandas
• dataframe
• validation
• polars