Repository Index

1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.

• pandas-profiling
• pandas-dataframe
• statistics
• jupyter-notebook

Evidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.

• data-drift
• jupyter-notebook
• pandas-dataframe
• machine-learning

A light-weight, flexible, and expressive statistical data testing library

• pandas
• validation
• schema
• dataframes

Deepchecks: Tests for Continuous Validation of ML Models & Data. Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling to thoroughly test your data and models from research to production.

• machine-learning
• ml
• model-validation
• data-validation

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

• data-science
• python
• dag
• data-engineering

Apache DataFusion Ballista Distributed Query Engine

• arrow
• big-data
• dataframe
• olap

Lightweight and extensible compatibility layer between dataframe libraries!

• cudf
• ibis
• pandas
• polars

Python Streaming DataFrames for Kafka

• kafka
• python
• stream-processing
• data-engineering

Clean APIs for data cleaning. Python implementation of R package Janitor

• pandas
• dataframe
• data
• cleaning-data

Jupyter magics and kernels for working with remote Spark clusters

• spark
• kernel
• cluster
• livy

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

• apache-spark
• big-data
• connected-components
• dataframe

Essential Spark extensions and helper methods ✨😲

• spark
• dataframe

[DEPRECATED] Tensorflow wrapper for DataFrames on Apache Spark

Easy pipelines for pandas DataFrames.

• pandas
• pandas-dataframe
• pipeline
• data

Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!

• python
• pandas
• spark
• data

Engine for AI/ML/Data tracking, visualization, explainability, drift detection, and dashboards for Polyaxon.

• pandas
• pandas-summary
• dataframes
• data-science

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

• spark
• hadoop
• webarchives
• analysis

Conflict-free merge for DataFrames, JSON, ML models & distributed agents — powered by CRDTs. The first merge library where every operation is mathematically guaranteed to converge.

• collaborative-editing
• conflict-resolution
• crdt
• data-engineering