Repository Index

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀

• dataframe
• python
• bigdata
• tabular-data

Koalas: pandas API on Apache Spark

• spark
• pandas
• pydata
• dataframe

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

• data-science
• python
• dag
• data-engineering

Apache DataFusion Ballista Distributed Query Engine

• arrow
• big-data
• dataframe
• olap

Clean APIs for data cleaning. Python implementation of R package Janitor

• pandas
• dataframe
• data
• cleaning-data

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

• apache-spark
• big-data
• connected-components
• dataframe

Essential Spark extensions and helper methods ✨😲

• spark
• dataframe

Easy pipelines for pandas DataFrames.

• pandas
• pandas-dataframe
• pipeline
• data

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

• spark
• hadoop
• webarchives
• analysis