Repository Index

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀

• dataframe
• python
• bigdata
• tabular-data

Koalas: pandas API on Apache Spark

• spark
• pandas
• pydata
• dataframe

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

• data-science
• python
• dag
• data-engineering

Apache DataFusion Ballista Distributed Query Engine

• arrow
• big-data
• dataframe
• olap

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

• apache-spark
• big-data
• connected-components
• dataframe

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

• spark
• hadoop
• webarchives
• analysis