the portable Python dataframe library
Repository Index
47,133 repositories indexed
1–21 of 21 · Most stars
Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark
State of the Art Natural Language Processing
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
Lightweight and extensible compatibility layer between dataframe libraries!
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Jupyter magics and kernels for working with remote Spark clusters
Hopsworks - Data-Intensive AI platform with a Feature Store
- feature-store
- aws
- azure
- data-science
GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs
- apache-spark
- big-data
- connected-components
- dataframe
PySpark + Scikit-learn = Sparkit-learn
Sparkling Water provides H2O functionality inside Spark cluster
- h2o
- spark
- machine-learning
- integration
PySpark test helper methods with beautiful error messages
pyspark methods to enhance developer productivity 📣 👯 🎉
A comprehensive Spark guide collated from multiple sources that can be referred to learn more about Spark or as an interview refresher.
Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!
A pure Python implementation of Apache Spark's RDD and DStream interfaces.
- data-science
- data-processing
- python
- apache-spark
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
Big Data essentials: Hadoop, MapReduce, Spark. Explore tutorials and demos in Jupyter notebooks—most are self-contained and live, ready to run with a click.
Helpers & syntactic sugar for PySpark.
Various examples of notebooks for working with web archives with the Archives Unleashed Toolkit, and derivatives generated by the Archives Unleashed Toolkit.
- notebooks
- pyspark-notebook
- juypter-notebook
- spark
A repository for deploying an AWS EMR cluster and submiting spark jobs on it. Boostrapping by default does inclues pysparkling so one can easily use h2o with python and spark.