:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Repository Index
132,241 visible repositories in the index
25–40 of 40 · Most stars
Concurrent Python made simple
- asyncio
- concurrency
- python
- threading
Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.
Automatically find issues in image datasets and practice data-centric computer vision.
- computer-vision
- data-centric-ai
- data-exploration
- data-quality
A privacy-first guardrail for LLMs and RAG pipelines. SurrogateShield detects, redacts, and reconstructs sensitive PII data on the fly while defending against adversarial attacks.
Diffprivlib: The IBM Differential Privacy Library
- differential-privacy
- data-privacy
- machine-learning
- python
Library for reading and processing ML training data.
- data-pr
- jax
- machine-learning
- python
Continuously updated paper list on advancements in Data Agents. Companion repo to our paper "A Survey of Data Agents: Emerging Paradigm or Overstated Hype?"
- data-analysis
- data-analytics
- data-cleaning
- data-discovery
Download numerical weather prediction datasets (HRRR, RAP, GFS, IFS, etc.) from NOMADS, NODD partners (Amazon, Google, Microsoft), ECMWF open data, and the University of Utah Pando Archive System.
- grib
- hrrr
- cfgrib
- xarray
All-in-one text de-duplication
- text-processing
- de-duplication
- nlp
- data-processing
Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/
Engine for AI/ML/Data tracking, visualization, explainability, drift detection, and dashboards for Polyaxon.
- pandas
- pandas-summary
- dataframes
- data-science
Compile organizational policy into verifiable decision training data.
- ai-agents
- data-provenance
- decision-data
- fine-tuning
The data transformation tool the open-source community deserves.
- data-engineering
- duckdb
- data-lineage
- data-pipeline
A pure Python implementation of Apache Spark's RDD and DStream interfaces.
- data-science
- data-processing
- python
- apache-spark
Delete yourself from data brokers for $0. An agent skill: paste the repo URL into Grok Bot, Claude, Codex, Cursor or any coding agent.
- agent-skill
- ai-agent
- ccpa
- claude