huggingface / datatrove
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
- Stars
- 3,419 stars
- Forks
- 326 forks
- Language
- Python
- License
- APACHE-2.0
- Created
- Jun 14, 2023
- Last push
- Nov 10, 2026
- Updated
- Nov 12, 2026
Why these Tags: an agent read the README and quoted it
- ETL from the readme:
process_common_crawl_dump.py full pipeline to read commoncrawl warc files, extract their text content, filters and save the resulting data to s3.
- Distributed computing from the readme:
DataTrove processing pipelines are platform-agnostic, running out of the box locally or on a slurm cluster.
An agent gave these Tags from the fixed Tag list in the AwesomeRepos source code. A code check kept each Tag only when its quote is in the text. A wrong Tag? Report an error with the form at the end of this page.
Admitted to the Repository Index on Nov 11, 2026: 3,419 stars on 2026-11-11.
Star History
189 days recorded, May 8, 2026 – Nov 12, 2026 (UTC). Net change +401 stars.
The star count of each day is the latest count minus the net stars of the later days.
Add this Star History to a README
The images follow the light or dark theme of the page that shows them, and update every hour.
Add ?theme=light or ?theme=dark
to the chart address to keep one theme.
Star History chart
[](https://awesomerepos.com/huggingface/datatrove) Badge
[](https://awesomerepos.com/huggingface/datatrove) Timeline
It gained stars on 149 of the 189 days recorded since May 8, 2026.
Its largest 7 days were Jul 15, 2026 to Jul 21, 2026: 63 stars.
In the 30 days to Nov 12, 2026 it got 39 stars, 28% fewer than in the 30 days before.
It had 3,018 stars before May 8, 2026 and has 3,419 stars now: 1,581 fewer than 5,000.
- Created on the Repository Host
- Release v0.8.0
- Release v0.7.0
- Release v0.9.0
- +17 stars in one day
- Largest star day recorded: +32 stars
- +19 stars in one day
- Release v0.10.0
- Release v0.10.1
- Latest push
- Entered the Repository Index: 3,419 stars on 2026-11-11
The timeline comes from the creation date and the releases on the Repository Host, the Admission, the Star History, and the Observations. Star milestones show only when the Star History records the day. The list shows up to 5 stable releases from the latest 10 releases recorded. When none of them is stable, one item counts the pre-releases. On the chart of net stars each day, a solid line marks each recorded release.
Forks and pushes
From 2 Observations, Nov 12, 2026 – Nov 13, 2026. Forks -1. AwesomeRepos saw a new push on 0 of 1 days.
- Nov 13, 2026: no new push
A filled day: the last push time changed since the Observation before it.
Health facts
Measured facts in four separate dimensions, each with a date and a source. AwesomeRepos does not add them into one score. AwesomeRepos reads the weekly commits, the changelog, and the community files each week for the 5,000 visible repositories with the most stars and for each rising repository. How AwesomeRepos defines each check
Maintenance
2 of 4 checks (2 without data)-
Yes: Not archived
GitHub does not mark the repository as archived. Source
-
Yes: A push in the last 90 days
Last push on 2026-11-10, 2 days ago. Source
-
No data: Commits in the last 13 weeks
No data: AwesomeRepos has not read the weekly commits.
-
No data: Commits in 26 or more of the last 52 weeks
No data: AwesomeRepos has not read the weekly commits.
Releases
2 of 3 checks (1 without data)-
Yes: A release in the last 365 days
Newest release v0.10.1 on 2026-09-30, 44 days ago. 5 releases in the last 365 days. Source
-
Yes: The newest release tag is a semantic version
The newest release tag is v0.10.1. Source
-
No data: A changelog file at the root
No data: AwesomeRepos has not scanned this repository.
Security practice
0 of 1 check (1 without data)-
No data: A security policy
No data: AwesomeRepos has not scanned this repository.
AwesomeRepos has not asked OpenSSF Scorecard about this repository.
Community files
1 of 3 checks (2 without data)-
Yes: A licence
GitHub finds the licence APACHE-2.0. Source
-
No data: A contributing guide
No data: AwesomeRepos has not scanned this repository.
-
No data: A code of conduct
No data: AwesomeRepos has not scanned this repository.
Dependencies in the index
This section shows the packages that connect this repository to other repositories in the index.
The GitHub dependency graph of this repository names 307 direct packages (scanned on Nov 13, 2026). Packages from 58 other repositories in the index are among them.
1 scanned repository names a package from this repository.
Uses (in the index) · 58
-
PyPI: vllm
- pallets/flask 75K
PyPI: flask
- Textualize/rich 58K
PyPI: rich
- astral-sh/ruff 51K
PyPI: ruff
-
PyPI: pandas
- ray-project/ray 44K
PyPI: ray
-
PyPI: sglang
- fxsjy/jieba 35K
PyPI: jieba
- explosion/spaCy 34K
PyPI: spacy
- tokio-rs/tokio 34K
Cargo: tokio, tokio-macros, tokio-util
Show all 58
- numpy/numpy 33K
PyPI: numpy
- tqdm/tqdm 31K
PyPI: tqdm
-
GitHub Actions: trufflesecurity/trufflehog
-
PyPI: fasttext-numpy2-wheel
- Delgan/loguru 24K
PyPI: loguru
-
PyPI: datasets
- apache/arrow 17K
PyPI: pyarrow
- clap-rs/clap 17K
Cargo: clap, clap_builder, clap_derive, clap_lex
- ipython/ipython 17K
PyPI: ipython
- hyperium/hyper 16K
Cargo: hyper
- encode/httpx 16K
PyPI: httpx
- nltk/nltk 15K
PyPI: nltk
-
PyPI: pytest
-
Cargo: windows-sys, windows-targets, windows_aarch64_gnullvm, windows_aarch64_msvc and 6 more
-
PyPI: tokenizers
- serde-rs/serde 11K
Cargo: serde, serde_derive
-
Cargo: js-sys, wasm-bindgen, wasm-bindgen-backend, wasm-bindgen-macro and 2 more
-
PyPI: hypothesis
-
GitHub Actions: actions/checkout
- getmoto/moto 8.7K
PyPI: moto
-
PyPI: bitsandbytes
- ijl/orjson 8.3K
PyPI: orjson
- sphinx-doc/sphinx 8.1K
PyPI: sphinx
- stanfordnlp/stanza 7.9K
PyPI: stanza
- rustls/rustls 7.8K
Cargo: rustls
- tokio-rs/mio 7.1K
Cargo: mio
- adbar/trafilatura 7.1K
PyPI: trafilatura
-
Cargo: tracing, tracing-attributes, tracing-core
- dtolnay/anyhow 6.7K
Cargo: anyhow
- rust-lang/futures-rs 5.9K
Cargo: futures-channel, futures-core, futures-sink, futures-task and 1 more
- console-rs/indicatif 5.2K
Cargo: indicatif
-
GitHub Actions: actions/github-script
- tower-rs/tower 4.4K
Cargo: tower-service
- briansmith/ring 4.1K
Cargo: ring
- rspeer/python-ftfy 4.1K
PyPI: ftfy
- rust-lang/regex 4.1K
Cargo: regex-lite
-
PyPI: huggingface-hub
- dtolnay/syn 3.5K
Cargo: syn
- Amanieu/parking_lot 3.5K
Cargo: lock_api, parking_lot, parking_lot_core
- awslabs/aws-sdk-rust 3.4K
Cargo: aws-sdk-s3, aws-sdk-sso, aws-sdk-ssooidc, aws-sdk-sts
- Tinche/aiofiles 3.3K
PyPI: aiofiles
- yaml/pyyaml 3K
PyPI: pyyaml
- ahupp/python-magic 2.9K
PyPI: python-magic
- RustCrypto/hashes 2.3K
Cargo: md-5, sha1, sha2
-
Cargo: p256
-
Cargo: ecdsa, rfc6979
-
PyPI: warcio
-
Cargo: base16ct, base64ct, const-oid, der and 3 more
Used by (in the index) · 1
-
PyPI: datatrove
Sources: the GitHub dependency graph of each repository, and deps.dev for the source repository of each package (checked on Nov 13, 2026). A dependency graph also names development packages. deps.dev takes most source repositories from package metadata. How the dependency graph is made
Observations
Latest Observation, Nov 13, 2026: 3,419 stars, 326 forks, last push Nov 10, 2026, not archived. One of 2 Observations since Nov 12, 2026.