huggingface /tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
- Stars
- 11,218 stars
- Forks
- 1,243 forks
- Language
- Rust
- License
- APACHE-2.0
- Created
- Nov 1, 2019
- Last push
- Nov 2, 2026
- Updated
- Nov 2, 2026
In the Repository Index from the Seed Dataset since Oct 22, 2026.
Star History
192 days recorded, Apr 26, 2026 – Nov 3, 2026 (UTC). Net change +620 stars.
The star count of each day is the latest count less the net stars of the later days.
Add this Star History to a README
The images follow the light or dark theme of the page that shows them, and update every hour. Add ?theme=light or ?theme=darkto the chart address to keep one theme.
Star History chart
[](https://awesomerepos.com/huggingface/tokenizers)Badge
[](https://awesomerepos.com/huggingface/tokenizers)Timeline
This repository got 28 stars on Sep 22, 2026, its largest day since Apr 26, 2026.
It gained stars on 179 of the 192 days with a record.
Its largest 7 days were Sep 21, 2026 to Sep 27, 2026: 85 stars.
In the 30 days to Nov 3, 2026 it got 80 stars, 47% fewer than in the 30 days before.
It had 10,598 stars before Apr 26, 2026 and has 11,218 stars now: 13,782 fewer than 25,000.
- Created on the Repository Host
- +17 stars in one day
- Largest star day recorded: +28 stars
- +12 stars in one day
- Entered the Repository Index from the Seed Dataset
- Latest push
From the Repository Host's creation date and releases, the Admission, the Star History, and the Observations. Star milestones show only when the Star History records the day. On the chart of net stars each day, a solid line marks a release.
Forks and pushes
From 13 Observations, Oct 22, 2026 – Nov 3, 2026. Forks +10. A new push was seen on 9 of 12 days.
- Oct 23, 2026: no new push
- Oct 24, 2026: new push
- Oct 25, 2026: no new push
- Oct 26, 2026: new push
- Oct 27, 2026: new push
- Oct 28, 2026: new push
- Oct 29, 2026: new push
- Oct 30, 2026: new push
- Oct 31, 2026: new push
- Nov 1, 2026: no new push
- Nov 2, 2026: new push
- Nov 3, 2026: new push
A filled day: the last push time changed since the Observation before it.
Dependencies in the index
AwesomeRepos has not scanned the dependency graph of this repository. The scan reads the 2,000 visible repositories with the most stars.
10 scanned repositories name a package from this repository.
Uses (in the index) 0
None.
Used by (in the index) 10
PyPI: tokenizers
PyPI: tokenizers
PyPI: tokenizers
PyPI: tokenizers
PyPI: tokenizers
Cargo: tokenizers
PyPI: tokenizers
PyPI: tokenizers
- astral-sh/uv91K
PyPI: tokenizers
PyPI: tokenizers
PyPI: tokenizers
Sources: the GitHub dependency graph of each repository, and deps.dev for the source repository of each package. A dependency graph also names development packages. deps.dev takes most source repositories from package metadata. How the dependency graph is made
Observations
What the Repository Host reported each day.
| Day (UTC) | Stars | Forks | Last push | Archived |
|---|---|---|---|---|
| Nov 3, 2026 | 11,218 | 1,243 | Nov 2, 2026 | No |
| Nov 2, 2026 | 11,217 | 1,243 | Nov 1, 2026 | No |
| Nov 1, 2026 | 11,217 | 1,240 | Oct 30, 2026 | No |
| Oct 31, 2026 | 11,216 | 1,238 | Oct 30, 2026 | No |
| Oct 30, 2026 | 11,213 | 1,239 | Oct 29, 2026 | No |
| Oct 29, 2026 | 11,208 | 1,240 | Oct 28, 2026 | No |
| Oct 28, 2026 | 11,204 | 1,237 | Oct 27, 2026 | No |
| Oct 27, 2026 | 11,203 | 1,238 | Oct 26, 2026 | No |
| Oct 26, 2026 | 11,201 | 1,237 | Oct 25, 2026 | No |
| Oct 25, 2026 | 11,194 | 1,235 | Oct 23, 2026 | No |
| Oct 24, 2026 | 11,193 | 1,234 | Oct 23, 2026 | No |
| Oct 23, 2026 | 11,192 | 1,233 | Oct 22, 2026 | No |
| Oct 22, 2026 | 11,191 | 1,233 | Oct 22, 2026 | No |