1.karpathy / minbpe
Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
+264 stars
from 10,749 · score 5.58
Trending repositories
Ranked by Attention Score: stars gained over the last 30 complete days
30 repositories by Attention Score · Oct 7, 2026 – Nov 5, 2026 (UTC)
2.openai / tiktoken
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
+229 stars
from 19,380 · score 5.44
3.explosion / spaCy
💫 Industrial-strength Natural Language Processing (NLP) in Python
- natural-language-processing
- data-science
- machine-learning
- python
- cython
- nlp
- +11 more
+101 stars
from 33,928 · score 4.62
4.argosopentech / argos-translate
Open-source offline translation library written in Python
- python
- machine-translation
- transformers
- translation
- language-models
- linux
- +2 more
+99 stars
from 6,527 · score 4.61
5.fxsjy / jieba
TranslatedJieba (结巴, "stutter") Chinese word segmentation
+65 stars
from 35,164 · score 4.19
6.microsoft / unilm
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
- nlp
- pre-trained-model
- unilm
- minilm
- layoutlm
- layoutxlm
- +14 more
+21 stars
from 22,219 · score 3.09
7.mozillazg / python-pinyin
TranslatedChinese characters to pinyin converter (pypinyin)
- python
- pinyin
- pypinyin
- hanzi
- hanzi-pinyin
- python2
- +2 more
+21 stars
from 5,370 · score 3.09
8.cjhutto / vaderSentiment
VADER Sentiment Analysis. VADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media, and works well on texts from other domains.
+17 stars
from 5,074 · score 2.89
9.spencermountain / compromise
modest natural-language processing
- nlp
- part-of-speech
- named-entity-recognition
+14 stars
from 12,158 · score 2.71
+13 stars
from 5,095 · score 2.64
11.fnnx-ai / scikit-llm
Seamlessly integrate LLMs into scikit-learn.
+9 stars
from 3,530 · score 2.30
12.facebookresearch / fairseqArchived
Facebook AI Research Sequence-to-Sequence Toolkit written in Python.
+8 stars
from 32,202 · score 2.20
13.cltk / cltk
The Classical Language Toolkit
+6 stars
from 923 · score 1.95
14.life4 / textdistance
📐 Compute distance between sequences. 30+ algorithms, pure python implementation, common interface, optional external libs usage.
- distance
- algorithm
- python
- textdistance
- hamming-distance
- levenshtein-distance
- +7 more
+5 stars
from 3,541 · score 1.79
15.axa-group / nlp.js
An NLP library for building bots, with entity extraction, sentiment analysis, automatic language identify, and so more
- nlp
- nlu
- nodejs
- sentiment-analysis
- natural-language-processing
- bot
- +7 more
+4 stars
from 6,573 · score 1.61
16.ikawaha / kagome
Self-contained Japanese Morphological Analyzer written in pure Go
- japanese
- tokenizer
- nlp-library
- japanese-language
- pos-tagging
- segmentation
- +3 more
+4 stars
from 983 · score 1.61
17.facebookresearch / vizseq
An Analysis Toolkit for Natural Language Generation (Translation, Captioning, Summarization, etc.)
+4 stars
from 460 · score 1.61
18.mozillazg / go-pinyin
TranslatedConvert Chinese characters to pinyin
- go
- pinyin
- golang
- hanzi
- hanzi-pinyin
- chinese
+3 stars
from 1,793 · score 1.39
+2 stars
from 10,880 · score 1.10
20.wooorm / franc
Natural language detection
+2 stars
from 4,417 · score 1.10
21.curiosity-ai / catalyst
🚀 Catalyst is a C# Natural Language Processing library built for speed. Inspired by spaCy's design, it brings pre-trained models, out-of-the box support for training word and document embeddings, and flexible entity recognition models.
+2 stars
from 859 · score 1.10
22.jpmckinney / tf-idf-similarity
Ruby gem to calculate the similarity between texts using tf*idf
+2 stars
from 790 · score 1.10
23.meta-toolkit / meta
A Modern C++ Data Sciences Toolkit
- nlp
- nlp-parsing
- search-engine
- inverted-index
- pos-tag
- text-analysis
- +6 more
+2 stars
from 719 · score 1.10
24.abadojack / whatlanggo
Natural language detection library for Go
+2 stars
from 695 · score 1.10
+1 stars
from 3,252 · score 0.69
26.retextjs / retext
natural language processor powered by plugins part of the @unifiedjs collective
- natural-language
- ast
- retext
- cst
- javascript
- unified
+1 stars
from 2,432 · score 0.69
27.patil-suraj / question_generation
Neural question generation using transformers
- nlp
- nlg
- deep-learning
- transformer
- t5
- question-generation
- +2 more
+1 stars
from 1,137 · score 0.69
28.diasks2 / pragmatic_segmenter
Pragmatic Segmenter is a rule-based sentence boundary detection gem that works out-of-the-box across many languages.
+1 stars
from 596 · score 0.69
29.minibikini / paasaa
🔤 Natural language detection for Elixir without AI
+1 stars
from 144 · score 0.69
+1 stars
from 55 · score 0.69