1.openai / tiktoken
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
+739 stars
from 18,853 · score 6.61
Trending repositories
Ranked by Attention Score: stars gained over the last 90 complete days
24 repositories by Attention Score · Aug 6, 2026 – Nov 3, 2026 (UTC)
2.karpathy / minbpe
Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
+363 stars
from 10,642 · score 5.90
3.argosopentech / argos-translate
Open-source offline translation library written in Python
- python
- machine-translation
- transformers
- translation
- language-models
- linux
- +2 more
+313 stars
from 6,310 · score 5.75
4.explosion / spaCy
💫 Industrial-strength Natural Language Processing (NLP) in Python
- natural-language-processing
- data-science
- machine-learning
- python
- cython
- nlp
- +11 more
+293 stars
from 33,724 · score 5.68
5.fxsjy / jieba
TranslatedJieba (结巴, "stutter") Chinese word segmentation
+215 stars
from 35,015 · score 5.38
6.hankcs / HanLP
TranslatedChinese word segmentation, part-of-speech tagging, named entity recognition, dependency parsing, constituency parsing, semantic dependency parsing, semantic role labeling, coreference resolution, style transfer, semantic similarity, new word discovery, keyword and keyphrase extraction, automatic summarization, text classification and clustering, pinyin and Simplified/Traditional Chinese conversion, natural language processing
- nlp
- natural-language-processing
- hanlp
- pos-tagging
- dependency-parser
- text-classification
- +2 more
+157 stars
from 36,379 · score 5.06
7.nltk / nltk
NLTK Source
+113 stars
from 14,653 · score 4.74
8.microsoft / unilm
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
- nlp
- pre-trained-model
- unilm
- minilm
- layoutlm
- layoutxlm
- +14 more
+96 stars
from 22,141 · score 4.57
9.PaddlePaddle / PaddleNLP
Easy-to-use and powerful LLM and SLM library with awesome model zoo.
- nlp
- embedding
- bert
- ernie
- paddlenlp
- pretrained-models
- +13 more
+66 stars
from 12,921 · score 4.20
10.stanfordnlp / stanza
Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages
+65 stars
from 7,841 · score 4.19
11.piskvorky / gensim
Topic Modelling for Humans
- gensim
- topic-modeling
- information-retrieval
- machine-learning
- natural-language-processing
- nlp
- +9 more
+64 stars
from 16,446 · score 4.17
12.cjhutto / vaderSentiment
VADER Sentiment Analysis. VADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media, and works well on texts from other domains.
+60 stars
from 5,030 · score 4.11
13.mozillazg / python-pinyin
TranslatedChinese characters to pinyin converter (pypinyin)
- python
- pinyin
- pypinyin
- hanzi
- hanzi-pinyin
- python2
- +2 more
+49 stars
from 5,340 · score 3.91
14.facebookresearch / fairseqArchived
Facebook AI Research Sequence-to-Sequence Toolkit written in Python.
+33 stars
from 32,177 · score 3.53
15.flairNLP / flair
A very simple framework for state-of-the-art Natural Language Processing (NLP)
+30 stars
from 14,366 · score 3.43
16.clips / pattern
Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.
- python
- machine-learning
- natural-language-processing
- web-mining
- wordnet
- sentiment-analysis
- +1 more
+20 stars
from 8,836 · score 3.04
17.life4 / textdistance
📐 Compute distance between sequences. 30+ algorithms, pure python implementation, common interface, optional external libs usage.
- distance
- algorithm
- python
- textdistance
- hamming-distance
- levenshtein-distance
- +7 more
+16 stars
from 3,530 · score 2.83
18.fnnx-ai / scikit-llm
Seamlessly integrate LLMs into scikit-learn.
+16 stars
from 3,520 · score 2.83
19.allenai / allennlpArchived
An open-source NLP research library, built on PyTorch.
+12 stars
from 11,871 · score 2.56
20.miso-belica / sumy
Module for automatic summarization of text documents and HTML pages.
- python
- lsa
- textteaser
- html-page
- summarizer
- pagerank-algorithm
- +8 more
+12 stars
from 3,688 · score 2.56
21.asyml / texar
Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/
- machine-learning
- natural-language-processing
- tensorflow
- deep-learning
- text-generation
- python
- +9 more
+5 stars
from 2,388 · score 1.79
22.facebookresearch / ParlAIArchived
A framework for training and evaluating AI models on a variety of openly available dialogue datasets.
+4 stars
from 10,612 · score 1.61
23.pytorch / textArchived
Models, data loaders and abstractions for language processing, powered by PyTorch
- nlp
- data-loader
- deep-learning
- pytorch
- dataset
- models
+2 stars
from 3,552 · score 1.10
24.minimaxir / textgenrnn
Easily train your own text-generating neural network of any size and complexity on any text dataset with a few lines of code.
- keras
- deep-learning
- text-generation
- tensorflow
- python
+1 stars
from 4,917 · score 0.69