💫 Industrial-strength Natural Language Processing (NLP) in Python
Repository Index
47,133 repositories indexed
1–24 of 41 · Most stars
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
- nlp
- pre-trained-model
- unilm
- minilm
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Offline, privacy-first grammar checker. Fast, open-source, Rust-powered
- developer-tools
- english-language
- grammar-checker
- rust
Easy-to-use and powerful LLM and SLM library with awesome model zoo.
- nlp
- embedding
- bert
- ernie
An open-source framework for detecting, redacting, masking, and anonymizing sensitive data (PII) across text, images, and structured data. Supports NLP, pattern matching, and customizable pipelines.
- python
- pii
- privacy
- data-anonymization
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
- nlp
- natural-language-processing
- natural-language-understanding
- language-model
Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
📐 Compute distance between sequences. 30+ algorithms, pure python implementation, common interface, optional external libs usage.
- distance
- algorithm
- python
- textdistance
A full spaCy pipeline and models for scientific/biomedical documents.
- scientific-documents
- spacy
- custom-pipes
- nlp
Neural question generation using transformers
An R package for the Quantitative Analysis of Textual Data
- r
- text-analytics
- natural-language-processing
- corpus
Ruby gem to calculate the similarity between texts using tf*idf
A Modern C++ Data Sciences Toolkit
- nlp
- nlp-parsing
- search-engine
- inverted-index
Implementation of various topic models
Natural Language Processing (NLP) library for Crystal
- string-distance
- stemmer
- inflector
- sentiment-analysis
Modular, fast NLP framework, compatible with Pytorch and spaCy, offering tailored support for French clinical notes.
- nlp
- medical
- text-mining
- clinical-data-warehouse
This repository is a proof of concept toolbox for using Deep Belief Nets for Topic Modeling in Python.
Lemmatizer for text in English. Inspired by Python's nltk.corpus.reader.wordnet.morphy
- rubynlp
- lemmatizer
- nlp
- wordnet
An NLP library for Uralic languages such as Finnish, Skolt Sami, Moksha and so on. Also supporting some non-Uralic languages such as Spanish, French, Arabic, Swedish, Norwegian, Russian and English. LLMs, FSTs and More!
- nlp-library
- uralic-languages
- finnish
- sami
A Multilingual Latent Dirichlet Allocation (LDA) Pipeline with Stop Words Removal, n-gram features, and Inverse Stemming, in Python.
- lda
- multilingual
- machine-learning
- latent-dirichlet-allocation
Transliterate Cyrillic → Latin in every possible way
- golang
- go
- transliteration
- translit
Golang port of Petrovich - an inflector for Russian anthroponyms.
Swift framework for document classification using a Core ML model.