47,133 repositories indexed
1–24 of 62 · Most stars
Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
💫 Industrial-strength Natural Language Processing (NLP) in Python
Easy-to-use and powerful LLM and SLM library with awesome model zoo.
Unsupervised text tokenizer for Neural Network-based text generation.
An open-source framework for detecting, redacting, masking, and anonymizing sensitive data (PII) across text, images, and structured data. Supports NLP, pattern matching, and customizable pipelines.
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
CoreNLP: A Java suite of core NLP tools for tokenization, sentence segmentation, NER, parsing, coreference, sentiment analysis, etc.
Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.
Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages
TranslatedChinese characters to pinyin converter (pypinyin)
Easily train your own text-generating neural network of any size and complexity on any text dataset with a few lines of code.
State of the Art Natural Language Processing
Module for automatic summarization of text documents and HTML pages.
TranslatedConvert Chinese characters to pinyin
Simple text to phones converter for multiple languages
Self-contained Japanese Morphological Analyzer written in pure Go
The Classical Language Toolkit
Fast vectorization, topic modeling, distances and GloVe word embeddings in R.
Parse human names into title, given, middle, family, suffix, nickname and maiden name. Typed, zero-dependency, CJK-aware.
Translated斑海豹 (Spotted Seal) — Phocinae-Largha-150M-v1: a 144.3M bilingual decision model. Tiny model. Big decisions.
CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, transliteration, verb-sense, and more.
A multilingual command line sentence tokenizer in Golang
An Analysis Toolkit for Natural Language Generation (Translation, Captioning, Summarization, etc.)