💫 Industrial-strength Natural Language Processing (NLP) in Python
Repository Index
111,821 visible repositories in the index
1–24 of 34 · Most stars
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Unsupervised text tokenizer for Neural Network-based text generation.
- neural-machine-translation
- natural-language-processing
- word-segmentation
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
- nlp
- natural-language-processing
- natural-language-understanding
- language-model
Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
CoreNLP: A Java suite of core NLP tools for tokenization, sentence segmentation, NER, parsing, coreference, sentiment analysis, etc.
- natural-language-processing
- nlp
- nlp-parsing
- named-entity-recognition
Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages
Language model tokenization at GB/s
🇨🇳 功能全面的汉字工具库 (拼音 笔画 偏旁 成语 语音 可视化等) (Chinese character util)
- spell-stroke
- draw
- chinese-characters
- pinyin
:book: A Golang library for text processing, including tokenization, part-of-speech tagging, and named-entity extraction.
Things you can do with the token embeddings of an LLM
Persian NLP Toolkit
- natural-language-processing
- python
- persian
- nlp
Developer friendly Natural Language Processing ✨
- natural-language-processing
- nlp
- tokenize
- sbd
Solves basic Russian NLP tasks, API for lower level Natasha projects
- nlp
- russian
- tokenizer
- sentence-segmentation
Local-first privacy guard: anonymize your documents before sharing with LLMs.
- anonymization
- gdpr
- italian
- legal-tech
JS port and JS/WASM bindings for openai/tiktoken
Self-contained Japanese Morphological Analyzer written in pure Go
- japanese
- tokenizer
- nlp-library
- japanese-language
Translated斑海豹 (Spotted Seal) — Phocinae-Largha-150M-v1: a 144.3M bilingual decision model. Tiny model. Big decisions.
- bilingual
- calibration
- cpu-inference
- decision-making
RNNLG is an open source benchmark toolkit for Natural Language Generation (NLG) in spoken dialogue system application domains. It is released by Tsung-Hsien (Shawn) Wen from Cambridge Dialogue Systems Group under Apache License 2.0.
- natural-language-processing
- natural-language-generation
- nlg-dataset
- deep-learning
CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, transliteration, verb-sense, and more.
- natural-language-processing
- nlp
- big-data
- data-mining
A multilingual command line sentence tokenizer in Golang
- sentence-tokenizer
- tokenizer
- cli
- sentences
:wink: :cyclone: :strawberry: TextRank implementation in Golang with extendable features (summarization, phrase extraction) and multithreading (goroutine).
- go
- golang
- textrank
- summarization
Statistical NLG for spoken dialogue systems
- natural-language-generation
- dialogue
- seq2seq
- python
Bayesian text classifier with flexible tokenizers and storage backends for Go