💫 Industrial-strength Natural Language Processing (NLP) in Python
Repository Index
55,140 visible repositories in the index
1–23 of 23 · Most stars
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Unsupervised text tokenizer for Neural Network-based text generation.
- neural-machine-translation
- natural-language-processing
- word-segmentation
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
- nlp
- natural-language-processing
- natural-language-understanding
- language-model
Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
CoreNLP: A Java suite of core NLP tools for tokenization, sentence segmentation, NER, parsing, coreference, sentiment analysis, etc.
- natural-language-processing
- nlp
- nlp-parsing
- named-entity-recognition
Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages
Self-contained Japanese Morphological Analyzer written in pure Go
- japanese
- tokenizer
- nlp-library
- japanese-language
Translated斑海豹 (Spotted Seal) — Phocinae-Largha-150M-v1: a 144.3M bilingual decision model. Tiny model. Big decisions.
- bilingual
- calibration
- cpu-inference
- decision-making
CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, transliteration, verb-sense, and more.
- natural-language-processing
- nlp
- big-data
- data-mining
A multilingual command line sentence tokenizer in Golang
- sentence-tokenizer
- tokenizer
- cli
- sentences
:wink: :cyclone: :strawberry: TextRank implementation in Golang with extendable features (summarization, phrase extraction) and multithreading (goroutine).
- go
- golang
- textrank
- summarization
Bayesian text classifier with flexible tokenizers and storage backends for Go
A multilingual tokenizer to split a string into tokens
Ruby port of the NLTK Punkt sentence segmentation algorithm
- rubynlp
- sentence-tokenizer
- ruby
- sentence-boundaries
Accurate Bayesian sentence tokenizer in Ruby.
- rubynlp
- ruby
- nlp
Beagle helps you identify keywords, phrases, regexes, and complex search queries of interest in streams of text documents.
- stream-search
- lucene
- luwak
- stored-query-engine
A simple tokenizer in Ruby for NLP tasks.
- ruby
- nlp
- natural-language-processing
- tokenizer
Multilingual library to easily parse date strings to java.util.Date objects.
- clojure
- parsing
- date-time
- date-parser
Naive Bayes text classification implementation as an OmniCat classifier strategy. (#ruby #naivebayes)
- naive-bayes-classifier
- ruby
- text-classification
- sentiment-analysis
Simple and customizable text tokenization gem.
- rubynlp
- ruby
- nlp
- tokenization
A tokenizer based on the dictionary and Bigram language models for Go. (Now only support chinese segmentation)
- tokenizer
- segmentation
- golang
The shamoji (杓文字) is a word filtering package
- go
- tokenizer
- word-filter