Repository Index

💫 Industrial-strength Natural Language Processing (NLP) in Python

• natural-language-processing
• data-science
• machine-learning
• python

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

Unsupervised text tokenizer for Neural Network-based text generation.

• neural-machine-translation
• natural-language-processing
• word-segmentation

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

• nlp
• natural-language-processing
• natural-language-understanding
• language-model

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

CoreNLP: A Java suite of core NLP tools for tokenization, sentence segmentation, NER, parsing, coreference, sentiment analysis, etc.

• natural-language-processing
• nlp
• nlp-parsing
• named-entity-recognition

Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages

• python
• nlp
• natural-language-processing
• machine-learning

Self-contained Japanese Morphological Analyzer written in pure Go

• japanese
• tokenizer
• nlp-library
• japanese-language

Translated斑海豹 (Spotted Seal) — Phocinae-Largha-150M-v1: a 144.3M bilingual decision model. Tiny model. Big decisions.

• bilingual
• calibration
• cpu-inference
• decision-making

CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, transliteration, verb-sense, and more.

• natural-language-processing
• nlp
• big-data
• data-mining

A multilingual command line sentence tokenizer in Golang

• sentence-tokenizer
• tokenizer
• cli
• sentences

:wink: :cyclone: :strawberry: TextRank implementation in Golang with extendable features (summarization, phrase extraction) and multithreading (goroutine).

• go
• golang
• textrank
• summarization

Bayesian text classifier with flexible tokenizers and storage backends for Go

A multilingual tokenizer to split a string into tokens

Ruby port of the NLTK Punkt sentence segmentation algorithm

• rubynlp
• sentence-tokenizer
• ruby
• sentence-boundaries

Accurate Bayesian sentence tokenizer in Ruby.

• rubynlp
• ruby
• nlp

Beagle helps you identify keywords, phrases, regexes, and complex search queries of interest in streams of text documents.

• stream-search
• lucene
• luwak
• stored-query-engine

A simple tokenizer in Ruby for NLP tasks.

• ruby
• nlp
• natural-language-processing
• tokenizer

Multilingual library to easily parse date strings to java.util.Date objects.

• clojure
• parsing
• date-time
• date-parser

Naive Bayes text classification implementation as an OmniCat classifier strategy. (#ruby #naivebayes)

• naive-bayes-classifier
• ruby
• text-classification
• sentiment-analysis

Simple and customizable text tokenization gem.

• rubynlp
• ruby
• nlp
• tokenization

A tokenizer based on the dictionary and Bigram language models for Go. (Now only support chinese segmentation)

• tokenizer
• segmentation
• golang

The shamoji (杓文字) is a word filtering package

• go
• tokenizer
• word-filter