Repository Index

Unsupervised text tokenizer for Neural Network-based text generation.

• neural-machine-translation
• natural-language-processing
• word-segmentation

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

Open-source offline translation library written in Python

• python
• machine-translation
• transformers
• translation

Naive linter for English prose

Module for automatic summarization of text documents and HTML pages.

• python
• lsa
• textteaser
• html-page

📐 Compute distance between sequences. 30+ algorithms, pure python implementation, common interface, optional external libs usage.

• distance
• algorithm
• python
• textdistance

Go efficient fast multilingual NLP and text segmentation; support English, Chinese, Japanese and others.

• go
• segment
• nlp
• gse

Self-contained Machine Learning and Natural Language Processing library in Go

• deep-learning
• machine-learning
• natural-language-processing
• neural-network

TranslatedConvert Chinese characters to pinyin

• go
• pinyin
• golang
• hanzi

A natural language date/time parser with pluggable rules

• natural-language
• datetime
• date
• time

The most accurate natural language detection library for Go, suitable for short text and mixed-language text

• natural-language-processing
• language-detection
• language-recognition
• language-classification

Self-contained Japanese Morphological Analyzer written in pure Go

• japanese
• tokenizer
• nlp-library
• japanese-language

A Modern C++ Data Sciences Toolkit

• nlp
• nlp-parsing
• search-engine
• inverted-index

Natural language detection library for Go

• language
• go
• nlp
• text-processing

Onnx transformer pipelines in Golang

Selected Machine Learning algorithms for natural language processing and semantic analysis in Golang

• go
• golang
• natural-language-processing
• nlp

A multilingual command line sentence tokenizer in Golang

• sentence-tokenizer
• tokenizer
• cli
• sentences

Fast and Portable Character String Processing in R (with the Unicode ICU)

• stringi
• icu
• icu4c
• r

:wink: :cyclone: :strawberry: TextRank implementation in Golang with extendable features (summarization, phrase extraction) and multithreading (goroutine).

• go
• golang
• textrank
• summarization

Automatically exported from code.google.com/p/berkeleyparser

Natural language detection package in pure Go

• nlp
• natural-language
• language-model

A Library to parse natural language in pure Clojure and ClojureScript

• natural-language-processing
• clojure
• clojurescript
• pos-tagger

Bayesian text classifier with flexible tokenizers and storage backends for Go

A Go port of the Rapid Automatic Keyword Extraction algorithm (RAKE)