Repository Index

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

Self-contained Japanese Morphological Analyzer written in pure Go

• japanese
• tokenizer
• nlp-library
• japanese-language

Parse human names into title, given, middle, family, suffix, nickname and maiden name. Typed, zero-dependency, CJK-aware.

• python
• text-processing
• text-parser
• python-module

Tagify produces a set of tags from a given source. Source can be either an HTML page, a Markdown document or a plain text. Supports English, Russian, Chinese, Hindi, Spanish, Arabic, Japanese, German, Hebrew, French and Korean languages.

• tagify
• keywords
• keyword-extraction
• html