VoxCPM2: Tokenizer-Free TTS for Multilingual Speech Generation, Creative Voice Design, and True-to-Life Cloning
- audio
- deeplearning
- minicpm
- python
Repository Index
47,133 repositories indexed
1–24 of 24 · Most stars
💫 Industrial-strength Natural Language Processing (NLP) in Python
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Unsupervised text tokenizer for Neural Network-based text generation.
- neural-machine-translation
- natural-language-processing
- word-segmentation
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
- nlp
- natural-language-processing
- natural-language-understanding
- language-model
Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
CoreNLP: A Java suite of core NLP tools for tokenization, sentence segmentation, NER, parsing, coreference, sentiment analysis, etc.
- natural-language-processing
- nlp
- nlp-parsing
- named-entity-recognition
Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages
Self-contained Japanese Morphological Analyzer written in pure Go
- japanese
- tokenizer
- nlp-library
- japanese-language
Translated斑海豹 (Spotted Seal) — Phocinae-Largha-150M-v1: a 144.3M bilingual decision model. Tiny model. Big decisions.
- bilingual
- calibration
- cpu-inference
- decision-making
CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, transliteration, verb-sense, and more.
- natural-language-processing
- nlp
- big-data
- data-mining
A multilingual command line sentence tokenizer in Golang
- sentence-tokenizer
- tokenizer
- cli
- sentences
:wink: :cyclone: :strawberry: TextRank implementation in Golang with extendable features (summarization, phrase extraction) and multithreading (goroutine).
- go
- golang
- textrank
- summarization
On-device Vision Model - a fully offline, free vision pipeline that gives text-only models (e.g. DeepSeek) image understanding and multilingual OCR through an OpenAI-compatible local API. Only tokens are needed: no API key, no subscription, no cloud cost. Recommended for native speakers of Chinese, English, Japanese and Korean.
Bayesian text classifier with flexible tokenizers and storage backends for Go
A multilingual tokenizer to split a string into tokens
Ruby port of the NLTK Punkt sentence segmentation algorithm
- rubynlp
- sentence-tokenizer
- ruby
- sentence-boundaries
Accurate Bayesian sentence tokenizer in Ruby.
- rubynlp
- ruby
- nlp
The Claude companion for Laravel: the Anthropic API behind a facade, with batches, token counting, files, refusal fallback, named connections, events and test fakes, beside laravel/ai.
Beagle helps you identify keywords, phrases, regexes, and complex search queries of interest in streams of text documents.
- stream-search
- lucene
- luwak
- stored-query-engine
A simple tokenizer in Ruby for NLP tasks.
- ruby
- nlp
- natural-language-processing
- tokenizer
Multilingual library to easily parse date strings to java.util.Date objects.
- clojure
- parsing
- date-time
- date-parser
Simple and customizable text tokenization gem.
- rubynlp
- ruby
- nlp
- tokenization
Trust metadata for AI agents — a stamp costs ~15 tokens, re-verifying costs 15,000. Agents stamp what they verify; the next agent runs 'akf check' and builds on it. pip install akf