tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Repository Index
142,273 visible repositories in the index
1–24 of 27 · Most stars
Seamlessly integrate LLMs into scikit-learn.
Kashgari is a production-level NLP Transfer learning framework built on top of tf.keras for text-labeling and text-classification, includes Word2Vec, BERT, and GPT2 Language Embedding.
- nlp
- sequence-labeling
- text-classification
- bert-model
Explain, analyze, and visualize NLP language models. Ecco creates interactive visualizations directly in Jupyter notebooks explaining the behavior of Transformer-based language models (like GPT2, BERT, RoBERTA, T5, and T0).
- nlp
- visualization
- explorables
- natural-language-processing
NLP DNN Toolkit - Building Your NLP DNN Models Like Playing Lego
- question-answering
- deep-learning
- pytorch
- natural-language-processing
In PyTorch Learing Neural Networks Likes CNN、BiLSTM
- cnn-model
- cnn-bilstm
- lstm-model
- gru
🛸 Use pretrained transformers like BERT, XLNet and GPT-2 in spaCy
- spacy
- spacy-pipeline
- spacy-extension
- nlp
Wiktionary dump file parser and multilingual data extractor
- wiktionary
- wiktionary-parser
- multilingual
- dictionary
Deep learning models trained to correct input errors in short, message-like text
ik-analyzer for solr 7.x-8.x
- ik-analyzer
- solr
- ik-analysis
- java
JS port and JS/WASM bindings for openai/tiktoken
Kuromoji is a self-contained and very easy to use Japanese morphological analyzer designed for search
- japanese
- morphological-analyser
- nlp-library
- part-of-speech-tagger
Self-contained Japanese Morphological Analyzer written in pure Go
- japanese
- tokenizer
- nlp-library
- japanese-language
Javascript library for detecting and transforming between Hiragana, Katakana, and Romaji
🔤 Natural language detection for Elixir without AI
- elixir
- language
- language-detection
- detect-language
An NLP library for Uralic languages such as Finnish, Skolt Sami, Moksha and so on. Also supporting some non-Uralic languages such as Spanish, French, Arabic, Swedish, Norwegian, Russian and English. LLMs, FSTs and More!
- nlp-library
- uralic-languages
- finnish
- sami
An elixir module to translate simplified Chinese to traditional Chinese, and vice versa, based on wikipedia data
The repository host gives no description.
Golang port of Petrovich - an inflector for Russian anthroponyms.
Framework for information extraction from tables
- table-information-extraction
- information-extraction
- literature
- text-mining
N-Gram generator in Ruby - http://en.wikipedia.org/wiki/N-gram
Text language identification using Wikipedia data
Elixir port of Nakatani Shuyo's natural language detector
- elixir
- language-detection
- language
- elixir-lang
A Elixir library for translating between hiragana, katakana, romaji, kanji and sound. It uses Mecab.
- elixir
- mecab