Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
- nlp
- pre-trained-model
- unilm
- minilm
107,098 visible repositories in the index
1–21 of 21 · Most stars
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Easy-to-use and powerful LLM and SLM library with awesome model zoo.
Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
TranslatedThe pkuseg toolkit for multi-domain Chinese word segmentation
Prompt Engineering | Prompt Versioning | Use GPT or other prompt based models to get structured output. Join our discord for Prompt-Engineering, LLMs and other latest research
FlagAI (Fast LArge-scale General AI models) is a fast, easy-to-use and extensible toolkit for large-scale model.
Rust native ready-to-use NLP pipelines and transformer-based models (BERT, DistilBERT, GPT2,...)
EasyNLP: A Comprehensive and Easy-to-use NLP Toolkit
NCRF++, a Neural Sequence Labeling Toolkit. Easy use to any sequence labeling tasks (e.g. NER, POS, Segmentation). It includes character LSTM/CNN, word LSTM/CNN and softmax/CRF components.
This tool provides an efficient implementation of the continuous bag-of-words and skip-gram architectures for computing vector representations of words. These representations can be subsequently used in many natural language processing applications and for further research.
Named-entity recognition using neural networks. Easy-to-use and state-of-the-art results.
Graph4nlp is the library for the easy use of Graph Neural Networks for NLP. Welcome to visit our DLG4NLP website (https://dlg4nlp.github.io/index.html) for various learning resources!
A collection of common regular expressions bundled with an easy to use interface.
Easy to use extractive text summarization with BERT
🛸 Use pretrained transformers like BERT, XLNet and GPT-2 in spaCy
Use Google's BERT for named entity recognition (CoNLL-2003 as the dataset).
Easy to use, state-of-the-art Neural Machine Translation for 100+ languages
Modular, fast NLP framework, compatible with Pytorch and spaCy, offering tailored support for French clinical notes.
Unicode text segmentation for OCaml
A Elixir library for translating between hiragana, katakana, romaji, kanji and sound. It uses Mecab.