🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
- nlp
- natural-language-processing
- pytorch
- pytorch-transformers
Repository Index
66,263 visible repositories in the index
1–21 of 21 · Most stars
A Python library for extracting structured information from unstructured text using LLMs with precise source grounding and interactive visualization.
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
- nlp
- pre-trained-model
- unilm
- minilm
Easy-to-use and powerful LLM and SLM library with awesome model zoo.
- nlp
- embedding
- bert
- ernie
Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
Open Source Neural Machine Translation and (Large) Language Models in PyTorch
- deep-learning
- pytorch
- machine-translation
- neural-machine-translation
Translatedpycorrector is a toolkit for text error correction. Text error correction, implementing applications of models such as Kenlm, T5, MacBERT, ChatGLM3, and Qwen2.5 in error correction scenarios, ready to use out of the box.
- spelling-errors
- error-correction
- pycorrector
- kenlm
Local-first healthcare AI: clinical NER and HIPAA PII de-identification on hardware you control. 2,200+ medical models, 35 model-backed PII languages, and Python, MLX, Android and browser runtimes. Apache-2.0 SDK.
- healthcare
- llm
- on-device
- on-premise
🦛 CHONK docs with Chonkie ✨ — The lightweight ingestion library for fast, efficient and robust RAG pipelines
- rag
- chonkie
- chunker
- chunking-algorithm
Prompt Engineering | Prompt Versioning | Use GPT or other prompt based models to get structured output. Join our discord for Prompt-Engineering, LLMs and other latest research
- chatgpt
- chatgpt-api
- gpt-3
- gpt-3-prompts
State of the Art Natural Language Processing
Language model tokenization at GB/s
Fengshenbang-LM(封神榜大模型)是IDEA研究院认知计算与自然语言研究中心主导的大模型开源体系，成为中文AIGC和认知智能的基础设施。
- chinese-nlp
- pretrained-models
- pytorch
- distributed-training
中文 NLP 预处理、解析工具包，准确、高效、易用 A Chinese NLP Preprocessing & Parsing Package www.jionlp.com
- nlp
- natural-language-processing
- python
- apache2
Seamlessly integrate LLMs into scikit-learn.
Open Source Pre-training Model Framework in PyTorch & Pre-trained Model Zoo
- bert
- pre-training
- fine-tuning
- gpt
KenLM: Faster and Smaller Language Model Queries
A small, fast decision model, or system one model, for agentic workflows. Pick between options or rate on a scale faster than an LLM, with a calibrated confidence on every decision.
- ai
- machine-learning
- natural-language-understanding
- python
Pretrained model hub for Keras 3.
JevStyle: local decision models for AI agents. Classify, route and score with a probability per option. systemone-compatible local server, MCP tools, Claude Code guard. Open weights, Apache-2.0.
- agent-skills
- calibration
- claude-code
- decision-model
An NLP library for Uralic languages such as Finnish, Skolt Sami, Moksha and so on. Also supporting some non-Uralic languages such as Spanish, French, Arabic, Swedish, Norwegian, Russian and English. LLMs, FSTs and More!
- nlp-library
- uralic-languages
- finnish
- sami