tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Repository Index
132,183 visible repositories in the index
1–24 of 42 · Most stars
Offline, privacy-first grammar checker. Fast, open-source, Rust-powered
- developer-tools
- english-language
- grammar-checker
- rust
An open-source framework for detecting, redacting, masking, and anonymizing sensitive data (PII) across text, images, and structured data. Supports NLP, pattern matching, and customizable pipelines.
- python
- pii
- privacy
- data-anonymization
Library for conversion between Traditional and Simplified Chinese
- chinese-translation
- traditional-chinese
- simplified-chinese
- chinese
Open Source Neural Machine Translation and (Large) Language Models in PyTorch
- deep-learning
- pytorch
- machine-translation
- neural-machine-translation
An open source library for deep learning end-to-end dialog systems and chatbots.
Open-source offline translation library written in Python
- python
- machine-translation
- transformers
- translation
Local-first healthcare AI: clinical NER and HIPAA PII de-identification on hardware you control. 2,200+ medical models, 35 model-backed PII languages, and Python, MLX, Android and browser runtimes. Apache-2.0 SDK.
- healthcare
- llm
- on-device
- on-premise
Translated:herb: Chinese synonyms: chatbot, intelligent question answering toolkit
A C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.
- address-parser
- machine-learning
- nlp
- address
An Open-Source Framework for Prompt-Learning.
Fast inference engine for Transformer models
- neural-machine-translation
- cpp
- mkl
- quantization
Prompt Engineering | Prompt Versioning | Use GPT or other prompt based models to get structured output. Join our discord for Prompt-Engineering, LLMs and other latest research
- chatgpt
- chatgpt-api
- gpt-3
- gpt-3-prompts
[EMNLP 2022] An Open Toolkit for Knowledge Graph Extraction and Construction
- knowledge-graph
- relation-extraction
- chinese
- named-entity-recognition
An Open-Source Package for Neural Relation Extraction (NRE)
- relation-extraction
FlagAI (Fast LArge-scale General AI models) is a fast, easy-to-use and extensible toolkit for large-scale model.
Open Source Pre-training Model Framework in PyTorch & Pre-trained Model Zoo
- bert
- pre-training
- fine-tuning
- gpt
TranslatedNon-autoregressive System 1 decision engine, designed for structured, typed decision-making scenarios. DeepOpen Multilingual, non-autoregressive System 1 decision engine.
OpenAI Privacy Filter
中文命名实体识别(包括多种模型:HMM,CRF,BiLSTM,BiLSTM+CRF的具体实现)
- named-entity-recognition
- nlp
- sequence-labeling
- pytorch-nlp
Baidu's open-source Sentiment Analysis System.
- natural-language-processing
- sentiment-analysis
- sentiment-classification
- paddlepaddle
An Open-source Neural Hierarchical Multi-label Text Classification Toolkit
Shared repository for open-sourced projects from the Google AI Language team.
Underthesea - AI Assistant