tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Repository Index
69,734 visible repositories in the index
1–5 of 5 · Most stars
Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
All In One Web Recon
- python3
- pentesting
- pentest-tool
- pentesting-tools
A lightweight suite of motion imitation methods for training controllers.
- animation
- reinforcement-learning
- robotics
- motion-imitation
Unsupervised Word Segmentation for Neural Machine Translation and Text Generation
- neural-machine-translation
- segmentation
- machine-translation
- nmt