INT4/INT5/INT8 and FP16 inference on CPU for RWKV language model
- deep-learning
- language-model
- llm
- machine-learning
100,822 visible repositories in the index
1–4 of 4 · Most stars
INT4/INT5/INT8 and FP16 inference on CPU for RWKV language model
FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.
TurboOCR, >200 img/s OmnidocBench. TensorRT FP16, PP-OCRv6, HTTP + gRPC
Fault-injected tool environments with side-effect grading for evaluating and post-training LLM agents