INT4/INT5/INT8 and FP16 inference on CPU for RWKV language model
- deep-learning
- language-model
- llm
- machine-learning
96,838 visible repositories in the index
1–3 of 3 · Most stars
INT4/INT5/INT8 and FP16 inference on CPU for RWKV language model
FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.
TurboOCR, >200 img/s OmnidocBench. TensorRT FP16, PP-OCRv6, HTTP + gRPC