a fast and user-friendly runtime for transformer inference (Bert, Albert, GPT2, Decoders, etc) on CPU and GPU.
- nlp
- transformer
- bert
- decoder
132,241 visible repositories in the index
49–72 of 106 · Most stars
a fast and user-friendly runtime for transformer inference (Bert, Albert, GPT2, Decoders, etc) on CPU and GPU.
Local inference engine optimized for NVIDIA Blackwell
NVIDIA Inference Xfer Library (NIXL)
KVMem virtualizes KV memory across inference backends to make ultra-long-context local AI accessible on limited hardware.
LLMs as Copilots for Theorem Proving in Lean
An Industrial Graph Neural Network Framework
solution of exercises of the book "probabilistic robotics"
Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.
FeatherCNN is a high performance inference engine for convolutional neural networks.
Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder and MiniCPM-V on a second engine written for PolyStrata. One-click install for Windows and Linux, OpenAI/Anthropic API on localhost.
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM
KVarN, KV cache precision tail, low-bit quants in llama.cpp for longer context of better precision in the same VRAM
High Performance LLM Inference Operator Library
TurboOCR, >200 img/s OmnidocBench. TensorRT FP16, PP-OCRv6, HTTP + gRPC
Generalized Random Forests
Cross-Platform Production-ready C++ inference engine for YOLO models (v5-v12, YOLO26). Unified API for detection, segmentation, pose estimation, OBB, and classification. Built on ONNX Runtime and OpenCV. Optimized for CPU/GPU with quantization support.
Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimizer. Any hardware, any OS, any model, any quantization, behind a local OpenAI-compatible server.
Cross-platform, customizable multimedia/video processing framework. With strong GPU acceleration, heterogeneous design, multi-language support, easy to use, multi-framework compatible and high performance, the framework is ideal for transcoding, AI inference, algorithm integration, live video streaming, and more.
LLM inference in C/C++
Low-bit LLM inference on CPU/NPU with lookup table
Deep learning inference nodes for ROS / ROS2 with support for NVIDIA Jetson and TensorRT
jevos: an open-source alternative to Jev and Microsoft-Decision-1. Yes/no decisions, choices and scores with probabilities, on your laptop's CPU.
A scalable inference server for models optimized with OpenVINO™
Bolt is a deep learning library with high performance and heterogeneous flexibility.