Distributed training & inference for multimodal perception: shared embeddings across text, images, video, audio, code & graphs from 64 to 1024 dimensions 🖼️
- huggingface-transformers
- language-vision
- multimodal
- pytorch
132,241 visible repositories in the index
121–144 of 233 · Most stars
Distributed training & inference for multimodal perception: shared embeddings across text, images, video, audio, code & graphs from 64 to 1024 dimensions 🖼️
The definitive vLLM runtime for dual RTX 2080 Ti 22GB + NVLink, delivering Qwen 27B local inference with maximum 200+ tok/s single-request decode with support of FP8 weight ( Join Discord :https://discord.gg/VFqVVySdMS )
[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.
Fine-tune the Whisper speech recognition model to support training without timestamp data, training with timestamp data, and training without speech data. Accelerate inference and support Web deployment, Windows desktop deployment, and Android deployment
Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder and MiniCPM-V on a second engine written for PolyStrata. One-click install for Windows and Linux, OpenAI/Anthropic API on localhost.
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM
KVarN, KV cache precision tail, low-bit quants in llama.cpp for longer context of better precision in the same VRAM
Your iPhone helps your Mac run a 27B model: faster prompt reading and more context over a USB-C cable
Run, serve, query and edit AI models — a Rust model runtime built on VINDEX3, an executable, inspectable model format.
EdgeRock: Strong Reasoning at Constant Memory. Multimodal models with O(N) generation and O(1) persistent history. Inference and evaluation code.
TurboOCR, >200 img/s OmnidocBench. TensorRT FP16, PP-OCRv6, HTTP + gRPC
Streamlines and simplifies prompt design for both developers and non-technical users with a low code approach.
Run Laya, the open-source Jev-compatible System-1 decision model, from Node.js / TypeScript via ONNX Runtime
Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimizer. Any hardware, any OS, any model, any quantization, behind a local OpenAI-compatible server.
Quantized inference code for LLaMA models
Minimal LLM inference in Rust
LLM inference in C/C++
A lightweight inference engine supporting speculative speculative decoding (SSD).
Low-bit LLM inference on CPU/NPU with lookup table
MLX Studio - Easiest way to run LLM's on your Mac. All in one engine.
A throughput-oriented high-performance serving framework for LLMs
Running Llama 2 and other Open-Source LLMs on CPU Inference Locally for Document Q&A
jevos: an open-source alternative to Jev and Microsoft-Decision-1. Yes/no decisions, choices and scores with probabilities, on your laptop's CPU.
A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM