Distribute and run LLMs with a single file.
- cross-platform
- gguf
- llama-cpp
- local-ai
Repository Index
47,133 repositories indexed
25–48 of 167 · Most stars
Faster Whisper transcription with CTranslate2
- deep-learning
- inference
- quantization
- speech-recognition
Official inference repo for FLUX.1 models
DeepInfer: High-performance, large-scale inference framework for DeepSeek models
TranslatedThis project aims to share the technical principles of large models and practical experience (large model engineering, large model application deployment).
- llm
- llm-inference
- llm-serving
- llm-training
Multi-lingual large voice generation model, providing inference, training and deployment full-stack ability.
- audio-generation
- gpt-4o
- text-to-speech
- tts
DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm
ncnn is a high-performance neural network inference framework optimized for the mobile platform
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
Run agents like Hermes, LangChain Deep Agents, and OpenClaw more securely inside NVIDIA OpenShell with managed inference
Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
- pytorch
- speech-recognition
- paraformer
- punctuation
Inference Llama 2 in one file of pure C
The repository provides code for running inference with the Meta Segment Anything Model 2 (SAM 2), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.
Machine Learning Engineering Open Book
A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations
High-performance In-browser LLM Inference Engine
Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We also show you how to solve end to end problems using Llama model family and using them on various provider services
MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.
- mnn
- convolution
- vulkan
- embedded-devices
Machine learning, in numpy
- machine-learning
- neural-networks
- topic-modeling
- gaussian-mixture-models
Nano vLLM
🎨 The exhaustive Pattern Matching library for TypeScript, with smart type inference.
- pattern-matching
- typescript
- ts
- pattern
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.
- tensorrt
- nvidia
- deep-learning
- inference