A blazing fast inference solution for text embeddings models
Repository Index
132,241 visible repositories in the index
49–72 of 233 · Most stars
Translatedcube studio is an open-source, cloud-native, one-stop machine learning / deep learning / large model AI platform. It covers the full mlops algorithm pipeline, a computing power rental platform, online notebook development, drag-and-drop task flow pipeline orchestration, multi-machine multi-GPU distributed training, hyperparameter search, inference services with VGPU virtualization, edge computing, a labeling platform with automated labeling, SFT fine-tuning / reward model / reinforcement learning training for large models such as deepseek, multi-machine large model inference with vllm/ollama/mindie, a private knowledge base, and an AI model marketplace. It supports domestic Chinese CPUs/GPUs/NPUs and the Ascend ecosystem, supports RDMA, and supports distributed frameworks such as pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/ray/volcano.
A fast inference library for running LLMs locally on modern consumer-class GPUs
LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.
An all-in-one, pure C++ inference engine for audio models, powered by ggml. Supports TTS, STT, VAD, voice conversion, music generation, and more, with highly optimized performance. No Python dependency.
- ai
- amd-gpu
- apple-silicon
- asr
Optimizing inference proxy for LLMs
- agent
- agentic-ai
- agentic-workflow
- agents
Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.
- apple-silicon
- local-llm
- macos
- mlx
Open-source library of optimized deep learning operations (matmul, convolution, attention) for CPUs (x64, AArch64, RISC-V) and Intel GPUs. Used by PyTorch, TensorFlow, OpenVINO, and ONNX Runtime.
- onednn
- oneapi
- performance
- cpp
CSGHub is a brand-new open-source platform for managing LLMs, developed by the OpenCSG team. It offers both open-source and on-premise/SaaS solutions, with features comparable to Hugging Face. Gain full control over the lifecycle of LLMs, datasets, and agents, with Python SDK compatibility with Hugging Face. Join us! ⭐️
- ai
- huggingface
- llm
- management-system
1 API key. 220,000+ AI models. Bytez is the largest serverless Model Inference API on the internet. Let's build ✨
High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle
Open-source inference server and production cluster for all the models your agent needs.
High-performance single-GPU inference for selected model checkpoints and GPUs.
A high-performance inference engine for AI models
Add a real-time analytics node to your operational database. Spice is a portable, accelerated SQL query, search, and LLM-inference engine in Rust for data-grounded AI apps and agents.
- artificial-intelligence
- developers
- machine-learning
- data
Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
Distributed inference for mobile, desktop and server.
Community maintained hardware plugin for vLLM on Huawei Ascend
RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.
- ai
- containers
- cuda
- hip
Run GLM-5.3 (744B), DeepSeek-V4.1-Flash (748B), MiniMax-M3.1-Flash (428B), GLM-5.3-Flash (321B) and Hy4-Flash (220B) on your own GPUs - from 8 GB of VRAM, NVIDIA and AMD, one dashboard, OpenAI- and Anthropic-compatible API.
- cuda
- glm
- llm
- llm-inference
Lucebox inference engine for heterogeneous hardware & consumer GPUs
High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.
TokenSpeed is a speed-of-light LLM inference engine.
The code used to train and run inference with the ColVision models, e.g. ColPali, ColQwen2, and ColSmol.
- colpali
- information-retrieval
- retrieval-augmented-generation
- vision-language-model