Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, tokenization caching, Responses API, embeddings, WASM plugins, MCP, and multi-tenant auth.
Repository Index
132,241 visible repositories in the index
601–624 of 748 · Most stars
[CVPR 2024 Highlight] DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models
- acceleration
- diffusion-models
- generative-ai
- generative-model
Inference Code for Polygon-RNN++ (CVPR 2018)
- cvpr2018
- tensorflow
- deep-learning
- instance-segmentation
Bayesian learning and inference for state space models
yolort is a runtime stack for yolov5 on specialized accelerators such as tensorrt, libtorch, onnxruntime, tvm and ncnn.
- libtorch
- yolov5
- inference
- torchscript
Privacy Meter: An open-source library to audit data privacy in statistical and machine learning algorithms.
- machine-learning
- data-privacy
- privacy
- inference
[ICML 2024] SqueezeLLM: Dense-and-Sparse Quantization
- efficient-inference
- large-language-models
- llm
- model-compression
[NeurIPS 2020] MCUNet: Tiny Deep Learning on IoT Devices; [NeurIPS 2021] MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning
- deep-learning
- neural-architecture-search
- pytorch
- tinyml
A curated list of AI tools, agent skills, libraries, and resources for economics research, teaching, and policy analysis. Maintained by the OpenEcon team.
- ai
- awesome
- awesome-list
- causal-inference
Serverless LLM Serving for Everyone.
- large-language-models
- model-as-a-service
- model-serving
- serverless-inference
Deep neural network library and toolkit to do high performace inference on NVIDIA jetson platforms
On-device Speech-to-Intent engine powered by deep learning
- natural-language-understanding
- voice-recognition
- nlu
- spoken-language-understanding
v objective diffusion inference code for PyTorch.
The repository host gives no description.
LLMFlows - Simple, Explicit and Transparent LLM Apps
- ai
- llm
- llm-inference
- llmops
Triton Python, C++ and Java client libraries, and GRPC-generated client examples for go, java and scala.
The example project of inferencing Pose Estimation using Core ML
Tools, ComfyUI workflows and benchmark configs from a 4x RTX 3090 local-inference rig
- comfyui
- local-inference
- lora
- minimax-h3
Bayesian Python: Bayesian inference tools for Python
USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference
- attention-is-all-you-need
- llm-inference
- llm-training
- pytorch
TensorFlow models accelerated with NVIDIA TensorRT
- tensorflow
- tensorrt
- nvidia
- jetson
Graph Data Science: an abstraction layer in Python for building knowledge graphs, integrated with popular graph libraries – atop Pandas, NetworkX, RAPIDS, RDFlib, pySHACL, PyVis, morph-kgc, pslpython, pyarrow, etc.
- knowledge-graph
- rdflib
- networkx
- skos
R toolkit for inference, visualization and analysis of cell-cell communication from single-cell and spatially resolved transcriptomics
- cell-cell-communication
- cell-cell-interaction
- microenvironment
- single-cell-analysis
Deep Learning on Flink aims to integrate Flink and deep learning frameworks (e.g. TensorFlow, PyTorch, etc) to enable distributed deep learning training and inference on a Flink cluster.
- flink
- tensorflow
- pytorch
- python