TranslatedOpen-source manuscript of 《深入理解 AI Infra:量化分析与系统设计》 (In-Depth Understanding of AI Infra: Quantitative Analysis and System Design), by 李博杰 (Li Bojie): starting from hardware constraints and model architecture, it quantitatively derives the system design of LLM inference and training. Includes the full text of the book, a PDF, and companion calculation tools and experiments
- accelerator
- ai-infra
- ai-infrastructure
- book
Infrastructure for continually self‑improving agents
A framework for efficient model inference with omni-modality models
Open-source implementation of AlphaEvolve
Everything about Web Application Firewalls (WAFs) from Security Standpoint! 🔥
Translated💎1MB lightweight face detection model
Home of StarCoder: fine-tuning & inference!
Infection Monkey - An open-source adversary emulation platform
Native MLX runtime for Laya typed decision models — 7–14 ms short decisions on M3 Max. No text generation, PyTorch, or cloud API.
[EMNLP'23, ACL'24] To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.
Modeling, training, eval, and inference code for OLMo
An anomaly detection library comprising state-of-the-art algorithms and features such as experiment management, hyper-parameter optimization, and edge inference.
- anomaly-localization
- anomaly-segmentation
- anomaly-detection
- unsupervised-learning
Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams
- python
- attack
- adversarial-machine-learning
- poisoning
Awesome autocompletion, static analysis and refactoring library for python
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
Uplift modeling and causal inference with machine learning algorithms
A list of RSS related stuff: tools, services, communities and tutorials, etc.
MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX.
- llava
- llm
- mlx
- vision-transformer
🔥🔥🔥AidLearning is a powerful AIOT development platform, AidLearning builds a linux env supporting GUI, deep learning and visual IDE on Android...Now Aid supports CPU+GPU+NPU for inference with high performance acceleration...Linux on Android or HarmonyOS
General plug-and-play inference library for Recursive Language Models (RLMs), supporting various sandboxes.
learn LLM inference system on Apple Silicon for systems engineers: build a tiny vLLM + Qwen
"Single-image Layer Decomposition for Anime Characters" (SIGGRAPH 2026 Conference Paper)
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
Framework agnostic sliced/tiled inference + interactive ui + error analysis plots