LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
Repository Index
142,273 visible repositories in the index
1–24 of 119 · Most stars
A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations
Sub-50mb agentic, voice and vision intelligence for consumer devices.
- cactus
- llm
- on-device-ai
- edge-ai
RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.
- attention-mechanism
- deep-learning
- gpt
- gpt-2
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
Semantic cache for LLMs. Fully integrated with LangChain and llama_index.
TranslatedOpen-source manuscript of 《深入理解 AI Infra:量化分析与系统设计》 (In-Depth Understanding of AI Infra: Quantitative Analysis and System Design), by 李博杰 (Li Bojie): starting from hardware constraints and model architecture, it quantitatively derives the system design of LLM inference and training. Includes the full text of the book, a PDF, and companion calculation tools and experiments
- accelerator
- ai-infra
- ai-infrastructure
- book
🔩 Like builtins, but boltons. 250+ constructs, recipes, and snippets which extend (and rely on nothing but) the Python standard library. Nothing like Michael Bolton.
- python
- utilities
- cache
- data-structures
[EMNLP'23, ACL'24] To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.
See what Claude Code and Codex actually send to the API — and what each part costs.
- claude-code
- codex-cli
- cost-tracking
- observability
TranslatedAutomatically deletes WeChat cache data on Windows and Mac computers, including the large amount of files, videos, images, and other content automatically downloaded from all chats, freeing up your space.
TranslatedPicACG comic PC client(Windows, Linux, MacOS)
- waifu2x
- picacomic
- bika
- picacg
Simple Linux Panel
Computing with Python functions.
- python
- parallel-computing
- caching
- multiprocessing
Translated18comic, a PC client implemented with Qt, supporting Windows, Linux, and MacOS
- 18comic
- jmcomic
- qt6
- pyside6
Open-source multimodal retrieval engine (Morphik Core)
- artificial-intelligence
- cache-augmented-generation
- colpali
- database
어느 플랫폼에서든 사용할 수 있는 system-ui 대체 글꼴 | A system-ui alternative font for all cross-platform
A GPU task manager for Windows, Linux, and WSL: per-process VRAM, GPU activity, and LLM model/KV-cache insights for NVIDIA, AMD, and Intel GPUs (plus Huawei Ascend on Linux).
Static site generator for designers. Uses Python and Django templates.
This tool generates gopher link for exploiting SSRF and gaining RCE in various servers
Full featured redis cache backend for Django.
Full-featured library for writing Alfred 3 & 4 workflows
- alfred-workflow
- alfred
- python
- keychain
Python disk-backed cache (Django-compatible). Faster than Redis and Memcached. Pure-Python.
- python
- cache
- persistence
- key-value-store
Compresses linked and inline javascript or CSS into a single cached file.