Sign in

Repository Index

142,273 visible repositories in the index

Tags: All Tags
Tags

A repository shows when it has any selected Tag.

Filters (1)

Filters

Clear all filters

1–24 of 119 · Most stars

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

  • apple-silicon
  • inference-server
  • llm
  • macos

RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.

TranslatedOpen-source manuscript of 《深入理解 AI Infra:量化分析与系统设计》 (In-Depth Understanding of AI Infra: Quantitative Analysis and System Design), by 李博杰 (Li Bojie): starting from hardware constraints and model architecture, it quantitatively derives the system design of LLM inference and training. Includes the full text of the book, a PDF, and companion calculation tools and experiments

  • accelerator
  • ai-infra
  • ai-infrastructure
  • book

🔩 Like builtins, but boltons. 250+ constructs, recipes, and snippets which extend (and rely on nothing but) the Python standard library. Nothing like Michael Bolton.

[EMNLP'23, ACL'24] To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.

TranslatedAutomatically deletes WeChat cache data on Windows and Mac computers, including the large amount of files, videos, images, and other content automatically downloaded from all chats, freeing up your space.

A GPU task manager for Windows, Linux, and WSL: per-process VRAM, GPU activity, and LLM model/KV-cache insights for NVIDIA, AMD, and Intel GPUs (plus Huawei Ascend on Linux).