Sign in

Repository Index

142,273 visible repositories in the index

Tags: LLM tools
Tags

A repository shows when it has any selected Tag.

Clear Tags
Filters (1)

Filters

Clear all filters

1–22 of 22 · Most stars

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

  • apple-silicon
  • inference-server
  • llm
  • macos

RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.

TranslatedOpen-source manuscript of 《深入理解 AI Infra:量化分析与系统设计》 (In-Depth Understanding of AI Infra: Quantitative Analysis and System Design), by 李博杰 (Li Bojie): starting from hardware constraints and model architecture, it quantitatively derives the system design of LLM inference and training. Includes the full text of the book, a PDF, and companion calculation tools and experiments

  • accelerator
  • ai-infra
  • ai-infrastructure
  • book

[EMNLP'23, ACL'24] To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.

The online version is temporarily unavailable because we cannot afford the key. You can clone and run it locally. Note: we set defaul openai key. If keys exceed plan and are invalid, please tell us. The response speed depends on openai. ( sometimes, the official is too crowded and slow)

  • chatgpt
  • information-extraction
  • zero-shot
  • event-extraction

Self-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default, cloud by choice. Offline AI memory (taOSmd), self-hosted multi-framework group chat, a full web desktop + app store, and auto-clustering across the consumer hardware you already have (Orange/Raspberry Pi, Mac mini, gaming PC).

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

  • cuda
  • flash-attention
  • fsdp
  • inference-engine