Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
Repository Index
47,133 repositories indexed
169–192 of 340 · Most stars
Agent Reinforcement Trainer: train multi-step agents for real-world tasks using GRPO. Give your agents on-the-job training. Reinforcement learning for Qwen3.6, GPT-OSS, Llama, and more!
The AI Compute Platform for frontier teams. SkyPilot turns fragmented AI compute into one AI supercomputer, so frontier AI teams build custom intelligence faster.
- cloud-computing
- deep-learning
- gpu
- hyperparameter-tuning
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
- bloom
- deep-learning
- distributed-systems
- language-models
Build, Manage and Deploy AI/ML Systems
- machine-learning
- model-management
- ai
- ml
Translated[CVPR 2024 Oral] InternVL Family: A Pioneering Open-Source Alternative to GPT-4o. An open-source multimodal dialogue model with performance close to GPT-4o
- image-classification
- image-text-retrieval
- llm
- semantic-segmentation
the LLM vulnerability scanner
- ai
- llm-evaluation
- llm-security
- security-scanners
~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
- academia
- arxiv
- brave
- deep-research
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
- data
- data-cleaning
- data-pipelines
- data-processing
A collection of libraries to optimise AI model performances
Semantic cache for LLMs. Fully integrated with LangChain and llama_index.
Infrastructure for continually self‑improving agents
Run LLMs with MLX
- llms
- mlx
TranslatedOpen-source manuscript of 《深入理解 AI Infra：量化分析与系统设计》 (In-Depth Understanding of AI Infra: Quantitative Analysis and System Design), by 李博杰 (Li Bojie): starting from hardware constraints and model architecture, it quantitatively derives the system design of LLM inference and training. Includes the full text of the book, a PDF, and companion calculation tools and experiments
- accelerator
- ai-infra
- ai-infrastructure
- book
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
- web-scraping
- text-extraction
- nlp
- html2text
TranslatedMedicalGPT: Training Your Own Medical GPT Model with ChatGPT Training Pipeline. Training medical large models, implementing incremental pre-training (PT), supervised fine-tuning (SFT), RLHF, DPO, ORPO, and GRPO.
ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.
- mlops
- machine-learning
- data-science
- production-ready
Superduper: End-to-end framework for building custom AI applications and agents.
- ai
- mlops
- torch
- transformers
Zero-modification Human-in-the-Loop adapter for OpenOPC's agentic DAG runtime.
AG2 (formerly AutoGen): The Open-Source AgentOS.Join us at: https://discord.gg/sNGSwQME3x
- a2a
- agent-framework
- llm
- python
[COLM 2024] OpenAgents: An Open Platform for Language Agents in the Wild
- llm
- code-generation
- executable-langauge-grounding
- language-model
Experience-Driven Black-Box Cost Control for Coding Agents
- agentic-ai
- agentic-coding
- ai-coding
- ai-coding-assistant
Long-term memory runtime for AI agents — plain Markdown as the source of truth, local ranked retrieval, and an independent sleep-time Manage layer. Claude Code and Codex share one store. No API key.
HarnessRouter Community Edition: the self-hosted, Apache-2.0 edition of the unified interface for agent harnesses. Run Codex, Claude Code, Hermes, PI, DSH, and more through one API, with sessions, streaming, files, cancellation, and failure handling. Implements the Unified Harness Protocol (UHP), an open standard. Your keys, your infrastructure.
- agent-harness
- ai-infrastructure
- open-protocol
- open-source