An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
Repository Index
61,121 visible repositories in the index
1–24 of 34 · Most stars
Supercharge Your LLM Application Evaluations 🚀
- llm
- llmops
- evaluation
Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced analytics with timeline and execution graph view
- agentneo
- agents
- ai-performance-optimization
- llm-testing
Open-source AI brand visibility and competitor reports. Official website: https://niubigeo.ai/ | Paid services: AI testing by real people and GEO optimization. Pricing: https://niubigeo.ai/pricing
the LLM vulnerability scanner
- ai
- llm-evaluation
- llm-security
- security-scanners
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
- data-drift
- jupyter-notebook
- pandas-dataframe
- machine-learning
A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation.
Framework for evaluating and improving agents
- evals
- rl-environments
- terminal-bench
Agent OS: the agent gets smarter on its own. We just hold the line: Interview-gated, staged evaluation, budgeted evolution loop. MCP server, 14 runtimes: Claude Code, Codex CLI, Gemini CLI, OpenCode, Copilot, Kiro and more.
- ai-agent
- agent-os
- llm-orchestration
- agentic-ai
A visual playground for agentic workflows: Iterate over your agents 10x faster
Next-generation AI Agent Optimization Platform: Cozeloop addresses challenges in AI agent development by providing full-lifecycle management capabilities from development, debugging, and evaluation to monitoring.
[ICLR'24 spotlight] An open platform for training, serving, and evaluating large language model for tool learning.
The open source research environment for AI researchers to seamlessly train, evaluate, and scale models from local hardware to GPU clusters.
Our library for RL environments + evals
35 production-grade agentic AI architectures (Reflexion, LATS, GraphRAG, MemGPT, Voyager, BrowserAgent, ...) — a Python library and runnable textbook with multi-provider LLM support and a 17-task benchmark leaderboard.
- agentic-ai
- ai-agents
- langchain
- langgraph
Align Anything: Training All-modality Model with Feedback
- large-language-models
- multimodal
- rlhf
- chameleon
One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks
- agi
- evaluation
- large-language-models
- multimodal
Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks
- gpt-4v
- large-language-models
- llava
- multi-modal
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
- ai-gateway
- observability
- simulation
- ai-agents
Evaluation code for various unsupervised automated metrics for Natural Language Generation.
- natural-language-generation
- natural-language-processing
- nlg
- nlp
A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.
FactCircuit: an auditable claim-and-evidence verification loop with immutable provenance, staged checks, bounded retrieval, and reproducible evaluation.
- accuracy-decline
- ai-safety
- audit-trail
- fact-checking
TranslatedSprix AI at 屿智同行 (Yuzhi Tongxing, "island wisdom traveling together") × 保研岛 (Baoyan Dao, "Baoyan Island") — an open-source benchmark for AI agents in Baoyan advising and application planning.
A unified framework for building, evaluating, and recursively improving embodied agents. 🚀 Code release coming soon.