An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
Repository Index
72,862 visible repositories in the index
1–21 of 21 · Most stars
A powerful tool for creating datasets for LLM fine-tuning 、RAG and Eval
- dataset
- javascript
- llm
- fine-tuning
AI Observability & Evaluation
- llmops
- ai-monitoring
- ai-observability
- llm-eval
OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.
- evaluation
- benchmark
- large-language-model
- llm
Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.
- ai
- chain-of-thought
- collaboration
- dataset-generation
Open-Source Agent Observability & Evals for AI Platform teams. Trace, test, route and govern every agent & coding assistant in the company.
Prompt Engineering | Prompt Versioning | Use GPT or other prompt based models to get structured output. Join our discord for Prompt-Engineering, LLMs and other latest research
- chatgpt
- chatgpt-api
- gpt-3
- gpt-3-prompts
TranslatedChinese Language Understanding Evaluation Benchmark: datasets, baselines, pre-trained models, corpus and leaderboard
- nlu
- benchmark
- chinese
- corpus
Comet: agent skill harness for turning ideas into evaluated workflows
Open-source tools for prompt testing and experimentation, with support for both LLMs (e.g. OpenAI, LLaMA) and vector databases (e.g. Chroma, Weaviate, LanceDB).
🤗 Evaluate: A library for easily evaluating machine learning models and datasets.
- evaluation
- machine-learning
AssetOpsBench - Industry 4.0: A unified benchmark and framework for building, orchestrating, and evaluating domain-specific AI agents for Industry 4.0 asset operations and maintenance, with 460+ scenarios, 5 specialist agents (IoT, FMSR, TSFM, Work Order,...), and multi-agent orchestration blueprints (MetaAgent, AgentHive) over MCP.
- llm-agents
- model-context-protocol
- time-series-forecasting
- condition-based-maintenance
UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.
- machine-learning
- experimentation
- llm-prompting
- llm-test
A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
- ai-gateway
- observability
- simulation
- ai-agents
Benchmark for tracking model capability after release.
A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.
BLEURT is a metric for Natural Language Generation based on transfer learning.
A unified framework for building, evaluating, and recursively improving embodied agents. 🚀 Code release coming soon.
How many bugs can LMs find & fix in large codebases?
Open-source simulation harness for testing AI agents. Simulate LLM and tool calls to test edge cases, failure paths, and agent logic without live API calls.
- ai-agents
- ai-testing
- llm
- llm-evaluation