Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring.
Repository Index
100,822 visible repositories in the index
1–18 of 18 · Most stars
OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.
- evaluation
- benchmark
- large-language-model
- llm
This repo powers my experiment where ChatGPT manages a real-money micro-cap stock portfolio.
Open-source AI red teaming toolkit from Microsoft. Test generative AI systems with automated scans, adaptive multi-turn attacks, and a web UI.
- ai-red-team
- generative-ai
- red-team-tools
- responsible-ai
One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks
- agi
- evaluation
- large-language-models
- multimodal
LLM-as-a-Verifier is a general-purpose framework that provides fine-grained feedback for any agent without requiring additional training. It achieves SOTA performance across coding, robotics, and medical agentic benchmarks.
Find, benchmark and install in CLI 170+ FREE coding LLM models across 15+ providers in real time
Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
- evaluation
- evaluation-framework
- evaluation-metrics
- huggingface
A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.
Large Language-and-Vision Assistant for Biomedicine, built towards multimodal GPT-4 level capabilities.
Responsible AI Toolbox is a suite of tools providing model and data exploration and assessment user interfaces and libraries that enable a better understanding of AI systems. These interfaces and libraries empower developers and stakeholders of AI systems to develop and monitor AI more responsibly, and take better data-driven actions.
- ui
- responsible-ai
- data-science
- fairness
CodeXGLUE
CLI for testing and evaluating AI agent workflows, custom agents, and skills with repeatable scenarios, graders, evidence, and CI integration.
Reference BLEU implementation that auto-downloads test sets and reports a version string to facilitate cross-lab comparisons
Open memory & reliability layer for AI agents + independent judging/verification (NACRE judge, caliber-bench meta-benchmark, preregistered criteria) — by Nautilus Platform
A lightweight Python SDK and CLI for evaluating recursive self-improvement (RSI) methods across agent benchmarks.
- ai-agents
- benchmarking
- llm-evaluation
- python
Open-source whiteboard AI tutor with animated lessons and interactive playback. Compare Claude, Gemini & DeepSeek across 288 generation runs and 64 blind-reviewed boards.
- ai-teacher
- ai-tutor
- ai-tutoring
- ai-whiteboard
VisSearch-Bench is a benchmark of 204 human-written tasks that test how well multimodal agents find, organize, and use visual evidence across multiple steps of web search.