Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.
- ai
- diagnostic-tool
- agent-evaluation
- ai-alignment
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.
Translated非线智能 NoneLinear (Non-linear Intelligence) - ReLE evaluation: capability evaluation of Chinese AI large models (continuously updated): it currently covers 374 large models, including commercial models such as chatgpt, gpt-5.4, Google gemini-3.1-pro, Claude-4.6, 文心 ERNIE-X1.1, ERNIE-5.0, qwen3.6-max, qwen3.6-plus, Baichuan, iFlytek Spark, and SenseTime senseChat, as well as open-source large models such as step3.5-flash, kimi-k2.6, ernie4.5, MiniMax-M2.7, deepseek-v4, Qwen3.6, llama4, Zhipu GLM-5.1, MiMo-V2, LongCat, gemma4, and mistral. It provides not only leaderboards but also a large model defect library with more than 2,000,000 entries! This makes it convenient for the broad community to research, analyze, and improve large models.
Next-generation AI Agent Optimization Platform: Cozeloop addresses challenges in AI agent development by providing full-lifecycle management capabilities from development, debugging, and evaluation to monitoring.
- agent
- ai
- agent-evaluation
- agent-observability
🧩 FrontierAgent, our agent framework, open-sourced alongside it — native command-line TUI, ReAct and Agent Team modes, one command on macOS and Linux, no preinstall, no hard Docker dependency.
35 production-grade agentic AI architectures (Reflexion, LATS, GraphRAG, MemGPT, Voyager, BrowserAgent, ...) — a Python library and runnable textbook with multi-provider LLM support and a 17-task benchmark leaderboard.
Sky-T1: Train your own O1 preview model within $450
Benchmarking large language models' complex reasoning ability with chain-of-thought prompting
Repository for benchmarking graph neural networks (JMLR 2023)
- graph-representation-learning
- graph-neural-networks
- benchmark-framework
- graph-deep-learning
Doing simple retrieval from LLM models at various context lengths to measure accuracy
拼好RAG:手搓并融合了GraphRAG、LightRAG、Neo4j-llm-graph-builder进行知识图谱构建以及搜索;整合DeepSearch技术实现私域RAG的推理;自制针对GraphRAG的评估框架| Integrate GraphRAG, LightRAG, and Neo4j-llm-graph-builder for knowledge graph construction and search. Combine DeepSearch for private RAG reasoning. Create a custom evaluation framework for GraphRAG.
- deepresearch
- deepsearch
- evaluation
- graphrag
Official repository of 'Visual-RFT: Visual Reinforcement Fine-Tuning' & 'Visual-ARFT: Visual Agentic Reinforcement Fine-Tuning'’
An on-premises, OCR-free unstructured data extraction, markdown conversion and benchmarking toolkit. (https://idp-leaderboard.org/)
- document
- document-analysis
- extraction
- llms
Beyond Accuracy: Behavioral Testing of NLP models with CheckList
An automatic evaluator for instruction-following language models. Human-validated, high-quality, cheap, and fast.
👾 nao is the #1 open source agentic BI. (1) Create context with nao-core cli, (2) deploy nao chat interface for everyone
- agentic-analytics
- analytics
- analytics-engineering
- bigquery
Cambrian-1 is a family of multimodal LLMs with a vision-centric design.
CORTEX RAG is an enterprise retrieval and knowledge assistant that helps teams find accurate answers from company data with citations, permission-aware retrieval, and fast deployment
BERT score for text generation
BullshitBench measures whether AI models challenge nonsensical prompts instead of confidently answering them, created by Peter Gostev.
AutoChain: Build lightweight, extensible, and testable LLM Agents
A collection of original, innovative ideas and algorithms towards Advanced Literate Machinery. This project is maintained by the OCR Team in the Language Technology Lab, Tongyi Lab, Alibaba Group.
A model-agnostic visual debugging tool for machine learning