Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.
- ai
- diagnostic-tool
- agent-evaluation
- ai-alignment
The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.
A framework for few-shot evaluation of language models.
Easily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!
"ClawWork: OpenClaw as Your AI Coworker - 💰 $15K earned in 11 Hours"
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
Translated非线智能 NoneLinear (Non-linear Intelligence) - ReLE evaluation: capability evaluation of Chinese AI large models (continuously updated): it currently covers 374 large models, including commercial models such as chatgpt, gpt-5.4, Google gemini-3.1-pro, Claude-4.6, 文心 ERNIE-X1.1, ERNIE-5.0, qwen3.6-max, qwen3.6-plus, Baichuan, iFlytek Spark, and SenseTime senseChat, as well as open-source large models such as step3.5-flash, kimi-k2.6, ernie4.5, MiniMax-M2.7, deepseek-v4, Qwen3.6, llama4, Zhipu GLM-5.1, MiMo-V2, LongCat, gemma4, and mistral. It provides not only leaderboards but also a large model defect library with more than 2,000,000 entries! This makes it convenient for the broad community to research, analyze, and improve large models.
The CLI and skills that turn any coding assistant into an expert at creating, evaluating, and deploying AI agents on Google Cloud.
🐢 Open-Source Evaluation & Testing library for LLM Agents
- mlops
- ml-validation
- ml-testing
- ai-testing
Democratizing Reinforcement Learning for LLMs
[ICLR'24 spotlight] An open platform for training, serving, and evaluating large language model for tool learning.
🧩 FrontierAgent, our agent framework, open-sourced alongside it — native command-line TUI, ReAct and Agent Team modes, one command on macOS and Linux, no preinstall, no hard Docker dependency.
Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.
- ai
- chain-of-thought
- collaboration
- dataset-generation
Open-Source Agent Observability & Evals for AI Platform teams. Trace, test, route and govern every agent & coding assistant in the company.
Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning
Open-source AI red teaming toolkit from Microsoft. Test generative AI systems with automated scans, adaptive multi-turn attacks, and a web UI.
Prompt Engineering | Prompt Versioning | Use GPT or other prompt based models to get structured output. Join our discord for Prompt-Engineering, LLMs and other latest research
Evaluation and Tracking for LLM Experiments and AI Agents
Comet: agent skill harness for turning ideas into evaluated workflows
Inspect: A framework for large language model evaluations
[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
🐧 Unified and Stable RSI Platform
Same agent, Smaller bill. The tools and infrastructure to get more work from your AI. Fewer tokens. Less compute. Lower cost.
Open-source tools for prompt testing and experimentation, with support for both LLMs (e.g. OpenAI, LLaMA) and vector databases (e.g. Chroma, Weaviate, LanceDB).