Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.
- ai
- diagnostic-tool
- agent-evaluation
- ai-alignment
The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
Supercharge Your LLM Application Evaluations 🚀
Gorilla: Training and Evaluating LLMs for Function Calls (Tool Calls)
Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring.
🐢 Open-Source Evaluation & Testing library for LLM Agents
- mlops
- ml-validation
- ml-testing
- ai-testing
Open-source AI red teaming toolkit from Microsoft. Test generative AI systems with automated scans, adaptive multi-turn attacks, and a web UI.
Leaderboard Comparing LLM Performance at Producing Hallucinations when Summarizing Short Documents
Inspect: A framework for large language model evaluations
Machine learning metrics for distributed, scalable PyTorch applications.
AssetOpsBench - Industry 4.0: A unified benchmark and framework for building, orchestrating, and evaluating domain-specific AI agents for Industry 4.0 asset operations and maintenance, with 460+ scenarios, 5 specialist agents (IoT, FMSR, TSFM, Work Order,...), and multi-agent orchestration blueprints (MetaAgent, AgentHive) over MCP.
UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.
拼好RAG:手搓并融合了GraphRAG、LightRAG、Neo4j-llm-graph-builder进行知识图谱构建以及搜索;整合DeepSearch技术实现私域RAG的推理;自制针对GraphRAG的评估框架| Integrate GraphRAG, LightRAG, and Neo4j-llm-graph-builder for knowledge graph construction and search. Combine DeepSearch for private RAG reasoning. Create a custom evaluation framework for GraphRAG.
- deepresearch
- deepsearch
- evaluation
- graphrag
Local typed decisions, contrastive data curation, and model evaluation.
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
Your Automatic Prompt Engineering Assistant for GenAI Applications
- ai
- prompt
- llm
- ai-experiments
:cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI
CORTEX RAG is an enterprise retrieval and knowledge assistant that helps teams find accurate answers from company data with citations, permission-aware retrieval, and fast deployment
BullshitBench measures whether AI models challenge nonsensical prompts instead of confidently answering them, created by Peter Gostev.
Responsible AI Toolbox is a suite of tools providing model and data exploration and assessment user interfaces and libraries that enable a better understanding of AI systems. These interfaces and libraries empower developers and stakeholders of AI systems to develop and monitor AI more responsibly, and take better data-driven actions.
DataComp for Language Models
An LLM-based autonomous agent controlling real-world applications via RESTful APIs
Llama-3 agents that can browse the web by following instructions and talking to you