Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
Supercharge Your LLM Application Evaluations 🚀
"ClawWork: OpenClaw as Your AI Coworker - 💰 $15K earned in 11 Hours"
AgentPilot: Make Your Agents Move Better and Faster
A visual playground for agentic workflows: Iterate over your agents 10x faster
Open-source observability for AI agents. Find where your agents fail, dispatch your coding agent to fix it, and verify the fix against real traces.
Sky-T1: Train your own O1 preview model within $450
Same agent, Smaller bill. The tools and infrastructure to get more work from your AI. Fewer tokens. Less compute. Lower cost.
mean Average Precision - This code evaluates the performance of your neural net for object recognition.
Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
- evaluation
- evaluation-framework
- evaluation-metrics
- huggingface
A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.
Your Automatic Prompt Engineering Assistant for GenAI Applications
- ai
- prompt
- llm
- ai-experiments
👾 nao is the #1 open source agentic BI. (1) Create context with nao-core cli, (2) deploy nao chat interface for everyone
- agentic-analytics
- analytics
- analytics-engineering
- bigquery
Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs
This repo contains the source code for RULER: What’s the Real Context Size of Your Long-Context Language Models?
Akto is the fastest growing AI Security platform for your teams to secure AI agents, MCPs, LLMs, Agent skills, Gen AI apps in your organization.
- ai-security
- ai-security-governance
- ai-security-prompt-injection-llm-security
- ai-security-testing
Break your AI before they do. Join Discord: https://discord.gg/n22d9s3xM
- agents
- ai-security
- ai-security-testing
- aisecurity
Stop your AI from making things up — it proposes, deterministic tools decide, every claim checked against ground truth with evidence. Grounded facts and context survive resets. Reverse engineering is the proving ground. MCP server + CLI.
Readymade evaluators for your LLM apps
Build AI phone agents on your own computer and test every change with an AI caller. Open source, runs free with no API keys, cents per call with your own keys. Imports Dialogflow CX.
- agents-as-code
- contact-center
- conversational-ai
- cx-agent-studio
:dog: The open-source Gauge CLI. Hillclimb your Skills, CLI, SDK, MCP, API, or Docs to make something agents want.
Catch your AI agents when they lie about what they shipped — verifies claims against git instead of believing the agent.
Framework-agnostic evaluation harness for Go — test your MCP servers and AI agents with scored, CI-ready checks.