Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
Repository Index
47,133 repositories indexed
1–24 of 27 · Most stars
Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced analytics with timeline and execution graph view
- agentneo
- agents
- ai-performance-optimization
- llm-testing
the LLM vulnerability scanner
- ai
- llm-evaluation
- llm-security
- security-scanners
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
- ai-gateway
- observability
- simulation
- ai-agents
Steering-first neuron explorer for Qwen3.5-4B: push every MLP neuron, keep every rollout, browse the results
- interpretability
- mechanistic-interpretability
- qwen
- steering
Model fingerprint detection with a shared Web and Bun CLI reference bank
Evaluation code for various unsupervised automated metrics for Natural Language Generation.
- natural-language-generation
- natural-language-processing
- nlg
- nlp
GVS5H: Five Qwen3.8-27B Models Match Claude Fable 5 on LiveCodeBench Hard | Fable 5 Level Coding for a Fifth the Price - or on a Single GPU
A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.
FactCircuit: an auditable claim-and-evidence verification loop with immutable provenance, staged checks, bounded retrieval, and reproducible evaluation.
- accuracy-decline
- ai-safety
- audit-trail
- fact-checking
This repository introduces PIXIU, an open-source resource featuring the first financial large language models (LLMs), instruction tuning data, and evaluation benchmarks to holistically assess financial LLMs. Our goal is to continually push forward the open-source development of financial artificial intelligence (AI).
- aifinance
- fintech
- llama
- machine-learning
Build AI phone agents on your own computer and test every change with an AI caller. Open source, runs free with no API keys, cents per call with your own keys. Imports Dialogflow CX.
- agents-as-code
- contact-center
- conversational-ai
- cx-agent-studio
A unified framework for building, evaluating, and recursively improving embodied agents. 🚀 Code release coming soon.
Build AI text detectors whose decisions are mechanistically grounded and human-auditable (EMNLP 2026)
- ai-text-detection
- explainable-ai
- interpretability
- mechanistic-interpretability
Scikit-learn for Recursive Self-improvement (RSI). A modular framework for recursive harness evolution.
- meta-learning
- reinforcement-learning
- rsi
- ai-agents
A lightweight Python SDK and CLI for evaluating recursive self-improvement (RSI) methods across agent benchmarks.
Aixio Layer v2 recovers editable layers from flat designs where possible. 150-design benchmark vs Canva Magic Layers, Qwen-Image-Layer and GPT-6 Astra Ultra.
- ai-design
- benchmark
- image-to-layers
- layer-decomposition
An agent that compiles git maintenance chores into reusable programs, then decides whether to replay them by running checks rather than by guessing the task looks familiar
- agent-architecture
- git-automation
- llm-agents
- program-synthesis
A Unified Framework for Efficient RSI Research
- agent
- recursive-self-improvement
:sunrise: A comprehensive, cutting-edge open-source framework for next-generation Transformers/LLM, with compute-matched benchmarking and a platform for autonomous architecture research.
- auto-research
- benchmark
- deep-learning
- language-models
TavernGym and TavernBench: a faithful, full-content Hearthstone Battlegrounds simulator with an RL interface, and a 960-task decision benchmark. Code and data for "Build, Don't Play".
- game-ai
- hearthstone
- reinforcement-learning
- rust
From a universe of models to Collective Model Intelligence.
- collective-model-intelligence
- model-capabilities
- model-collaboration
Runtime defense against memory poisoning in AI agents (OWASP ASI06). Drop-in for LangChain, OpenAI Agents, AutoGen, CrewAI and mem0.
DecisionPivot: a diagnostic benchmark for decision models in LLM systems. Contrast pairs that differ in one decisive detail, generators with independent solvers, scorer and runners.
- benchmark
- calibration
- decision-making
- evaluation