Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
Repository Index
142,273 visible repositories in the index
1–24 of 84 · Most stars
Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced analytics with timeline and execution graph view
- agentneo
- agents
- ai-performance-optimization
- llm-testing
A framework for few-shot evaluation of language models.
- evaluation-framework
- language-model
- transformer
Generates original ARC-AGI-1-style tasks distribution-matched to the public eval set.
Agent OS: the agent gets smarter on its own. We just hold the line: Interview-gated, staged evaluation, budgeted evolution loop. MCP server, 14 runtimes: Claude Code, Codex CLI, Gemini CLI, OpenCode, Copilot, Kiro and more.
- ai-agent
- agent-os
- llm-orchestration
- agentic-ai
Democratizing Reinforcement Learning for LLMs
- agent-framework
- agentic-workflow
- distributed-training
- llm-training
Open-Source Agent Observability & Evals for AI Platform teams. Trace, test, route and govern every agent & coding assistant in the company.
Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning
Open-source AI red teaming toolkit from Microsoft. Test generative AI systems with automated scans, adaptive multi-turn attacks, and a web UI.
- ai-red-team
- generative-ai
- red-team-tools
- responsible-ai
Prompt Engineering | Prompt Versioning | Use GPT or other prompt based models to get structured output. Join our discord for Prompt-Engineering, LLMs and other latest research
- chatgpt
- chatgpt-api
- gpt-3
- gpt-3-prompts
The Ruby AI framework.
Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks
- gpt-4v
- large-language-models
- llava
- multi-modal
LLM-as-a-Verifier is a general-purpose framework that provides fine-grained feedback for any agent without requiring additional training. It achieves SOTA performance across coding, robotics, and medical agentic benchmarks.
[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
- agent
- artificial-intelligence
- benchmark
- multimodal
Same agent, Smaller bill. The tools and infrastructure to get more work from your AI. Fewer tokens. Less compute. Lower cost.
- observability
- llmops
- ai-observability
- openai
LLMRouter: An Open-Source Library for LLM Routing
Together Mixture-Of-Agents (MoA) – 65.1% on AlpacaEval with OSS models
mean Average Precision - This code evaluates the performance of your neural net for object recognition.
- machine-learning
- pascal-voc
- average-precision
- ground-truth
Formal verification for AI: a Lean 4 + Dafny proof checker that catches fake AI proofs (reward hacking), and verified recursive self-improvement (RSI) for coding agents. AI for math, physics and software engineering.
- proof-assistant
- lean4
- ai-agents
- ai-for-math
Machine learning metrics for distributed, scalable PyTorch applications.
- python
- data-science
- machine-learning
- pytorch
An open-source Harness Engineering platform for coding agents—define harnesses as code, run controlled experiments, inspect evidence, and compare outcomes. Turn task evidence into actionable team and organization insights.
- agent-plugin
- claude-code
- codex
- cursor
Automatic Video Generation from Scientific Papers
Build tested agent skills and govern their lifecycle through a user-defined marketplace: evidence, discovery, updates, rollback, quarantine, and 17-platform distribution.
UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.
- machine-learning
- experimentation
- llm-prompting
- llm-test