🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
Repository Index
47,133 repositories indexed
1–24 of 93 · Most stars
Open-source LLM knowledge platform: turn raw documents into a queryable RAG, an autonomous reasoning agent, and a self-maintaining Wiki.
Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
- apple
- cua
- lume
- macos
The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.
- machine-learning
- ai
- ml
- mlflow
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
🚀 A curated collection of AI engineering resources, roadmaps, tools, and projects covering LLMs, RAG, AI Agents, MCP, fine-tuning, evaluations, and production AI systems.
- agentic-ai
- ai-agents
- ai-engineering
- ai-projects
The LLM Evaluation Framework
- evaluation-metrics
- evaluation-framework
- llm-evaluation
- llm-evaluation-framework
High performance and SEO friendly lazy loader for images (responsive and normal), iframes and more, that detects any visibility changes triggered through user interaction, CSS or JavaScript without configuration.
- javascript
- lazyload
- lazy-evaluation
- lazysizes
Supercharge Your LLM Application Evaluations 🚀
- llm
- llmops
- evaluation
Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced analytics with timeline and execution graph view
- agentneo
- agents
- ai-performance-optimization
- llm-testing
A framework for few-shot evaluation of language models.
- evaluation-framework
- language-model
- transformer
AI Observability & Evaluation
- llmops
- ai-monitoring
- ai-observability
- llm-eval
BISHENG is an open LLM devops platform for next generation Enterprise AI applications. Powerful and comprehensive features include: GenAI workflow, RAG, Agent, Unified model management, Evaluation, SFT, Dataset Management, Enterprise-level System Management, Observability and more.
:metal: awesome-semantic-segmentation
- semantic-segmentation
- benchmark
- evaluation
- deeplearning
the LLM vulnerability scanner
- ai
- llm-evaluation
- llm-security
- security-scanners
Expression language and expression evaluation for Go
- configuration-language
- expression
- evaluator
- go
Open-Source Agent Observability & Evals for AI Platform teams. Trace, test, route and govern every agent & coding assistant in the company.
Python package for the evaluation of odometry and SLAM
- slam
- odometry
- evaluation
- metrics
TranslatedOpen-source AI API relay benchmark, rankings & switcher for Claude Code / Codex: measured availability, latency and price
Fast, portable, non-Turing complete expression evaluation with gradual typing (Go)
- golang
- go
- cel
- expression
Klipse is a JavaScript plugin for embedding interactive code snippets in tech blogs.
- clojure
- clojurescript
- interactive-snippets
- javascript
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
- ai-gateway
- observability
- simulation
- ai-agents
Interactive evaluation for Neovim (Clojure, Fennel, Scheme, Python, JavaScript, PHP, R, Lua, Rust and more!)
- clojure
- clojurescript
- clojure-repl
- neovim