AI Observability & Evaluation
- llmops
- ai-monitoring
- ai-observability
- llm-eval
108,094 visible repositories in the index
1–13 of 13 · Most stars
AI Observability & Evaluation
Open Source LLM Gateway 🦚
Self-hosted TypeScript agent runtime with durable approvals and verifiable run records. Own it, approve it, audit it.
Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI
⚡️ Open-source AI Gateway — Use any SDK to call 100+ LLMs. Built-in failover, load balancing, cost control & end-to-end tracing.
The missing DevTools for Claude Code — inspect session logs, tool calls, token usage, subagents, and context window in a visual UI. Free, open source.
Local gateway for Claude Code, Codex and other AI clients on macOS, Windows and Linux: switch upstreams without touching clients, keep API keys from relays, cut off malicious tool calls, trace every request.
OrcaReplay — Time travel for AI agents. Record, replay, fork, and debug any agent run with any model. Built by the OrcaRouter.ai team.
Track AI API quotas across Synthetic, Z.ai, Anthropic (Claude Code), Codex, GitHub Copilot & Antigravity in real time. Lightweight background daemon (<50MB RAM), SQLite storage, Material Design 3 dashboard. Zero telemetry.
Ultra-fast token & cost tracker for LLM Token Usage (e.g. Claude Code)
Claude Code dashboard: see where your limits are headed before you hit them. Calibrated usage forecasts, live rate-limit gauges, session costs, memory browser. Single binary, zero config, plus a VS Code extension.
Native macOS app for tracking Claude Code usage — tokens, cost, rate-limit pacing, per-project breakdowns. SwiftUI + SwiftData.
Publish Claude Code usage telemetry to MQTT for Home Assistant (Clawdmeter-compatible mood/burn-rate sensors)