AI Observability & Evaluation
- llmops
- ai-monitoring
- ai-observability
- llm-eval
Repository Index
47,133 repositories indexed
25–48 of 91 · Most stars
BISHENG is an open LLM devops platform for next generation Enterprise AI applications. Powerful and comprehensive features include: GenAI workflow, RAG, Agent, Unified model management, Evaluation, SFT, Dataset Management, Enterprise-level System Management, Observability and more.
Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring.
the LLM vulnerability scanner
- ai
- llm-evaluation
- llm-security
- security-scanners
My Python scripts to make high-quality figures for publications in top AI conferences and journals.
- figures
- python
- icml
- nature-machine-intelligence
Open-source AI brand visibility and competitor reports. Official website: https://niubigeo.ai/ | Paid services: AI testing by real people and GEO optimization. Pricing: https://niubigeo.ai/pricing
JiuwenSwarm is an intelligent AI Agent built on openJiuwen. It extends the powerful capabilities of large language models directly to your fingertips through various communication apps you use daily.
Observability and enforcement for AI agent harnesses. Capture every run and runtime reliability with policy enforcement.
- claude-code
- reliability
- agent-reliability
- hooks
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
- data-drift
- jupyter-notebook
- pandas-dataframe
- machine-learning
Open Source LLM Gateway 🦚
- openclaw
- cost-tracking
- llm-observability
- token-tracking
A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation.
PDF GPT allows you to chat with the contents of your PDF file by using GPT capabilities. The most effective open source solution to turn your pdf files in a chatbot!
- pdfgpt
- chatpdf
- chatwithpdf
AI-powered bug bounty hunting toolkit that works with or without subscription.
- ai-security
- bug-bounty
- bugcrowd
- claude-ai
Open-Source Agent Observability & Evals for AI Platform teams. Trace, test, route and govern every agent & coding assistant in the company.
Open-source observability for AI agents. Find where your agents fail, dispatch your coding agent to fix it, and verify the fix against real traces.
- llm-observability
- agent-monitoring
- agent-observability
- ai-error-monitor
Switch between multiple Claude Code accounts, with automatic rate-limit rotation, usage dashboard, and parallel sessions
- claude
- claude-code
- account-manager
- account-switcher
TranslatedOpen-source AI API relay benchmark, rankings & switcher for Claude Code / Codex: measured availability, latency and price
A curated, battle-tested AI-coding toolkit: Claude Code plugins, subagent orchestration, quality gates, and ready-to-copy prompts, extracted from real production use.
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
- ai-gateway
- observability
- simulation
- ai-agents
Steering-first neuron explorer for Qwen3.5-4B: push every MLP neuron, keep every rollout, browse the results
- interpretability
- mechanistic-interpretability
- qwen
- steering
🦎 A lightweight, streaming-native, multi-protocol gateway for AI, APIs, and real-time data. Route LLM requests across providers, govern MCP tools, and connect Kafka, MQTT, and HTTP through one high-performance engine with shared security and observability.
Open decision models from 0.8B to 27B: a calibrated probability for every option in one pass, and a short thought only when unsure. Engine for vLLM, llama.cpp and MLX.
- agents
- calibration
- classification
- decision-making
Turn any LLM into a Jev-style decision model: typed decisions, real probabilities, no training. (continue updating, welcome any issue and PR request)
- calibration
- decision-model
- jev
- jev-model
A Utility CLI for AI Coding Agents