AI Observability & Evaluation
- llmops
- ai-monitoring
- ai-observability
- llm-eval
Repository Index
47,133 repositories indexed
25–48 of 98 · Most stars
BISHENG is an open LLM devops platform for next generation Enterprise AI applications. Powerful and comprehensive features include: GenAI workflow, RAG, Agent, Unified model management, Evaluation, SFT, Dataset Management, Enterprise-level System Management, Observability and more.
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
- bloom
- deep-learning
- distributed-systems
- language-models
Automate your mobile devices with natural language commands - an LLM agnostic mobile Agent 🤖
the LLM vulnerability scanner
- ai
- llm-evaluation
- llm-security
- security-scanners
Open-source AI brand visibility and competitor reports. Official website: https://niubigeo.ai/ | Paid services: AI testing by real people and GEO optimization. Pricing: https://niubigeo.ai/pricing
World's first open-source enterprise world model.
- agent-memory
- knowledge-base
- knowledge-graph
- llm
JiuwenSwarm is an intelligent AI Agent built on openJiuwen. It extends the powerful capabilities of large language models directly to your fingertips through various communication apps you use daily.
Observability and enforcement for AI agent harnesses. Capture every run and runtime reliability with policy enforcement.
- claude-code
- reliability
- agent-reliability
- hooks
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
- data-drift
- jupyter-notebook
- pandas-dataframe
- machine-learning
AI-Powered Dark Web OSINT Tool
- ai-tool
- darkweb
- darkweb-osint
- investigation-tool
Open Source LLM Gateway 🦚
- openclaw
- cost-tracking
- llm-observability
- token-tracking
🥷 Engineering habits you already know, turned into skills AI assistants like Claude Code, Codex and Cursor can run.
- claude
- claude-code
- design
- engineer
A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation.
PDF GPT allows you to chat with the contents of your PDF file by using GPT capabilities. The most effective open source solution to turn your pdf files in a chatbot!
- pdfgpt
- chatpdf
- chatwithpdf
TranslatedMedicalGPT: Training Your Own Medical GPT Model with ChatGPT Training Pipeline. Training medical large models, implementing incremental pre-training (PT), supervised fine-tuning (SFT), RLHF, DPO, ORPO, and GRPO.
AI-powered bug bounty hunting toolkit that works with or without subscription.
- ai-security
- bug-bounty
- bugcrowd
- claude-ai
Open-Source Agent Observability & Evals for AI Platform teams. Trace, test, route and govern every agent & coding assistant in the company.
Open-source observability for AI agents. Find where your agents fail, dispatch your coding agent to fix it, and verify the fix against real traces.
- llm-observability
- agent-monitoring
- agent-observability
- ai-error-monitor
TranslatedOpen-source AI API relay benchmark, rankings & switcher for Claude Code / Codex: measured availability, latency and price
Fully featured & enhanced replacement for copilot.vim complete with API for interacting with Github Copilot
- copilot
- github-copilot
- lsp-server
- lua
[CVPR2024 Highlight][VideoChatGPT] ChatGPT with video understanding! And many more supported LMs such as miniGPT4, StableLM, and MOSS.
The Swiss Army Knife of Offline AI. Chat, see, speak, and generate images on your phone or Mac — GGUF LLMs, vision, Whisper speech-to-text, Stable Diffusion, tool calling, and local-network servers. Runs on your CPU, GPU, or NPU. No account, no API key, zero data leaves your device.
- privacy-first
- gguf
- llama-cpp
- local-ai
🛡️ BLA (Blind LLM Agents) — Adversarial defense shield that protects binaries, Electron apps, JS bundles, Python, and WebAssembly against AI reverse-engineering agents like REA. Poison -> Exhaust -> Hallucinate.
- adversarial-ai
- claude-code
- cursor
- electron-security