Supercharge Your LLM Application Evaluations 🚀
- llm
- llmops
- evaluation
Repository Index
77,353 visible repositories in the index
25–48 of 180 · Most stars
Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced analytics with timeline and execution graph view
- agentneo
- agents
- ai-performance-optimization
- llm-testing
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
- deep-learning
- document-parsing
- machine-learning
- nlp
A powerful tool for creating datasets for LLM fine-tuning 、RAG and Eval
- dataset
- javascript
- llm
- fine-tuning
Unified framework for building enterprise RAG pipelines with small, specialized models
- parsing
- retrieval-augmented-generation
- agents
- generative-ai-tools
A framework for few-shot evaluation of language models.
- evaluation-framework
- language-model
- transformer
[MLsys2026 Best Paper]: https://arxiv.org/abs/2506.08276. RAG on Everything with LEANN. Enjoy 97% storage savings while running a fast, accurate, and 100% private RAG application on your personal device.
💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows
- python
- search
- nlp
- semantic-search
Gorilla: Training and Evaluating LLMs for Function Calls (Tool Calls)
🏄 Scalable embedding, reasoning, ranking for images and sentences with CLIP
- bert
- sentence-encoding
- deep-learning
- clip-model
Retrieval and Retrieval-augmented LLMs
- embeddings
- information-retrieval
- llm
- sentence-embeddings
BISHENG is an open-source enterprise Agent platform for building, deploying, and managing AI applications. It combines AI agents, workflows, RAG and knowledge management, desktop agent integration, and vibecoding app hosting with unified model management, evaluation, observability, access control, and enterprise administration.
AI Observability & Evaluation
- llmops
- ai-monitoring
- ai-observability
- llm-eval
Generates original ARC-AGI-1-style tasks distribution-matched to the public eval set.
the LLM vulnerability scanner
- ai
- llm-evaluation
- llm-security
- security-scanners
~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
- academia
- arxiv
- brave
- deep-research
Easily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!
KAG is a logical form-guided reasoning and retrieval framework based on OpenSPG engine and LLMs. It is used to build logical reasoning and factual Q&A solutions for professional domain knowledge bases. It can effectively overcome the shortcomings of the traditional RAG vector similarity calculation model.
- knowledge-graph
- large-language-model
- logical-reasoning
- multi-hop-question-answering
Observability and enforcement for AI agent harnesses. Capture every run and runtime reliability with policy enforcement.
- claude-code
- reliability
- agent-reliability
- hooks
TranslatedBELLE: Be Everyone's Large Language model Engine (open-source Chinese conversational large language model)
- bloom
- instruction-set
- llama
- open-models
SoTA production-ready AI retrieval system. Agentic Retrieval-Augmented Generation (RAG) with a RESTful API.
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
- data-drift
- jupyter-notebook
- pandas-dataframe
- machine-learning
OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.
- evaluation
- benchmark
- large-language-model
- llm
A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation.