🪢 Open source AI engineering platform: AI Gateway, agent observability, evals, and prompt/skill management, built on OpenTelemetry. One open platform for every AI call and every agent. Self-host or use Langfuse Cloud.
TranslatedBELLE: Be Everyone's Large Language model Engine (open-source Chinese conversational large language model)
- bloom
- instruction-set
- llama
- open-models
This repo powers my experiment where ChatGPT manages a real-money micro-cap stock portfolio.
Translated非线智能 NoneLinear (Non-linear Intelligence) - ReLE evaluation: capability evaluation of Chinese AI large models (continuously updated): it currently covers 374 large models, including commercial models such as chatgpt, gpt-5.4, Google gemini-3.1-pro, Claude-4.6, 文心 ERNIE-X1.1, ERNIE-5.0, qwen3.6-max, qwen3.6-plus, Baichuan, iFlytek Spark, and SenseTime senseChat, as well as open-source large models such as step3.5-flash, kimi-k2.6, ernie4.5, MiniMax-M2.7, deepseek-v4, Qwen3.6, llama4, Zhipu GLM-5.1, MiMo-V2, LongCat, gemma4, and mistral. It provides not only leaderboards but also a large model defect library with more than 2,000,000 entries! This makes it convenient for the broad community to research, analyze, and improve large models.
🧩 FrontierAgent, our agent framework, open-sourced alongside it — native command-line TUI, ReAct and Agent Team modes, one command on macOS and Linux, no preinstall, no hard Docker dependency.
One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks
Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
- evaluation
- evaluation-framework
- evaluation-metrics
- huggingface
The Fable Workflow: how Claude Fable 5 worked, distilled into skills any model can run, with the eval that keeps it honest. Think / act / prove.
A desktop app to prototype agent ideas, inspect every harness step, replay failures, and evaluate performance, all in one place. Local-first, cloud-ready for managed agents.
- agent
- deer-flow
- develop
- electrobun
An on-premises, OCR-free unstructured data extraction, markdown conversion and benchmarking toolkit. (https://idp-leaderboard.org/)
- document
- document-analysis
- extraction
- llms
👾 nao is the #1 open source agentic BI. (1) Create context with nao-core cli, (2) deploy nao chat interface for everyone
- agentic-analytics
- analytics
- analytics-engineering
- bigquery
Build, Evaluate, and Deploy GUI Agents — online RL training, standardized benchmarks, and real-device deployment in one framework.
A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.
Build AI phone agents on your own computer and test every change with an AI caller. Open source, runs free with no API keys, cents per call with your own keys. Imports Dialogflow CX.
- agents-as-code
- contact-center
- conversational-ai
- cx-agent-studio
TavernGym and TavernBench: a faithful, full-content Hearthstone Battlegrounds simulator with an RL interface, and a 960-task decision benchmark. Code and data for "Build, Don't Play".
LLM-powered Text-to-SQL evaluation toolkit: schema ingestion, retrieval eval, and standalone SQL judge/re-scorer.
DecisionPivot: a diagnostic benchmark for decision models in LLM systems. Contrast pairs that differ in one decisive detail, generators with independent solvers, scorer and runners.
- benchmark
- calibration
- decision-making
- evaluation
Evolved DSH plugin harnesses for five agent benchmarks, with versioned artifacts, evolution curves and Life-Harness comparisons
- agent-benchmark
- dsh
- harness
- leaderboard