A framework for few-shot evaluation of language models.
- evaluation-framework
- language-model
- transformer
142,273 visible repositories in the index
1–21 of 21 · Most stars
A framework for few-shot evaluation of language models.
This repo powers my experiment where ChatGPT manages a real-money micro-cap stock portfolio.
The open source research environment for AI researchers to seamlessly train, evaluate, and scale models from local hardware to GPU clusters.
Open-source observability for AI agents. Find where your agents fail, dispatch your coding agent to fix it, and verify the fix against real traces.
Prompt Engineering | Prompt Versioning | Use GPT or other prompt based models to get structured output. Join our discord for Prompt-Engineering, LLMs and other latest research
Together Mixture-Of-Agents (MoA) – 65.1% on AlpacaEval with OSS models
BullshitBench measures whether AI models challenge nonsensical prompts instead of confidently answering them, created by Peter Gostev.
The WeightWatcher tool for predicting the accuracy of Deep Neural Networks
A powerful tool for automated LLM fuzzing. It is designed to help developers and security researchers identify and mitigate potential jailbreaks in their LLM APIs.
HOTA (and other) evaluation metrics for Multi-Object Tracking (MOT).
A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.
Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.
A benchmark to evaluate language models on questions I've previously asked them to solve.
AgentSims is an easy-to-use infrastructure for researchers from all disciplines to test the specific capacities they are interested in.
Aixio Layer v2 recovers editable layers from flat designs where possible. 150-design benchmark vs Canva Magic Layers, Qwen-Image-Layer and GPT-6 Astra Ultra.
An agent that compiles git maintenance chores into reusable programs, then decides whether to replay them by running checks rather than by guessing the task looks familiar
Open-source whiteboard AI tutor with animated lessons and interactive playback. Compare Claude, Gemini & DeepSeek across 288 generation runs and 64 blind-reviewed boards.
Feather agent harness, training integrations, and evaluation tools for knowledge work
Enterprise-grade benchmark evaluating whether AI agents deliver verifiable business outcomes in real business systems — general domain, finance, supply chain, and manufacturing.
Open-source, local-first observability and evals for AI agents: see what Claude Code, Codex, Cursor and others did and what it cost.
See inside any GGUF: where two quantizations part ways, token by token, and what each sampler cuts. Live in your browser, powered by llama.cpp (CUDA or Vulkan).