All projects

Markscheme

A benchmark for how often a model invents a spec the source never stated.

Jul 2026 – PresentPersonal ProjectAI · WebActivePrivate

Overview

  • Tests how reliably language models pull structured specifications out of real AI model-release announcements, scored against a hand-written answer key.
  • Running a model and scoring it are kept apart, so a change to the scoring rules re-scores every past run without paying for new calls.
  • Every prompt is versioned and recorded with each run, so a score can always be traced to the exact prompt that produced it.
  • A response that fails validation is recorded as a failure, never quietly turned into an empty answer, so a crash can't pass for the model rightly declining to guess.

Built with

Next.jsReactTypeScriptZodTailwind CSSGitHubVercelPrompt EngineeringContext EngineeringClaude CodeCodex