Methodology · Lending Desk Bench

The task

The fixture is an equipment-lending app. /members already works. The agent implements the /items slice and fills existing stub files only. The agent must not add files. The agent must not change dependencies. The agent must not rename files. Conventions must match the members slice.

Next.js 16 App Router, React 19, TypeScript strict, Tailwind v4
Fixed clock via now() (no Date.now())
Server Actions for checkout/return, REST route for loans
Phased Measure plan with closeout metadata

Four tracks

Build (the ranking above): fill the stubs of the /items slice.
Bug fix: repair undisclosed defects in the finished app without regressions.
Test writing: write a test suite for the finished app. The grader runs it against the app, against a second correct implementation, and against defective versions. A good suite passes on both correct apps and fails on the defects.
Change request: add four requested behaviors to the finished app, and keep every existing behavior.

Isolation

Each agent runs in a container. The container shows only the task files. Candidate source is writable. Fixture dependencies are read-only during the agent phase. The harness, the reference overlay, the host home directory, and unrelated workspace paths are absent.

No Skills (primary)

Fixture + prompt only

No skills and no extra context files. This gives a direct comparison.

Skills

+ AGENTS.md + 5 skills

The task is the same. Do not mix these scores into the No Skills ranking.

Executor change. The benchmark changed from Pi to OpenCode 2 on 2026-10-25. The reason was to align benchmark traffic with the OpenCode program users run and remove Go traffic from the former Pi executor. OpenCode 2 uses the host Console login through an isolated credential store. The model can read the run credential and send it over the network.

Grading pipeline

Canary: before any model run, the harness grades the reference solution. The batch stops unless it scores 100.
Candidate filesystem contract (writable set + public API surface). A symlink or a special file makes the run invalid. An added file, a protected edit, or a changed export costs adherence: the harness grades the model's writable files on a clean tree.
npm run typecheck and npm run build (gate)
Unit tests, API tests against a live Next server
Playwright browser suite + separate error-boundary boot. A browser that cannot start makes the run invalid, never a low score.
Score: completion (55%), adversarial probes (20%), adherence (15%), quality (10%)

Weights inside completion. Seventeen checks pass on almost every run (pure status logic, the item card, the catalogue filters). They still gate a broken submission through the tiers, but each weighs a quarter of a normal check, so the forms, the mutations, and the probes decide the order.

Adherence. Half the axis is the file contract: each class of violation (an added file, a protected edit, a changed export) costs half of it. The other half is the Measure closeout the task asks for.

Repeats. The build track runs each model three times a week per arm. The published total is the lower median of the week; the worst run is shown beside it. The Skills effect is one paired value: the mean over models of the Skills median minus the No Skills median.

Soft tier gates. Higher tiers always contribute. The harness scales their credit by min(1, lower_rate / 0.9). A previous hard binary rule compressed mid-pack models into a 17-point band. That rule reduced score separation.

Domains

Executable check groups. Each group maps to one product surface.

Advisory (unscored). ADV.useActionState and ADV.useOptimistic detect preferred React 19 form idioms. They never change the total. They show one signal only: use of modern APIs.

Sources behind the harness

The test checks and the quality scoring come from two external sources. The harness pins both sources. Pinned versions keep grades comparable across weeks.

Vercel React Best Practices

Vercel maintains this guide for React and Next.js performance. The guide holds 70 rules across 8 categories. Each rule has a priority. The critical categories are request waterfalls and bundle size. The harness turns the rules that fit this task into executable probes:

Parallel data fetching. Independent reads must use Promise.all. Check C.parallel-reads.
Per-request deduplication. The rule is React.cache() for repeated item reads. Probes P.dedup-item and P.n-plus-one.
Auth on server actions. Server actions need the same auth checks as API routes. Probes F.auth-checkout and F.auth-return.
Streaming with Suspense. The loading shell must stream while data loads. Check H.loading.
Race handling. Two desks that check out one item at the same moment get one loan and one clear error. Probes P.action-race, P.concurrent-checkout, and P.concurrent-return.
Accessibility. Named form controls, announced errors, and valid ARIA. Checks E.labels, E.error-announced, and probe P.axe-clean (pinned axe-core).
Modern React 19 idioms. useActionState and useOptimistic for form state. These probes are advisory. They never change the score.

The Skills arm receives the same guide as a mounted skill (vercel-react-best-practices). It also receives measure, next-best-practices, vercel-composition-patterns, and build-graph. The No Skills arm sees none of them.

react-doctor

react-doctor is a static analyzer for React and Next.js code. The harness pins version 0.9.2. The analyzer reads the candidate source without running it. It emits diagnostics in five categories. Each category has a weight: Security 2.0, Bugs 1.5, Performance 1.0, Accessibility 1.0, Maintainability 0.5. Each severity also has a weight: error 3, warning 1.

The quality axis does not use the raw health score. An empty repository scores near 100, so the raw score rewards writing no code. Instead, the harness counts weighted diagnostics above the reference baseline. The baseline is the reference solution's own diagnostic set. Diagnostics already present in the reference are subtracted. A candidate that adds 30 weighted penalty points gets zero quality credit. Quality is 10% of the total score. Deterministic source signals (explicit any, non-null assertions, leftover console calls, unused imports, repeated blocks) join the analyzer's diagnostics.

The analyzer scans a relocated copy of the candidate source. The candidate tree lives in a git-ignored directory during grading, and the analyzer skips git-ignored files, so an in-place scan reports zero findings. The copy lives outside every repository, so every source file is scanned. In the current week the analyzer finds real warnings in most candidates (missing button types, raw image elements, loading flags not reset in a finally block). The warnings differentiate the quality axis from the completion axis.

Cost & tokens

The harness extracts usage from the OpenCode session export. The published cost is a percent of the monthly OpenCode Go Usage for that model. The harness recomputes the cost from tokens when prices change.

Limits

Scores are publishable only with current provenance and agent isolation. Host load is recorded as metadata.
Host load can affect timing-sensitive browser assertions.
Three repeats a week reduce single-run noise but do not remove it. The 4-week range over weekly medians shows stability. The range is not a confidence interval.
Fixture revision r2 (from week 2026w40) states the export rule and the spec-edit rule plainly. Earlier weeks ran on r1; the harness regraded their work on the r2 tree, so all weeks share one ruler.
The reference score is a ceiling. The reference is not a model.