How the benchmark works

The task

The fixture is an equipment-lending app. /members already works. The agent implements the /items slice and fills existing stub files only. The agent must not add files. The agent must not change dependencies. The agent must not rename files. Conventions must match the members slice.

• Next.js 16 App Router, React 19, TypeScript strict, Tailwind v4
• Fixed clock via now() (no Date.now())
• Server Actions for checkout/return, REST route for loans
• Phased Measure plan with closeout metadata

Isolation

Each agent runs in a container. The container shows only the task files. Candidate source is writable. Fixture dependencies are read-only during the agent phase. The harness, the reference overlay, the host home directory, and unrelated workspace paths are absent.

Grading pipeline

1. Candidate filesystem contract (writable set + public API surface)
2. npm run typecheck and npm run build (gate)
3. Unit tests, API tests against a live Next server
4. Playwright browser suite + separate error-boundary boot
5. Score: soft-tier completion (60%), adversarial probes (20%), react-doctor quality (20%)

Domains

Executable check groups. Each group maps to one product surface.

Sources behind the harness

The test checks and the quality scoring come from two external sources. The harness pins both sources. Pinned versions keep grades comparable across weeks.

Vercel React Best Practices

Vercel maintains this guide for React and Next.js performance. The guide holds 70 rules across 8 categories. Each rule has a priority. The critical categories are request waterfalls and bundle size. The harness turns the rules that fit this task into executable probes:

• Parallel data fetching. Independent reads must use Promise.all. Probe C.parallel-reads and P.no-waterfall.
• Per-request deduplication. The rule is React.cache() for repeated item reads. Probes P.dedup-item and P.n-plus-one.
• Auth on server actions. Server actions need the same auth checks as API routes. Probes F.auth-checkout and F.auth-return.
• Streaming with Suspense. The loading shell must stream while data loads. Probes P.streams-shell and H.loading.
• Modern React 19 idioms. useActionState and useOptimistic for form state. These probes are advisory. They never change the score.

The Skills arm receives the same guide as a mounted skill (vercel-react-best-practices). It also receives measure, next-best-practices, vercel-composition-patterns, and build-graph. The No Skills arm sees none of them.

react-doctor

react-doctor is a static analyzer for React and Next.js code. The harness pins version 0.9.2. The analyzer reads the candidate source without running it. It emits diagnostics in five categories. Each category has a weight: Security 2.0, Bugs 1.5, Performance 1.0, Accessibility 1.0, Maintainability 0.5. Each severity also has a weight: error 3, warning 1.

The quality axis does not use the raw health score. An empty repository scores near 100, so the raw score rewards writing no code. Instead, the harness counts weighted diagnostics above the reference baseline. The baseline is the reference solution's own diagnostic set. Diagnostics already present in the reference are subtracted. A candidate that adds 30 weighted penalty points gets zero quality credit. Quality is 20% of the total score.

The analyzer scans a relocated copy of the candidate source. The candidate tree lives in a git-ignored directory during grading, and the analyzer skips git-ignored files, so an in-place scan reports zero findings. The copy lives outside every repository, so every source file is scanned. In the current week the analyzer finds real warnings in most candidates (missing button types, raw image elements, loading flags not reset in a finally block). The warnings differentiate the quality axis from the completion axis.

Cost & tokens

The harness extracts usage from the agent transcript. The published cost is a percent of the monthly OpenCode Go Usage for that model. The harness recomputes the cost from tokens when prices change.

Limits

• Scores are publishable only with current provenance, agent isolation, and a quiet host.
• Host load can affect timing-sensitive browser assertions.
• One weekly run is a point estimate. The 4-week range shows stability. The range is not a confidence interval.
• The reference score is a ceiling. The reference is not a model.