Agentic development evaluation taxonomy: v0

Research cut and estate inspection: 2026-09-23. Status: proposed vocabulary freeze; no runner, enforcement change, or new measurement. Parent: #299. Scope: #300. Companion authority contract: #304.

Scope and vocabulary

INFER: Freeze five observation layers: unit, tool-use, trajectory, scenario, and online/prod. These are not a maturity ladder or increasing levels of authority. Tool-use makes an implicit seam in the earlier four-layer draft explicit. Prod-like-local is an environment qualifier, not a sixth layer.

FACT means directly supported by an identified source. INFER marks this estate’s proposed synthesis. OVERSOLD marks a conclusion beyond the evidence. A classification example is not an executed experiment.

Only an applicable, repo-owned, contract-valid deterministic oracle may acquire automated merge authority. LLM semantic judgments, review-bot verdicts, vendor scores, telemetry, and control-plane index coverage may not. Eligibility is not enrollment: nothing in this file makes a check required or changes existing tier/review rules. See the existing oracle practice and companion #304.

Five-layer taxonomy

INFER: Classify checks, not entire tools. Conditional eligibility below requires a valid contract, controlled/reproducible execution, revision binding, and explicit repository adoption. The online/prod row has no direct v0 merge authority.

Mixed and overlapping cases

A smoke suite can contain tool-use and scenario checks. A scenario can include a trajectory and several tool calls. Split a mixed report’s hard assertion results from its semantic scores; neither inherits authority from the other.

Online/prod is the cross-cutting live-observation layer, not a claim that a unit test becomes a new kind of unit test in production. A live permission failure can be described as online/prod evidence at a tool-use boundary. State the primary question and secondary boundary rather than forcing one exclusive label.

An exact comparator over a newly sampled model run does not make the whole check reproducible. Such a capability trial stays advisory in v0. A fixed patch generated by an agent can still be tested by the repository’s ordinary deterministic suite. A recorded model response used as a fixture tests the replayed software boundary, not the live model’s reliability.

Environment qualifiers

INFER: Environment is a separate qualifier for evidence, not an authority grant.

A local laptop doing real work on real repositories is production for that agent loop. A disposable local sandbox is not production merely because the same binary runs there. A canary supplements controlled tests rather than proving the laboratory represents every live condition; see R5. Separate rollout/incident policies are outside this merge-evaluation freeze.

Estate map and maximum claims

Public-safe naming (2026-10-04). This repository is public. The map names public surfaces by path or link and private ones by role only (a private product repository, a private control plane, a local swarm). Their identities and pinned revisions live in the adopter’s private evidence record. A role row claims no more than a linked row: inspected source, not results.

FACT: The following source inventory was inspected at agent-harness b586bbf10629b50bcdb08d8b5f99bc60b06795ec. INFER: Its layer assignments and maximum-claim descriptions apply only to the named boundaries. Source inspection does not prove a check ran or that GitHub branch protection requires it.

Proposed and unqualified surfaces

FACT, 2026-09-23 inspection: Open PRs #295, #296, and #298 describe reviewer-packet diagnostics, logical-job accounting, and mapped authority-document drift. They are not dependencies of this document or evidence of shipped capability. Their deterministic diagnostic reports do not acquire product merge authority.

INFER: Local swarm, OpenCode, Claude Code and Codex runs use the same vocabulary, but no native run was qualified here. Proposed examples include an approval-before-write sequence (trajectory), an adapter denial (tool-use), a complete issue-resolution task (scenario), and a live failed/retried job (online/prod). Do not invent captured spans, tool support, task success, or measured coverage.

Control-plane indexing without authority

FACT: A private control-plane evidence-index issue, created 2026-09-22, specifies descriptive coverage with authority: "none" and gate_eligible: false. It prohibits treating completeness as correctness, approval, or promotion.

INFER: An eval index may describe subject/revision, producer, source result, layer, environment, metric denominator, synthetic/recorded/live status, availability, and limitations. These are conceptual indexing needs, not additions to the control plane’s current schemas. Keep payloads and sensitive source data at their owner.

A producer verdict remains source data. Even complete index coverage remains gate_eligible: false; metadata completeness is not evidence of semantic success.

Reliability and confidence limits

pass@k and pass^k

FACT: R1 defines pass@k as the chance that at least one of k independent, identically distributed trials succeeds. R2 defines pass^k as the chance that all k such trials succeed, averaged across tasks. They answer different questions: search yield versus consistent success.

INFER: Report pass@1 beside either aggregate. Retain task population, per-task attempts, successes, retry/selection protocol, model/tool/check versions, environment, costs, missing outcomes, and uncertainty. Never discard failed attempts to make the denominator more favorable.

For one task with true success probability p under those assumptions, the corresponding probabilities are 1-(1-p)^k and p^k; these are not instructions to substitute an empirical pooled success rate. For n >= k trials with c successes, the papers give per-task estimators 1-C(n-c,k)/C(n,k) and C(c,k)/C(n,k), respectively, then average across tasks. C(a,k) is the binomial coefficient, zero when a < k.

Do not raise a pooled success rate across heterogeneous tasks to k and call it measured pass^k. Correlated trials, adaptive retries, and missing outcomes need an explicit protocol; the formulas alone do not validate those assumptions.

Neither aggregate gates an individual PR. Repetition can investigate suspected flakiness, but identical deterministic fixture reruns are not independent new evidence of agent capability. Do not freeze a universal k or invent precision from a tiny sample.

Golden journeys and fixture oracles

INFER: A golden journey is a curated scenario, not privileged truth. It helps when initial state is pinned, the actual boundary is exercised, and relevant durable effects and forbidden side effects are observed. A model-assisted journey does not inherit deterministic eligibility from an exact final-state comparator.

Use a known-good control, candidate, and named-broken/negative control where appropriate. A control must fail for the intended reason after reaching the boundary; import errors, dead services, and missing credentials are not proof of defect sensitivity. Record unsafe or infeasible historical reproduction and its weaker substitute. See existing control practice.

Fixtures help with parser, policy, serialization, migration, and historical regressions. They fake confidence when assertions repeat implementation logic, a no-op still passes, persistence is unobserved, or synthetic coverage is presented as production quality. A schema/file-existence assertion can prove its narrow contract; it cannot prove the artifact’s semantic truth.

LLM judges and LGTM theater under PR flood

FACT: R4 studies useful judge agreement alongside systematic limitations; its conversational evaluation is not proof of coding correctness. INFER: Apply the following failure analysis inside the existing review budget.

LGTM theater means review-shaped activity without an independently falsifiable correctness claim. Comment count, agreement, and bot silence are not acceptance. A confirmed defect can inform existing severity/review handling; the bot verdict itself cannot veto or authorize a merge.

Benchmark marketing boundary

FACT: SWE-bench evaluates issue resolution in repository snapshots [R3]. The 2026-02-23 first-party retrospective [R6] reports test-design and contamination problems in a selected audit. It is not an independent audit of all benchmarks.

OVERSOLD: The interpretations below are excluded, not attributed as quotations to every benchmark author. INFER: Borrow protocols without borrowing authority.

Ownership, gaps, and next slices

FACT: The earlier E1 comment identifies production trajectory observability as the gap beyond the oracle/flood scavenge. INFER: This freeze addresses classification, not that instrumentation.

The machine-local agent-eval-scavenge-2026-09-23 and agent-eval-obs-2026-09-23 handoff packs were not directly inspected in this connector-based slice. The gap comparison above is issue-reported, not a file diff. A local maintainer should reconcile ORACLES/REVIEW_BOTS/PATTERNS and the handoff drafts before closing #300; record any disagreement in that issue. No new child issue is justified merely by exposing tool-use or an environment qualifier.

Sources and inspection record

Estate sources

The source links above are bounded by the inspected harness revision b586bbf10629b50bcdb08d8b5f99bc60b06795ec; relative links follow the checkout being read. Pinned source snapshot. The private product workflow was inspected at a pinned commit recorded privately; job calls are source facts, not test results. Issue contents are mutable and were inspected on 2026-09-23.

BENCHMARKS labels its snapshot 2026-07-31. No result there was rerun or promoted to a current baseline. The completed 2026-09-23 eval-taxonomy research and approved two-document handoff informed this synthesis; that does not resolve the separate older brief’s B0 provenance item in docs/review-evidence/SOURCES.md.

Primary sources

Accessed 2026-09-23. Publication dates and supported claims are separate from this document’s policy recommendations. No vendor performance score establishes a gate.