Outcome ledger

Status: implemented, experimental, 2026-09-27. Parent: #299 (E1 taxonomy: this is the development outcome dataset). Tool: scripts/outcome_ledger.py.

The local swarm produces a JSON receipt per job and a coordinator item table per lane, but neither answers “which recipe finds real defects”, “which effort level wastes turns” or “did the PR stick”. The coordinator also prunes decided items after 30 days and keeps only its last 200 turns, so its own history is lossy. The ledger is the durable, normalised record the swarm’s learning (a Thompson-sampling posterior keyed repo x recipe, with runtime and model only as per-model sub-counts in its score, later GEPA prompt optimisation) is judged against. It lives outside both producers so neither can grade itself.

Contract

• Read-only over the swarm. Inputs are <root>/<lane>/state/waves/<NNN>/run/<repo>--<entry>/result.json and <root>/<lane>/coordinator/state.json. extract refuses an --out inside the runs root.
• A data contract, not an import. Nothing from the private control plane is loaded (AH-9). The one producer rule restated here is the coordinator’s item id, "f-" + sha1(repo|file|claim[:200])[:10], pinned by a test against a live item. If the producer changes it, the summary’s unjoined_items rises instead of the join silently failing.
• Deterministic. The same inputs and --observed-at give byte-identical JSONL, sorted by record type then id.
• Private output. The ledger carries claim and evidence text from private repositories. Keep it outside every repository (suggested: %USERPROFILE%\.estate\outcome-ledger\); never commit it.
• Labels are judgments, not proof. A verdict is the coordinator judge’s triage decision. Every finding keeps judge and judge_verified; precision is “judge-agreed precision” until the reviewer-calibration corpus exists.

Records (outcome-ledger/v1)

Every record has provenance {producer, source, source_sha256, observed_at}. With --prior, a record the runs root no longer holds is carried forward with carried: true. A finding whose receipt survives but whose coordinator item was pruned keeps the prior ledger’s richer label (a decided item beats a turn verdict, which beats a pending item, which beats none) and its PR fields, marked labels_carried: true. A live item-table observation is never overridden, and PR states are re-joined after the merge so carried PR URLs are observed too. A record whose content changed names the prior content digest in supersedes. Reusing --prior on every run is what makes the ledger outlive the coordinator’s 30-day item pruning and 200-turn window.

Verdicts from coordinator status: new/classified -> pending, fixing -> confirmed, dropped -> refuted, deferred -> deferred; any other status is kept in status_raw and labelled unknown, never guessed. An item already pruned from the table recovers its verdict from the retained turn outcomes (status_raw: pruned) and still joins its worktree and PR. Findings from lanes with no coordinator are unjudged. A receipt or state file that is unreadable, or whose coordinator overlay fails, is listed under problems and the lane falls back to receipt-only findings; wrong-typed fields degrade to nulls rather than aborting the run.

fingerprint is autonomy-v2 C8’s identity: sha256 of repo | recipe | normalised path | line bucket (int(line) // 40, or - when the line is not a number) | first 12 normalised claim words, truncated to 12 hex. The coordinator computes the same value (its fingerprint()); the ledger restates it as a data contract and the tests pin the coordinator’s vectors. cluster is the same without the recipe. Each finding names its key_rule; a carried record keyed before that rule is re-keyed from its stored fields and names the digest it supersedes, while a current-rule record keeps its key. The bucket was line // 20 before 2026-09-29, so the split was redrawn then; the B-015 figures below predate it. MESH’s claim key (no recipe, full hash) is a different identity and does not join on fingerprint.

Sealed hold-out

split is a salted hash of cluster, 20% hold-out. Keying on the recipe-free cluster keeps a finding re-reported by another recipe with the same path, line bucket and first twelve normalised claim words on one side. It does not catch the same defect described in different words or in another 40-line window; that leakage is bounded, not excluded. metrics reads only dev unless --split holdout|all comes with --unseal REASON, which is echoed in the output. Every metrics run prints a holdout_manifest (cluster count and digest of the hold-out cluster set, no labels), so a tuning run can show which hold-out it never read. The set grows as the swarm finds more; compare digests only between runs over the same ledger.

Metrics

• Precision = confirmed / (confirmed + refuted), with beta = [1 + confirmed, 1 + refuted] for a Thompson sampler, by recipe, by repo x recipe x runtime x effort, and by judge.
• Coordinated fraction = findings that reached a coordinator (any status) / all findings; decided fraction = confirmed + refuted / all findings. The precision_by_* groups cover coordinated findings and show pending/deferred counts beside the decided ones.
• Closure rate = terminal findings / coordinated findings. Terminal: refuted, or confirmed with its PR merged or closed unmerged.
• Matured merge: reverted: false observed at least 7 days after the merge (each PR entry records its own observed_at). Merged less than 7 days before --as-of is merged_maturing; anything else unproven is merged_revert_unchecked, never matured.
• Rediscovery: repeat sightings, and sightings of a refuted finding in jobs that started after the verdict (the waste C8 verdict memory removes).
• Jobs by mode x recipe x runtime x effort: failure rate, elapsed p50/p90, findings per job, top terminal reasons. Turns by kind x runtime: not-ok, timeouts, seconds p50.
• Not measured, stated in the output: avoidable idle (needs supervisor idle intervals) and seeded-fault review recall (needs a seeded corpus).

Learning metrics

Add repeatable --learning PATH files or recursive directories to metrics, with or without --ledger. Records live under %USERPROFILE%\.estate\learning\, never in a repository. The five learning contracts are validated; invalid records and lifecycle fold errors appear in learning.problems. Experiences fold through the contract’s fold_experiences (latest observed_at; a later observation that changes the source or split key is a problem, not applied); memory-use observations keep the latest observed_at. A candidate loaded twice counts once. The default is sealed dev; --split holdout|all still requires --unseal REASON. The learning hold-out manifest counts experiences and hashes sorted IDs joined by newlines. Counts cover loaded records after re-observation folding.

py -3 scripts\outcome_ledger.py metrics --learning "$env:USERPROFILE\.estate\learning"

Task outcomes: memory_assisted_task_delta compares success with supplied memory against success without it. This is observational, confounded by task selection, and not a causal estimate. Success means completed, published or merged, without a matured revert or an explicit regression. memory_bytes_per_successful_task divides supplied bytes by successful tasks with a memory-use record, reporting unknown bytes separately.

Memory quality: memory_harm_rate counts helped, harmed and neutral effects judged by an evaluator other than self. retrieval_waste counts supplied items with measured read or cited signals that were neither read nor cited. Unmeasured items do not enter its denominator.

Reuse: lesson_reuse counts semantic and consolidation candidates that reached active and landed at a memory ref, and subsequent experiences supplied that ref. skill_reuse_success counts measured skill completions, overall and for promoted skill refs. cross_runtime_transfer splits lesson uses by whether the runtime appears in the lesson’s loaded evidence experiences. Refs compare without their @version suffix. Memory-use records take precedence over the experience’s memory_used fallback, including when no memory was supplied.

Learning lifecycle: candidate_to_promoted_ratio counts candidates that ever reached active, grouped by kind and current folded state. promotion_to_revert_rate counts subsequent reverts, with pre-promotion canary/probation reverts separately. time_to_learn measures hours from the earliest evidence experience to activation, using the ledger’s quantile rule; promoted candidates with any unloaded evidence are counted as missing. These three candidate metrics are not split, because candidates are not experiences. Only the applied lifecycle chain contributes activation, revert and landed refs. --as-of does not replace observed learning outcomes.

Recurrence: owner_correction_recurrence groups keyed owner corrections, and same_failure_recurrence groups failure keys, ordered by experience time. Every occurrence after the first is a recurrence, attributed to that recurring experience’s runtime. Every metric reports its denominator or population count; rates with an empty denominator are null.

PR state and reverts

fetch-pr-states is the one networked step: GitHub REST through gh api, one pull read per PR and one search per merged PR, paced under the search rate limit. A revert is recognised only by GitHub’s own revert-PR body (“Reverts owner/repo#N”, searched in:body) on a merged PR. A failed probe or a non-object response leaves the state null with the error. extract --pr-states joins that file on PR URL. Replace <swarm-runs-root> with the local swarm’s runs directory (its real path is machine-specific and not published).

$L = "$env:USERPROFILE\.estate\outcome-ledger"
py -3 scripts\outcome_ledger.py extract --runs-root "<swarm-runs-root>" --out "$L\ledger.jsonl" --prior "$L\ledger.jsonl"
py -3 scripts\outcome_ledger.py fetch-pr-states --ledger "$L\ledger.jsonl" --out "$L\pr-states.json"
py -3 scripts\outcome_ledger.py extract --runs-root "<swarm-runs-root>" --out "$L\ledger.jsonl" --prior "$L\ledger.jsonl" --pr-states "$L\pr-states.json"
py -3 scripts\outcome_ledger.py metrics --ledger "$L\ledger.jsonl"

First baseline (B-015, 2026-09-27)

Measured over the owner’s live runs root on 2026-09-27 at about 18:00Z; the ledger’s provenance label is --observed-at 2026-09-27T18:15:00Z, and PR states were fetched in the same run. The ledger stays private; its sha256 was adb1b90fb1ed045e107dbf5f3521034ae3b7c4dca33704069b3ea389e69b3da1. Numbers are over all splits (--unseal "B-015 baseline"), because this is the baseline record, not a tuning run.

• 1,148 jobs, 18 coordinator turns across 3 coordinated lanes, 3,417 findings. Every coordinator item joined to its receipt (unjoined_items 0).
• 88.6% of findings never reached a coordinator (3,029 in lanes without one). Coordinated fraction 11.4%; decided fraction 2.0% (67 findings), with 319 still pending triage.
• Judge-agreed precision 0.343 (23 confirmed / 44 refuted). By recipe: bug-hunt 0.818 (9/11), test-gaps 0.333 (12/36), review-range 0.25 (1/4), doc-drift 0.062 (1/16).
• By judge: local-swarm 0.526, codex 0.286, grok 0.25. Confounded by which lanes each judged, so it is a calibration question, not a ranking.
• Job failure rate by effort: high 1.6% (16/1,013), xhigh 3.6% (3/84), max 21.2% (7/33), mostly stream idle timeouts and missing terminal events.
• Closure 0.142 (55/388). All 11 swarm PRs merged on 2026-09-27 while the coordinator still recorded them as published; all 11 are merged_maturing.

What this means for routing: doc-drift lenses cost triage turns for almost no confirmed defects, and max effort buys a failure rate thirteen times that of high. Both are small samples; the Beta parameters say how small. The largest lever is not precision but throughput of judgment: 98% of findings carry no verdict yet.