Reproducible reliability benchmark

A repair only counts when
the evidence closes the loop.

Thirty seeded Python failures. Real pytest runs before and after every candidate. One supervisor policy, one single-attempt baseline, and zero self-reported fixes.

Explore all cases
Seeded failures — Across three small repositories
Evidence coverage — Reproduction and agent trace
False verified — Claims without passing evidence
Measured comparison

Verification feedback changes the outcome.

Both systems receive the same first repair candidate. The supervisor can use failed verification evidence for one bounded retry.

ProofPatch supervisorEvidence gated
—verified repair rate
Verified
—
Escalated
—
Median
—
Single-attempt baselineNo retry
—verified repair rate
Verified
—
Escalated
—
Median
—
Evidence explorer

Inspect every seeded incident.

IncidentProjectSupervisorAttemptsBaseline
Evaluation contract

What “verified” means here.

  1. 01
    Observe a real failure

    The selected pytest suite must execute and report at least one failing node ID.

  2. 02
    Reproduce before editing

    The same node IDs must fail again on an untouched copy of the repository.

  3. 03
    Verify the candidate

    Formerly failing tests pass and the selected regression suite retains every test identity.

  4. 04
    Escalate uncertainty

    Two failed candidates stop safely with logs, diffs, and reasons for a human reviewer.

Read this result correctly.

The repair candidates in this benchmark are scripted fixtures. The comparison measures the value of reproduction, verification feedback, bounded retries, and escalation. It is not a claim about any language model’s repair accuracy. LLM cost is therefore $0 for both systems.