A repair only counts when
the evidence closes the loop.
Thirty seeded Python failures. Real pytest runs before and after every candidate. One supervisor policy, one single-attempt baseline, and zero self-reported fixes.
Verification feedback changes the outcome.
Both systems receive the same first repair candidate. The supervisor can use failed verification evidence for one bounded retry.
- Verified
- —
- Escalated
- —
- Median
- —
- Verified
- —
- Escalated
- —
- Median
- —
Inspect every seeded incident.
| Incident | Project | Supervisor | Attempts | Baseline |
|---|
What “verified” means here.
- 01Observe a real failure
The selected pytest suite must execute and report at least one failing node ID.
- 02Reproduce before editing
The same node IDs must fail again on an untouched copy of the repository.
- 03Verify the candidate
Formerly failing tests pass and the selected regression suite retains every test identity.
- 04Escalate uncertainty
Two failed candidates stop safely with logs, diffs, and reasons for a human reviewer.
The repair candidates in this benchmark are scripted fixtures. The comparison measures the value of reproduction, verification feedback, bounded retries, and escalation. It is not a claim about any language model’s repair accuracy. LLM cost is therefore $0 for both systems.