Case studies
Three runs you can reproduce
One for each outcome. Each comes with the signed Passport, the public key to check it, and the exact commands to get the same verdict.
VERIFIED
A real fix, independently re-run
A refund bug fix with a new test. RelayOps re-ran it in a clean sandbox: the new test fails on the base branch and passes on the change, so the fix fixes the bug it claims to. The tests also caught every bug RelayOps planted in the changed lines.
- Checks
- tests 4/4 · reproduce: fails on base, passes on head · test strength: strong (1 of 1 planted bugs caught) · test integrity: no weakening · secrets, dependencies, code scan: 0 findings
- Passport
- 01M3X324D464V00VR0CE42BZYN · verify it in your browser
- Limits
- A small planted repository: it shows the mechanism, not performance on a large codebase.
FAILED · test integrity
Tests weakened to get green
A "fix" that breaks legacy charges, then deletes one assertion and loosens another so the tests pass. The commit carries an AI co-author trailer. The change's own tests pass, so CI is green. RelayOps re-ran the base branch's test against the new code: it fails. The Passport names the exact line.
- Checks
- tests 4/4 (green) · test integrity: fail, assertion removed at
test/refunds.test.js:7 plus one loosened, confirmed by execution · test strength: weak (2 of 3 planted bugs caught) - Passport
- 01M3X32GRHNX67MKG2W2TH9WKY
- Limits
- A planted repository. In the 15 real merged pull requests measured so far, none weakened an assertion to get green.
INCOMPLETE
A real agent pull request that skips test suites
e2b-dev/E2B #1863 made 13 JavaScript test files and one Python suite skip unless an environment variable is set. The maintainers merged it on purpose (it tests a deprecated API). RelayOps listed all three skip mechanisms with file and line, and refused to call the change verified or failed: the suites need an API key the sandbox doesn't have, and the Passport names it.
- Checks
- test integrity: 3 skip signals, can't be confirmed without the key · tests: 9 passed, 4 failed with authentication errors, the same 4 failing on the base branch, so recorded as "needs secret" and not as the change failing
- Passport
- 01M3P5KYZZWS0P76WX88204E31
- Limits
- An earlier engine version found only 1 of the 3 skip mechanisms; the current one finds all 3. RelayOps doesn't inject secrets into tests yet (planned).
Re-run them yourself
relayops verify fixtures/.generated/node-refunds --base main --head clean-fix # case 1
relayops verify fixtures/.generated/node-refunds --base main --head agent-weakened # case 2
relayops-passport verify passport.dsse.json --key passport-keys.json --allow-dev-key
The full method for the 15-PR measurement (which changes, what counted as correct, and what it doesn't show) is shared with design partners on request. All three Passports here are signed with development keys.