Reading a verification result

This page helps you read the comment Spectrace posts on a verified pull request and decide what to do about it.

Prerequisites

The pull request has been verified at least once. See How PR verification works.

Spectrace keeps one comment per pull request, updated in place on every run and written to be read in a few seconds. The first line names the requirement; the second says what the reader has to do. Everything else is optional detail, and the full report is one click away in Spectrace.

The heading is always the same shape: a mark, the requirement's reference in a code span, and the requirement's title — for example ✗ sss-050 — Consolidate webhook routing. The reference is omitted when the requirement has none.

The comment's shape — its tier — is chosen by what you have to do, never by a percentage. Confidence values, scores, model names, and rubric text are deliberately absent from the comment; they are on the report page.

The "Since the last push" line

From the second run onward, a line directly under the heading says what moved — for example Since the last push: 2 more met, 1 regressed. It counts three things: more met for criteria that are now satisfied, improved for criteria that climbed without reaching satisfied yet, and regressed for criteria that were satisfied on the previous push and no longer are. When nothing moved, the line reads No change since the last push. and the rest of the comment still renders in full, because the comment must stand alone.

The line is absent on the first verification, and when the pull request was re-pointed at a different requirement between pushes — a delta across two different criteria lists would be meaningless.

Tier 1 — contradiction (`✗`)

Used when at least one acceptance criterion is contradicted: the diff actively breaks or removes something the requirement needs. The second line reads <met> of <total> criteria met, **N contradicted**.

The body leads with **Broken:** followed by the criterion, then the verifier's reasoning and, where evidence exists, See <file:lines>. If more than one criterion is contradicted, a line reads …and N more — see the full breakdown. A short Not yet: list of up to two outstanding criteria may follow.

What to do: read the Broken line and the evidence before anything else. Either the diff needs to change, or the criterion is wrong — in which case open the report page and record an override with a reason. As a required check, this is the only tier that blocks a merge; see Check runs and branch protection.

Tier 2 — progress (`✓` or `◐`)

Used when at least one criterion is satisfied or partial and none is contradicted. The mark is ✓ when every criterion is met and ◐ otherwise. The second line is one of:

All <total> criteria met.
<met> of <total> met, <partial> in progress.
<met> of <total> met. — when nothing is in progress
<partial> of <total> in progress, none complete yet.

The body has two lists. Done: names up to three met criteria, each followed by an evidence link where the verifier recorded evidence. Outstanding: names up to three criteria that are not met, each tagged *(in progress)* for partial, *(not in this diff)* for not_found, or *(unclear from the diff)* for unclear. A criterion that was satisfied on the previous push and regressed is listed first and tagged *(regressed)* instead. Longer lists end with …and N more.

What to do: nothing is broken. Use the Outstanding list to see what remains for the requirement; not in this diff is expected on an incremental pull request and is not a defect in this change.

Tier 3 — focused (`○`)

Used when no criterion is met or in progress and none is contradicted. The second line reads This diff covers 0 of <total> criteria., followed by a sentence saying this is the normal shape for a focused change against a broader requirement and that nothing here works against it.

What to do: check that the pull request is linked to the requirement you intended. If it is, no action is needed. If the change was meant to implement one of the criteria, the verifier found no evidence of it in the code files of the diff.

Evidence links

An evidence reference points at the first piece of evidence the verifier recorded for that criterion. It renders as a link labelled with the file's basename and line range — for example route.ts:118-127 — pointing at those exact lines on the verified commit (https://github.com/<owner>/<repo>/blob/<head sha>/<path>#L118-L127). Because the link is pinned to the commit, it keeps pointing at what was verified after later pushes.

Every tier ends with a <sub> footer that can carry two things. Confirmed by file traceability appears when the linked requirement was also the one the changed files are linked to. Full breakdown → is the comment's single navigation link and opens the report page in Spectrace.

The Full breakdown link is omitted when the Spectrace deployment's public app URL is not configured for the background worker; the report page still exists and is reachable from the requirement. See Environment variables.

A drift note can appear above the footer on any tier. See Spec drift.

The report page

The full result lives on the requirement in Spectrace. Open the requirement and choose the PR verification tab; the tab appears once the requirement has at least one pull request or is in the In Review or Done stage. The Full breakdown link opens the requirement on its Specification tab; open the PR verification tab and the panel expands and scrolls to that pull request's card, using the pull request number carried in the link.

Each card shows the run's overall status — satisfied, partial, not_addressed, or unclear, or running, pending, or failed for runs that have not completed — and, per criterion, its verdict, confidence, evidence (each with the verifier's explanation and a snippet), and suggestions.

Most evidence entries point at code — a file, a line range, and a snippet. A criterion about a test-run outcome, such as "existing tests pass", can instead cite a CI check: the entry names the check and its conclusion (for example, check Test: success), because the proof lives in the commit's check runs rather than in the diff. Check citations appear only for test- and CI-outcome criteria; verification runs before CI finishes on a fresh push, so they typically show up after a Re-verify or on the next push.

When the diff exceeded the verifier's size budget, the card's coverage section carries a line like "Verified against 61% of the diff (3 of 7 files truncated)." Criteria whose evidence may sit in the invisible part come back unclear — a verification gap, not an implementation gap — so they are excluded from the risk figures and listed as "not visible" among the risk drivers instead of being priced as remediation work. A fully visible diff shows no coverage line.

A diff too large for one run is split into up to 4 chunks and verified in full; the coverage line then reports the run count — "Verified against 100% of the diff in 3 runs." — and the pull request comment carries the same note in a small footer. The result is still one card with one verdict per criterion; the stored run records how the diff was carved so any combined verdict can be traced to the chunk that produced its evidence.

From the card you can:

Re-verify the pull request against the same requirement. This is refused while a run is already pending or running.
Override a criterion's verdict with your own, with a reason, or remove an override.

Failed runs are listed with their error message. Compliance-control grades (ASVS, CWE, OWASP Top 10, SSDF) are recorded with the run but are not shown in the comment or on the report page today.

What done looks like

You have read the result when you know which tier the comment is and have acted on it: fixed or overridden a contradiction, noted what remains outstanding, or confirmed a focused change is linked to the right requirement.

If the comment is headed "Spectrace — no requirement linked" or "Spectrace — which requirement?", it is not a verification result: the run could not resolve a requirement. See Linking a PR to a requirement. If a run shows as failed on the report page, use Re-verify or see Troubleshooting.