How PR verification works
What triggers a verification, what Spectrace compares, the five per-criterion verdicts, and where the result lands.
This page explains what happens between a pull request opening on GitHub and a verification result appearing on it, so you know what Spectrace checked and what it did not.
Prerequisites
- The Spectrace GitHub App is installed for your organization and the repository is mapped to a project. See GitHub App.
- The requirement the pull request implements has acceptance criteria. See Acceptance criteria.
- The pull request names its requirement, or the changed files are already linked to one. See Linking a PR to a requirement.
What triggers a run
Spectrace verifies a pull request when GitHub sends one of three pull_request events for a mapped repository: opened, synchronize (a new push to the branch), and reopened. Every push produces a fresh run against the new head commit.
Editing the title or description alone does not start a run. The edited event is not handled, so a reference added to the title takes effect on the next push or when the pull request is reopened.
When a pull request is merged, Spectrace reads the latest verification for that pull request that is bound to a requirement and moves that requirement to the Done stage with status verified. If the project syncs status to Linear, Jira, or GitHub Issues, the source issue is updated as well.
The pipeline
The webhook validates the payload signature, resolves the installation to your Spectrace organization and the repository to its project, and enqueues a background job. An unmapped repository is acknowledged and skipped.
The job creates a check run named Spectrace verification in the in-progress state, fetches the diff, and resolves which requirement to verify against. If no requirement resolves, the job posts an explanatory comment, concludes the check as neutral, and stops. See Linking a PR to a requirement.
The job then runs the AI verifier, stores the result, posts a comment, and concludes the check run. If the verifier throws, the run is recorded as failed with its error message, and the check run is concluded as neutral with the title "Verification could not complete" — a pipeline error is never reported as a failing pull request.
What is compared
The verifier compares the code changes in the diff against the requirement's acceptance criteria. Alongside the criteria it receives the requirement's title, description, and user story, plus up to five related code chunks from the project's traceability index.
It also receives the head commit's completed CI check runs — names and conclusions only, with Spectrace's own check excluded — as GitHub-reported context. A criterion that asserts a test-run outcome, such as "existing tests pass", can be satisfied by a green Test check, because no diff can prove a claim about a test run. Check conclusions never support criteria about code content, and when no check data is available — no checks configured, all still pending, or the fetch failed — such criteria come back unclear and are excluded from the risk figures, and everything else behaves exactly as if the feature did not exist.
Only code files are sent to the verifier. Spectrace keeps files with source extensions such as .ts, .tsx, .js, .py, .go, .rs, .java, .rb, .swift, .kt, and .vue, and drops paths that start with a dot (such as .github/), node_modules, package-lock.json, pnpm-lock.yaml, yarn.lock, minified and source-map files, dist/ and build/ output, and .d.ts declarations.
The first 20 code files are included, sharing one total budget of 40,000 characters of patch text. Files whose changes fit keep them whole; what they don't use is split across the larger files — so a 3-file pull request gets roughly 13,000 characters per file and a 7-file one roughly 5,700, rather than a flat 2,000 each.
A diff that exceeds one run's budget is split into up to 4 chunks of whole files — a test stays with the file it tests, and same-directory files travel together. Each chunk is verified in full against all the acceptance criteria, and a final step combines the per-chunk verdicts into one result: evidence split across chunks (an implementation in one, its test in another) can combine to satisfied, a criterion contradicted in any chunk stays contradicted, and the combined result cites only evidence some chunk actually produced. The pull request still gets exactly one comment and one check run, whose coverage note reports the run count — for example, "Verified against 100% of the diff in 3 runs."
When part of the diff is still not visible — a file cut short inside its chunk, or files beyond the 4-chunk cap — the verifier is told exactly which files it cannot fully see, criteria whose evidence may sit in the invisible part come back unclear rather than partial, and the run records how much of the diff was visible. See Reading a verification result.
The pull request description is read for requirement references but is not passed to the verifier. Verification is based on the code, not on what the description says the code does.
Spectrace does not check dependency vulnerabilities, license compliance, lint, performance, or style. An OSV dependency scan runs when the diff touches package.json, pnpm-lock.yaml, or package-lock.json, but its findings are not currently stored or displayed anywhere.
The five verdicts
Every acceptance criterion receives exactly one status:
| Status | Meaning |
|---|---|
satisfied | The diff clearly and completely implements the criterion, with evidence in the changed code. |
partial | The diff implements some of the criterion, but the implementation is incomplete. |
not_found | Nothing in this diff relates to the criterion. This is the normal answer for a focused pull request against a broad requirement and is not a failure. |
unclear | The verifier cannot tell whether the criterion is met — either the visible code is ambiguous, or the evidence lies in a part of the diff that exceeded the size budget. The run's coverage note says which parts were not visible, and criteria that were merely invisible are excluded from the risk figures. |
contradicted | The diff actively breaks or removes something the criterion requires. The verifier assigns this only when it can point at a specific change. |
Each verdict carries a confidence value and, where applicable, evidence: the file, the line range, and a snippet. When any criterion is not satisfied, a second pass adds a short reasoning for every criterion and one to three suggestions for each criterion that is not satisfied. A pull request is verified against exactly one requirement per run.
Where the result lands
Each pull request has one Spectrace comment, updated in place on every run. The first run posts it; every later run edits the same comment to describe the new head commit, opening with what changed since the previous push. If someone deletes the comment, the next run posts a fresh one and updates that from then on. See Reading a verification result.
The per-commit history is not lost to the editing: every run is stored in Spectrace, one per head commit, on the requirement's PR verification tab.
Each run also concludes the Spectrace verification check run on the head commit. See Check runs and branch protection.
The full result — every criterion with its reasoning, evidence, and suggestions, plus manual overrides and a Re-verify button — is on the requirement's page in Spectrace. Open the requirement and choose the PR verification tab. The tab appears once the requirement has at least one pull request or is in the In Review or Done stage.
Per-project controls
Project owners and admins can tune how loudly Spectrace speaks, under Project settings → Verification. Three modes: Comments and checks — the default, everything this page describes. Checks only — no PR comments, including the unmatched nudge; check runs and stored results are unchanged, and the check's summary still carries the full verdict. Silent — nothing on GitHub and no Slack; results are still recorded in Spectrace exactly as always. A separate toggle turns off just the unmatched-PR nudge without changing anything else.
Path filters (include and exclude globs, one per line) decide which changed files count: filtered files are invisible to requirement resolution and to the verifier. When every changed file is excluded, the run is recorded with status skipped, the check concludes neutral, and nothing is verified. Dependency security scanning always reads the full file list, so excluding lockfiles cannot switch it off.
What done looks like
A successful run leaves a comment on the pull request and a concluded check run, and the requirement's PR verification tab shows the run with a status of satisfied, partial, not addressed, or unclear.
If the pull request gets a comment headed "Spectrace — no requirement linked" or "Spectrace — which requirement?", the run stopped before verification because it could not decide which requirement to check. Fix the reference as described in Linking a PR to a requirement and push again.
If the check run concludes with "Verification could not complete", the run failed. The failed run and its error message appear in the PR verification tab; use Re-verify to run it again, or see Troubleshooting.