Check runs and branch protection

This page shows you how to use the Spectrace verification check run as a merge gate, and exactly when it blocks a merge.

Prerequisites

• The Spectrace GitHub App is installed for your organization with the Checks: Read & write permission, and the repository is mapped to a project. See GitHub App.
• You are a repository administrator on GitHub, or can ask one to edit branch protection.

What the check run is

A comment is advice; a check run is a gate. Every verification run creates a check run named Spectrace verification on the pull request's head commit, so reviewers see verification is happening, and concludes it when the run finishes.

The check is created as in progress as soon as the job has GitHub credentials, with the title "Verifying against the linked requirement…". When the run finishes, the check's title reads <met> of <total> criteria met, with · 1 or more contradicted appended when a criterion is contradicted. From the second run onward the title also carries the delta against the previous completed run — (+2, -1 since last push), or (no change since last push) when nothing moved. The check's summary is the same markdown as the pull request comment, so one result never reads two ways, and its Details link opens the report page in Spectrace when the worker knows the app's public URL, and otherwise points back at the pull request. One addition the comment does not carry: when the diff exceeded the verifier's size budget, the summary ends with a coverage note — e.g. "Verified against 61% of the diff (3 of 7 files truncated)."

Check runs are only created through a GitHub App installation token. If your organization has not installed the App and Spectrace falls back to a member's OAuth token, GitHub refuses the check and the run continues with the comment only.

What each conclusion means

GitHub treats neutral as passing for a required check. That means a required Spectrace verification check blocks a merge only when a criterion is contradicted. An incremental pull request that covers part of a requirement, a dependency bump with no requirement, and a run that threw inside Spectrace never block a merge. A regression that is not a contradiction — a criterion satisfied on the previous push and no longer — shows in the title's delta and the comment, but the conclusion stays neutral.

For an unmatched pull request the check title is "No linked requirement" or "Several requirements referenced — pick one", and its summary ends "Not blocking." For a run that threw, the title is "Verification could not complete" — a pipeline error is Spectrace's problem, not the pull request's.

Add it as a required check

The check appears in GitHub's list of available status checks after it has run at least once on the repository, so open one pull request on a mapped repository first.

1. Open branch protection

   In the repository on GitHub, go to Settings → Branches and add a branch protection rule for your default branch, or edit the existing one.

2. Require status checks

   Enable Require status checks to pass before merging.

3. Select the check

   In the search box, type Spectrace verification and select it.

4. Save

   Save the rule. From now on, a pull request whose latest verification concluded failure cannot be merged until a new push produces a passing conclusion.

Keeping the check from staying pending

A check created as in progress and never concluded stays pending forever, and a required check that is stuck pending blocks every merge on the repository. Spectrace concludes the check on every exit path it knows about: a check is only created once the job holds a token to conclude it with, and the job's error path concludes any open check as neutral before recording the failure. One exception exists today: a pull request that names a requirement id which no longer exists in the project records a failed verification and leaves the check in progress. Every check-run call is best-effort — if GitHub refuses to create or update the check, the verification is still stored and the comment still posts.

Permissions and the 403 hint

Creating a check requires the App installation to carry Checks: Read & write. If your installation predates that permission, GitHub returns 403 when Spectrace creates the check, the job logs the hint "installation may lack checks:write — re-accept the app's permissions", and the run continues without a check.

To fix it, an organization owner accepts the App's updated permissions on GitHub: open Organization settings → GitHub Apps, choose Configure next to Spectrace, and approve the pending permission request. Spectrace records the acceptance from GitHub's installation.new_permissions_accepted event, and the next verification run creates its check. See GitHub App permissions for the full list.

What done looks like

The check is working when a pull request on a mapped repository shows Spectrace verification in its checks list, first as in progress and then concluded, and — with branch protection configured — a contradicted criterion turns the merge button off.

If no check appears while a comment does, the installation lacks the Checks permission or the run used a user token; see the section above. If the check appears but branch protection does not list it, run one verification on the repository first and then reopen the branch protection rule.

Related

• How PR verification works
• Reading a verification result
• GitHub App
• GitHub App permissions