Classifying the risk of a change: a practical matrix for deciding what a person reviews and what a machine does
In our article on verification debt, one of the mitigation levers was allocating by risk: full human review where failure is expensive, automated verification everywhere else. The table summarized four levels but left the practical question open: how do you decide, change by change, which level it falls into?
This article proposes a way to do it: an ordered list of questions, the controls each level calls for, borderline cases to calibrate judgment, and a rule for erring in the safe direction. At the end we apply it to our own 141 PRs, and the result does not flatter us.
Why classify
Human attention is the scarce resource. If it is spread evenly across all changes, the important ones get the same as a typo fix. The model in the first article puts it in terms of expected loss:
An hour of review pays off most where the product of failure probability and cost is highest. Classifying risk means estimating that product before deciding how much attention to give.
The matrix: questions in order
Answer from top to bottom. The first "yes" decides the level.
| # | Question | If yes |
|---|---|---|
| 1 | Could a failure harm people's safety or health, breach a regulation, or affect the calculation core of a critical system? | Critical |
| 2 | Does it touch money (charges, prices, rounding, balances)? | High |
| 3 | Does it touch who can see or change what (authentication, permissions, personal data)? | High |
| 4 | Does it change persistent data (migrations, deletions, schemas)? | High |
| 5 | Does it change a public contract (an API or event others consume)? | High |
| 6 | Does it touch security or infrastructure (secrets, networking, cloud permissions, CI/CD, the agent's controls)? | High |
| 7 | Does it process external input, or AI output that reaches customers? | High |
| 8 | Does it change business logic? | Medium |
| 9 | Is it only text, styles, documentation, new tests, or a refactor covered by tests? | Low |
Modifiers that raise the level by one:
- The change is irreversible, or rolling it back has not been rehearsed (an email sent, a charge made, a deletion with no copy).
- The blast radius is the whole system or all users, with no gradual rollout.
- Nobody on the team knows that code: the risk lies in the missing context.
- The code was generated by AI without tests written first.
One non-negotiable rule: the level can go up, never down. A reviewer can raise it; nobody lowers it to move faster.
Who reviews what at each level
| Low | Medium | High | Critical | |
|---|---|---|---|---|
| Automated verification (tests, types, static analysis, secrets) | Yes | Yes | Yes | Yes |
| Agent review with a different model from the one that generated the code | Optional | Yes | Yes | Yes |
| Human review | By sampling | Tests and acceptance criteria | Line by line | Line by line, two independent people |
| Approvals | 1 | 1 | 2, one experienced | 2 independent + area owner |
| Evidence | Verification green | + one test per acceptance criterion | + mutation testing on the change, threat model, rehearsed rollback | + formal or property-based verification of the core |
| Deployment | Normal | Normal | Gradual, with automatic rollback | Gradual, with a window and contingency plan |
| Sampling audit (deep review after the fact) | 5% of changes | 20% | Not applicable: everything was reviewed | Not applicable |
The sampling rates are a proposal. They estimate how much slips through at the lower levels without reviewing everything; if sampling finds many defects, the level was miscalibrated.
Borderline cases
The matrix is learned through examples. These ten help calibrate it:
| Change | Level | Why |
|---|---|---|
| Fixing a typo on the help page | Low | Text only |
| Changing the wording of the terms and conditions | Medium | It is text, but it has legal effect: a person reviews it |
| Changing how a price that is only displayed is rounded | High | It touches money even without charging: the customer sees a price and expects to pay it |
| Adding an optional column to a table | High | Persistent data, even if the migration is additive |
| Changing a timeout from 30 to 60 seconds | Medium, or high if it is in the infrastructure | It can change costs or failure cascades |
| Adding a new dependency | High | Supply chain: verify that it exists, who publishes it and since when |
| Adjusting the prompt of an assistant that answers customers | High | External input and AI output that reaches customers |
| Refactoring a module with no tests | Medium, raised to high if nobody knows it | The "no behavior change" refactor is not verified |
| Changing styles in a way that hides the cancel-subscription button | High | By file it is styling; by effect it touches money and customer rights |
| Changing the rule that computes a dose or an official grade | Critical | Harm to people or regulated effect |
The second-to-last is the most important: classify by effect, not by file. A rule that only looks at paths would call it low.
What happened when we applied it to our 141 PRs
We applied our own repository's risk guide to the 141 PRs merged between September 17 and October 17, 2026, the same ones as in the previous article. The classification is heuristic: it looks at the paths each PR touches, with the limitation shown by the borderline case above.
| Level | PRs | Approvals the guide requires | Received | Median time from opening to merge |
|---|---|---|---|---|
| Low | 28 | 1 | 0 | 18 min |
| Medium | 50 | 1 | 0 | 5 min |
| High | 63 | 2 | 0 | 5 min |
Of the 63 high-risk PRs, 29 only touch CI, hooks or infrastructure: building the controls themselves. The other 34 touch permissions (23), migrations (12), secrets (11), money (3) or a public contract (1); one PR can touch several.
Three uncomfortable readings:
- Almost half the changes were high risk. That is not unusual in a month spent setting up pipelines and controls, but it is far more than we had assumed.
- Review intensity did not follow risk. High-risk PRs were merged as fast as medium-risk ones, and faster than low-risk ones. None got the second approval the guide requires.
- The guide existed, but no mechanism enforced it. The check that requires approvals is in warning mode because the team is one person. A rule that does not fail on its own is a good intention.
What is measured and what is not
| Claim | Status |
|---|---|
| 63 of 141 PRs touched high-risk areas according to the guide | Measured with a path-based heuristic |
| No PR received the approvals its level requires | Measured |
| High-risk PRs were reviewed less than low-risk ones | Not measured: time to merge does not measure review time |
| The proposed matrix reduces defects reaching production | Not measured: it is a proposal |
| The sampling rates are the right ones | Not measured: they need to be calibrated with data |
Risks of classifying
- Classifying by file instead of by effect. It is the easiest way to automate and the easiest to game, as the hidden button shows.
- Classification turning into paperwork. If "low" is always picked to go faster, the matrix only adds bureaucracy. That is why the level can only go up, and why someone other than the author should confirm it.
- Calibrating once and never looking again. Sampling exists to correct the matrix with what it finds.
- Inflating everything to high. If everything is high, nothing is: everything gets reviewed the same way again.
What this article does not claim
- It does not claim this matrix fits every team equally. The levels for a payments system and for a blog look nothing alike; each team adapts the questions to its domain.
- It does not claim low-risk changes go unreviewed. They are reviewed by sampling, which is not the same as not reviewing.
- It does not claim our high-risk PRs had defects. It claims they did not receive the controls our own guide requires.
Sources
- The repository's risk-based review guide (
11-quality/risk-based-review.md), on which the matrix is based. - The repository's PR history, retrieved October 17, 2026, and the classification script used.
- Dixi-Project: Verification debt, Scope, specify, audit, Do AI-written tests catch AI-written bugs?, One maintainer, one agent and 141 PRs with no human review and The broken pipeline (2026).