Prioritization criteria

You will never have time to run every test for every change. Prioritization decides which tests sit in which tier, and which ones run first when a window is short. Done well, it is explicit and repeatable: two test leads looking at the same data reach the same ranking. Done badly, it reflects whoever shouts loudest or whichever tests are easiest to run. The aim is to spend scarce test time where a failure would hurt most and is most likely. That means some factors should never drive priority: who wrote the test, how easy it is to run, or how long it has been in the suite. Those are convenience, not risk. Five criteria do the real work.

Five criteria

A simple scoring model

Score each test (or each process the test covers) from 1 (low) to 5 (high) on the five criteria, then weight them. Criticality matters most, because it measures impact; the others mostly measure likelihood. Let the process owners score criticality, because they know what a failure costs, and let the test lead score the other four from data, because those are facts about the system rather than opinions. Where data is missing, for example usage of a rarely run report, score conservatively and note the gap so it can be filled before the next review. A workable starting formula is: priority = 3 × criticality + 2 × change frequency + 2 × usage + 2 × defect history + 1 × integration. The maximum is 50. The weights are a starting point, not a law; a regulated business might weight compliance-related criticality even higher, and a company with a history of interface failures might raise the integration weight. What matters is that the formula is written down and applied the same way to every test. The worked example below shows five Contoso tests scored with these weights.

Two rules keep the model honest. First, a criticality override: any process scored 5 for criticality goes into the core tier regardless of its total. The payment run changes rarely, but paying suppliers wrongly is not a risk Contoso will accept, so it sits in core at 36. Second, re-score after every release. Change frequency and defect history move; a test that scored 34 last quarter may score 42 after three defects. Present the model to the process owners once, agree the weights, thresholds and override rule, and record the agreement. After that, priority discussions become discussions about data, not opinions, and a new test lead can pick up the ranking without starting again.

Risk-based ranking plus change-based selection

The scoring model gives you a standing ranking: which tests matter most in general. Each change then adds a second layer: which tests are affected by this change. A transport that changes output for billing documents pulls in the billing and output tests, even if they normally sit in extended. In practice, the regression set for a change is: the smoke tier, plus the core tests if the change is significant, plus every test linked to the affected process steps. Lesson 5 shows how to find those affected steps. Change-based selection also works in reverse: if a stack touches nothing in an area for a whole release, its extended tests can safely wait, freeing time for the areas that did change. The order of execution follows the same logic: when a window is short, run smoke first, then the affected tests in descending score, then the remaining core tests. If time runs out, the tests left unrun are the lowest-risk ones, and you can say exactly which they were. That list belongs in the sign-off record, because a process owner is entitled to know what was not tested before accepting the residual risk.