Lesson 2 of 6
Independent quality assessment
12 min
When indicators point to trouble, the program needs a shared, credible picture of where it really stands. People inside the program are too close to provide it, and the steering committee no longer fully trusts the reports. An independent quality assessment provides that picture: a short, evidence-based review by people who are not responsible for the work being reviewed.
Independence and scope
Independence means the reviewers do not report to the people whose work they assess and have nothing to gain from a particular conclusion. It can be an internal assurance team from another part of the company, an experienced program leader from another business unit, or an external reviewer. Public sector project assurance offers a useful model: the UK government's gate review process provides independent guidance to the senior responsible owner, based on evidence that should be available before the review starts, including documents and interviews, and gives a realistic view of the program's chances of success.
Set the scope in a short terms of reference: the questions to answer, the areas in scope, the evidence the reviewers can access, the people they can interview, the time box and who receives the report. For a troubled SAP program the core question is usually simple: can this program reach a defensible go-live, and if so, when and on what conditions?
- Days 1–2Terms of referenceQuestions, scope, access, interviewees, sponsor agreed
- Days 3–7Evidence collectionDocuments, tool data, interviews, walkthroughs
- Days 8–10AnalysisFindings by area; root causes; options
- Day 11Read-out to program leadsFactual accuracy check; no surprises
- Day 12Report to sponsorDelivery confidence rating, findings, recommendations
Evidence, not opinion
The assessment's strength is its evidence. Reviewers look at what exists and what it shows, rather than at what people say about it. Interviews are evidence too, but they are used to explain the data and find causes, not to replace it.
Sampling and re-execution
A small, well-chosen sample tells you a lot. In the what-if, the reviewer picks 20 test cases marked passed in SIT, across order-to-cash, procure-to-pay and finance, and re-traces each from requirement to evidence. Six have no evidence attached, three were passed with a workaround nobody had approved, and two fail when re-executed with Maria because a pricing condition for Great Lakes Fleet Services had changed since. A pass rate reported at 88% now has a credible error bar.
| Sample result | Count of 20 | What it means |
|---|---|---|
| Passed with complete evidence | 9 | Reliable |
| Passed, evidence missing | 6 | Status unproven; re-test or retrieve evidence |
| Passed with unapproved workaround | 3 | Hidden risk; process owner must decide |
| Failed on re-execution | 2 | Regression or data change; reported status is wrong |
Re-tracing a sample of "passed" cases. Fewer than half were fully reliable.
Findings and the confidence rating
The report should be short: an overall delivery confidence rating (for example on a red, amber, green scale with defined meanings), five to ten findings ranked by impact, the root cause behind each, and recommendations with owners. Each finding links to the evidence that supports it. Avoid long lists of minor observations; they bury the few issues that decide the outcome.
| Rating | Meaning | Typical response |
|---|---|---|
| Green | Likely to deliver on time and to quality; no major open issues | Proceed; address minor recommendations |
| Amber | Achievable, but significant issues need management action now | Targeted recovery actions; re-check at the next gate |
| Red | Not achievable on the current plan without major change | Re-baseline scope, plan or date; consider pausing |
What would you do?
The systems integrator's account lead proposes that his own quality team performs the assessment, because they know the solution best. What should the sponsor decide?
- An independent assessment gives a credible picture when internal reporting has lost trust.
- Write terms of reference: questions, scope, access, interviewees, time box, recipient.
- Prefer tool data, documents, walkthroughs and re-execution; use interviews to explain causes.
- Report a defined confidence rating and a few evidence-linked findings; protect the messengers.
Check your understanding · not graded
Lesson check
1. What makes a quality assessment independent?
2. Which is the strongest form of evidence in an assessment?
3. What should the terms of reference define?
4. Why should confidence rating meanings be defined before the review?