Lesson 3 of 6
Documentation and evidence standards
17 min
Listen to this lesson
Evidence is the TCoE's product. A test that passed but cannot be shown to have passed is worth very little at a go/no-go and nothing in an audit. This lesson standardizes three things: how test assets are named, how they trace to processes and requirements, and what a passed test must show. It then maps the documents of ISO/IEC/IEEE 29119-3 onto SAP Cloud ALM so the TCoE does not run a second document system beside it.
Naming conventions
Names are how people find, reuse and report on tests. A good convention puts the process first, then the scenario, then the variant, so that sorting by name groups tests by process. Contoso uses:
| Asset | Pattern | Example |
|---|---|---|
| Test case | <process>-<number> <scenario>, <variant> | O2C-010 Standard order to billing, CMC-ALT-100, sales area 1710/10/00 |
| Test plan | <release or project> <cycle> <level> | FPS update 2027-09 regression cycle 1 |
| Defect | <process>: <what fails> when <condition> | O2C: billing document not created when delivery is partially picked |
| Automated test | Same identifier as the manual test case it implements | O2C-010 (automated) |
| Data recipe | DR-<process>-<number> <what it creates> | DR-O2C-003 Customer with credit limit and stock in 171A |
Contoso's naming conventions. The automated test reuses the manual identifier so results trace to one design.
Traceability
Traceability answers the auditor's question, "how do you know this requirement was tested?", and the process owner's question, "what did you test in my process?" In SAP Cloud ALM, test cases in Test Preparation can be linked to requirements and user stories, and the traceability pages show coverage per requirement, feature, user story and solution process. The standard says every test case links to a process and to at least one requirement or user story, and every defect links to the run that found it.
What a passed test must show
The evidence standard is a checklist that any reviewer can apply to any result. A result that fails one line is "passed without evidence," which the TCoE treats as not passed.
- Who executed it, and in which role (not a shared user).
- When, with date and time.
- Where: system and client, for example CMQ client 100, and the build or transport level.
- With what data: the master data and document numbers used, such as customer, material CMC-ALT-100, plant 1710.
- Expected versus actual for each verification step, not just a final status.
- Proof: screenshots, logs or output documents for the steps that matter, such as the billing document number and its accounting document.
- Linked defects for any deviation, even if accepted.
- Can you tell who ran it, when, and in which system, client and build?Yes → next question · No → passed without evidence: return to the tester
- Can you see the data used and the document numbers produced?Yes → next question · No → add data and documents from the run
- Does each verification show expected versus actual?Yes → next question · No → the result shows a status, not a check
- Are deviations linked to defects, even accepted ones?Yes → counts as passed · No → raise or link the defect first
Evidence should be produced by the tool where possible, not assembled by hand. Automated runs capture who, when, where, data and screenshots automatically; manual runs in SAP Cloud ALM Test Execution record step results and attachments. The standard also says how long evidence is kept. Agree the retention period with internal audit; for financially relevant processes, it usually follows the retention rules for financial records.
Mapping 29119-3 documents to SAP Cloud ALM
| ISO/IEC/IEEE 29119-3 document | Where it lives at Contoso |
|---|---|
| Test policy; organizational test practices | TCoE playbook, versioned, linked from Cloud ALM |
| Test plan (including the test strategy) | Strategy or release plan document stored with the Cloud ALM project; test plans in Cloud ALM for execution |
| Test model specification; test case specification | Test cases in Test Preparation, linked to requirements and user stories |
| Test procedure specification | Activities and actions inside the Cloud ALM test case, or the automated test that implements it |
| Test data and environment requirements and readiness reports | Data recipe references in the test case; readiness checklist confirmed before a test plan goes to In Testing |
| Test execution log; actual results; test results | Test Execution step results and attachments; automation provider results |
| Incident report | Defects, linked to the test run |
| Test status report; test completion report | Cloud ALM analytics plus a short narrative; completion report attached at the quality gate |
ISO/IEC/IEEE 29119-3:2021 documentation mapped to SAP Cloud ALM. The standard defines information content, not file formats, so a tool record can satisfy it.
What would you do?
The AMS partner submits regression results for a stack update: every test is Passed, each with a single screenshot of the final screen, all executed by the user TEST01 in CMR client 100. Go/no-go is in two days. What should Priya do?
- Name assets process first, then scenario, then variant; automated tests reuse the manual identifier.
- Every test case links to a process and a requirement or user story; every defect links to its run.
- A passed test shows who, when, where, with what data, expected versus actual, proof and linked defects.
- Map ISO/IEC/IEEE 29119-3 documents to SAP Cloud ALM records instead of keeping a second document system.
Check your understanding · not graded
Lesson check
1. Why does Contoso's naming convention put the process first?
2. A result is marked passed but has no record of the system and client. How does the standard treat it?
3. Where does an ISO/IEC/IEEE 29119-3 incident report live at Contoso?
4. Why should an automated test reuse the manual test case identifier?