Lesson 4 of 6
Guardrails for AI in testing
16 min
Listen to this lesson
A TCoE exists to provide evidence that a change is safe. AI that produces test assets, scope proposals or results must not weaken that evidence. Guardrails are the rules that let the TCoE use AI and still sign off with confidence: who reviews what, which data may be used, how outputs are traced, how quality is measured, and what people must understand. They should be short enough to follow and strict enough to defend in an audit.
Seven guardrails
| Guardrail | What it requires | Contoso example |
|---|---|---|
| Human accountability | A named reviewer approves every AI output before it becomes a test asset or a decision input | Process test leads approve generated test cases; Luis approves scope |
| Traceability | AI-generated or AI-changed assets are marked, with the tool, version and date | A custom field "AI-assisted" on test cases; self-healing events kept in reports |
| Data protection | No production personal data in prompts or AI inputs; only approved tools for confidential data | Masked or synthetic data only; consumer chat tools banned for Contoso data |
| Evaluation | A golden set of known cases measures AI output quality before and after adoption and after model changes | Thirty reviewed enhancement test cases used to score generated drafts |
| Least privilege for agents | Agents act with the narrowest rights, and cannot approve their own output | Agents may draft in SAP Cloud ALM, never set a test to passed |
| Transparency | Stakeholders know where AI was used in the evidence they rely on | Readiness assessments state AI-assisted scope and test design |
| AI literacy | People who use or review AI outputs understand its limits | Short training before tool access; refreshed yearly |
Contoso's AI guardrails for testing. They apply equally to SAP features, automation platforms and general assistants.
Measure the AI, not just the testing
Evaluation is the guardrail most often skipped. Before a pilot, Contoso's TCoE builds a small golden set: thirty enhancement test cases already written and reviewed by people, with their requirements. The AI drafts cases for the same requirements, and reviewers score them: correct, correctable with minor edits, or wrong. The pilot's acceptance criteria use those scores, for example at least 70% correct or correctable and no wrong expected results in critical processes. The same golden set is rerun when the vendor changes the model or the tool version, because generative behavior can shift without notice.
Frameworks to lean on
- NIST AI Risk Management Framework 1.0 (January 26, 2023) organizes AI risk work into four functions: Govern, Map, Measure and Manage. NIST's Generative AI Profile (NIST AI 600-1, July 26, 2024) adds risks specific to generative AI.
- ISO/IEC 42001:2023 specifies an AI management system, useful if Contoso's IT already runs ISO management systems.
- ISTQB CT-GenAI v1.1 covers the risks of generative AI in testing and how to deploy it in a test organization.
- The EU AI Act applies its AI literacy article (Article 4) from February 2, 2025. The Digital Omnibus on AI, Regulation (EU) 2026/1744, in force since July 27, 2026, reworded it so that providers and deployers take measures that support AI literacy without having to guarantee a specific individual level, and moved stand-alone high-risk obligations to December 2, 2027 and those embedded in regulated products to August 2, 2028. Contoso has no EU operations, but uses the Act as a reference, and companies with EU staff or customers should check the consolidated text with counsel.
| NIST AI RMF function | Question for the TCoE | Contoso guardrail |
|---|---|---|
| Govern | Who decides, and under which policy? | AI use in testing policy approved by the quality council |
| Map | Where is AI used and what could go wrong? | Use case cards with risk ratings (lesson 2) |
| Measure | How good are the outputs, and are they drifting? | Golden set evaluation; acceptance rates; healing reviews |
| Manage | What happens when risk is too high? | Review gates, least privilege, rollback to manual |
What would you do?
Wei finds that a test analyst pasted a CMQ extract with real customer names and addresses into a consumer chat assistant to "quickly generate test data variations." The extract came from a copy refreshed before masking was automated. What should happen?
- Seven guardrails: accountability, traceability, data protection, evaluation, least privilege, transparency and AI literacy.
- Nothing AI-generated enters the regression pack without a named reviewer; model changes trigger re-evaluation.
- Measure AI quality with a golden set and acceptance criteria, before and after adoption.
- Lean on NIST AI RMF, ISO/IEC 42001, CT-GenAI and the EU AI Act as amended by the 2026 Digital Omnibus.
Check your understanding · not graded
Lesson check
1. Why does Contoso rerun its golden set when an AI tool changes its model?
2. What did the 2026 Digital Omnibus change about EU AI Act Article 4?
3. Which NIST AI RMF function covers golden-set evaluation of AI outputs?
4. What should an agent drafting test cases in SAP Cloud ALM never be allowed to do?