Lesson 7 of 7
Limits, failure modes and governance
11 min
Listen to this lesson
Both worked examples ended well because people caught what the agent got wrong: a field name it made up, a Prepare entry it forgot, a test that tested nothing, a signature change that would have broken unseen callers, an API it couldn't name. None of these is rare. This lesson collects the failure modes you will meet with Claude Code on ABAP, explains why generated code still needs review, and ends with the approval pack Contoso's security, Basis and audit teams signed.
Failure modes and how each one surfaces
| Failure mode | What it looks like in ABAP | What catches it |
|---|---|---|
| Hallucinated API | A class, method, function module or CDS view that doesn't exist, or exists with other parameters | Activation or syntax check; a search for the object; the reviewer asking "where did this come from?" |
| Hallucinated field or syntax | CustomerID instead of Customer; BDEF syntax from an older RAP version | Activation errors (deterministic); Claude usually fixes these itself |
| Unreleased object | A classic function module or SAP table read in new code | ABAP Cloud packages: syntax error. Classic packages: nothing at activation; ATC, clean core runs and review |
| Locks | "Object locked in request ..."; an editor lock while someone has the object open; inactive versions left after a failed activation | The error itself; SM12 for editor locks; a person coordinates |
| Scope creep | Reformatted methods, a "while I was here" fix, a changed access control | Unified diff of the request; pull request diff; ask rules on edits |
| Test tampering | Existing assertions changed to match new behavior; tests that pass without the logic | CLAUDE.md rule plus review; break-the-behavior check |
| Prompt injection | A comment or long text that reads like an instruction, and the agent follows it | Treat content as data; deny rules; human approval; least privilege |
| Truncated context | A large class or ATC result cut off, so the agent reasons from part of it | Claude Code warns above 10,000 tokens of MCP output and saves results over 25,000 tokens to a file; ask for narrower reads |
| Tooling drift | Renamed MCP tools after an IDE update break permission rules and hooks | Recheck settings after each ADT release; SAP renamed the tools once already, in version 1.2 |
What goes wrong with an agent on ABAP, and the check that finds it.
Hallucinated APIs and unreleased objects
Models write the ABAP they have seen most, and most public ABAP is classic: function modules, direct reads of SAP tables, statements ABAP Cloud forbids. So two different errors look alike. A hallucinated API doesn't exist; activation catches it. An unreleased object exists and works today, but SAP doesn't promise it will survive an upgrade. In ZCMC_CLOUD the ABAP Cloud language version turns unreleased use into a syntax error. In classic packages it compiles and runs, which is exactly why it's dangerous.
Make the agent show its sources. Contoso's CLAUDE.md requires a search with abap_repository_search_object before any SAP object is used, and the reviewer checks the API state of each new SAP object in ADT (the API State tab in the Properties view) or in SAP's Cloudification Repository on GitHub, which lists released objects and successors. An agent that can't name the object it wants to call has probably invented it.
Why generated code still needs review
- It is plausible by construction. It compiles, reads well and follows naming rules, which makes shallow reviews feel safe.
- The same agent wrote the code and the tests. If it misunderstood the rule, both are wrong in the same way, and the tests pass.
- It isn't repeatable. The same prompt can give a different change tomorrow, so the reviewed diff is the only record of what was approved.
- It acts through tools. A mistake can reach objects nobody asked about, which only the diff of the whole request shows.
- Accountability stays with people. The developer who pushes and the reviewers who approve vouch for the change; the agent can't.
Review agent changes with the checklist from sap-aid-204 (scope, authorization logic, data access, hard-coded values, tests, metadata) and add two agent-specific questions: did a person confirm the business rule, and does every SAP object the change uses exist and have a release state you accept?
Governance: the approval pack
Before the pilot widened from Kenji to his whole team, Nadia (security), Sam (Basis) and Luis (release) asked for one pack they could review in an hour. Contoso's organization-wide policy, tiers and audit trail are in sap-aid-302; this pack documents one repository's setup against it:
- Route register: Route A (ADT MCP server, VS Code version) and Route B (community abapGit in CMD); no community bridges; the smoke-test result that shows which ADT tools work on CMD.
- Configuration files:
.mcp.json(token as a variable),.claude/settings.json, the hook scripts, the reviewer subagent and CLAUDE.md, all in the repository so any auditor can read them. - Identity and access: the agent users, role ZCMC_AGENT_DEV, ADT access restrictions (project and package), and the absence of CMQ and CMP destinations on agent laptops.
- Compliance check: the API Policy questions per route from lesson 1, with counsel's answer and date, and the license position for SAP's MCP tool support.
- Data: Anthropic's commercial terms (no training on code or prompts sent under commercial terms unless the customer opts in; 30-day standard retention), local transcripts under
~/.claude/projectswith a shortenedcleanupPeriodDays, feeds denied. - Evidence: five pilot changes with their requests, diffs, pull requests, ATC and ABAP Unit results, and one sampled audit trail.
Measure the pilot honestly. Contoso tracks four numbers per agent change: time from ticket to reviewed pull request, review rounds, defects found in CMQ, and changes reverted. If agent changes need more review rounds or produce more CMQ defects than the team's baseline, the setup isn't ready to widen, whatever the time saved.
What would you do?
Grace, the PMO lead, has seen the pilot numbers and asks Kenji to let Claude Code work through the 340 open ATC findings in ZCMC_SD over a weekend, unattended, with bypassPermissions mode "so it doesn't stop for prompts". What should Kenji recommend?
- Expect hallucinated APIs and fields, unreleased objects, locks, scope creep, test tampering, injection and truncated context.
- ABAP Cloud catches unreleased use at activation; classic packages need ATC, clean core runs and review.
- Make the agent show where each SAP object came from; check its API state.
- Review is still required because agent code is plausible, self-tested and not repeatable.
- Document the setup in one approval pack and widen it only on measured quality, not time saved alone.
Check your understanding · not graded
Lesson check
1. In a classic package such as ZCMC_SD, what happens when an agent adds a call to an unreleased SAP function module?
2. Why can't passing ABAP Unit tests written by the same agent stand in for review?
3. Claude Code warns that an MCP result was large and saved it to a file. What risk does this point to?
4. Which item belongs in the approval pack for one repository's Claude Code setup?