Now accepting guest editorial from SAP, Oracle, Workday and ServiceNow practitioners.Pitch a guide
The Sign-Off Table

Lesson 6 of 6

Testing tools and releasing them

9 min

Listen to this lesson

0:00 / 6:17

An MCP server has two audiences that both need testing. The protocol audience is any MCP client: it must connect, authenticate, list tools and get valid results. The model audience is the AI that reads your descriptions and decides what to call. A server can pass every protocol test and still fail the model test, because the agent keeps choosing the wrong tool. Priya, Contoso's test lead, plans both before anything moves from CMC_TEST to CMC_PROD.

Connect a real MCP client

Use the AI application your users will use, or a developer client, against the test tenant only. For example, Claude Code can register a remote server with claude mcp add --transport http <name> <url>. For servers that don't support dynamic client registration, which is the case when credentials come from a Developer Hub subscription, Claude Code documents pre-configured credentials: --client-id, --client-secret (which prompts for the secret and stores it outside the config file) and --callback-port, whose http://localhost:PORT/callback redirect you register as the subscription's callback URL. Inside Claude Code, /mcp shows server status and runs the OAuth sign-in. Whatever client you use, never point it at production with test credentials, and never share a secret in a chat or ticket.

Priya's test connection: from subscription to a first tool call in the test tenant.
The Sign-Off Table · thesignofftable.com/academy/sap-pim-302

What to test

AreaTestExpected
DiscoveryList tools, resources and promptsExactly the curated set, with the intended names and descriptions; no hidden attributes leak
SchemaCall each tool with valid, missing and wrong-type inputsValid calls succeed; invalid inputs are rejected with a clear error
AuthenticationCall with no token, an expired token and a token from another IdPHTTP 401 each time; nothing reaches the backend
AuthorizationCall as a user without the scope, and as a user outside sales area 1710/10/00HTTP 403 from the policy; or S/4HANA returns only data the user may see
TrafficExceed Surge Protection and Quota deliberately in testHTTP 429 with the documented behavior; the backend is not flooded
Model behaviorRun 20 to 30 realistic user questions through the agentThe right tool is chosen with correct parameters in an agreed share of cases; failures analyzed
SynchronizationChange the source API in test and synchronizeRefinements kept; newly required attributes reappear; regression set rerun

Test areas for an MCP server before release (Contoso plan).

The model-behavior test deserves care. Write the questions in the users' own words, including awkward ones ("what's going on with the Great Lakes order from last Tuesday?"). Record which tool the agent called and with which parameters, not just whether the final answer looked right. When the agent picks wrongly, fix the server first: a sharper description or a hidden parameter usually helps more than any instruction to users.

  1. Build in CMC_DEVDeveloper changes tools, descriptions or policies and deploys to the development tenant
  2. Transport to CMC_TESTContent Transport Management imports automatically into the test tenant
  3. TestProtocol, security, traffic and model-behavior tests; evidence stored with the release
  4. Sign-offPriya signs the test result; Nadia signs security; Maria accepts the model-behavior results
  5. Forward to CMC_PRODPriya forwards after sign-off; Luis imports in the approved release window
  6. HypercareWatch Health and policy-error charts daily for two weeks
Release flow for an MCP server change, with Priya's gate before production.
The Sign-Off Table · thesignofftable.com/academy/sap-pim-302

Keep the design reviewable

Record, for each MCP server, the API Policy version you reviewed against, each tool's backing Published API, the identity model, the policy values and the test evidence. When SAP ships a roadmap feature, such as composition across APIs, you will revisit designs; when the API Policy is revised, you will re-review. A short design record makes both cheap.

  1. Each releaseAPI Policy versionConfirm the current version and section numbering on SAP Help Portal
  2. Each releaseWhat's NewCheck MCP Gateway features and limits, such as the tool limit and supported MCP specification versions
  3. QuarterlyRoad Map ExplorerTrack composition across APIs, the AI Gateway and federation
  4. QuarterlyService plan and regionCheck SAP Note 2903776 and SAP Note 3769634 if you add tenants or regions
Things to re-verify at every release, because they change faster than the course.
The Sign-Off Table · thesignofftable.com/academy/sap-pim-302

What would you do?

Testing shows the agent answers order-status questions correctly 92% of the time; in the rest it calls <code>list_open_orders_for_customer</code> with the order number as a customer number. The release window is tomorrow. What should Priya do?

Key points
  • Test both audiences: the protocol (connect, authenticate, list, call) and the model (does the agent choose and fill tools correctly?).
  • Use a real MCP client against the test tenant with Developer Hub credentials; never test against production.
  • Negative tests for 401, 403 and 429 are part of the release evidence.
  • Keep a design record per server and re-verify the API Policy, What's New and the roadmap at every release.

Check your understanding · not graded

Lesson check

  1. 1. Why test model behavior separately from protocol behavior?

  2. 2. In Claude Code, what does the --callback-port option help with when connecting to a Developer Hub subscription?

  3. 3. Which result shows the Quota policy works in a test?

  4. 4. Who forwards an MCP server change from CMC_TEST to CMC_PROD at Contoso?