Service levels and metrics

A service without measures is a promise nobody can check. The data and environment services need a small set of service levels that requesters can plan around, and metrics that show whether the services are getting better. They also feed the TCoE scorecard and the value reporting covered in sap-tcoe-406.

The measures

Measure	Definition	Contoso baseline	Target after 12 months
Time to test-ready after refresh	Working days from copy complete to readiness check passed	6 days	2 days
Environment availability in test windows	Test-ready hours ÷ booked hours, per environment	Not measured	95% or better
Refreshes on schedule	Refreshes completed in their calendar slot ÷ refreshes planned	1 of 2	All
Data request lead time	Working days from request to test-ready data, by request type (median and 85th percentile)	About 4 days, all requests bespoke	Golden: immediate; recipe: 2 days; bespoke: 5 days
Data-caused failures	Failed tests whose root cause is data ÷ all failed tests	About 30%	Under 10%
Blocked hours from data and environments	Tagged blocked or repeated hours ÷ execution hours per release event	22%	8%
Masked before access	Refreshed copies masked and checked before testers logged on ÷ all refreshes	50%	100%, with no exceptions unapproved

Contoso's data and environment service measures. Baselines come from lesson 1; targets are agreed with Basis, integration and security.

Two of these deserve care. Lead time should be reported as a median and a high percentile, not an average, because a few bespoke requests can make the average meaningless; the 85th percentile tells requesters what they can count on. Data-caused failures need the root-cause tag on every failed test, which is also what separates real defects from noise in the regression report.

The records behind the measures. If any of these is not captured, the matching measure cannot be calculated.

The Sign-Off Table · thesignofftable.com/academy/sap-tcoe-405

Readiness reports as evidence

ISO/IEC/IEEE 29119-3 pairs requirements with readiness reports: test environment requirements with a test environment readiness report, and test data requirements with a test data readiness report. They are useful formats for the TCoE. Before each release regression, Priya issues one short readiness note per environment: the refresh date and source, the transport level, masking log reference, connected systems status, data recipes run and golden data checked, and any conditions. That note becomes part of the test basis statement in the evidence pack (sap-tst-307), so results always say what they were obtained on.

Day −15Refresh completeCopy, runbook and masking log filed
Day −13Readiness check passedSmoke tests per value stream; connected systems proven
Day −10Data readyRecipes run; golden data checked and reserved
Day −8Stack in CMRFeature pack stack imported; technical checks
Day 0Readiness note issuedEnvironment and data readiness for regression, with conditions
Day 0 to 10Regression windowBlocked hours tagged by cause; daily status

The readiness milestones before a stack regression in CMR. Days are counted back from the start of the window.

The Sign-Off Table · thesignofftable.com/academy/sap-tcoe-405

Reporting and improving

Report the measures monthly in the TCoE service review with Sam, Theo and Nadia, and quarterly to the quality council as part of the TCoE scorecard. Use them to decide investments: if data-caused failures stay high in one value stream, that is where the next recipes go; if time to test-ready is stuck, the runbook needs automation or the contract with SAP needs a different post-copy split. Publish the numbers to requesters too. A project that can see the data service meets its two-day level 90% of the time plans around it instead of creating its own data on the side.

What would you do?

The data service meets its two-day level for recipes 95% of the time, but data-caused failures in procure to pay are still 25%. What does this tell Priya?

Hands-on lab · ACCELQ

Run one test against two environments with App Environments

This lesson's readiness note says what a result was obtained on. In ACCELQ, App Environments and environment-specific Global Properties let one Scenario run against different systems, and the Test Report records which App Environment was used. You practice the same idea as running one regression in CMQ or CMR, using two public hosts of the same SAPUI5 demo app as stand-ins.

Practice on The SAPUI5 Shopping Cart demo and the same demo on OpenUI5 (both public, no login)

If you haven't set up ACCELQ yet, do the setup lab first.
As a Project Admin, open Project Administration and the Application Environments section. Create two environments, for example "Env-A" and "Env-B". If your trial user isn't a Project Admin, read along and use an existing environment pair.
In Resources, create a Global Property "Cart URL" marked as environment-specific, with the SAPUI5 address as its value for Env-A and the OpenUI5 address for Env-B.
Record a short read-only Scenario on the cart demo: open a product category, open any product you see, and verify in Intent mode that its detail page shows a price.
On the Scenario's Initialize step (Initiate Browser Session), point the URL to the "Cart URL" Global Property instead of a literal address.
Make the product category a Run Property so the Run modal asks for it, just as a tester would request data for a run. Enter any category you see in the app.
Click Run twice, once with Env-A and once with Env-B selected as the App Environment, with the same Purpose text plus the environment name.
Open both Test Reports and check the sidebar: each shows its App Environment. Write a two-line readiness note for each run: environment, date and the data value used.

Done when Two Test Reports from one unchanged Scenario, each showing a different App Environment, plus a short readiness note per run. Keep both report links with the notes.

Learn more in ACCELQ’s docs: Setting up Application Environments · Understanding Global Properties in ACCELQ · Run Properties

Start the free ACCELQ trial Matt Angerer, who runs The Sign-Off Table, works at ACCELQ. Disclosure

Key points

Measure time to test-ready, availability, refreshes on schedule, data lead time, data-caused failures, blocked hours and masking before access.
Report lead times as a median and a high percentile; tag every failed test with a root cause.
Issue an environment and data readiness note before each regression, after the readiness reports in ISO/IEC/IEEE 29119-3.
Pair speed measures with quality measures, and use the numbers to direct the next investment.

The measures

Readiness reports as evidence

Reporting and improving

Lesson check