Lesson 4 of 5
Performance tests
9 min
Listen to this lesson
Contoso nearly found the fleet rebate problem too late. It surfaced only because Priya's team timed the batch chain after the second mock conversion. A performance problem found in a dress rehearsal threatens the cutover; one found after go-live threatens the business. The answer is to treat performance as an acceptance criterion for critical custom code, tested like any other requirement.
Which code needs a performance test
- Batch jobs in a time window: the nightly chain must finish before shipping starts; month-end jobs before close deadlines.
- High-frequency dialog steps: order entry, delivery processing, anything Maria's team does hundreds of times a day.
- Interfaces with volume or latency requirements: inbound orders from fleet customers, outbound invoices.
- Code that runs during cutover: conversion follow-up reports and data loads that must fit in the 60-hour window.
- Anything changed by a pushdown or remediation, to prove it got faster and stayed correct.
Writing a performance acceptance criterion
A useful criterion names the operation, the data volume, the environment and the threshold, and says how it will be measured. "Should be fast" cannot be tested. Compare these for the fleet rebate run:
| Weak | Testable |
|---|---|
| The rebate job should run quickly | The fleet rebate run completes in under 30 minutes in CMQ for 120,000 open items and 40 fleet customers, measured as job runtime over three runs |
| Order entry must not be slower than ECC | Saving a 20-item order for Great Lakes Fleet Services in sales area 1710 / 10 / 00 completes in under 2 seconds server time at the 90th percentile in CMQ, compared with the EP1 baseline |
| The deposit report must perform | The open deposits report for company code 1710 returns in under 5 seconds and issues no more than 10 database statements, measured with an SQL trace |
Performance criteria for Contoso. Thresholds come from the business window and the ECC baseline, not from guesses.
- Before conversionBaseline in ECCRecord runtimes of critical jobs and transactions in EP1 and from job logs
- Mock conversion 1First comparisonTime the batch chain and critical steps in CMQ; list regressions
- Between mocksFix and unit-testPushdown and anti-pattern fixes, with unit tests proving unchanged results
- Mock conversion 2Performance acceptanceRe-run against criteria with production-like data; record evidence
- Dress rehearsalWindow checkThe batch chain and cutover jobs fit their windows end to end
Making measurements trustworthy
- Production-like data: volumes and distributions close to production. After a conversion, CMQ built from a production copy is ideal.
- Repeat runs: measure at least three times and report the median or a percentile; the first run often pays for cold caches.
- Stable conditions: agree a window with Sam so other load does not distort the result, and note what else was running.
- Measure the right thing: for batch, total job runtime; for dialog, server response time; for database work, statements and database time from a trace.
- Check correctness too: a faster job that produces different totals is a defect. Compare results with the baseline.
- Does the change touch a critical job, transaction or interface?Yes → performance criterion required · No → normal regression only
- Is there a baseline from ECC or a previous release?Yes → compare against it · No → measure now and set the threshold from the business window
- Did the measured runs meet the threshold and match results?Yes → attach evidence to the feature · No → fix, re-measure, or escalate the risk to Grace
What would you do?
Kenji's team optimized the fleet rebate run. In CMQ it now takes 18 minutes, against a 30-minute criterion. Priya notices the rebate total for one fleet customer differs by $212.40 from the ECC baseline. What should happen?
- Treat performance as an acceptance criterion for critical jobs, dialog steps, interfaces and cutover code.
- A testable criterion names operation, volume, environment, threshold and measurement method.
- Baseline in ECC, compare after each mock, measure repeatedly with production-like data.
- Faster but different results is a defect: reconcile outputs as part of the test.
Check your understanding · not graded
Lesson check
1. Which is a testable performance criterion?
2. Why measure a job at least three times?
3. Where should Contoso get the baseline for comparing performance after conversion?
4. An optimized job meets its time threshold but produces different totals. What is it?