A selection framework and a fair proof of concept

Tool selection goes wrong in a predictable way: two teams see two impressive demos, each picks its favorite, and the decision becomes political. A selection framework prevents that. It fixes the requirements, weights and knock-out criteria before anyone sees a demo, and it replaces demos with a proof of concept on your own system with your own scenarios.

Start from requirements, not features

CTAL-TM v3.0 recommends evaluating tools against objective criteria and ending the evaluation with a proof of concept, then rolling out through a pilot with a named tool owner and administrator. It distinguishes commercial, open-source and custom tools, and advises a long-term tool strategy. For an SAP TCoE, the requirements come from the regression pack and the operating model:

Technology reach: SAP Fiori and UI5 apps, SAP GUI for HTML (WebGUI), SAP GUI for Windows, BTP apps, OData and SOAP APIs, IDoc-driven flows, and the non-SAP systems in Contoso's end-to-end processes.
Maintainability across SAP releases: how tests survive a feature pack stack update, UI5 changes and browser updates.
Authoring and readability: who will build tests (engineers, test analysts, key users) and who must be able to review them.
Integration: the SAP Cloud ALM Test Automation API, CI/CD, defect tracking.
Execution: running inside Contoso's network against CMQ without opening inbound firewall ports; parallel runs.
Evidence: reports that meet the TCoE evidence standard without manual assembly.
Security: credential handling, roles, audit trail of test changes.
Commercials and viability: license model, total cost, vendor roadmap, support, and an exit path.

Weights and knock-outs

Criterion	Weight	Knock-out?	How it is scored
SAP technology reach (Fiori, WebGUI, SAP GUI, API)	20	Yes: must cover Fiori and SAP GUI	PoC scenarios built and passing
Maintainability across a change event	20	No	Hours to repair after the PoC change event
Readability for key users	10	No	Key users review two tests and score them
SAP Cloud ALM integration	10	Yes: results must reach Cloud ALM	Shown working in the PoC
Run stability	10	No	False failures in 10 unchanged runs
Execution behind the firewall and CI/CD	10	Yes: no inbound ports to CMQ	Architecture review plus PoC
Evidence quality	10	No	Sample checked against the evidence standard
Five-year total cost of ownership	10	No	TCO model (lesson 6)

Contoso's weighted criteria (total 100). Knock-outs are pass or fail before any scoring.

Weeks 1–2Requirements and weightsAgreed with test architects, Ravi, Theo, Nadia and two key users before any vendor contact
Week 3Knock-outsOptions that cannot reach Fiori and SAP GUI, Cloud ALM or run behind the firewall drop out
Weeks 4–7Proof of conceptFive scenarios, a change event and ten unchanged runs per option
Week 8Scoring and TCOScore sheet and five-year model completed with PoC measurements
Weeks 9–10Decision and pilot planQuality council decides; pilot booked on the next release

A ten-week selection at Contoso. Criteria are fixed before the first demo, so the PoC tests them rather than inventing them.

The Sign-Off Table · thesignofftable.com/academy/sap-tcoe-404

A fair proof of concept

A proof of concept is fair when every option builds the same scenarios on the same system with the same data in the same timebox, and is scored by the same people against the same sheet. Contoso's protocol uses five scenarios that cover its technology mix:

#	Scenario	Technology
1	Sales order to billing for CMC-ALT-100, sales area 1710/10/00, with the billing document number captured and verified	Fiori apps in the launchpad
2	Goods receipt for a purchase order from Lakeside Components	SAP GUI for Windows
3	A custom transaction run in SAP GUI for HTML	WebGUI
4	Read and verify a sales order through an OData service	API
5	Core return agreement created in the BTP extension app	UI5 on BTP

Contoso's PoC scenarios. The same five go to every option.

The most important part of the protocol is the change event. Building a test is a one-time cost; keeping it alive is a cost every release. Halfway through the PoC, Contoso applies a realistic change in a sandbox client (a field added to the sales order app, a renamed label, an extra confirmation dialog), then measures how many hours each option needs to get all five scenarios green again. Then each option runs the suite ten times without changes; any failure not caused by the application counts against run stability.

Contoso's proof-of-concept protocol. The change event and repeat runs measure what demos never show: maintenance and stability.

The Sign-Off Table · thesignofftable.com/academy/sap-tcoe-404

Hands-on lab · ACCELQ

Run one proof-of-concept task and score it

A fair PoC measures build hours, readability and stability, not demo polish. In this lab you run one PoC-style task in ACCELQ on a public SAPUI5 app and fill in three lines of the lesson's score sheet with your own measurements.

Practice on SAPUI5 Shopping Cart demo app (public, no login)

If you haven't set up ACCELQ yet, do the setup lab first. Start a stopwatch. Create a Context for the cart app and an Action "Add product to cart" with input parameters for the category and product.
Record the steps with the Web Recorder in Fluid mode, then use Intent mode to add Verify Element Exists for the cart entry. Replace the hard-coded values with Action Parameters.
Build a Scenario starting with Initiate Browser Session on the app URL. In the Test Cases tab, add three rows with different categories and products you can see in the app. Stop the stopwatch when all three pass: that is your build time.
Notice how each element was identified. Open one statement's element options and compare Locator-Free with Smart Locator; note which you would standardize for UI5 lists and why.
Run the Scenario five times without changes from the Run modal on your Local Agent. Count any failure that was not caused by the app: that is your false-failure count.
Show the Scenario to a colleague who does not code. Ask them to explain what it tests from the statements alone, and score readability 1 to 5.
Fill in three score-sheet lines: build hours for one data-driven scenario, false failures in five runs, readability score.

Done when All three Test Cases pass, you have run the suite five times, and you have three measured score-sheet lines. Keep the Test Report links with the sheet; they are the evidence behind the scores.

Learn more in ACCELQ’s docs: Working with Test Cases in ACCELQ · Locator-Free and Smart Locator · Recording Fundamentals

Start the free ACCELQ trial Matt Angerer, who runs The Sign-Off Table, works at ACCELQ. Disclosure

What would you do?

Two Contoso teams each saw a vendor demo and want different tools. Ravi suggests letting each vendor run its own best demo on Contoso's system and choosing the most impressive. What should Priya do?

Key points

Fix requirements, weights and knock-outs before any demo; CTAL-TM ends evaluation with a proof of concept and a pilot.
Score SAP technology reach, maintainability, readability, Cloud ALM integration, stability, execution, evidence and TCO.
A fair PoC uses the same scenarios, system, data, timebox and scorers for every option.
Include a change event and repeat runs: they measure maintenance and stability, which demos never show.

Start from requirements, not features

Weights and knock-outs

A fair proof of concept

Lesson check