Week 7 of 8
AI agents and Otto features
In August 2026 ServiceNow renamed its Now Assist branding to ServiceNow Otto. Whatever the label, generative features and AI agents are in scope for upgrade testing, and they do not behave like a business rule. The same request can produce different results.
AI Agent Studio separates manual tests, which run a single execution, from automated agentic evaluations, which score many executions on task completeness, tool calling correctness and tool choice accuracy. Generated evaluation runs execute real workflows and change real records.
This week: give every agent an evaluation, not a demo
- List every AI agent and agentic workflow in use or in pilot, with its tools, the records it can change and its owner.
- Write typical and edge-case requests for each. Include requests the agent should refuse.
- Run manual tests as a user with the agent’s real roles, then automated evaluations at a meaningful sample size.
- Run evaluations only in sub-production. Generated runs modify records.
- Repeat the evaluation after each family upgrade and each AI app update, and compare scores before you sign off.
Number of the week
100–300
samples ServiceNow suggests for a production-readiness agentic evaluation, against 10–50 for initial testing.
Source: ServiceNow Community, Agentic Evaluations (December 2025)Use this week
- AI Agent Studio testing (ServiceNow docs) →
How manual tests and automated evaluations differ.
- Release Regression Plan →
Add AI agents as their own scope line in every release.
Next week, the last of the playbook: the production upgrade and the sign-off.
Get the playbook in your inbox, one week at a time
Subscribe free and every template in the library is yours.