Skip to main content
Free PDF · Checklist

AI Agent Evaluation Checklist

Test one task against an AI agent on your own data, with the accuracy bar set before you start.

What You Get

The AI Agent Evaluation Checklist is a three-page PDF for testing one business task against an AI agent before you commit to it. It follows our guide on how to evaluate an AI agent for your business, step by step, with space to write your answers down.

What Is Inside

  • A task definition box, with the hours spent today and the cost of one mistake
  • Seven tick-box tests of whether the task suits an agent or a simpler automation
  • A place to write the accuracy threshold before testing, against the human baseline
  • What to record when you test on real data: the type of mistake, whether the agent flags uncertainty, and whether errors can be put right
  • The four oversight models, with a box for who does the reviewing
  • Six cost lines, most of which never appear on the API bill
  • A comparison with rule-based automation and with a person doing the job

Who This Is For

Operations leads, founders and team leads who have one task in mind, such as sorting support emails, pulling data out of documents or drafting replies, and want to know whether an agent would earn its cost. If you are still choosing between agents and fixed-rule automation in general, read AI agents vs traditional automation first.

How to Use This

Use one copy per task. Fill in the task definition and the threshold before anyone runs a test, because a bar set afterwards moves to wherever the results landed. Then run the agent on a representative sample of real inputs, not the examples in a demo, and record the mistakes as well as the score. The final page asks you to compare the agent with the two alternatives. For some volumes and tasks, the person wins.

Why We Made This

We build agents for clients, and the projects that work start with a narrow task, a written threshold and a plan for the errors. This checklist sets those down. It may tell you not to use an agent at all, and that is a useful answer. If it points the other way, the AI agents page covers what we build, or you can get in touch to talk it through.

Get the Checklist

Fill in the short form and the PDF opens straight away. We email you the link too. Everything else we have published is in the resource library.

Portrait of Alexander De Sousa, founder of Digital Royalty
Founder-led
“I’ve put everything I know into how this company works — the standards, the method, the care on every project. It runs through the whole team, and I hold us all to it.”

Alexander De Sousa · Founder LinkedIn

Featured on BBC Radio Solent

Get started

Tell us what you need

A few quick questions, then a straight answer from a real person — usually within a few hours.

What happens next

  1. 1 We reply A real person reads your answers and gets back to you, usually within a few hours.
  2. 2 Questions or a short call If anything isn't clear, we ask by email or book a short call. Before any call we audit your website and make the quick fixes it finds, free.
  3. 3 Your fixed quote A clear, fixed price in writing, within 48 hours of any call. There is no obligation to go ahead.

Tell us what you're working on

Whether it's a new site, a platform, or a process that shouldn't be manual any more — we'll tell you honestly if we can help.