The eight-scenario booking reliability test to run before you buy any AI phone agent

A demo call shows a vendor at their best. A reliability test shows what happens on the days that go wrong. This test is short enough to run in an afternoon with a colleague, and it works on any AI receptionist, AI answering service or AI phone agent, including ours once it exists. Notaray is in development at Agnotiq and has no measured results to show yet, so we publish the test we intend to be held to.

Why run a test before you buy?

Every vendor can book a simple appointment on a quiet line. The costly failures are the ones a demo skips: a caller told “you are booked” when nothing is on the calendar, two people given one slot, a callback that nobody owns. The US Federal Trade Commission has said there is no AI exemption from the laws against deceptive practices, so a claim of reliability is something you can ask a vendor to demonstrate.

For every scenario below, judge the result by the calendar and by what the caller was told, in that order. Guide one in this series explains how to check a booking.

What are the eight scenarios?

1. A normal booking. Ask for a service you offer, on a day you have room. Pass mark: an event with the right time, name, contact and service, and a confirmation sent after the event exists.
2. A taken slot. Fill a slot yourself, then ask for it by phone. Pass mark: the caller is offered other times or a person, and nothing is double-booked.
3. A request outside your hours or your services. Ask for a time you are closed or a service you do not offer. Pass mark: a clear, polite no with an alternative, and no event.
4. A change of mind. Book, then say “actually, make it Thursday” before the call ends. Pass mark: one event on Thursday, none left behind on the original day.
5. A repeated request. Call back with the same request, or send it twice. Pass mark: the second attempt returns the original outcome and creates no second event.
6. A calendar that fails. Ask the vendor to simulate a calendar outage or a slow reply. Pass mark: the caller hears that the request is pending, a named person gets the details with a deadline, and nobody hears “booked”.
7. A caller who hangs up mid-call. End the call after the agent offers a time but before you accept. Pass mark: no event, and a record that the request was left open.
8. A caller who speaks another language. Call in French or Spanish if your customers do. Pass mark: the same standard on every earlier scenario, in that language.

Score each scenario as pass, fail or not shown. A vendor who cannot demonstrate a scenario has told you something too.

What should you ask the vendor after the test?

Ask to see the record for each of your eight calls: the request, the time the calendar answered, the event reference, what the caller was told and the final state. If the vendor can show you a record for the successes alone, you are looking at a summary and not a log.

Ask how failures are handled, who is notified and how fast. An uncertain result should always reach a named person with a deadline. Notaray is being built so that a booking it cannot confirm becomes a callback with an owner and a deadline.

Where is the printable checklist?

The checklist is a one-page printable version of the eight scenarios with space to record each result. It comes by email after you sign up on our updates page with Get notified. The summary above is complete without it, and you can run the whole test from the summary.

Frequently asked questions

How long does the reliability test take?

About an afternoon for the eight calls, plus time to open the calendar after each one.

Do I need to be technical to run it?

No. You need a phone, your calendar and a colleague. The pass marks are written in terms of what you can see.

Can I run it on a vendor’s free trial?

Yes, if the trial connects to a calendar. Use a test calendar or clearly marked test events, and remove them afterwards.

Does Notaray pass this test today?

We do not have measured results. Notaray is in development, and we will publish results, with denominators and a period, when they exist.