Seven Questions to Ask an AI Vendor Before You Buy
Claimed: [THESIS] This page claims that seven questions, asked in order and scored on the kind of evidence offered, give a buyer a better view of a vendor than a demo alone.
Tested: [OBSERVED] What was actually tested: nothing. No buyer was surveyed, no vendor was put through this checklist, and no purchase decision has been compared against the results of using it. The checklist was designed at a desk. [UNKNOWN] Whether these seven are the seven best for every buyer is not measured. They are our best argued selection, and the closing question on this page asks you to tell us which one is missing. [UNKNOWN] Whether scoring a vendor with this worksheet predicts a better outcome than not scoring them is not measured. [THESIS] Because nothing was tested, the correct status for this whole page is a guide and an argument, not a finding.
Free page. No signup, no email, no gate of any kind.
Part 1. BLUF
[THESIS] This page gives you seven questions to put to any vendor who sells you an AI system that does work on its own, plus a printable worksheet to score the answers.
[THESIS] It is for practical buyers and operators, from a one-person business to a large enterprise, who have to decide whether a thing that "finished" has actually finished.
[THESIS] It does not cover model quality benchmarks, legal contract terms, security certification or pricing comparison, and it is not an audit.
Part 2. The claim and the stakes
What appears true.
[THESIS] A modern AI system can produce a convincing report that a task is complete: a green tick, a summary, a log with no errors in it.
[THESIS] That report is real. It is also written by the same system whose work is in question.
The missing condition.
[THESIS] The missing condition is an acceptance test that was agreed before the work started and checked by something other than the thing that did the work.
[THESIS] Without it, "finished" means "the system believes it finished", which is a statement about the system and not about your outcome.
Why it costs you.
[THESIS] The cost shows up later and somewhere else: a wrong figure in a quote, a customer emailed twice, a record changed that nobody noticed, a staff member quietly redoing the work and never saying so.
[THESIS] The expensive failures are not the loud ones, which get caught on day one. They are the quiet ones that look like success and are paid for in rework, trust and time.
[UNKNOWN] We have not measured how often AI systems report completion that turns out to be wrong across the market. We do not cite a rate, and we do not need one for the argument: one quiet failure that reaches a customer is enough to justify asking.
The sharpest defensible contradiction.
[THESIS] The usual way to buy this kind of software is a demo and a sales deck. The better way is an acceptance test written first. The demo shows what the vendor chose to show; the acceptance test shows whether it does what you need on your work.
Part 3. What was claimed versus what was actually tested
[THESIS] This page claims that seven questions, asked in order and scored on the kind of evidence offered, give a buyer a better view of a vendor than a demo alone.
[OBSERVED] What was actually tested: nothing. No buyer was surveyed, no vendor was put through this checklist, and no purchase decision has been compared against the results of using it. The checklist was designed at a desk.
[UNKNOWN] Whether these seven are the seven best for every buyer is not measured. They are our best argued selection, and the closing question on this page asks you to tell us which one is missing.
[UNKNOWN] Whether scoring a vendor with this worksheet predicts a better outcome than not scoring them is not measured.
[THESIS] Because nothing was tested, the correct status for this whole page is a guide and an argument, not a finding.
Part 4. The method: a worked exercise
How to run it.
- [THESIS] Pick one real workflow you are thinking of handing to an AI system. Not "our operations". One workflow, such as matching supplier invoices to purchase orders.
- [THESIS] Decide the risk tier (the three tiers are defined in Part 8). The tier decides how hard you push and when you should stop using a checklist and pay for an independent assessment.
- [THESIS] Ask the seven questions below, in this order, in writing if you can. Written answers are harder to walk back than spoken ones.
- [THESIS] For each answer, note two things: the score from 0 to 3, and the evidence rung (A, B, C or D, defined below). A confident answer with no evidence is rung A, however good it sounds.
- [THESIS] Apply the decision rules at the bottom of the worksheet. Sign it. Keep it with the contract.
The evidence rungs (our own scale for this checklist, not a standard).
- [THESIS] Rung A: the vendor said it.
- [THESIS] Rung B: the vendor showed you a document, screenshot or recording they made.
- [THESIS] Rung C: the vendor showed it working on a sample of your own work, which you chose and they had not seen.
- [THESIS] Rung D: someone independent of the vendor showed it, such as a reference customer you found yourself, or a paid assessor.
The seven questions, with how to read the answers.
Question 1. "What exactly counts as finished, and who decides?"
[THESIS] This is the question the other six depend on. If "finished" is not defined in a form that a stranger could check, nothing after it can be tested.
- Good answer: [THESIS] A written definition of done for your workflow, in terms of the outcome (the right invoice is matched to the right order, with the amount within a stated tolerance) and not in terms of activity (the process ran, the job exited cleanly). It names who signs off, and it is agreed before work starts.
- Bad or evasive answer: [THESIS] "It will tell you when it is done." "Our customers are very happy." "That depends on the use case" with no follow-up offer to define it for yours. A tick on a dashboard offered as the definition.
- Why it is hard to fake: [THESIS] A real definition of done forces the vendor to commit to something that can later be shown wrong. A fake one stays vague, because vague cannot fail.
- Where it matters most: [THESIS] All tiers. It is the cheapest question to ask and the most expensive to skip.
Question 2. "Show me the outcome, not the log. Who checks it, and is it anyone but the system?"
[THESIS] A system's own report of success is self-attestation. A check performed by a separate step, a human sample, or a count against a source of truth is verification.
- Good answer: [THESIS] A named verification step that does not rely on the agent's own report: a reconciliation against your records, a human spot check at a stated sample rate, a second system with different access. The vendor can show you the output of that check from a real run.
- Bad or evasive answer: [THESIS] "The agent confirms its own work." "We have logging." "It has never reported an error." Logs prove the process ran, not that the result is right.
- Why it is hard to fake: [THESIS] Ask to see a check that caught something. A verification step that has never caught anything is either unneeded or not really running.
- Where it matters most: [THESIS] Material and high stakes, where an unverified wrong result becomes money, a customer message or a record that others rely on.
Question 3. "What did you actually test, on what data, and when?"
[THESIS] A demo and a deployment are different things. A demo runs on prepared inputs on a good day. A deployment meets your messy, odd, out-of-date real work.
- Good answer: [THESIS] A specific statement: the task, the sample size, where the sample came from, the date, the software version, and what was not tested. Ideally an offer to run your own sample before you sign.
- Bad or evasive answer: [THESIS] "Extensively tested." "Thousands of runs" with no description of the runs. A benchmark with no date or version. Results on the vendor's data only.
- Why it is hard to fake: [THESIS] Specificity is expensive. A vendor who did the testing can say what they ran and when. A vendor who did not will stay general, and will resist running your sample.
- Where it matters most: [THESIS] Material and high stakes, and any time your data is unusual (your own formats, your own jargon, your own edge cases).
Question 4. "Tell me about a time it failed. What happened, and what did you change?"
[THESIS] Every real system has failed in some way. A vendor who cannot describe a failure has either not looked, not run it long enough, or is choosing not to say.
- Good answer: [THESIS] A concrete, dated failure told plainly: what went wrong, how it was noticed, who was affected, what was fixed, and what check now stops it happening again. Some discomfort in the telling is a good sign.
- Bad or evasive answer: [THESIS] "We haven't had any failures." "Nothing material." A failure described only as something the customer did wrong. A story with no change made afterwards.
- Why it is hard to fake: [THESIS] An invented failure story tends to be tidy and to end well. A real one has loose ends and a specific fix. Ask a follow-up about the fix, and a made-up story runs out quickly.
- Where it matters most: [THESIS] All tiers, and especially high stakes, where how a vendor behaves after a failure is part of what you are buying.
Question 5. "When it is wrong, what is the worst it can do, and how do I stop it?"
[THESIS] This question is about the blast radius: what the system can touch, what it can change, and whether you can undo it.
- Good answer: [THESIS] A plain list of what the system can read and write, a statement of the worst realistic outcome, the controls that limit it (least access, approval steps for anything irreversible, a spend or volume limit), a stop switch you control, and a way to roll back. Ideally they have rehearsed it.
- Bad or evasive answer: [THESIS] "It can't go wrong." "Our model is very safe." Broad access granted by default. No stop switch, or a stop that needs the vendor's help. Rollback described as "restore from backup" with no evidence it has been tried.
- Why it is hard to fake: [THESIS] Ask them to demonstrate the stop, on a test copy, while you watch. A control that exists only as a sentence cannot be demonstrated.
- Where it matters most: [THESIS] High stakes first, then material. In the low tier, with reversible and cheap actions, a lighter answer can be enough.
Question 6. "What does one accepted result cost me, all in?"
[THESIS] The price on the quote is not the price of your outcome. The number that matters is the total cost divided by the results you actually accept.
- Good answer: [THESIS] The vendor works through the cost with you, including the subscription or usage fee, your own review time, the rework on results that fail, and any extra checking you will need. They accept that cost per accepted result is the fair comparison, and they are willing to measure it on a trial.
- Bad or evasive answer: [THESIS] Only a per-run or per-seat price. "It pays for itself" with no arithmetic. Savings stated as a percentage with no baseline, no date and no method.
- Why it is hard to fake: [THESIS] A vendor cannot honestly state your cost per accepted result without your accepted-result rate, which only a trial on your work can produce. Anyone who quotes it before the trial is guessing.
- Where it matters most: [THESIS] Material tier, where the money is real but the decision is still yours to make with a spreadsheet.
Question 7. "Where should I not use this, and what do I keep if I leave?"
[THESIS] The best signal of an honest vendor is that they can tell you where their own product is the wrong answer. The second part is exit: your data, your configuration and your freedom to go.
- Good answer: [THESIS] A real list of poor fits, including cases where a simple rule-based automation or a person would do the job better or more safely. A clear account of what you can export on exit, in what format, how fast, and what is deleted and when.
- Bad or evasive answer: [THESIS] "It works for everything." Nothing offered as a poor fit. Exit terms that say your data stays with them, or your workflow only runs on their platform with no export.
- Why it is hard to fake: [THESIS] A vendor who sells only to people they should sell to will happily name the people they should not. Pressure on this question tends to reveal whether the sale or the fit comes first.
- Where it matters most: [THESIS] All tiers. Poor-fit honesty matters most in low and material work, where a cheaper tool often wins. Exit rights matter most at high stakes, where you cannot afford to be stuck.
A worked example (fictional).
[EXAMPLE] This workflow is invented for illustration. It is not a TITANOS customer, it is not a case study, and no part of it happened.
[EXAMPLE] A fictional distributor, Harbour Lane Supplies, is offered an AI agent to match supplier invoices to purchase orders. About 400 invoices arrive a week. Material tier: a wrong match means a wrong payment.
[EXAMPLE] Question 1: the owner and the vendor agree that "finished" means every invoice is either matched to exactly one purchase order within a 1 percent amount tolerance, or placed in an exceptions list with a reason. "Processed" does not count.
[EXAMPLE] Question 2: the vendor proposes that the agent's own "all matched" message is the check. The owner declines this and instead asks for a weekly reconciliation of matched totals against the accounting ledger, plus a human check of 20 random matches. The agent's report is rung A. The ledger reconciliation is rung C.
[EXAMPLE] Question 3: the vendor's evidence is a demo on tidy sample invoices (rung B). The owner supplies 50 real invoices from last quarter, including awkward ones, and asks for a trial run before signing (rung C).
[EXAMPLE] Credible failure case: in the trial, the agent reports "400 of 400 matched". The ledger reconciliation shows 31 invoices matched to the wrong supplier, because two suppliers share near-identical trading names. The log shows no errors, because from the agent's point of view nothing went wrong.
[EXAMPLE] This is the case the seven questions exist for. Without Question 2 the report is accepted and 31 wrong payments follow. With it, the failure is found in the trial, the vendor adds a supplier-ID check (their answer to Question 4), and the owner decides whether the fix is good enough.
[EXAMPLE] Question 5: the owner asks what the agent can do. It can write to the payment queue. The owner insists it write to a holding list that a person releases, so the worst case is a bad suggestion and not a bad payment. A stop switch is demonstrated on a test copy.
[EXAMPLE] Question 6, illustrative arithmetic only, with all figures invented and in Australian dollars: 1,000 runs in a month at 50 cents a run is $500. Of those, 820 results are accepted and 180 need rework at 6 minutes each, at $45 an hour of staff time: 180 x 6 minutes = 18 hours x $45 = $810. Reviewing every run for 30 seconds is 500 minutes, about 8.33 hours, which is about $375. Total about $1,685. Divided by 820 accepted results, that is about $2.05 per accepted result.
[EXAMPLE] The point of that sum is the shape, not the number: the quote said 50 cents, and the cost per accepted result was about four times that. The figures are made up and predict nothing about any real vendor.
[EXAMPLE] Question 7: the owner asks where the agent is a poor fit. The honest answer is the 12 invoices a week with handwritten amendments, which a person should handle. The owner also asks about exit and learns the matching rules are exportable, which is a point in the vendor's favour.
Part 5. The unique asset: the seven-question worksheet
[THESIS] Copy this out or print it.
THE SEVEN-QUESTION PROOF TEST: BUYER WORKSHEET
A guide, not an audit. It cannot detect a determined liar and it is weaker than a paid independent assessment. Free to copy and print. Not a certification of any vendor or product.
Vendor / product: ______________________ Date: ____________
Workflow being considered (one only): ______________________________
Risk tier (circle): LOW MATERIAL HIGH STAKES
Completed by: ______________________ Second reviewer: ______________
EVIDENCE RUNGS (record one per answer)
A = the vendor said it
B = the vendor showed a document or recording they made
C = shown working on a sample of MY work, chosen by me
D = shown by someone independent of the vendor
SCORE: 0 = no answer or evasive 1 = answer, no evidence
2 = answer with evidence 3 = answer with evidence I could check myselfQ1 What exactly counts as finished, and who decides?
Good: written, outcome-based, agreed before work starts, named sign-off.
Red flag: "it tells you when it's done"; a dashboard tick as the definition.
Their definition of done, in their words: ______________________
______________________________________________________________
Rung: A B C D Score: 0 1 2 3 Notes: ____________Q2 Show me the outcome, not the log. Who checks it, and is it anyone
but the system?
Good: a named check the agent does not control; output of a real
check that caught something.
Red flag: "the agent confirms its own work"; "we have logging".
The check, who runs it, how often: _____________________________
______________________________________________________________
Rung: A B C D Score: 0 1 2 3 Notes: ____________Q3 What did you actually test, on what data, and when?
Good: task, sample size, source, date, version, and what was NOT tested;
willing to run MY sample before I sign.
Red flag: "extensively tested"; results on their data only; no date.
What was tested / sample / date / version: _____________________
______________________________________________________________
Rung: A B C D Score: 0 1 2 3 Notes: ____________Q4 Tell me about a time it failed. What happened, and what did you change?
Good: a dated, specific failure, a fix, and a check that now stops it.
Red flag: "no failures"; blames the customer; no change made.
The failure they described: ____________________________________
The fix and the new check: _____________________________________
Rung: A B C D Score: 0 1 2 3 Notes: ____________Q5 When it is wrong, what is the worst it can do, and how do I stop it?
Good: list of what it can read and write; limits on access, spend and
volume; approval step for irreversible actions; a stop I control;
a tested rollback; stop demonstrated while I watch.
Red flag: "it can't go wrong"; broad default access; stop needs the vendor.
Worst realistic outcome: _______________________________________
Stop switch demonstrated? YES NO Rollback tried? YES NO
Rung: A B C D Score: 0 1 2 3 Notes: ____________Q6 What does one accepted result cost me, all in?
Good: fee + my review time + rework on failures, divided by results I
accept; measured in a trial on my work.
Red flag: per-run price only; "pays for itself" with no arithmetic;
savings percentage with no baseline, date or method.
Fee: ________ My review time: ________ Rework: ________
Accepted results: ________ Cost per accepted result: ________
Rung: A B C D Score: 0 1 2 3 Notes: ____________Q7 Where should I not use this, and what do I keep if I leave?
Good: a real list of poor fits, including where a simple tool or a
person is better; export of my data and settings, format and speed stated.
Red flag: "works for everything"; no poor fits; data stays with them.
Poor fits they named: __________________________________________
What I can export, how, how fast: ______________________________
Rung: A B C D Score: 0 1 2 3 Notes: ____________DECISION RULES (our suggestions; adjust to your own situation)
1. Any score of 0 on Q1, Q2 or Q5 at MATERIAL or HIGH STAKES: stop.
Do not proceed until the vendor answers it with evidence.
2. Q2 and Q3 only at rung A at MATERIAL: ask for a trial on my own
sample (rung C) before any commitment.
3. HIGH STAKES: this sheet is a first filter only. Commission a paid
independent assessment and take legal and insurance advice first.
4. A vendor who tells me not to buy (Q7) has given me information.
Thank them. Do not punish it.
5. Two vendors: compare on rung first, then score, then cost per
accepted result (Q6). A cheaper vendor on rung A can be dearer.
TOTAL SCORE: ____ / 21 Lowest rung on Q2, Q3 or Q5: ____
Proceed to trial / Ask for more evidence / Stop (circle one)
Owner sign-off: ______________________ Date: ____________
Second reviewer sign-off: ______________ Date: ____________
LIMITS: this worksheet is a guide, not an audit. It cannot detect a
determined liar. It is weaker than a paid independent assessment. It is
not legal, financial or compliance advice. Black Ice is a company-run
editorial challenge from TITANOS and is not an independent certifier.Part 6. The strongest fair objection
The objection, steelmanned.
[THESIS] "This is too much ceremony. For most business tasks, a plain scripted automation, a spreadsheet formula or a person doing the work is cheaper, easier to understand and easier to check than an autonomous AI agent. A buyer who runs seven questions on an agent has already made the mistake of choosing an agent. The right first question is whether you need one at all."
[THESIS] The objection has more force than it first appears. A rule-based tool has a fixed, testable behaviour. A human reviewer can notice something odd that nobody specified. Complex systems add failure modes, ongoing cost and a dependency on the vendor. A checklist can also create false comfort: someone who ticks seven boxes may feel safer than they are.
Our answer, and where the objection wins.
[THESIS] The objection wins on its core point. If a simple automation or a human can do the job at acceptable cost and risk, that is usually the better buy, and Question 7 is built to bring that out. Where a vendor's honest answer to Question 7 is "you do not need us", the checklist has worked.
[THESIS] The objection also wins on false comfort. That is why Part 7 and Part 8 exist, and why the worksheet records the evidence rung and not just a score.
[THESIS] Where we disagree: the objection treats "use a simple tool" and "run an acceptance test" as alternatives. They are not. A simple tool also needs a definition of done and a check that someone other than the tool performs. The seven questions work on a person's process or a script as well, with small adjustments.
[THESIS] There are also tasks where the work is too variable for rules and too large for a person, such as reading thousands of free-text documents. There, an AI system may be the only sensible option, and the questions are how you buy it without trusting its own report.
Part 7. What this does not prove
- [THESIS] A vendor who scores well on this worksheet is not therefore good. A vendor who scores badly is not therefore dishonest.
- [THESIS] The checklist does not prove the system works. It shows what the vendor has and has not shown you.
- [THESIS] It cannot detect a determined liar. A vendor willing to fabricate a failure story, a reference customer or a test result can pass every question on paper. The rungs help, since rung C (your own sample) and rung D (independent) are much harder to fake than rung A, but they reduce the risk and do not remove it.
- [THESIS] It is not an audit, and it is weaker than a paid independent assessment. If the stakes are high, hire the assessment.
- [THESIS] It does not tell you whether the vendor will be around in two years, whether their security is adequate or whether the contract protects you. Those need their own checks.
- [UNKNOWN] It does not show that using the checklist improves outcomes. We have not tested that, and the page says so in Part 3.
- [THESIS] It does not show that TITANOS is better than anyone. We have not published our own answers to the seven questions. That is a gap, and Part 12 addresses it.
Part 8. Limitations and exclusions
- Risk tiers. [THESIS] These tier definitions are our own working definitions for this checklist, not a standard.
- [THESIS] Low: the action is cheap, internal and easy to reverse. A wrong result costs minutes. Examples: drafting an internal summary a person reads before use, tagging support tickets for later review. A lighter pass through the questions is reasonable, with Questions 1, 4 and 7 as the minimum.
- [THESIS] Material: a wrong result costs real money, reaches customers or suppliers, or changes records others rely on, but can be corrected with effort. Examples: matching invoices, drafting customer replies that are sent, updating inventory. Run all seven, and insist on rung C for Questions 2, 3 and 6.
- [THESIS] High stakes: a wrong result could cause harm to a person, a legal or regulatory breach, a large or irreversible financial loss, or a safety consequence. [THESIS] At this tier a seven-question checklist is a first filter only. Get a paid, independent assessment, involve your lawyers and your insurer, and keep a qualified human in the loop for the decision itself.
- When not to automate: [THESIS] When the task is rare, cheap and quick for a person. The setup and checking effort will exceed the saving.
- When not to automate: [THESIS] When you cannot define "finished" in terms someone else could check. Fix that first, because no vendor can do it for you.
- When not to automate: [THESIS] When a rule-based tool covers it. Prefer the thing you can fully understand.
- When not to automate: [THESIS] When the decision needs judgement, empathy or accountability that must sit with a named person, such as a decision about someone's employment, care or legal rights.
- When not to automate: [THESIS] When you have no one who can do the verification. An unchecked agent is not a saving, it is a deferred cost.
- Scope limits of this page: [THESIS] The seven questions are written for systems that do multi-step work with some autonomy. They apply less well to a simple chat assistant used for brainstorming, where "finished" has no real meaning.
- Scope limits of this page: [UNKNOWN] We have not tested the checklist on any specific industry. Regulated sectors will have requirements this page does not know about.
- Scope limits of this page: [THESIS] Nothing here is legal, financial or compliance advice.
Part 9. Evidence and sources
- THESIS: [THESIS] An argument we make and stand behind. It would change if a better argument or a measured counter-example showed it wrong. Most of this page is THESIS, and that is the honest label for a guide.
- EXAMPLE: [THESIS] The Harbour Lane Supplies workflow, the invoice figures, the dollar arithmetic and the failure case are fictional. They would stop being labelled EXAMPLE only if a real, consented, documented case replaced them, and then they would be relabelled and dated.
- OBSERVED: [THESIS] Used on this page only to say that nothing was tested. It would apply to any future section reporting something we actually saw, with what, when and how.
- REPRODUCED: [THESIS] Not used. Would apply only to results seen repeatedly under stated conditions.
- UNVERIFIED: [THESIS] Not used. Would apply to a claim we state but have not checked.
- UNKNOWN: [THESIS] Used where we lack evidence: market-wide failure rates, whether the checklist improves outcomes, industry fit, and our own published answers. Each would change with a measurement we make or a source we can name and date.
- Sources: [THESIS] This page cites no external statistic, study, benchmark or vendor price. That is deliberate. Where a figure would have helped, we preferred to write UNKNOWN. Readers who know a good, dated, citable source for any point on this page are invited to send it; we will add it with its date and label.
Part 10. Buyer reading and builder reading
If you are a buyer
1. [THESIS] Pick one workflow you are considering. Write its definition of done in two sentences. If you cannot, that is your first finding.
2. [THESIS] Send the vendor Questions 1, 2 and 3 in writing and ask for a reply with evidence. Note the rung.
3. [THESIS] Ask for a trial on 50 of your own real, awkward items, with you choosing the sample.
4. [THESIS] Decide the tier before you decide the vendor.
If you are a builder or vendor
1. [THESIS] Answer the seven questions about your own product, in writing, in public if you can. A buyer who finds your answers before asking is a buyer who already trusts you more.
2. [THESIS] Replace every activity-based "done" in your product (process ran, job exited) with an outcome-based one, and show the check.
3. [THESIS] Write down one real failure, what you changed, and the check that now stops it. Be ready to tell it.
4. [THESIS] Write the list of poor fits for your product. If it is empty, you have not thought hard enough.
Part 11. Corrections
No corrections yet. Material changes will be listed here with their date.
Part 12. Which question would you add?
[THESIS] The seven here are our best argued selection, not a proven set. If you have bought, sold or built AI systems and know a question that would have caught something these miss, we want it. [UNKNOWN] We have not yet published our own answers to these seven questions for TITANOS. If that is useful to you, you are welcome to ask us to answer them in public in the same format, and a no thanks is just as welcome. --- ---
Image: Illustration only. A buyer checks a vendor's claim of completion against a written seven-question list. The diagram shows three steps: define what finished means, check the outcome independently, then decide.