How to Evaluate an AI Vendor: The 20-Question Scorecard
By the Flon team · Published July 11, 2026 · Last updated July 11, 2026
Most AI vendor pitches sound identical: fast, custom, powered by the latest model. What actually matters — what happens if it's late, who owns the system, who fixes it when it breaks, whether the price is even published — rarely comes up unless you ask directly. This scorecard is 20 questions across five categories, built to surface exactly those differences before you sign anything.
Score any vendor's actual answers: 0 if they can't answer clearly, 1 if partial or vague, 2 if specific and in writing. Flon's own answers sit under each question as a working benchmark — not because every vendor should match us, but because a benchmark makes it obvious when an answer is dodging the question.
How to use it: Ask the questions in order, on a call or in writing, and score each response as you get it. The running total updates at the top out of 40. Vague answers to the guarantee and ownership questions are the biggest red flags.
0 / 40
Score each answer 0 (unclear), 1 (vague), or 2 (specific + in writing)
The scorecard
Guarantees
1. Do you guarantee an install date, or just estimate one?
Flon: Scope, price, and timeline are fixed together after the Blueprint, before any code.
2. What happens if the system doesn't hit its metric after launch?
Flon: One agreed number, reported in a single page every month after launch.
3. Is there a paid discovery step, and does it credit toward the build?
Flon: Yes — the $1,900 Blueprint is credited to any install.
4. Does the discovery process guarantee its own return?
Flon: If the Blueprint doesn't identify ROI ≥ 5× its own cost, it's free.
Ownership
5. Do I own the system outright, or am I renting access to yours?
Flon: What we build is yours outright, in your own stack, no lock-in.
6. Can I take this system elsewhere later — another vendor, or in-house?
Flon: Yes. It lives in your stack from day one.
7. If I stop paying for management, does the system keep working?
Flon: Yes — what we build keeps running whether or not we do.
8. Where does the data the system uses actually live?
Flon: Your own accounts and your own stack.
Evaluation and governance
9. Do you test the system before launch, and how?
Flon: Every build includes an evaluation suite run before go-live.
10. Is there human approval before consequential actions (sending outbound, approving refunds)?
Flon: Yes where it matters — on outbound, human approval before send is on by default.
11. Do you provide audit trails of what the AI did?
Flon: Core to governed builds; on anything else, ask during scoping what gets logged.
12. Is the system built to be GDPR/HIPAA-aware if my industry needs it?
Flon: Yes for Custom Systems and governed builds in regulated industries — raise it early if it applies to you.
Operations after launch
13. Who operates the system after it launches — you or me?
Flon: Flon Managed by default — the first month is included with every install; clients opt out, not in.
14. What does "managed" actually include?
Flon: Monitoring, evaluation, and one agreed metric in a monthly one-page report. Flon Managed, from $1,490/mo.
15. What happens when the underlying AI model changes or degrades?
Flon: That's what Managed exists for — deprecation and prompt drift are handled as ongoing operation, not billed as emergencies.
16. Is there one metric we're both accountable to, or is success left vague?
Flon: Every build agrees one activity number up front (answer rate, speed to lead, draft quality, exception clear time, grounded answers) and reports against it monthly. Conversion, meetings, and hours saved are reported against baseline.
Pricing and timeline
17. Are your prices published, or do I need a call to find out?
Flon: Published. Blueprint $1,900, credited. Custom AI System $25k–$150k, scoped after it. Managed from $1,490/mo.
18. Is there a lower-commitment way to try before I own a system outright?
Flon: Yes — the Blueprint is $1,900 and credited in full to whatever gets built, so the first step is small and never wasted.
19. What's the fixed price for what I actually need, and what changes it?
Flon: Fixed per scoped install; the Blueprint ($1,900, credited to any install) scopes custom needs before a price is quoted on larger builds.
20. Can you show me proof, not just claims?
Flon: Ask what happens if the discovery step finds nothing worth building — ours is free if it can't identify 5x its own cost.
How to score it
| Total | What it means |
|---|---|
| 32–40 | Exceptional. Clear, specific, written answers across guarantees, ownership, and operations — proceed with confidence. |
| 22–31 | Solid, with real gaps. Press for specifics on whatever scored 0 or 1 before signing, especially guarantees and ownership. |
| 12–21 | Meaningful risk. Get the vague answers in writing before committing any budget, or treat them as unresolved. |
| Below 12 | Walk away, or renegotiate from scratch. A vendor that can't answer most of these clearly is asking you to trust judgment it hasn't demonstrated. |
FAQ
Should I use this scorecard on Flon too? Yes — that's exactly the point of including our answers as a benchmark. Score us the same way you'd score anyone else, and press us on anything that reads as vague.
What's the single most important question on this list? Question 2 (what happens if it doesn't hit its metric). It's the question most vendors dodge, because most don't offer any real answer beyond "we'll keep working on it" — which isn't a guarantee, it's a hope.
A vendor scored well but has no published pricing. Is that disqualifying? Not automatically, but ask about it directly. Published pricing is largely a proxy for confidence — vendors sure of their delivery timelines are usually willing to put a number on the page.
Compare us against this scorecard
Every answer in the scorecard above is on our pricing page, next to all three guarantees, in writing. Related reading: Flon guarantees for what operating a system means month to month, the Blueprint explained for what the discovery step covers, and AI agent evaluation and human-in-the-loop in the glossary for the governance questions above.