Questions to ask any AI vendor, us included
Two checklists for buying AI, whoever the vendor is. Use them in a shortlisting meeting or attach them to a request for proposal.
Free to copy and forward. No sign-up.
Eleven questions every AI vendor should answer in writing
Ask every vendor the same questions, in writing, and compare the answers side by side.
Where will our data be stored and processed, part by part?
The AI model, the search index, the logs, the backups and the support tools all hold data. A server address answers for only one of them.
Which model runs the system, under what terms, and is any of our data used for training?
Model terms change, so the answer belongs in the contract.
What can the system do inside our systems, and with which logins?
Each connection should allow only the actions the task needs. The passwords and keys should stay in your own secure store.
Which actions wait for a person to approve them, and can the system skip that step?
Approval points belong in the design and in writing, agreed with the people who will approve.
How is the system tested before it goes live, and after every change?
Ask for a set of test cases built from your real work, and ask whether you keep it.
Is every run recorded, and where are those records kept?
Without a record of each step, nobody can explain a wrong result to an auditor or a regulator.
Who owns the code, the instructions given to the AI, the connections and the server setup?
If they sit in the vendor’s accounts, so does your ability to change or leave.
What keeps running if we stop working with you?
The right answer is everything. Anything less means you are locked in.
Who will build it, and who reviews the work before it ships?
Ask which roles write the code, which review it, and what checks run before a change goes live.
How well does it handle your own documents, including scanned ones?
Test it with your own documents during the review. A vendor’s demo set tells you little about yours.
When would you advise us not to use AI for this?
A vendor that gives honest advice can name a case where a simple script or a better form would do the job.
One record, followed through the system, shows where your data goes
Follow one record from the moment it enters the system to the moment the AI answers. For each place below, write down the region, how long the data is kept, and who at the vendor can read it.
Data intake and storage
Include the region where copies are kept, set when the storage account was created.
The model that indexes documents, and the index itself
To make documents searchable, the system turns every piece of text into numbers. If an outside online service does this, every piece of every document has been sent there.
Where the AI model runs
Where the model does its work, and whether the provider keeps the questions or the answers.
App logs and error reports
These often hold the full question and the full answer.
Analytics and monitoring tools
Often added for a demo and never checked again.
Backups
They are copied to wherever the account was set up on day one.
Vendor support access
Support tools can open a path to your data each time a support ticket is raised.
Models trained on your data, and test sets
Once a model is trained on your data, the model file itself is data you must place and protect.
For a security reviewer, our Security page answers these points for our own work
Put these questions to us
Send the checklist with your request. We will answer each question in writing before any meeting.
Start a project