How to Vet an AI Vendor: The Questions to Ask Before You Sign

Choosing an AI vendor is also a decision about who can access your data, how your work depends on their service, and what happens if the tool gets something wrong. Before you sign, get clear answers in writing about privacy, security, performance, cost, support, and how you can leave.

A demo can show what a tool might do. It can’t prove the vendor’s claims fit your use case or your risk tolerance. Use the questions below to evaluate the service, not just the sales presentation.

1. WHAT HAPPENS TO YOUR DATA?

Start with the data you expect to enter: prompts, uploaded files, generated outputs, usage logs, and feedback. Ask the vendor:

• Is any of this data used to train, fine-tune, evaluate, or improve models?
• Does the answer change based on your plan, product settings, or use of the API?
• Where is the data processed and stored, including backups and support systems?
• How long are each type of data and its logs retained?
• Which subprocessors can access it?
• How do you request deletion, and what happens to backups after deletion?
• Can the vendor sign a data processing agreement (DPA) and list its subprocessors?

Don’t settle for a general promise such as “we take privacy seriously.” Ask for the contract, privacy terms, and product settings that apply to your specific plan. Confirm whether you can disable training, limit retention, or restrict processing locations. “Data stored in your region” may not mean all processing, support access, and backups stay there.

Until you have answers you can accept, keep confidential, regulated, personal, and trade-secret information out of the tool. The FTC’s guidance on AI privacy commitments is a useful reminder: a vendor’s promises about customer data need to be honored in practice, not just in a sales conversation.

2. WHAT SECURITY EVIDENCE CAN THE VENDOR PROVIDE?

Ask for evidence that applies to the service you’re buying. Depending on your risk and the product, that may include a current SOC 2 report, an ISO/IEC 27001 certificate, a penetration-test summary, or details about incident response and access controls.

Then check the details:

• Does the evidence cover this product, or only another part of the company?
• When was it issued, and when does it expire?
• What systems and locations are in scope?
• Were exceptions found, and how is the vendor addressing them?
• Are customer data encrypted in transit and at rest?
• Can your administrators use multifactor authentication, single sign-on, and role-based access?
• How and when will the vendor notify you of a security incident?

A badge or certification is useful evidence, not a guarantee that a product is secure for every purpose. Check what it covers and whether it addresses your requirements. The Cloud Security Alliance’s vendor questionnaire guidance offers practical topics to include in your review.

For additional structure, the NIST AI Risk Management Framework recommends managing AI risks throughout the system’s life cycle, including risks that come from third parties.

3. HOW DOES THE VENDOR HANDLE WRONG ANSWERS?

AI tools can produce confident-sounding answers that are incomplete or incorrect. Ask how the vendor measures accuracy and what its claims are based on. “Highly accurate” is not enough. Find out what was tested, on which tasks, and whether the results resemble your work.

Test the product with realistic, low-risk examples. Look for:

• Incorrect facts, calculations, or references
• Answers that fail to acknowledge uncertainty
• Responses to questions outside the tool’s intended purpose
• Exposure of information that should stay private
• Whether users can verify answers against approved sources
• Whether you can review, report, and investigate problematic outputs

Decide in advance when a person must review an output. For example, you may allow AI to draft an internal summary but require an employee to verify it before it goes to a customer or affects a business decision. The FTC’s guidance on AI claims is a useful standard: claims about a product’s capabilities should be truthful and supported by evidence.

4. WILL IT FIT YOUR SYSTEMS, AND CAN YOU LEAVE?

A tool that works well in a standalone demo may be difficult to connect to your business systems. Ask what integrations are available, what setup requires, and whether additional fees apply. Confirm what data moves between systems, who manages the connection, and how access is controlled.

Plan for the end of the relationship before it begins:

• Can you export your inputs, outputs, settings, and usage records?
• What format will the export use?
• Can the vendor help you transition to another service?
• What happens to your data after termination?
• Are there export fees, minimum terms, or cancellation penalties?

Don’t assume you own or can retrieve everything just because you uploaded it. Make data ownership, export, deletion, and transition assistance explicit in the agreement.

5. WHAT WILL YOU ACTUALLY PAY?

AI pricing can be based on seats, usage, tokens, or a combination. A low starting price may not reflect what you’ll pay once your team uses the tool regularly.

Ask for a written breakdown of:

• Per-seat charges and minimum seat commitments
• Usage limits and overage rates
• Charges for tokens, integrations, storage, premium models, or support
• Additional fees for security features, single sign-on, or data controls
• Price increases, renewal terms, and cancellation conditions

Then estimate the cost using your expected usage, not just the vendor’s example. Include likely add-ons, extra usage, and any staff time needed to review outputs. Ask how you can monitor usage and set spending limits.

There is no reliable universal price range for AI vendors: costs depend on the product, plan, usage, and contract. If the vendor can’t explain how your bill is calculated, treat that as a reason to pause.

6. WHAT DOES THE SERVICE PROMISE WHEN SOMETHING BREAKS?

Ask for the service-level agreement (SLA) and support terms. Look for uptime commitments, planned maintenance rules, support hours, response targets, and how the vendor communicates outages. Confirm whether the SLA covers the service you plan to use, including any underlying model or third-party dependencies.

A percentage alone is not the full story. Find out how uptime is measured, what exclusions apply, and what remedy you receive for repeated failures. Ask what happens if the service is unavailable during a critical workflow. You may need a manual fallback, another approved process, or the ability to pause use without disrupting operations.

7. CAN YOU TEST THE VENDOR BEFORE MAKING A BIG COMMITMENT?

Run a bounded pilot before signing a long-term agreement. Keep it narrow: one use case, a small group of users, limited access, and low-risk or appropriately protected test data.

Set success criteria before the pilot starts. For example:

• Does it complete the defined task to your required standard?
• How often do users need to correct its work?
• Can users verify important answers?
• Does it work with your existing systems and access controls?
• Does usage stay within an agreed budget?
• Can the vendor meet the support and reliability needs you identified?

A practical pilot might run two to four weeks, depending on the use case and how quickly you can gather meaningful results. That’s a planning suggestion, not a guarantee that every tool can be evaluated in that timeframe. Agree on the end date, the test data, the success measures, and what happens if the vendor falls short. Don’t expand access or upload sensitive production data just because the demo went well.

8. MAKE THE DECISION WITH A SHORT VENDOR CHECKLIST

Before signing, make sure you can answer these questions clearly:

1. What data can the vendor access, and is it used for training?
2. Where is data processed, how long is it retained, and how can it be deleted?
3. What security evidence covers the product you’re buying?
4. How will you check accuracy and review important outputs?
5. Can the tool integrate with your systems and export your data?
6. Can you explain the full pricing model and likely additional charges?
7. What support, uptime, and incident-notification commitments are in writing?
8. What will make the pilot a success, and what will make you walk away?

If important answers are vague, unavailable, or only given verbally, pause. A vendor that is a good fit should be able to explain its controls, limits, and commercial terms clearly.

CHOOSE A VENDOR THAT FITS YOUR BUSINESS

AI can support digital transformation when it solves a real operational problem and fits your risk, budget, and team. Five 9 helps businesses assess AI opportunities, security considerations, and implementation needs through our AI capabilities, IT consulting services, and security services.

If you’re comparing vendors, we can help you assess the questions that matter without turning the first conversation into a sales pitch. Let’s have an honest conversation about your requirements, your risks, and whether we’re the right fit. Book a free 30-minute call with a real consultant.