How to Evaluate an AI Consultant (A Founder's Honest Checklist)

The AI consulting market is flooded with firms that will happily take your money and deliver a 50-slide strategy deck. Six months later, you have a beautiful presentation gathering dust in a shared drive and zero automated processes. I know this because I've talked to dozens of founders who've been through exactly this experience before they found us. Here's the checklist I'd use to evaluate any AI consultant — including myself.

1. Ask to See Production Systems, Not Demos

Anyone can build a demo. Demos work perfectly because they're designed for perfect conditions — clean data, predictable inputs, no edge cases. What matters is whether the system works when your intern uploads a scanned invoice that's slightly rotated, or when a client submits a contract in a format you've never seen before. Ask your potential consultant: "Can you show me a system running in production today?" If the answer involves caveats, excuses, or "we're still in pilot phase with most clients" — that's a red flag.

2. Who Actually Does the Work?

This is the question that separates boutique consultancies from agency-model firms. At most agencies, the impressive people you meet during the sales process are not the people who build your system. The senior AI architect who wowed you in the pitch meeting hands your project to a team of juniors or offshore developers they've never met. Ask directly: "Will the person I'm talking to right now be the person writing code for my project?" At MDS, the answer is always yes — because there is no junior team. I architect and build every system personally.

3. Check for Privacy Awareness

If your consultant's first suggestion is "let's upload all your data to OpenAI's API" without discussing data residency, privacy implications, or on-premise alternatives — they're not thinking about your business. They're thinking about their convenience. A good AI consultant should be able to articulate: where your data flows, who can access it, what happens if the AI provider changes their terms, and what on-premise or privacy-first alternatives exist.

4. Demand Specific ROI Projections

Vague promises like "AI will transform your operations" mean nothing. Push for specifics: How many hours per week will this save? What's the expected accuracy rate? What's the payback period? A consultant who can't give you numbers either hasn't done the analysis or doesn't want to be held accountable. Our free AI assessment delivers specific projections — because we believe you deserve to know what you're investing in before you spend a dollar.

5. Ask About Failure

Every consultant has had projects that didn't go as planned. The good ones will tell you about it honestly and explain what they learned. The bad ones will claim a perfect track record. Ask: "What's the hardest project you've worked on, and what went wrong?" The quality of that answer tells you everything about how they'll handle problems on your project — because problems always come up.

6. Look for Fixed-Scope Pricing

Hourly billing in consulting creates a perverse incentive: the longer the project takes, the more the consultant earns. Fixed-scope pricing aligns incentives — the consultant is motivated to deliver efficiently because the price is locked regardless of how many hours it takes. Ask: "Is this a fixed price or hourly?" and "What happens if the project takes longer than expected?" At MDS, we use fixed-scope pricing for exactly this reason.

The Bottom Line

Good AI consultants are rare. Most are either pure strategists (great ideas, no code) or pure engineers (great code, no business context). What you need is someone who understands both — who can identify the highest-ROI automation opportunity and then actually build the system to capture it. Use this checklist, ask uncomfortable questions, and don't settle for slide decks when you need production systems.