Interviewing an agency is an art. You need to pierce through the sales script to find the engineering reality. Use these questions to find the truth:
1. "Can you walk me through a project where your initial model failed, and how you fixed it?"
Why it’s useful: It reveals their problem-solving resilience and honesty about the iterative nature of AI.
2. "How do you handle data drift after the model is deployed?"
Why it’s useful: AI models degrade over time. If they don't have a maintenance plan, you’re buying a depreciating asset.
3. "Do you fine-tune existing models or build from scratch, and why?"
Why it’s useful: This determines if they are efficient pragmatists or unnecessary perfectionists who will burn your budget.
4. "Who owns the IP of the trained model and the dataset after the project ends?"
Why it’s useful: You need to ensure you aren't locked into their ecosystem forever.
5. "How do you optimize for inference costs for AI in app personalization?"
Why it’s useful: Integration of AI for app personalization is expensive. A good partner builds efficient models that don't bankrupt you on server costs.
6. "What is your approach to explainable AI (XAI)?"
Why it’s useful: You need to know why the AI made a decision, especially in regulated industries like finance or health.
7. "Can you provide references from clients with a similar AI development cost structure to ours?"
Why it’s useful: Confirming the AI development cost is useful if you want to know whether they can deliver quality within your specific financial constraints.
8. "How do you integrate security best practices into your ML pipeline?"
Why it’s useful: AI systems are new attack vectors. Security cannot be an afterthought.
9. "What is your team's experience with the specific future of AI trends like agentic workflows?"
Why it’s useful: It ensures they are looking ahead and building a solution that won't be obsolete in six months.
10. "How do you ensure data privacy when using third-party APIs?"
Why it’s useful: Critical for protecting your users' sensitive information from leaking into public models.