How to vet an AI developer
Many developers can call a language model API; far fewer can build an AI feature that stays accurate, safe and affordable in production. Ask candidates how they grounded answers in company data with retrieval-augmented generation, how they measured quality and what they did when the model produced wrong or harmful outputs. Specific stories about failures and fixes are the best signal.
Engineering discipline matters as much as model knowledge. Look for experience with prompt versioning, evaluation sets that run on every change, structured outputs, function calling with permission checks, rate limits, cost tracking per feature and logging that respects privacy. Developers should also know when traditional software or a simple rule beats an AI model.
Data handling deserves direct questions. Ask how they protected sensitive data sent to model providers, handled personal information, enforced document permissions in retrieval and chose between hosted APIs and self-hosted open models for compliance reasons. Clear answers here protect both customers and the business.
- Has shipped AI features used by real customers.
- Builds evaluation sets and measures quality continuously.
- Designs retrieval pipelines that respect permissions.
- Controls latency and cost per request.
- Applies guardrails and human review where needed.
- Understands data privacy with model providers.
- Writes clean, tested code in Python or TypeScript.


