LLM Development for Production Workloads

Proprietary models offered through APIs by providers such as OpenAI, Anthropic and Google usually lead on broad reasoning and writing quality, require no infrastructure and improve frequently. They suit most applications, especially early on, as long as contracts covering data use and retention meet your requirements.

Open-weight models, such as the Llama, Mistral, Qwen, Gemma, DeepSeek and gpt-oss families, can run on your own servers or private cloud. They give full control over data location, predictable costs at high volume and freedom to fine-tune deeply. In exchange, you manage GPUs, scaling, updates and security, and may need more engineering to match the quality of leading hosted models.

Many production systems use both. A smaller open model handles high-volume, narrow tasks such as classification or extraction, while a hosted frontier model handles complex reasoning. Designing the application with a model abstraction layer keeps this choice flexible as models and prices change.

Licensing deserves attention as well. Open-weight models come with different licenses, some permissive and some with usage restrictions for large companies or specific applications. Review the license terms, the provider's acceptable use policy and any obligations on derived models before building a product around a particular model.

Public leaderboards are a starting point, but they rarely reflect your documents, languages and output formats. The reliable approach is to build an evaluation set from real examples and test several candidate models against it, measuring the criteria that matter to the business rather than general benchmark scores.

Run the comparison again whenever a promising new model appears, because the landscape changes quickly. Keeping the evaluation set, scoring scripts and prompts in version control makes each comparison a matter of hours rather than weeks, and gives you evidence when negotiating with vendors or justifying a switch to stakeholders.

LLM costs grow with tokens, so the first lever is prompt design. Long system prompts, unnecessary examples and entire documents pasted into every request add up quickly. Retrieving only relevant passages, trimming instructions and limiting output length often reduce cost noticeably without hurting quality. Structured outputs, such as JSON schemas, also reduce wasted tokens from rambling responses.

The second lever is routing. Not every request needs the most capable model. Simple questions, classifications and formatting tasks can go to smaller, cheaper models, with escalation to a larger model only when confidence is low. Caching answers to repeated questions and using batch processing for non-urgent jobs reduce spending further.

Finally, measure cost per business outcome, such as cost per resolved ticket or per processed document, rather than total monthly spend. That framing helps teams decide where better quality justifies a higher price and where a cheaper approach is good enough. Report these unit costs monthly alongside quality metrics.