AI Costs Are the New Cloud Bill
Most AI ops systems route every task through one expensive model. The cheaper path is task-model fit: rules, cache, small models, then escalate.
By Ramakrishnan Ayyappan · · AI · Cost Optimization · LLMs · Operations
View on LinkedIn →The problem
Teams often solve a one-dollar question with a hundred-dollar model. One agent, one frontier model, every request routed through it works in a demo. In production it becomes another untracked cloud bill.
The issue is not that models cost money. It is that the same expensive path is used for routine classification, known playbooks, and rare design questions.
Approach
Match capability to the job:
- Rules and cache for known patterns and repeated questions
- Small models for structured analysis and simple recommendations
- Mid-tier models when reasoning or novelty increases
- Frontier models only when the cost of being wrong is high
A cascade is useful when the right tier is unclear: try the cheap path first, then escalate if confidence is low.
Architecture
Routing sits in front of inference. It scores complexity and criticality, checks exact and semantic cache, and only then calls a model. Batching covers weekly reviews and other async work.
This is the same idea as right-sizing compute: do not run every query on the largest instance.
Lessons
- Cost control starts with routing, not with a cheaper vendor
- Quality can improve when routine work stays on rules
- Track cost per outcome, not just tokens
- Revisit the matrix as new small models appear
Read the original post on LinkedIn for the longer argument.