Token prices have fallen every year. It would be reasonable to assume inference cost is becoming a non-issue. It isn't — because usage is growing faster than prices fall, and the new techniques (agents, reasoning, big-context RAG) burn tokens an order of magnitude faster than a simple chat call. Inference is now a serious line item, and it shows up in your gross margin.

The metric that matters

Stop tracking cost-per-token. Track cost-per-successful-task. A cheaper model that fails more often and triggers retries or escalations can cost more per completed job than an expensive model that nails it first time. The token price is an input; the unit economics of the actual task are the output, and that's what you manage.

Cost-per-token is a vendor's number. Cost-per-successful-task is your number — and it's the one that decides whether the product has a business.

The levers, in order of impact

  • Route to smaller models for the easy majority of requests. Usually the single biggest win.
  • Cache aggressively — prompt caches, semantic caches, and plain old "we've answered this before".
  • Trim context. Every token you don't send is a token you don't pay for, on every request, forever.
  • Cap reasoning with budgets so a single hard query can't run away.
  • Batch and stream where latency allows.

Design for margin from day one

The expensive mistake is shipping a feature that delights users and quietly loses money on every interaction, then discovering it at scale. The teams that avoid this put a cost number next to every AI feature in the spec — "this should cost under X per use" — and treat blowing it as a bug, not a finance problem to fix later.

Make it observable

You can't manage what you can't see. Instrument cost per request, per feature, per user tier. A dashboard that shows cost-per-successful-task trending over time turns inference spend from a quarterly surprise into a number you steer week to week.

The takeaway

In 2026, the best AI PMs and engineers reason about token economics the way good builders have always reasoned about latency and reliability — as a first-class constraint that shapes the design, not an afterthought to optimize once the bill hurts.