Inference Cost Engineering: Make AI Apps Cheaper Without Making Them Worse

Your AI app is not expensive because AI is magic. It is expensive because every request leaks tokens, retries, and sloppy context.

That is the uncomfortable truth behind inference cost engineering for AI apps: most teams do not need to make the product dumber. They need to stop paying premium-model prices for work that should have been routed, cached, trimmed, batched, or skipped entirely.

The good news: you can usually cut inference spend by 30-70% before touching model quality. The bad news: you have to measure. Vibes are useless here.

This tutorial walks through a practical cost-engineering system for AI apps: how to instrument usage, classify tasks, route models, shrink prompts, cache stable context, control outputs, batch background jobs, and set guardrails so the bill does not punch you in the face later.

Prerequisites

You do not need to be a hardcore ML engineer. You do need basic control over how your app calls an AI API.

Before you start, make sure you have:

• Access to your AI provider usage data, including input tokens, output tokens, model name, and request count
• A list of your app’s main AI workflows, such as chat, summarization, extraction, classification, search, writing, or support automation
• The ability to change prompts, model selection, timeout behavior, and retry logic
• A staging environment where you can test output quality before pushing changes
• A small evaluation set of real or realistic user inputs

Expected result: by the end, you should have a cost map, a routing policy, a cheaper prompt structure, and a repeatable process for reducing spend without randomly breaking quality.

Step 1: Measure Cost Per Workflow, Not Just Total Spend

If your only metric is “the API bill went up,” you are flying blind with expensive wings.

Start by logging every AI call with a consistent event shape. You want to know what workflow triggered the call, which model handled it, how many tokens went in, how many came out, whether the response was accepted, and whether the call was retried.

Example event:

{
  "event": "ai_inference_completed",
  "workflow": "support_ticket_summary",
  "model": "gpt-4o-mini",
  "input_tokens": 1840,
  "output_tokens": 260,
  "cached_input_tokens": 0,
  "latency_ms": 1320,
  "status": "success",
  "retry_count": 0,
  "user_tier": "pro"
}

Then calculate cost by workflow:

function estimateCost({ inputTokens, outputTokens, inputPricePerMillion, outputPricePerMillion }) {
  const inputCost = (inputTokens / 1_000_000) * inputPricePerMillion;
  const outputCost = (outputTokens / 1_000_000) * outputPricePerMillion;
  return inputCost + outputCost;
}

Do not stop at averages. Averages hide the monsters. Track:

• p50, p90, and p99 input tokens
• p50, p90, and p99 output tokens
• retry rate
• timeout rate
• cache hit rate
• cost per successful user action
• cost per retained customer, if this is a paid product

Expected result: you should be able to say, “Workflow X is 8% of requests but 43% of inference spend.” That is where you start.

Step 2: Split AI Work Into Cost Classes

Most AI apps use one model like a hammer. That is how you end up paying frontier-model rates to classify whether an email is “billing” or “technical support.” Ridiculous.

Create four cost classes:

Class 1: Tiny Deterministic Tasks

Examples:

• Intent classification
• Sentiment labels
• Routing decisions
• Simple extraction
• Language detection
• Moderation pre-checks

Use the cheapest reliable model, a rules engine, embeddings, or even normal code. If the task has a tiny label space, do not let a large model write an essay about it.

Expected result: low-cost models or non-LLM code handle obvious work before expensive inference gets involved.

Class 2: Structured Transformation

Examples:

• Extracting invoice fields
• Turning meeting notes into action items
• Converting user text into database filters
• Summarizing a support thread into fixed sections

Use structured outputs or function calling so the model returns only the fields you need. Free-form text creates parsing errors, retry loops, and bloated output tokens.

Expected result: fewer retries, shorter responses, and cleaner downstream automation.

Class 3: User-Facing Generation

Examples:

• Drafting emails
• Writing recommendations
• Answering support questions
• Producing personalized reports

This is where quality matters. Still, quality does not automatically mean biggest model every time. Many apps can use a mid-tier model by default and escalate only when confidence is low or the request is complex.

Expected result: premium models are reserved for work users can actually feel.

Class 4: Offline or Delayed Work

Examples:

• Nightly document processing
• Dataset labeling
• Evaluation runs
• Bulk summaries
• CRM enrichment
• Report generation that does not need instant response

Use batch processing where available. OpenAI’s Batch API, for example, is designed for asynchronous jobs and can return results within a 24-hour completion window at a discounted rate.

Expected result: non-urgent work stops competing with real-time user interactions and gets cheaper.

Step 3: Add a Router Before the Model Call

A model router is just a decision layer. It looks at the task and picks the cheapest acceptable path.

Here is a simple version:

function chooseModel({ workflow, inputLength, userTier, requiresReasoning }) {
  if (workflow === "intent_classification") {
    return "small-fast-model";
  }

  if (workflow === "data_extraction" && inputLength < 4000) {
    return "small-structured-model";
  }

  if (requiresReasoning || inputLength > 20000) {
    return "large-reasoning-model";
  }

  if (userTier === "free") {
    return "small-general-model";
  }

  return "mid-general-model";
}

This looks basic because it should be basic at first. Do not start with a grand orchestration cathedral. Start with obvious routing rules, measure failure cases, then improve.

Add an escalation path:

async function runWithEscalation(request) {
  const firstModel = chooseModel(request);
  const first = await callModel({ ...request, model: firstModel });

  if (first.confidence >= 0.85 && first.valid) {
    return first;
  }

  return callModel({
    ...request,
    model: "large-reasoning-model",
    previous_attempt: first.output
  });
}

Expected result: cheap paths handle routine work, expensive paths handle ambiguity, and the user does not see the plumbing.

Step 4: Stop Sending the Same Prompt Junk Every Time

Long prompts feel responsible. They are often just expensive.

Audit your system prompt. Look for:

• Repeated policy text that could be shortened
• Examples that no longer improve results
• Formatting instructions repeated in multiple places
• Large product docs included for every user, whether relevant or not
• Conversation history copied forever

A bad prompt says:

You are an expert customer support assistant for our company. You should always be helpful, accurate, friendly, concise, professional, empathetic, and aligned with our brand values...

A tighter version says:

You are a support assistant. Answer using the provided policy context only. If the answer is missing, say what information is needed. Return concise, customer-ready text.

The model does not need a corporate horoscope. It needs constraints.

For examples, keep only the ones that change behavior. If removing an example does not hurt your eval score, it was decoration with a token tax.

Expected result: lower input tokens with the same pass rate on your evaluation set.

Step 5: Cache Stable Context

Prompt caching is one of the cleanest wins in inference cost engineering because it attacks repeated input tokens directly.

OpenAI applies prompt caching automatically for supported models when long prompts share a prefix. Anthropic lets developers mark cacheable prompt sections with cache controls. The implementation details vary by provider, but the principle is the same: keep stable content stable, and put variable content after it.

Bad structure:

User request: {{new_user_message}}

Company policy:
{{long_static_policy}}

Output rules:
{{formatting_rules}}

Better structure:

Company policy:
{{long_static_policy}}

Output rules:
{{formatting_rules}}

User request:
{{new_user_message}}

Why? Caches usually work on repeated prefixes. If you put the unique user message first, you destroy the shared prefix before the expensive reusable context even appears.

For apps with large static context, split your prompt like this:

const staticPrefix = `
You are a support assistant for Acme.

Policy:
${policyText}

Style:
- Direct
- Accurate
- No invented policy
- Escalate when policy is missing
`;

const requestSuffix = `
Customer message:
${customerMessage}

Return:
{
  "answer": string,
  "needs_human": boolean,
  "policy_refs": string[]
}
`;

const prompt = staticPrefix + requestSuffix;

Expected result: repeated calls with the same prefix should show cached input tokens or lower billed input, depending on provider reporting.

Step 6: Retrieve Less Context, But Better Context

Retrieval-augmented generation can save money or burn it faster. The difference is whether you retrieve relevant chunks or shovel half your knowledge base into every prompt.

Bad retrieval pattern:

• Fetch top 20 chunks
• Insert all chunks
• Ask the model to figure it out
• Wonder why the bill is ugly

Better retrieval pattern:

1. Fetch top 12 candidates with embeddings or search.
2. Re-rank them with a cheap model or keyword scoring.
3. Keep the top 3-5 chunks.
4. Remove boilerplate, navigation text, and duplicate passages.
5. Include source titles and dates so the model can reason about freshness.

Example context budget:

const MAX_CONTEXT_TOKENS = 3000;

function buildContext(chunks) {
  const selected = [];
  let used = 0;

  for (const chunk of chunks) {
    if (used + chunk.tokenCount > MAX_CONTEXT_TOKENS) continue;
    selected.push(`[${chunk.title}]\n${chunk.text}`);
    used += chunk.tokenCount;
  }

  return selected.join("\n\n---\n\n");
}

Expected result: the model sees fewer tokens, but the tokens are more relevant. Quality often improves because the model has less junk to reconcile.

Step 7: Cap Output Like You Mean It

Output tokens are usually more expensive than input tokens. Letting the model ramble is not generosity. It is poor engineering.

Set output limits per workflow:

const outputBudgets = {
  intent_classification: 20,
  ticket_summary: 250,
  email_draft: 500,
  research_answer: 900
};

Then enforce them in the API call:

await callModel({
  model,
  input,
  max_output_tokens: outputBudgets[workflow]
});

Also write prompts that specify the shape of the answer:

Return exactly:
- Summary: 2 sentences max
- Customer issue: 1 sentence
- Recommended action: 3 bullets max
- Escalate: yes or no

For structured tasks, use JSON schemas. Structured outputs reduce wandering, make validation easier, and prevent the classic “Here’s the JSON you requested:” wrapper nonsense.

Expected result: lower output-token spend and fewer parser failures.

Step 8: Kill Retry Loops

Retries quietly murder AI budgets.

A single failed request that retries twice has tripled cost. If your retry logic resends the same giant prompt each time, congratulations: you built a money shredder.

Separate retry types:

• Network failure: retry with backoff
• Rate limit: retry after the provider’s suggested delay
• Invalid JSON: repair locally first, then retry with a smaller correction prompt
• Low confidence: escalate model once, not forever
• Safety refusal: do not retry unless the user changes the request

Example retry policy:

const retryPolicy = {
  network_error: { maxRetries: 2, backoffMs: [500, 1500] },
  rate_limit: { maxRetries: 3, respectRetryAfter: true },
  invalid_json: { maxRetries: 1, useRepairPrompt: true },
  low_confidence: { maxRetries: 1, escalateModel: true },
  refusal: { maxRetries: 0 }
};

For JSON repair, do not resend the original 40-page context. Send the invalid output and the schema:

Fix this JSON so it matches the schema. Do not add new facts.

Schema:
{{schema}}

Invalid JSON:
{{bad_output}}

Expected result: retry costs become visible, bounded, and boring.

Step 9: Batch Anything That Does Not Need Real-Time Output

Users need instant answers. Your nightly enrichment job does not.

Move delayed work to batch processing:

• Product catalog tagging
• Lead enrichment
• Backfill summaries
• Large eval runs
• Import-time document analysis
• Offline moderation queues

Create JSONL requests for batch jobs:

{"custom_id":"ticket-1001","method":"POST","url":"/v1/responses","body":{"model":"small-general-model","input":"Summarize this ticket: ..."}}
{"custom_id":"ticket-1002","method":"POST","url":"/v1/responses","body":{"model":"small-general-model","input":"Summarize this ticket: ..."}}

The key product decision is latency tolerance. If a result can arrive in minutes or hours instead of seconds, it should not use the same path as an interactive chat response.

Expected result: real-time capacity is reserved for users, and background inference moves to a cheaper operating mode where your provider supports it.

Step 10: Add Cost Budgets Per User, Team, and Feature

Cost engineering without budgets is just optimism wearing a hoodie.

Set budgets at three levels:

• Per request: maximum tokens and maximum retries
• Per user or workspace: daily or monthly inference allowance
• Per feature: budget caps for beta features and high-risk workflows

Example:

function enforceBudget({ workspaceSpendToday, projectedRequestCost, dailyLimit }) {
  if (workspaceSpendToday + projectedRequestCost > dailyLimit) {
    return {
      allowed: false,
      fallback: "Use cached answer, shorter model path, or ask user to narrow scope."
    };
  }

  return { allowed: true };
}

Do not just throw an error when a user hits a limit. Offer a cheaper path:

This request is too large to process as-is. Try narrowing it to one document, one date range, or one question.

Expected result: your app degrades gracefully instead of silently torching margin.

Common Pitfalls

Pitfall 1: Optimizing Before Measuring

Shrinking prompts feels productive. It may save nothing if the real problem is retries, background jobs, or one enterprise customer uploading 600-page PDFs.

Fix: rank workflows by monthly spend and optimize the top three first.

Pitfall 2: Using the Cheapest Model Everywhere

Cheap models are not cheap if they fail twice as often, require longer prompts, or create support tickets.

Fix: measure cost per accepted result, not cost per API call.

Pitfall 3: Caching the Wrong Thing

Caching only works when repeated content stays in the same position. If the first line of your prompt includes timestamps, user IDs, random request IDs, or fresh messages, cache hits drop.

Fix: put stable instructions and context first. Put volatile request data last.

Pitfall 4: Letting Conversation History Grow Forever

Chat history is useful until it becomes landfill.

Fix: summarize old turns, keep recent turns verbatim, and retrieve specific facts when needed.

A simple policy:

const conversationPolicy = {
  keepLastTurns: 6,
  summarizeOlderThanTurns: 6,
  maxSummaryTokens: 500,
  retrieveFactsOnDemand: true
};

Pitfall 5: Treating Latency and Cost as Separate Problems

They are linked. Long prompts cost more and usually take longer. Long outputs cost more and make users wait. Cache hits can reduce both.

Fix: track latency and cost together per workflow.

A Practical Cost Review Checklist

Run this checklist before launching any AI feature:

• Does this workflow need an LLM at all?
• Can a smaller model handle the first attempt?
• Is there an escalation path for hard cases?
• Are input and output tokens logged?
• Is the prompt stripped of dead examples and repeated policy text?
• Is stable context placed before variable user input?
• Are max output tokens set per workflow?
• Are structured outputs used where parsing matters?
• Are retries capped by failure type?
• Can offline work move to batch processing?
• Are user, workspace, and feature budgets enforced?
• Is quality measured against real examples, not demo prompts?

If you cannot answer these, the feature is not production-ready. It is a bill with a UI.

What Good Looks Like

A cost-engineered AI app does not feel cheaper to the user. It feels faster, more consistent, and less weird.

Behind the scenes, it looks like this:

• Small tasks go to small models or normal code
• Hard tasks escalate only when needed
• Stable prompt context gets cached
• Retrieval sends only relevant chunks
• Outputs are capped and structured
• Retries are rare and bounded
• Batch jobs handle delayed work
• Dashboards show cost per workflow, not just total spend

The best version of inference cost engineering is invisible. Users get the same or better result. You stop paying for waste.

Takeaway

Do not start by asking, “Which model is cheapest?”

Ask: “Which parts of this request are actually worth expensive inference?”

That one question changes the architecture. You route better. You cache better. You retrieve less junk. You stop retrying blindly. You reserve serious models for serious work.

Make the app cheaper by making it sharper. Then keep measuring, because token waste always tries to crawl back in.