Build an LLM Cost Calculator: Forecast Tokens, Latency, and Margin Before You Ship

The cheapest model on a pricing page can produce the most expensive product in production. Retries, oversized context, uncached prefixes, tool loops, and verbose outputs routinely matter more than the headline price.

An LLM cost calculator should estimate the cost of a successful user outcome, not merely one model request. Build it early enough to influence architecture and pricing.

Define the Unit You Sell

Choose a product unit before writing code. It might be one support resolution, one analyzed document, one generated video script, or one completed research report.

For that unit, map the complete request path:

1. classification or routing
2. retrieval and reranking
3. primary model generation
4. tool calls
5. validation
6. retries or fallback models
7. storage and background processing

If an “answer” requires three model calls, the calculator must represent all three.

Create separate scenarios for median, heavy, and failure cases. Averages hide the customers who upload enormous documents or trigger repeated tool loops.

Model Token Cost Correctly

Store provider prices as configuration with an effective date. Never hard-code a pricing-page snapshot deep inside application logic.

The basic formula for one call is:

input_cost = input_tokens / 1,000,000 × input_price
cached_input_cost = cached_tokens / 1,000,000 × cached_price
output_cost = output_tokens / 1,000,000 × output_price
call_cost = input_cost + cached_input_cost + output_cost

Keep cached and uncached input separate. Some providers also price long contexts, batch requests, reasoning tokens, images, audio, or web search differently. Represent those as line items rather than forcing everything into one token rate.

A minimal TypeScript model:

type ModelPrice = {
  inputPerMillion: number;
  cachedInputPerMillion?: number;
  outputPerMillion: number;
};

type Usage = {
  inputTokens: number;
  cachedInputTokens: number;
  outputTokens: number;
};

export function modelCost(price: ModelPrice, usage: Usage) {
  const uncached = Math.max(0, usage.inputTokens - usage.cachedInputTokens);
  return (
    (uncached / 1_000_000) * price.inputPerMillion +
    (usage.cachedInputTokens / 1_000_000) *
      (price.cachedInputPerMillion ?? price.inputPerMillion) +
    (usage.outputTokens / 1_000_000) * price.outputPerMillion
  );
}

Use actual usage returned by the provider in production. Tokenizers are useful for forecasts, but provider billing records are the source of truth.

Add Retries, Tools, and Failure Probability

Suppose 8% of requests need one retry and 1% need two. Expected model cost is not simply the base call.

expected_calls = 1 + retry_once_rate + 2 × retry_twice_rate
expected_model_cost = base_call_cost × expected_calls

This approximation works when retries resemble the original call. If fallback calls use larger models or longer prompts, model them as separate branches.

Tool calls can dominate cost even when they are not billed in tokens. Add search fees, browser time, database queries, third-party API charges, image generation, and compute for code execution. Include failure cleanup and moderation where relevant.

Do not forget free-but-expensive infrastructure: vector databases, queues, traces, object storage, and egress.

Forecast Latency Alongside Cost

Users experience a chain, not a ledger. Track:

• queue time
• time to first token
• generation time
• tool latency
• retry delay
• post-processing time

Parallel calls should contribute their maximum latency, not their sum. Sequential calls add together. A cheaper architecture that takes 45 seconds may reduce conversion enough to destroy the apparent savings.

Model latency with percentiles. The median demonstrates the normal experience; p95 reveals whether heavy requests break the product promise.

sequential_total = route_p95 + retrieval_p95 + generation_p95
parallel_total = max(search_p95, database_p95) + synthesis_p95

Convert Cost Into Gross Margin

Once you have expected cost per unit, connect it to revenue.

gross_margin =
  (revenue_per_unit - variable_cost_per_unit) / revenue_per_unit

Variable cost should include payment fees, support burden that scales with use, and non-LLM services. Then stress-test the assumptions:

• output tokens double
• cache hit rate falls by half
• the fallback model activates more often
• a provider raises price
• power users consume ten times the median

For subscription products, simulate a distribution of customer usage. “100 requests included” means little if one request can contain a two-page email and another a 500-page PDF.

Instrument the Calculator With Real Data

Forecasts should graduate into an operating dashboard. Log model, prompt version, input and output usage, cache usage, tool charges, latency, retry reason, and customer or plan identifier. Avoid storing raw sensitive prompts when aggregates are enough.

Compare forecast to actual weekly. If the difference grows, identify whether usage patterns, routing, or prices changed.

Set alerts for cost per successful task rather than cost per request. A request that returns malformed JSON is cheap in the provider dashboard and worthless to the customer.

Make Architecture Decisions With It

The calculator becomes valuable when it changes design:

• summarize old context instead of replaying it
• route simple tasks to smaller models
• cache stable prompt prefixes
• cap tool loops
• validate outputs before an expensive downstream step
• batch background work
• ask users to narrow huge inputs

Do not optimize blindly. A larger model that succeeds on the first try can be cheaper than a smaller model that retries, escalates, and generates support tickets.

Build the spreadsheet or service before launch, attach every assumption to an observable metric, and revisit it whenever prompts, providers, or product limits change. Unit economics should be an engineering input, not a surprise discovered after growth.