Model Routing in Production: Send Every AI Task to the Right Model

Your AI bill is not high because users ask hard questions. It is high because your app treats every task like a board exam.

That is the core problem behind model routing production ai systems: most products send password-reset copy, JSON cleanup, support triage, legal-ish reasoning, coding help, and executive summaries to the same model. That is lazy architecture. It works in the demo. Then traffic arrives, latency spikes, invoices get ugly, and everyone pretends “AI is expensive” instead of admitting the router is missing.

Model routing fixes that. You classify the job, pick the cheapest model that can do it well, escalate when needed, and log enough data to improve the system instead of guessing.

Prerequisites

You do not need to be a full-time ML engineer to build this. You do need a few pieces in place.

You should have:

• A production app that sends tasks to one or more AI models
• Basic logging for request type, latency, failures, and token usage
• Access to at least two model tiers, such as fast/cheap and slower/stronger models
• A small test set of real prompts from your product
• A way to compare outputs, even if the first version is human review in a spreadsheet

The examples below use TypeScript-style pseudocode because the pattern matters more than the vendor SDK. You can apply the same design with OpenAI, Anthropic, Gemini, Amazon Bedrock, LiteLLM, an internal gateway, or your own queue workers.

What Model Routing Actually Means

Model routing is the decision layer between your application and the model provider.

Instead of this:

const response = await ai.generate({
  model: "best-expensive-model",
  input: userPrompt
});

You do this:

const route = chooseRoute({
  taskType,
  riskLevel,
  inputSize,
  userTier,
  latencyBudgetMs
});

const response = await ai.generate({
  model: route.model,
  input: userPrompt,
  options: route.options
});

That tiny layer becomes one of the highest-leverage parts of your AI stack.

A good router considers:

• Task type: classification, extraction, summarization, reasoning, writing, coding, retrieval, tool use
• Risk: whether a bad answer is annoying, costly, unsafe, or legally dangerous
• Latency: whether the user is waiting in the UI or a background job can take longer
• Cost: input tokens, output tokens, retries, tool calls, cache hits, and batch opportunities
• Context size: whether the model needs a long window or only a compact prompt
• Confidence: whether the first model is good enough or should escalate

The point is not to worship cheap models. The point is to stop wasting premium models on low-stakes chores.

Step 1: Split Your AI Workloads Into Task Classes

Start brutally simple. Do not build a “smart” router before you know what you are routing.

Create a task taxonomy like this:

Task class	Example	Default route
classify	Detect intent, label ticket, score sentiment	Cheap fast model
extract	Pull fields from invoice, parse resume	Cheap or mid model
rewrite	Improve tone, shorten copy, draft email	Mid model
summarize	Summarize meeting, article, thread	Mid model
reason	Analyze tradeoffs, diagnose issue	Strong model
code	Generate patch, explain stack trace	Strong coding model
safety_review	Policy, legal, medical, financial review	Strong model plus stricter checks
fallback	Unknown or failed classification	Strong model

Then encode it.

type TaskClass =
  | "classify"
  | "extract"
  | "rewrite"
  | "summarize"
  | "reason"
  | "code"
  | "safety_review"
  | "fallback";

type AiRequest = {
  taskClass: TaskClass;
  input: string;
  userTier: "free" | "pro" | "enterprise";
  latencyBudgetMs: number;
  riskLevel: "low" | "medium" | "high";
};

Expected result: every AI request now has a label before it hits a model. Even if the labels are manual at first, you have a routing surface.

Step 2: Create Model Tiers, Not Model Chaos

Do not scatter raw model names across your codebase. That is how migrations become a haunted spreadsheet.

Create capability tiers:

type ModelTier = "fast" | "balanced" | "strong" | "specialist";

const MODEL_REGISTRY: Record<ModelTier, string> = {
  fast: process.env.AI_MODEL_FAST!,
  balanced: process.env.AI_MODEL_BALANCED!,
  strong: process.env.AI_MODEL_STRONG!,
  specialist: process.env.AI_MODEL_SPECIALIST!
};

Your environment might look like this:

AI_MODEL_FAST=gpt-5.6-luna
AI_MODEL_BALANCED=gpt-5.6-terra
AI_MODEL_STRONG=gpt-5.6-sol
AI_MODEL_SPECIALIST=your-coding-or-domain-model

Use your actual provider models. The names above are examples of tiering, not a commandment carved into stone.

Expected result: product code asks for fast, balanced, or strong. Your infrastructure decides which provider/model currently backs that tier.

Step 3: Write the First Routing Rules

Your first router should be boring. Boring is good. Boring can be debugged at 2 a.m.

function chooseRoute(request: AiRequest): {
  tier: ModelTier;
  reason: string;
  maxOutputTokens: number;
} {
  if (request.riskLevel === "high") {
    return {
      tier: "strong",
      reason: "high_risk_request",
      maxOutputTokens: 2000
    };
  }

  if (request.taskClass === "classify" || request.taskClass === "extract") {
    return {
      tier: "fast",
      reason: "simple_structured_task",
      maxOutputTokens: 500
    };
  }

  if (request.taskClass === "rewrite" || request.taskClass === "summarize") {
    return {
      tier: "balanced",
      reason: "language_quality_needed",
      maxOutputTokens: 1200
    };
  }

  if (request.taskClass === "reason" || request.taskClass === "code") {
    return {
      tier: "strong",
      reason: "complex_reasoning_needed",
      maxOutputTokens: 2500
    };
  }

  return {
    tier: "strong",
    reason: "fallback_unknown_task",
    maxOutputTokens: 2000
  };
}

Then call the model through the registry.

async function runAi(request: AiRequest) {
  const route = chooseRoute(request);
  const model = MODEL_REGISTRY[route.tier];

  const startedAt = Date.now();

  const response = await ai.generate({
    model,
    input: request.input,
    maxOutputTokens: route.maxOutputTokens
  });

  await logAiRoute({
    taskClass: request.taskClass,
    riskLevel: request.riskLevel,
    model,
    tier: route.tier,
    routeReason: route.reason,
    latencyMs: Date.now() - startedAt,
    inputChars: request.input.length
  });

  return response;
}

Expected result: you can now answer the most important production question: “Which tasks are using which models, and why?”

Step 4: Add Escalation When Cheap Is Not Good Enough

A router without escalation is just a cost-cutting machine. That is dumb. Cheap wrong answers are still wrong.

For structured tasks, ask the first model to return confidence and validation fields.

type ExtractionResult = {
  customerName?: string;
  invoiceTotal?: number;
  dueDate?: string;
  confidence: number;
  missingFields: string[];
};

function shouldEscalateExtraction(result: ExtractionResult) {
  return (
    result.confidence < 0.82 ||
    result.missingFields.length > 0 ||
    typeof result.invoiceTotal !== "number"
  );
}

Then retry with a stronger model only when needed.

async function extractInvoice(input: string) {
  const first = await ai.generateJson<ExtractionResult>({
    model: MODEL_REGISTRY.fast,
    input,
    schema: "invoice_extraction"
  });

  if (!shouldEscalateExtraction(first)) {
    return {
      result: first,
      modelTierUsed: "fast",
      escalated: false
    };
  }

  const second = await ai.generateJson<ExtractionResult>({
    model: MODEL_REGISTRY.strong,
    input,
    schema: "invoice_extraction"
  });

  return {
    result: second,
    modelTierUsed: "strong",
    escalated: true
  };
}

Expected result: easy cases stay cheap. Messy cases get the expensive brain. This is where real savings show up without wrecking quality.

Step 5: Route by Latency Budget

Not every request deserves the same wait time.

A user staring at a chat box needs a fast answer. A nightly report can wait. A compliance review should take the time it takes.

Add a latency budget to the route.

function chooseLatencyAwareRoute(request: AiRequest) {
  if (request.latencyBudgetMs < 1500 && request.riskLevel !== "high") {
    return {
      tier: "fast" as ModelTier,
      reason: "interactive_latency_budget",
      maxOutputTokens: 700
    };
  }

  return chooseRoute(request);
}

For long-running work, push it to a job queue instead of punishing the user interface.

async function handleReportGeneration(input: string) {
  const job = await queue.add("generate_report", {
    input,
    taskClass: "reason",
    riskLevel: "medium",
    latencyBudgetMs: 60000
  });

  return {
    status: "queued",
    jobId: job.id
  };
}

Expected result: your product stops pretending that “AI request” is one performance category. Interactive and background workloads behave differently because they are different.

Step 6: Add Provider Failover

Model routing is not only about cost. It is also about uptime.

Providers rate-limit. Regions wobble. New model releases occasionally behave weirdly. If your app has one model path, your app has one throat to choke.

Add fallback providers or deployments behind each tier.

const MODEL_POOLS: Record<ModelTier, string[]> = {
  fast: [
    process.env.AI_FAST_PRIMARY!,
    process.env.AI_FAST_BACKUP!
  ],
  balanced: [
    process.env.AI_BALANCED_PRIMARY!,
    process.env.AI_BALANCED_BACKUP!
  ],
  strong: [
    process.env.AI_STRONG_PRIMARY!,
    process.env.AI_STRONG_BACKUP!
  ],
  specialist: [
    process.env.AI_SPECIALIST_PRIMARY!,
    process.env.AI_SPECIALIST_BACKUP!
  ]
};

Try the primary, then fail over on retryable errors.

function isRetryableAiError(error: unknown) {
  const message = String(error);
  return (
    message.includes("rate_limit") ||
    message.includes("timeout") ||
    message.includes("overloaded") ||
    message.includes("5")
  );
}

async function generateWithFailover(tier: ModelTier, input: string) {
  const models = MODEL_POOLS[tier];
  let lastError: unknown;

  for (const model of models) {
    try {
      return await ai.generate({ model, input });
    } catch (error) {
      lastError = error;

      if (!isRetryableAiError(error)) {
        throw error;
      }

      await logAiFailover({ tier, model, error: String(error) });
    }
  }

  throw lastError;
}

Expected result: one provider hiccup does not automatically become your outage.

If you do not want to build this yourself, tools like LiteLLM can handle routing, load balancing, cooldowns, retries, and fallbacks across deployments. Amazon Bedrock also offers intelligent prompt routing for supported model families, where routing can optimize for response quality and cost inside Bedrock’s constraints.

Step 7: Measure Quality, Not Just Spend

The fastest way to ruin model routing is to optimize only for cost.

Track these metrics per task class:

Metric	Why it matters
Cost per successful request	The only cost number that matters
P95 latency	Average latency lies to your face
Escalation rate	Shows whether the cheap tier is overloaded with hard tasks
Retry/failover rate	Reveals provider or region instability
User correction rate	Good proxy for bad answers
Human review rejection rate	Critical for internal workflows
Output validation failure rate	Best signal for structured tasks

Your log shape can be simple:

type AiRouteLog = {
  requestId: string;
  taskClass: TaskClass;
  tier: ModelTier;
  model: string;
  routeReason: string;
  escalated: boolean;
  latencyMs: number;
  inputTokens?: number;
  outputTokens?: number;
  estimatedCostUsd?: number;
  validationPassed?: boolean;
  userAccepted?: boolean;
};

Expected result: you can compare routes using evidence instead of vibes. If fast handles 92% of extraction tasks with a 1.8% validation failure rate, keep it. If it fails 18%, stop being cheap.

Step 8: Build an Evaluation Set

Before changing routes, freeze a test set.

Use 50 to 200 real examples per major task class. Strip private data. Keep the nasty edge cases. The edge cases are the point.

Your eval table should include:

Field	Example
taskClass	extract
input	Redacted invoice text
expected	JSON object
mustPass	Total must match, date must be ISO
riskLevel	medium
notes	Vendor uses European date format

Run each candidate model against the set.

async function runEvalSet(examples: EvalExample[], tier: ModelTier) {
  const results = [];

  for (const example of examples) {
    const output = await ai.generate({
      model: MODEL_REGISTRY[tier],
      input: example.input
    });

    results.push({
      id: example.id,
      taskClass: example.taskClass,
      tier,
      passed: validateOutput(example, output)
    });
  }

  return summarizeEvalResults(results);
}

Expected result: routing changes become boring release decisions. You compare pass rate, latency, and cost, then ship the route that wins.

Step 9: Add Guardrails for High-Risk Tasks

Some requests should never be routed purely by price.

For high-risk categories, use stricter routing:

function classifyRisk(input: string): "low" | "medium" | "high" {
  const riskyPatterns = [
    "diagnose",
    "lawsuit",
    "tax filing",
    "investment advice",
    "wire money",
    "delete production"
  ];

  return riskyPatterns.some(pattern =>
    input.toLowerCase().includes(pattern)
  )
    ? "high"
    : "low";
}

Then force a stronger path.

if (request.riskLevel === "high") {
  return {
    tier: "strong",
    reason: "forced_high_risk_route",
    maxOutputTokens: 3000
  };
}

For regulated or sensitive workflows, add human approval, retrieval from approved sources, and output constraints. Do not let the router improvise its way through legal, medical, financial, or security-sensitive answers.

Expected result: your router saves money where mistakes are cheap and spends more where mistakes matter.

Common Pitfalls

Pitfall 1: Routing by Prompt Length Alone

Long prompts are not always hard. Short prompts are not always easy.

“Summarize this 40-page transcript” may be straightforward. “Should we terminate this enterprise contract?” is short and dangerous.

Use prompt length as one signal, not the whole brain.

Pitfall 2: No Fallback Path

If your cheap model fails validation and you return garbage, that is not routing. That is sabotage with a lower invoice.

Always define escalation criteria for structured and high-value tasks.

Pitfall 3: Hiding Model Names From Logs

You do not need to expose model names to users, but your internal logs must include the actual model used. Without that, debugging quality regressions is basically archaeology.

Log tier, provider, model, route reason, latency, token usage, and escalation status.

Pitfall 4: Changing Routes Without Evals

A model that feels better in five manual tests can still fail your weird production cases.

Before replacing a route, run the eval set. After replacing it, monitor production drift.

Pitfall 5: Ignoring Provider-Specific Features

Some providers offer native routing, caching, batch APIs, prompt routers, or deployment-level load balancing. Use them when they fit. Just do not surrender your entire product policy to a black box.

Your app still needs to know which tasks are risky, which customers paid for premium latency, and which outputs require validation.

A Practical Routing Policy You Can Steal

Start with this:

Workload	Route	Escalation trigger
Intent classification	Fast	Confidence below 0.85
Data extraction	Fast	Schema failure or missing required fields
Short rewrite	Balanced	User asks for legal/medical/financial claims
Long summarization	Balanced	Summary fails coverage checks
Deep analysis	Strong	No downgrade
Coding task	Strong or specialist	Tests fail or patch incomplete
Safety-sensitive answer	Strong	Human review for high impact
Unknown task	Strong	Add new task class later

This is not fancy. It is useful. Fancy comes later.

Expected Production Results

After you deploy model routing, you should see four changes.

First, cost per request drops because simple work stops hitting premium models.

Second, latency improves because fast models handle high-volume lightweight tasks.

Third, reliability improves because failover gives you more than one path through provider trouble.

Fourth, model upgrades become less terrifying because you can swap the model behind a tier and compare results by task class.

The first version will be imperfect. That is fine. A simple explicit router with logs beats a magical one-model blob every day of the week.

Final Takeaway

Model routing is not an optimization trick. It is production architecture.

Send cheap tasks to cheap models. Send hard tasks to strong models. Escalate when confidence drops. Fail over when providers choke. Measure everything.

That is how you build AI systems that do not torch budget, latency, or user trust just because every prompt got treated like the most important prompt in the company.