LLM Load Testing in Production: Find the Breaking Point Before Users Do

Your chatbot’s prettiest demo is worthless if ten real users can turn it into a timeout machine with a burn rate. llm load testing production systems is how you find that ugly truth before your customers do.

Traditional load tests ask: “How many requests per second can this endpoint survive?”

LLM load tests ask nastier questions:

• What happens when prompts get huge?
• What happens when the model stalls for 40 seconds?
• What happens when rate limits kick in mid-checkout?
• What happens when retries multiply your bill?
• What happens when one customer runs a 200-document analysis while everyone else waits?

That is the game now. Latency is variable. cost is variable. output size is variable. provider behavior is variable. Your app needs to stay boring anyway.

This tutorial shows how to run controlled production load tests for LLM apps without torching your budget, leaking user data, or accidentally DDoSing your own AI vendor.

Prerequisites

You do not need to be a full-time performance engineer, but you need access to a few moving parts.

You should have:

• A production or production-like LLM endpoint behind authentication
• A staging environment that mirrors production closely enough to test safely
• Access to logs, metrics, and traces
• A way to set provider spend limits or per-project budgets
• A load testing tool such as k6, Locust, Artillery, or Gatling
• A small set of realistic prompts
• Permission from your team to run controlled load in production

For this tutorial, the examples use k6 because it is simple, scriptable, and has first-class threshold support. The same strategy works with Locust or any other runner.

We will test an imaginary endpoint:

POST https://api.example.com/v1/assistant/respond

It accepts:

{
  "conversationId": "test-conversation-001",
  "message": "Summarize this customer complaint and suggest the next action.",
  "userTier": "pro"
}

And returns:

{
  "answer": "The customer is frustrated about delayed onboarding...",
  "model": "gpt-4.1-mini",
  "usage": {
    "inputTokens": 218,
    "outputTokens": 94,
    "totalTokens": 312
  }
}

Expected result after this setup: you know what endpoint you are testing, what data shape it expects, and what response fields matter for latency, errors, and cost.

Step 1: Define What “Broken” Means

Do not start by throwing traffic at the endpoint. That is amateur hour with a credit card attached.

Start by writing down your failure criteria.

For an LLM feature, “broken” usually includes more than HTTP 500s. Use this checklist:

• p95_latency_ms: 95% of requests finish under your target
• p99_latency_ms: the slow tail stays tolerable
• error_rate: failed requests stay below an agreed threshold
• rate_limit_rate: provider throttling stays rare
• timeout_rate: user-facing timeouts stay near zero
• tokens_per_minute: total token pressure stays under provider limits
• cost_per_1k_requests: test traffic stays inside budget
• queue_depth: async jobs do not pile up
• fallback_rate: fallback models or cached responses do not trigger too often

A sane first target for a user-facing chat workflow might look like this:

success_criteria:
  p95_latency_ms: 8000
  p99_latency_ms: 20000
  error_rate: 0.01
  timeout_rate: 0.005
  rate_limit_rate: 0.005
  max_test_cost_usd: 25
  max_duration_minutes: 20

That does not mean those numbers are universally correct. A coding assistant, customer support bot, legal document analyzer, and AI search box have different tolerance levels.

Expected result: you have a measurable definition of failure before the test starts. No vibes. No “seems slow.” No dashboard staring contest.

Step 2: Add Production Observability First

If your LLM endpoint only logs 200 OK, your load test will be mostly theater.

You need to capture the pieces that make LLM workloads weird:

• Request ID
• User tier or traffic class
• Route name
• Model name
• Provider
• Input tokens
• Output tokens
• Total tokens
• Cache hit or miss
• Retry count
• Timeout count
• Provider status code
• Provider request ID when available
• End-to-end latency
• Provider latency
• Queue wait time
• Moderation or guardrail latency if used

OpenAI’s API responses include headers that can help with production debugging, including request IDs and rate limit information. Log them. Do not log raw user prompts unless you have a clear data policy and a reason.

A minimal Node-style logging wrapper might look like this:

async function callModel({ traceId, messages, model }) {
  const startedAt = Date.now();

  const response = await client.responses.create({
    model,
    input: messages,
    metadata: {
      trace_id: traceId
    }
  });

  const latencyMs = Date.now() - startedAt;

  logger.info({
    event: "llm_request_completed",
    traceId,
    model,
    latencyMs,
    inputTokens: response.usage?.input_tokens,
    outputTokens: response.usage?.output_tokens,
    totalTokens: response.usage?.total_tokens,
    providerRequestId: response._request_id
  });

  return response;
}

If your stack supports OpenTelemetry, add spans around the application handler, retrieval step, model call, cache lookup, and response streaming. The point is not to worship tracing diagrams. The point is to know where the time went when the graph turns red.

Expected result: one test request creates useful metrics and logs. You can answer: did the app slow down, did the provider slow down, did retrieval slow down, or did your queue turn into soup?

Step 3: Use Synthetic Production Users

Never use raw customer conversations as load test fixtures. That is how compliance meetings get spicy.

Create synthetic prompts that match production shape without exposing private data. You need variety, because LLM cost and latency are driven by token volume and task type.

Create a file called prompts.json:

[
  {
    "name": "short_support_question",
    "message": "A customer says their invoice is wrong. Ask two clarifying questions and suggest the next action.",
    "expectedClass": "short"
  },
  {
    "name": "medium_summary",
    "message": "Summarize this onboarding call transcript into decisions, blockers, and next steps: [synthetic transcript with 1200 words]",
    "expectedClass": "medium"
  },
  {
    "name": "long_document_analysis",
    "message": "Analyze this synthetic vendor contract and identify renewal terms, termination risk, and payment obligations: [synthetic contract with 5000 words]",
    "expectedClass": "long"
  },
  {
    "name": "retrieval_heavy_question",
    "message": "Find the policy that applies to enterprise refunds and write a response for a finance admin.",
    "expectedClass": "rag"
  }
]

Your fixture set should match your actual traffic mix. If 80% of production requests are short support questions and 5% are long document analysis, your load test should reflect that.

Expected result: you have representative prompts that stress the system without using private user data.

Step 4: Create a Safe Test Identity

Production load testing should be clearly labeled.

Create a dedicated test user, project, organization, API key, or tenant depending on your architecture. Then make every test request easy to identify.

Use headers like:

X-Load-Test: llm-production-2026-07-31
X-Test-Run-Id: run_2026_07_31_001
X-Client-Request-Id: loadtest-run-001-000042

Your app should store those values in logs and traces.

Also add guardrails:

• Block test users from sending email, billing customers, or changing real records
• Route destructive actions to a sandbox
• Disable external webhooks for test traffic
• Put a hard cap on concurrent virtual users
• Put a hard cap on total requests
• Put a hard cap on test duration
• Put a hard cap on LLM spend

Expected result: you can run traffic in production and isolate it from real customer behavior, billing, and side effects.

Step 5: Write the First k6 Test

Install k6, then create llm-load-test.js.

import http from "k6/http";
import { check, sleep } from "k6";
import { Rate, Trend, Counter } from "k6/metrics";

const errorRate = new Rate("llm_errors");
const timeoutRate = new Rate("llm_timeouts");
const tokenTrend = new Trend("llm_total_tokens");
const rateLimitCounter = new Counter("llm_rate_limits");

const prompts = [
  {
    name: "short_support_question",
    message: "A customer says their invoice is wrong. Ask two clarifying questions and suggest the next action."
  },
  {
    name: "medium_summary",
    message: "Summarize this synthetic onboarding call transcript into decisions, blockers, and next steps."
  },
  {
    name: "retrieval_heavy_question",
    message: "Find the policy that applies to enterprise refunds and write a response for a finance admin."
  }
];

export const options = {
  scenarios: {
    steady_load: {
      executor: "ramping-vus",
      stages: [
        { duration: "2m", target: 5 },
        { duration: "5m", target: 10 },
        { duration: "2m", target: 0 }
      ]
    }
  },
  thresholds: {
    http_req_failed: ["rate<0.01"],
    http_req_duration: ["p(95)<8000", "p(99)<20000"],
    llm_errors: ["rate<0.01"],
    llm_timeouts: ["rate<0.005"],
    llm_total_tokens: ["p(95)<3000"]
  }
};

export default function () {
  const prompt = prompts[Math.floor(Math.random() * prompts.length)];
  const requestId = `loadtest-${__VU}-${__ITER}`;

  const payload = JSON.stringify({
    conversationId: `load-test-${__VU}`,
    message: prompt.message,
    userTier: "pro"
  });

  const res = http.post(
    `${__ENV.BASE_URL}/v1/assistant/respond`,
    payload,
    {
      headers: {
        "Content-Type": "application/json",
        "Authorization": `Bearer ${__ENV.LOAD_TEST_TOKEN}`,
        "X-Load-Test": "llm-production-2026-07-31",
        "X-Client-Request-Id": requestId
      },
      timeout: "30s",
      tags: {
        prompt_name: prompt.name
      }
    }
  );

  const ok = check(res, {
    "status is 200": (r) => r.status === 200,
    "has answer": (r) => Boolean(r.json("answer")),
    "has usage": (r) => Boolean(r.json("usage.totalTokens"))
  });

  const wasTimeout = res.error && String(res.error).includes("timeout");
  const wasRateLimited = res.status === 429;

  errorRate.add(!ok);
  timeoutRate.add(wasTimeout);
  rateLimitCounter.add(wasRateLimited);

  const totalTokens = Number(res.json("usage.totalTokens") || 0);
  if (totalTokens > 0) {
    tokenTrend.add(totalTokens);
  }

  sleep(1);
}

Run it against staging first:

BASE_URL="https://staging-api.example.com" \
LOAD_TEST_TOKEN="your-test-token" \
k6 run llm-load-test.js

Expected result: staging receives controlled traffic, k6 reports latency percentiles, failed checks, HTTP failures, and token trends.

Step 6: Run a Baseline Test

Your first production test should be boring on purpose.

Use a small load:

stages: [
  { duration: "2m", target: 2 },
  { duration: "5m", target: 2 },
  { duration: "1m", target: 0 }
]

This test answers one question: does the production path behave as expected under tiny synthetic load?

Watch:

• Error rate
• p95 latency
• provider 429s
• app CPU and memory
• database connections
• vector database latency
• queue depth
• total token usage
• cost estimate
• logs for test request IDs

Expected result: the test should pass comfortably. If it does not, stop. You do not need a bigger test. You already found a production problem.

Step 7: Increase Load Until Something Bends

Now ramp carefully.

A useful sequence:

Run 1: 2 virtual users for 5 minutes
Run 2: 5 virtual users for 5 minutes
Run 3: 10 virtual users for 10 minutes
Run 4: 25 virtual users for 10 minutes
Run 5: 50 virtual users for 10 minutes

Between runs, inspect dashboards. Do not stack tests without reading the results. That is how you confuse your own incident timeline.

At each level, capture:

run_id: run_2026_07_31_003
virtual_users: 10
duration: 10m
total_requests: 486
success_rate: 99.4%
p95_latency_ms: 6200
p99_latency_ms: 14100
rate_limit_count: 0
timeout_count: 2
estimated_cost_usd: 8.40
notes: "RAG latency rose after 8 minutes; app remained stable."

The breaking point is not always a crash. For LLM systems, the first real failure is often one of these:

• p99 latency becomes absurd
• provider rate limits start appearing
• retries create a traffic multiplier
• token usage spikes because prompts expand
• streaming starts but never completes
• queues drain too slowly
• vector search becomes the bottleneck
• fallback model quality degrades the answer
• cache hit rate collapses under varied prompts

Expected result: you find the first constraint and can name it precisely.

Bad result: “The AI got slow.”

Good result: “At 25 virtual users, p99 hit 34 seconds because provider latency rose and our retry policy sent duplicate model calls after 15 seconds.”

Step 8: Test the Failure Modes on Purpose

A load test that only checks the happy path is a demo with extra steps.

Create targeted scenarios.

Rate Limit Scenario

Simulate provider throttling by forcing your model wrapper to return 429 for a percentage of test traffic.

Expected behavior:

• The app backs off
• The user sees a useful message
• Retries are capped
• No request retries forever
• Metrics show rate limit events clearly

Example wrapper logic:

if (isLoadTestRequest(req) && req.headers["x-force-provider-429"] === "true") {
  throw new ProviderRateLimitError("Synthetic load test rate limit");
}

Slow Model Scenario

Force artificial delay before returning the model response.

Expected behavior:

• The frontend does not hang silently
• Streaming endpoints send heartbeat events if needed
• Server timeouts are longer than provider timeouts
• User-facing timeouts are graceful
• Background jobs can continue without blocking the web request

Huge Prompt Scenario

Send long synthetic prompts that approximate your largest real requests.

Expected behavior:

• The app rejects oversized prompts early
• Token estimates are logged
• Expensive requests require the right plan or workflow
• The system does not discover the problem after sending the request to the model

Partial Outage Scenario

Make the primary model unavailable and test fallback logic.

Expected behavior:

• Fallbacks activate only when intended
• Quality-sensitive workflows do not silently downgrade
• The fallback path is observable
• Alerts fire when fallback rate crosses a threshold

Expected result: you know how the system behaves when the provider, prompt, retrieval layer, or timeout policy stops being friendly.

Step 9: Watch Cost Like a Production Metric

LLM load testing has a special failure mode: the test passes and the bill punches you later.

Before every run, estimate cost:

estimated_requests = virtual_users * requests_per_user_per_minute * duration_minutes
estimated_tokens = estimated_requests * average_tokens_per_request
estimated_cost = estimated_tokens * model_price_per_token

Use your provider’s current pricing page for the actual model price. Do not paste stale prices into your test plan and pretend that is engineering.

Add a runtime cap where possible:

load_test_budget:
  max_requests: 2000
  max_tokens: 3000000
  max_duration_minutes: 20
  max_estimated_cost_usd: 50

Your application can enforce a test-only budget:

if (isLoadTestRequest(req)) {
  const usage = await getLoadTestUsage(req.headers["x-test-run-id"]);

  if (usage.totalTokens > 3000000) {
    return res.status(429).json({
      error: "Load test token budget exceeded"
    });
  }
}

Expected result: your test cannot become an unbounded spend event.

Step 10: Turn Results Into Capacity Rules

The output of a production LLM load test should not be a screenshot in Slack. It should become operating rules.

Write down:

• Maximum safe concurrent requests by endpoint
• Maximum safe concurrent long-context requests
• Token-per-minute budget by model
• Timeout settings by route
• Retry policy by error type
• Queue limits
• Fallback activation rules
• Alert thresholds
• Customer-facing degradation behavior

Example:

capacity_rules:
  assistant_chat:
    safe_concurrent_requests: 40
    p95_target_ms: 8000
    p99_target_ms: 20000
    max_retries: 1
    retry_on:
      - "429"
      - "provider_timeout"
    do_not_retry_on:
      - "400"
      - "content_policy_error"
      - "prompt_too_large"
  long_document_analysis:
    safe_concurrent_requests: 8
    execution_mode: "background_job"
    user_timeout_behavior: "return_job_id"

Expected result: product, engineering, and support know what the system can handle and what happens when demand crosses the line.

Common Pitfalls

Testing the Model Instead of Your Product

Sending raw prompts directly to an LLM API tells you provider latency. That is useful, but it is not your product.

Your product includes auth, routing, prompt assembly, retrieval, moderation, logging, retries, streaming, database writes, and frontend timeout behavior. Test the whole path.

Ignoring Token Mix

Two requests per second can mean almost nothing if one request uses 300 tokens and another uses 30,000. Track tokens as aggressively as request count.

Retrying Too Hard

Retries feel responsible until they multiply failure. Under load, a naive retry policy can turn 100 requests into 250 provider calls.

Use capped retries, exponential backoff, jitter, and clear retry rules. Never retry client errors. Be careful retrying long-running completions after the provider may already be processing them.

Using Real Customer Data

Do not do it casually. Synthetic data is usually enough for performance testing. If you truly need production samples, anonymize them, minimize them, and get the right approvals.

Forgetting Streaming Behavior

A streaming endpoint can look fast because the first token arrives quickly while the full response still takes forever. Measure both:

• time to first token
• time to final token
• stream interruption rate
• client disconnect rate

Running Big Tests Without a Kill Switch

Every production load test needs a stop button. Ideally several:

• k6 max duration
• provider budget limit
• app-level test budget
• feature flag to block test traffic
• dashboard alert on cost or error spike

Treating 429s as “Unexpected”

Rate limits are not weird. They are part of the contract. Your app should handle them like weather, not like a meteor strike.

A Practical Production Load Test Plan

Use this as your first real runbook.

1. Confirm production dashboards are working.
2. Confirm test identity cannot mutate real user data.
3. Confirm provider and app budgets are set.
4. Run staging baseline.
5. Run production baseline with 2 virtual users.
6. Inspect latency, errors, tokens, and cost.
7. Ramp to 5, 10, 25, then 50 virtual users.
8. Stop at the first failed threshold.
9. Run one targeted failure scenario.
10. Document the breaking point and update capacity rules.

The winning move is not to run the biggest test. The winning move is to learn exactly where the system fails, cheaply and safely.

Final Takeaway

LLM load testing in production is not about proving your app is invincible. It is about finding the point where latency, rate limits, retries, queues, and cost start working against you.

Start small. Instrument first. Use synthetic prompts. Set budgets. Ramp slowly. Break one thing at a time.

Your users do not care that the model is nondeterministic, the provider has limits, or long prompts are expensive. They care that the product works when they need it. Find the breaking point before they do.