Why Is a 32B Model Cheaper to Serve Than a 7B One?

A 32B model should cost more to serve than a 7B one. Four times the parameters, four times the FLOPs per token. But on real serving stacks, the 32B model often wins on cost per useful answer. The gap isn’t marketing.

The FLOPs math is only the first line

Raw compute scales with parameter count. A 7B model at FP16 needs about 14 GB of weights; a 32B needs about 64 GB. That’s why the 7B fits on a single 24 GB GPU and the 32B doesn’t. But serving cost is not weights divided by throughput.

Serving cost is GPU-hours per completed task. A 7B model that fails 40% of the time and needs retries, repair loops, or a bigger fallback model is not cheap. A 32B model that one-shots the task can be cheaper per success even at 4x the per-token price.

Quantization changes the denominator

A 32B model at 4-bit quantization needs roughly 16–20 GB of weights, depending on group size and whether you keep some layers at higher precision. That fits on a single 24 GB card. Llama 3.1 70B at 4-bit needs about 40 GB, which is why it lands on 48 GB cards or splits across two 24 GB GPUs.

Once a 32B model fits on one GPU, the cost comparison changes. You’re no longer paying for multi-GPU orchestration, NVLink overhead, or tensor-parallel latency. A single-card deployment with vLLM or SGLang can hit far better tokens-per-dollar than a smaller model that needs a fallback chain.

The bill is dominated by utilization, not parameter count

An idle 7B GPU costs the same per hour as an idle 32B GPU if both sit on the same card class. The real number is cost per token at realistic batch sizes. A 7B model at batch 1 on an H100 might generate 200 tokens/sec; a quantized 32B might do 60–80 tokens/sec. That’s a 3x throughput gap, not a 4x price gap.

But add a verifier, a retry loop, or a second model for structured output repair, and the 7B pipeline’s effective cost climbs. The 32B model’s advantage is fewer moving parts. Fewer calls, fewer tokens, fewer failures.

Where the 7B still wins

High-volume, low-complexity tasks: classification, routing, extraction from clean text, short chat replies. At batch sizes above 64, a 7B model on a single A10G or L4 can serve thousands of requests per dollar. The 32B model’s latency and memory footprint make it a poor fit for edge or bursty workloads.

The rule of thumb: compute cost per successful task, not per token. If your 7B model needs a 32B fallback for 30% of traffic, you’re paying for both. If your 32B model handles 95% of traffic alone, the math flips.

What to measure before you pick

Log tokens per request, retry rate, fallback rate, and GPU utilization. A 7B model at 20% utilization is more expensive than a 32B model at 70%. The parameter count is the least interesting number in the bill.