LLM inference now has two ways to get cheaper

LLM inference is not a single optimization problem. It is a set of competing objectives: lower per-user latency, higher aggregate throughput, better model quality and lower cost per token. Baseten’s technical breakdown of LLM inference efficiency separates the engineering choices that move a deployment along those tradeoffs from the changes that improve the overall frontier.

The distinction matters for production systems. A coding agent may justify low latency for an interactive user, while a batch pipeline may accept slower responses to generate more tokens per GPU. The same model and hardware can therefore need very different configurations depending on the traffic pattern.

Baseten frames the discussion around an LLM such as GLM-5.3 or Kimi K3 running agentic coding workloads, with KV-cache reuse and KV-aware routing enabled. Its “efficient frontier” is the set of useful latency-throughput combinations available to that deployment—not a single best setting.

The knobs that trade one outcome for another

These controls do not necessarily make the serving system universally faster. They let operators choose where to sit on the latency, throughput and cost curve.

Batch size is the most direct example. Token-level continuous batching removes the need to wait for a conventional batch to begin, but the configured concurrency still affects how much work each GPU can process at once. Small batches leave more capacity available to each request, while larger batches raise utilization and aggregate throughput but increase the latency experienced by individual users.

Parallelism creates a similar choice at a larger scale. Tensor Parallelism splits model work across GPUs and can reduce latency when the interconnect is fast enough to handle its all-to-all communication. Expert Parallelism offers a different balance for models with expert layers: a smaller degree can help response time, while spreading experts across a full GPU rack is generally aimed at throughput.

Attention Data Parallelism keeps attention computation replicated rather than distributing it in the same way as the rest of the model. That can raise system-level throughput, but the gain comes with lower speed per request. These are configuration decisions, not free performance improvements.

Quantization is the exception in Baseten’s categorization because it can improve both latency and throughput by reducing the precision of weights, activations and/or KV-cache values. It also shifts the quality frontier: lower precision can reduce model quality, although the company says formats such as MXFP4 and NVFP4 can deliver substantial serving gains with little to no quality loss in some cases.

The practical warning is that the frontier is “jagged.” Small changes in configuration can produce disproportionately large effects, so the right batch size, parallelism degree or quantization format has to be found through empirical sweeps rather than assumed from a smooth latency-throughput curve.

The optimizations that improve the whole system

A second class of techniques pushes the entire frontier outward. These changes can be spent on lower latency, higher throughput or a combination of both. They also compound: a twofold improvement from hardware and a separate twofold improvement from software would produce a fourfold serving improvement, assuming the gains apply to the same workload.

Kernel and runtime optimization works at the lowest layers of the inference stack. CUDA kernels execute individual operations such as matrix multiplication, and improving those kernels—or the end-to-end forward pass—reduces the resources needed to generate each token. Gains repeated across the stack can therefore improve both cost and performance without requiring the operator to accept a direct quality tradeoff.

Speculative decoding takes a different route. A smaller or specialized mechanism predicts likely next tokens, and the main model validates those predictions. Accepted guesses allow the system to skip some forward passes. Earlier implementations were constrained by expensive speculation, short sequences and low acceptance rates, but Baseten identifies EAGLE-3, DSpark and DFlash as techniques that are now particularly useful for code generation, where output sequences are relatively predictable.

Speculation is not free. The speculative process competes with the main model loop for resources and can limit the maximum batch size. The benefit is therefore broader than raw response latency: when acceptance is strong, the system can generate more tokens per second per user while also reducing the number of full forward passes.

Prefill and decode should not always share workers

P/D disaggregation separates prefill—the phase that processes the input prompt—from decode, which generates output tokens. Dedicated worker pools can then be tuned to the different compute and memory behavior of each phase.

The ratio between prefill and decode workers can be adjusted to match incoming input and output lengths, as well as KV-cache hit rates. In high-volume deployments, Baseten describes this as primarily a throughput optimization that can preserve latency or make it slightly better, rather than a guaranteed dramatic reduction in response time.

That distinction is relevant to systems already using KV-cache reuse. Cache hits change how much prefill work is required, so a fixed worker split can waste capacity when traffic patterns shift. Disaggregation provides a control surface for adapting the serving fleet, but it also adds operational complexity and does not remove the need to measure the workload.

The deployment lesson

The key division is between tradeoff management and frontier expansion. Batch size, parallelism and quantization help an operator target a particular point; kernel work, speculative decoding and prefill/decode disaggregation aim to make more points achievable at once.

That is a different optimization model from treating inference as a race for the lowest latency number. A system serving interactive coding agents may spend speculative-decoding gains on faster responses, while a batch service may use the same gains to increase throughput. Quantization can reduce the hardware needed per token, but only after its quality impact is measured for the target model and workload.

This also complements the deployment constraints we reported for Nvidia’s TensorRT Model Connect preview: packaging checkpoints for native C++ inference does not by itself determine the serving point. Runtime version, platform support, parallelism, cache behavior and traffic shape still decide whether the resulting deployment is fast, cheap or both. Baseten’s engineering point is that production efficiency comes from measuring the jagged parts of the frontier, then choosing which gains to spend.