LLM Inference Explained: How It Works, What It Costs, and How to Optimize It

LLM inference is what happens when you send a prompt to a language model and it generates a response. Every chatbot reply, every autocomplete suggestion, every AI-generated summary is the result of an inference call. Despite being the core operation in deployed AI systems, the mechanics of LLM inference are poorly understood outside of ML infrastructure teams -- which leads to systematic underestimates of cost, misdiagnoses of latency problems, and missed optimization opportunities.

This post covers the full picture: what LLM inference is, how the pipeline works step by step, what metrics actually matter, what determines speed, how providers price it, and how to optimize it.

What Is LLM Inference? (Plain-English Definition)

LLM inference is the process of running a trained language model on new input to produce output. The model's weights are fixed -- learning already happened during training. Inference is the act of using those fixed weights to generate tokens in response to a prompt.

How Inference Differs from Training

Training and inference share the same underlying model architecture but are completely different workloads.

Training runs for days or weeks on large GPU clusters and is measured in GPU-hours. Inference runs in milliseconds and happens continuously. Hardware tuned for training is often suboptimal for inference because the bottlenecks are different.

Why Inference Is Now the Dominant AI Workload

For most organizations running AI systems in production, inference costs exceed training costs by a large margin. A model trained once over several months will serve inference requests for years. Every new user adds inference load, not training load. As AI gets embedded into more products -- coding assistants, customer support, document processing, agents -- inference becomes the primary ongoing compute expense.

The Economics: Why Inference Costs More Than Expected

Teams often underestimate inference costs for a few reasons.

Output tokens are expensive. Generating each token requires a full sequential forward pass through the model. A 1,000-token response required 1,000 such passes during the decode phase.

Context window costs accumulate silently. In a multi-turn conversation, you send the full prior context with each turn. By turn 10, even short exchanges have grown into thousands of input tokens.

Long-tail requests dominate the bill. The median request cost might be $0.001, but p99 requests with huge prompts or long responses can be 50-100x more expensive. Cost estimates built on average behavior miss these outliers.

Inside the LLM Inference Pipeline Step by Step

When a request arrives at an inference server, it goes through five stages. Understanding each one tells you where latency comes from.

Step 1: Tokenization

Before anything reaches the GPU, text has to be converted into token IDs. A tokenizer -- usually a byte-pair encoding (BPE) tokenizer like Tiktoken or SentencePiece -- breaks the input string into subword units and maps each to an integer from a fixed vocabulary.

Tokenization runs on the CPU and is fast, but token count is not word count. English text averages roughly 1.3 tokens per word. Code and foreign-language text can be denser. A 500-word prompt is typically 600-700 tokens.

Step 2: The Prefill Phase

With tokens ready, the model processes the entire input sequence in a single forward pass. All input tokens are processed in parallel, which means the GPU can apply massive matrix multiplications across the whole sequence at once.

Prefill is compute-intensive and uses GPU hardware efficiently. It produces two outputs: the first generated token, and a fully populated KV cache -- the data structure that stores the model's internal attention states for every input token.

Time-to-first-token (TTFT) is determined almost entirely by how long prefill takes, plus any time the request spent waiting in a queue. Longer prompts and larger models both increase TTFT because more matrix operations have to complete before that first token can be sampled.

Step 3: The Decode Phase

After prefill, the model generates output one token at a time. For each step: take the last generated token, retrieve the KV cache for all prior tokens, run a forward pass, sample the next token. Repeat until a stop condition is met.

This is inherently sequential. Token 50 cannot be generated before token 49. Unlike prefill -- where the GPU works on many tokens in parallel -- decode reads the full model weights once per generated token. For a 70B model stored in BF16, that means reading 140 GB of data from memory once per token. At 100 tokens/second, that is 14 TB of memory reads per second.

This is why decode is memory-bandwidth-bound rather than compute-bound, and why GPUs with higher memory bandwidth produce faster decode throughput.

Step 4: Detokenization

After each token is generated, its ID gets converted back to text. For streaming responses this happens incrementally, token by token. Detokenization is fast and runs on the CPU, typically in parallel with the decode loop so it does not add to latency.

The KV Cache: Why Storing Attention States Changes Everything

During the attention computation, the model generates key (K) and value (V) vectors for each token. Without caching, generating token N would require recomputing K and V vectors for all N-1 prior tokens from scratch -- making generation quadratically expensive in sequence length.

The KV cache stores those vectors after they are first computed and reuses them at every subsequent decode step. This turns a quadratic process into a linear one and makes decode viable at all.

The tradeoff is memory. The KV cache grows with both sequence length and batch size. For a 70B model serving long contexts to many concurrent users, the KV cache can consume more GPU memory than the model weights. This memory pressure is what motivates prefix caching, multi-query attention, and KV cache compression.

Approximate KV cache memory formula:

KV cache (bytes) = 2 × n_layers × n_kv_heads × head_dim × seq_len × batch_size × dtype_bytes

For Llama 3 70B at BF16 with 32 KV heads, 128 head dim, 80 layers, sequence length 8192, batch 16:

2 × 80 × 8 × 128 × 8192 × 16 × 2 bytes ≈ 43 GB

That is 43 GB of KV cache alone for a single batch, on hardware where the model itself already occupies 140 GB.

LLM Inference Performance Metrics You Must Understand

One latency number is not enough to operate a production system. Inference performance has multiple dimensions.

TTFT

TTFT is how long a user waits before seeing any output. For chat and voice AI, this is the latency that determines whether the product feels fast. A 500ms TTFT feels sluggish; under 200ms feels snappy. For voice AI, the target is often below 150ms because any longer and the conversation starts to feel broken.

Long prompts push TTFT up. Large models push TTFT up. A busy server with high queue depth pushes TTFT up. All three factors compound.

TPS (Tokens Per Second)

TPS during decode determines how quickly text streams to the user. Human reading speed is roughly 250 words per minute, or about 5 words per second. At 1.3 tokens per word, you need roughly 6-7 tokens per second to keep pace with a fast reader. In practice, 30+ TPS is the target for interactive use -- it ensures the model is never the bottleneck.

Per-request TPS and server-aggregate TPS are different numbers. A server delivering 5,000 tokens/second across 100 concurrent users might only be giving each user 50 tokens/second. Both numbers are meaningful but for different questions.

End-to-End Latency

E2E latency is the total time from sending a request to receiving the final token. For batch processing -- document summarization, classification pipelines, offline jobs -- this is the number that determines throughput. For streaming chat, TTFT and per-token rate are more informative.

GPU Memory Utilization

GPU VRAM is the primary capacity constraint in LLM serving. Model weights, KV cache, and activations all compete for the same pool of memory. When utilization climbs above ~90%, new requests can be queued or rejected because there is no room to allocate their KV cache. Monitoring memory utilization is essential for sizing your deployment.

Cost per Million Tokens

This is the unit providers use for billing. Input tokens and output tokens are priced separately, and output typically costs 2x to 5x more. Understanding your token mix -- ratio of input to output, average context length -- is necessary to estimate costs accurately.

What Determines LLM Inference Speed?

Model Size

Parameter count is the most direct predictor of inference speed. More parameters means more weight data to load per forward pass, more floating-point operations per token, and longer TTFT because prefill scales with model size.

A 7B model running on the same hardware as a 70B model will typically be 5-10x faster at decode. Choosing the smallest model that meets your quality bar is usually the highest-leverage optimization available, before touching any infrastructure.

Quantization

Quantization reduces the numerical precision of model weights from the training default (BF16 or FP16) to smaller formats. Less memory per weight means faster memory reads during decode and a smaller model footprint overall.

FP8 has become the production standard because it halves memory and speeds up decode by 1.5-2x with almost no quality loss on modern hardware that supports it natively.

Hardware: GPU, LPU, and ASIC Trade-offs

GPUs are the default inference hardware. They are well-suited to the large matrix operations in prefill and handle decode at scale, but they were designed for training and are not specifically optimized for token generation's sequential memory-access pattern.

LPU architectures (like Groq's) optimize for the decode pattern specifically. They can achieve very low per-token latency on individual requests.

ASICs designed for inference (like the hardware powering General Compute) can tailor the entire memory hierarchy and compute pipeline to token generation. The result is higher throughput and lower TTFT at a given cost compared to general-purpose hardware adapted to the task.

Serving Framework: vLLM, SGLang, TensorRT-LLM

The serving framework that sits between hardware and the API layer has measurable performance impact.

vLLM is the most widely deployed open-source option. Its paged attention mechanism treats KV cache like virtual memory, eliminating fragmentation and supporting larger effective batch sizes. It is production-tested across most major model families.

SGLang performs well on multi-turn and agentic workloads via RadixAttention, which reuses KV cache across requests sharing a common prefix. Good choice if your workload is agent-heavy.

TensorRT-LLM is NVIDIA's highly optimized library. It achieves the best raw performance on NVIDIA hardware but requires more engineering to operate and is hardware-specific.

Managed providers handle framework selection, tuning, and operational burden as part of the service.

Concurrency and Batch Size

Batching multiple requests together improves GPU utilization because one memory read of the model weights serves multiple users. This improves aggregate throughput at the cost of potentially higher per-request latency.

Modern servers use continuous batching (iteration-level scheduling). Rather than waiting for an entire batch to complete before accepting new requests, new requests join the active batch at each decode step. This significantly improves throughput under real workloads where requests arrive and complete at different times.

LLM Inference Pricing: How It Actually Works

Input vs Output Tokens

Every mainstream LLM API charges for input and output tokens separately. Input tokens are what you send -- the prompt, system message, conversation history, retrieved context. Output tokens are what comes back.

Output costs more because the model must run a sequential forward pass for each output token. Reading an input token during prefill is cheaper because the entire prefix processes in parallel.

Context Window Pricing

In a multi-turn conversation, you re-send the entire conversation history with each request. A chatbot with 20 turns and 300 tokens per turn is sending roughly 6,000 tokens of context on the final turn -- even if the user's latest message was five words. This is a significant source of cost growth that does not scale linearly with the number of messages.

Some providers offer tiered pricing for long contexts, with tokens beyond a threshold costing more or less than the base rate.

Estimate Your Monthly Bill

For a single request:

cost = (input_tokens × input_price_per_token) + (output_tokens × output_price_per_token)

For a production chatbot at 50,000 requests/day, 800 input tokens and 400 output tokens on average, using a 70B-class model:

daily_input_tokens  = 50_000 * 800  # = 40,000,000
daily_output_tokens = 50_000 * 400  # = 20,000,000

input_price_per_m   = 0.12   # $/M tokens
output_price_per_m  = 0.40   # $/M tokens

daily_cost = (40 * input_price_per_m) + (20 * output_price_per_m)
           = 4.80 + 8.00
           = $12.80/day
           # $384/month

Switch to a provider charging $0.20/M input and $0.60/M output and the same workload costs $640/month. Provider choice is a real lever.

Provider Pricing Comparison Table (2026)

Approximate rates for a strong 70B-class open-source model as of mid-2026:

The self-hosted numbers exclude engineering overhead, on-call burden, idle capacity costs, and hardware procurement. At moderate traffic volumes, managed APIs are typically cheaper in total cost of ownership.

5 Ways to Optimize LLM Inference

1. Quantization: FP16 to INT4

Switching from BF16 to FP8 halves model memory and speeds up decode by 1.5-2x with minimal quality loss. Going to INT4 cuts memory by 4x versus BF16 but requires more careful evaluation of output quality on your specific task. Start with FP8 -- it offers most of the benefit with low risk.

2. Speculative Decoding: 2-3x Faster

Speculative decoding uses a small draft model to predict several tokens ahead, then verifies those predictions in parallel with the target model. If the target model agrees with the draft, you get N tokens in roughly the same wall-clock time as one. The speedup on decode-heavy workloads is consistently 2-3x. It requires pairing models of appropriate sizes and works best when the draft model's distribution closely matches the target.

3. KV Cache Tuning and Prefix Caching

Prefix caching stores the computed KV cache for common prefixes (like a fixed system prompt) and reuses it without recomputation. If 90% of your requests share the same 2,000-token system prompt, prefix caching eliminates the prefill cost for those tokens on every request after the first. TTFT improvements of 50-80% on the cached portion are common.

KV cache quantization (storing cache values in INT8 rather than BF16) can roughly double the number of concurrent requests you can serve at the same memory budget, at a small cost to output quality.

4. Model Routing

Send simple requests to small models and complex requests to large ones. A 7B model can handle many classification, extraction, and simple Q&A tasks that do not require the capacity of a 70B model. A routing layer -- a small classifier or rules-based filter -- that dispatches requests to the right model can reduce average cost by 60-80% with acceptable quality trade-offs on the simple requests.

5. Choosing the Right Inference Provider

If you are using a managed API, the provider itself has a larger impact on cost and latency than most application-level changes. Evaluate providers on TTFT under your expected concurrency (not just idle benchmarks), sustained TPS at load, pricing for your specific model and token ratio, and uptime/reliability. Testing under realistic load before committing is worth the effort.

LLM Inference in Production: Common Pitfalls

Rate Limits and Retry Storms

Every managed API enforces rate limits. When your application hits one, naive retry logic often makes things worse: failed requests trigger immediate retries, which compound the overload. Use exponential backoff with jitter, set a maximum retry count, and handle rate limit errors at the queue level rather than inline. Circuit breakers help contain cascades when a provider degrades.

Latency Spikes Under Load

A system that performs well at 10 concurrent requests will often show very different tail latency at 100 concurrent requests. Queue depth grows, batch composition shifts, and memory pressure increases. A setup that looks fast in isolated testing can have a p99 that is 5-10x the p50 under real traffic. Load-test at expected peak concurrency before deploying.

Cold Start Penalties

Some providers load models into GPU memory on demand. If a model has not been requested recently, the first request has to wait for the model to load, which can add seconds of latency. For low-traffic applications where the model unloads between requests, this creates unpredictable spikes. Ask your provider whether models are kept warm and what cold start behavior looks like for your use case.

FAQ

What is the difference between LLM inference and AI inference?

AI inference is the general term for using any trained model to generate predictions on new data. LLM inference is a specific case: running a large language model to generate text tokens. The term LLM inference typically implies the prefill/decode pipeline, token-by-token generation, and the associated metrics (TTFT, TPS). AI inference is broader and covers image classifiers, audio models, recommendation systems, and anything else that runs a trained model on new input.

How long does LLM inference take?

It depends on model size, hardware, and response length. For a 7B model on modern hardware, TTFT is typically under 100ms and decode runs at 100-200 tokens/second. For a 70B model, TTFT is commonly 200-500ms and decode runs at 20-80 tokens/second depending on hardware and concurrent load. A 300-token response from a 70B model on shared infrastructure takes roughly 3-15 seconds end-to-end.

What hardware is best for LLM inference?

For most teams, managed inference APIs are the practical starting point -- they abstract away hardware selection and handle operational burden. If you need to self-host: NVIDIA H100s are the current standard, with broad software support and well-understood performance characteristics. H200s and B100s offer higher memory bandwidth, which improves decode throughput for large models. ASICs and LPUs (like those from General Compute and Groq) offer better efficiency per dollar for token generation at scale, available through their respective managed APIs.

---

If you want to see these numbers in practice, General Compute's API runs 70B-class models on custom ASIC infrastructure with TTFT under 100ms. The API is OpenAI-compatible, so existing integrations work without code changes.