Fast AI Inference: 8 Proven Techniques That Deliver Real-World Speedups

Getting fast inference from an LLM requires more than picking a fast provider. The underlying techniques -- quantization, speculative decoding, continuous batching, and others -- each address a different bottleneck in the inference pipeline. This guide covers all eight, with concrete speedup numbers, practical code, and guidance on how to combine them effectively.

Why Inference Speed Is the Defining Competitive Advantage

The Shift from Training to Inference as the Core AI Cost

Training a model is a one-time cost. Running it in production is ongoing. As organizations move from prototypes to deployed applications, inference spend typically exceeds training spend within a few months. For high-traffic applications, inference compute is the dominant budget line, and its cost scales directly with latency.

A faster inference stack means either lower costs at the same throughput, or higher throughput at the same cost. Both outcomes matter in production.

How Speed Compounds in Agentic Workflows

Single-turn chat applications are relatively forgiving of latency. Agentic systems are not. A workflow that chains 10 LLM calls, each taking 2 seconds, has a minimum end-to-end latency of 20 seconds. Cut each call to 500ms and you reach 5 seconds total -- a 4x improvement in user-perceived responsiveness from the same improvement at each step.

This compounding effect makes inference speed a multiplicative factor in agentic AI. For more on this, see our breakdown of the agentic inference tax.

Every 100ms Reduction Improves Completion Rates

Latency research from web and mobile consistently shows that slower interfaces lead to higher abandonment rates. The same applies to AI applications. Users who encounter a slow coding assistant or a chatbot with noticeable delays will query it less often or switch to faster tools. Fast inference is a product requirement before it is an infrastructure preference.

The Eight Techniques at a Glance

Technique 1 -- Quantization: Trading Precision for Speed

INT4/INT8/FP8/BF16 -- Which Format for Which Use Case

Quantization reduces the number of bits used to represent model weights. Less memory per weight means faster memory transfers, which translates directly into higher token generation throughput.

• BF16: Standard training precision, baseline speed, full quality.
• FP8: Strong quality retention with 1.5x -- 2x speedup over BF16. Requires Hopper-class hardware (H100) or ASIC support. The current production sweet spot for most deployments.
• INT8: Broader hardware support with slightly more quality loss than FP8.
• INT4: Maximum compression. 3x -- 4x faster than BF16. Noticeable quality degradation on smaller models; more acceptable on 70B and above.

GPTQ, AWQ, SmoothQuant -- Practical Methods

• GPTQ: Post-training quantization using second-order information. Effective for INT4 on large models.
• AWQ: Activation-aware weight quantization. Identifies and protects the most important weight channels before quantizing, achieving better perplexity than GPTQ at INT4.
• SmoothQuant: Migrates quantization difficulty from activations to weights, enabling INT8 without per-channel overhead.

For production, AWQ at INT4 or FP8 (via vLLM's native support) is the most practical starting point.

Expected Speedup: 1.5x -- 4x with Minimal Quality Loss

FP8 typically costs less than 0.5 points on standard benchmarks. INT4 with AWQ on a 70B model usually stays within 1-2% of FP16 performance on most tasks. The tradeoff is acceptable in almost all production scenarios.

Code Example -- Enabling FP8 in vLLM

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    quantization="fp8",           # Enable FP8 weight quantization
    max_model_len=8192,
    gpu_memory_utilization=0.9,
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = llm.generate(["Explain KV cache in one paragraph."], sampling_params)
print(outputs[0].outputs[0].text)

For more on quantization formats and their tradeoffs, see our quantization deep dive.

Technique 2 -- Speculative Decoding: 2-3x Faster Without Changing the Model

How Draft Models Generate Candidate Tokens

LLM decoding is sequential by design: each token depends on all previous tokens. Speculative decoding breaks this constraint by running a small draft model to generate a batch of candidate tokens, then verifying all of them with the main model in a single forward pass.

When the draft model guesses correctly -- which it does for common phrases and code patterns at a high rate -- you generate multiple tokens in the time it would normally take to generate one.

The Verification Step -- Why It's Almost Free

The verification pass runs all candidate tokens through the main model simultaneously, costing roughly the same compute as a single-token decode. When the draft model gets k tokens correct, you receive k tokens for the price of one verification pass. The overall speedup is proportional to the average acceptance length.

Choosing the Right Draft Model Size Ratio

A good draft model is roughly 10-30x smaller than the target model. For Llama 3.1 70B, a 7B or 8B model from the same family works well. Draft and target models must share a tokenizer and vocabulary. In practice, the same model family at a smaller parameter count is the right choice.

See our speculative decoding explainer for implementation details and acceptance rate benchmarks by task type.

Technique 3 -- Continuous Batching: Eliminate GPU Idle Time

The Problem with Static Batching

Traditional inference servers process a fixed batch of requests together and wait until all finish before accepting new ones. Because LLM outputs vary in length, short requests finish early and the GPU sits idle while long requests complete. This wastes substantial compute.

How Iteration-Level Scheduling Works

Continuous batching (also called iteration-level scheduling) inserts new requests into the batch as slots open, after each decode iteration. The GPU never waits. This is how production systems like vLLM, SGLang, and TGI all operate by default.

Throughput Gains Up to 36x (ORCA Paper)

The original ORCA paper demonstrated up to 36x throughput improvement over static batching for LLM serving. In practice, the gain depends on request length variance: the higher the variance, the larger the improvement. For mixed-length production workloads, continuous batching is not optional.

Technique 4 -- KV Cache Optimization

What the KV Cache Stores and Why It Matters

The KV cache stores the key and value tensors from the attention mechanism for each already-processed token. Without it, every decode step would re-process the entire prompt. With it, only the new token requires computation. KV cache size scales linearly with sequence length and batch size, making memory management a central challenge in production LLM serving.

Prefix Caching -- Reusing System Prompts Across Requests

When many requests share a common system prompt or prefix (common in multi-tenant applications), you can cache the KV tensors for that prefix and reuse them across requests. This avoids redundant prefill computation and reduces TTFT by the length of the shared prefix. For applications with a fixed system prompt, enabling prefix caching typically cuts TTFT by 30-60% on the second and subsequent requests.

vLLM and SGLang both support prefix caching natively. Our prefix caching post covers the setup in detail.

KV Cache Compression -- MLA and Beyond

DeepSeek introduced Multi-Head Latent Attention (MLA), which compresses the KV cache by projecting key/value heads into a shared latent space before storing. This reduces KV cache memory by 5-13x depending on configuration, enabling larger batch sizes or longer context at the same memory footprint. For a full technical breakdown, see our KV cache deep dive.

Technique 5 -- Flash Attention

Why Standard Attention Is Memory-Bound

Standard attention requires materializing the full attention score matrix (sequence length x sequence length) in GPU HBM (high-bandwidth memory). At long context lengths, this becomes the bottleneck, not the compute itself.

How Flash Attention Reorders Computation

Flash Attention uses tiling to keep the attention computation in fast SRAM rather than writing intermediate results back to HBM. It computes the same result as standard attention but avoids the memory round-trips. The speedup grows with sequence length: minimal at 512 tokens, significant at 4K, and essential at 32K and above.

Flash Attention 2 vs 3 -- What Changed

Flash Attention 2 improved parallelism across the sequence dimension and fixed work-partitioning inefficiencies in the original version. Flash Attention 3, targeting Hopper GPUs, adds FP8 support, WGMMA instruction usage, and asynchronous softmax, pushing peak FLOP utilization to 75%+ on H100s. All major serving frameworks include Flash Attention 2 by default; FA3 support is rolling out for H100 deployments.

The full technical details are in our Flash Attention explainer.

Technique 6 -- Disaggregated Prefill and Decode

Why Prefill and Decode Have Different Hardware Needs

Prefill (processing the input prompt) is compute-bound: it processes all input tokens in parallel and benefits from high FLOP/s hardware. Decode (generating output tokens one at a time) is memory-bandwidth-bound: it reads all model weights for each token and benefits from high HBM bandwidth.

These are fundamentally different hardware profiles. Running both phases on the same GPU is a compromise that serves neither well.

Routing Prefill to Compute-Optimized Hardware

Disaggregated serving (as described in the Splitwise and DistServe papers) routes prefill requests to compute-optimized nodes and decode requests to bandwidth-optimized nodes. This reduces TTFT by 20-30% and increases overall throughput by better matching workloads to hardware. At scale, it also enables independent autoscaling of prefill and decode capacity based on request patterns.

This is an infrastructure-level optimization, relevant primarily for teams running their own clusters or working with providers that expose disaggregated serving control.

Technique 7 -- Model Routing and Cascade Inference

Routing Simple Requests to Smaller Models

Not every request requires a 70B model. Simple classification tasks, short Q&A, and code completion for common patterns are well within the capability of 7B -- 13B models, which run at 4-5x higher token throughput and substantially lower cost. A routing layer classifies each incoming request by complexity and directs it to the appropriate model.

The FrugalGPT Approach -- 98% Cost Reduction

The FrugalGPT paper from Stanford demonstrated that a cascade approach -- trying cheaper models first and escalating only on failures -- can achieve near-equivalent accuracy to the best available model at 2% of the cost on many task types. The key finding is that a large fraction of real-world requests do not require large-model capability.

Implementing a Simple Router in Python

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_GC_API_KEY",
    base_url="https://api.generalcompute.com/v1",
)

def classify_complexity(prompt: str) -> str:
    """Use a fast small model to route the request."""
    response = client.chat.completions.create(
        model="llama-3.1-8b-instruct",
        max_tokens=5,
        messages=[
            {
                "role": "user",
                "content": (
                    "Is this request complex (requiring detailed reasoning) "
                    "or simple? Reply with only 'complex' or 'simple'.\n\n"
                    f"Request: {prompt}"
                ),
            }
        ],
    )
    return response.choices[0].message.content.strip().lower()


def route_request(prompt: str) -> str:
    complexity = classify_complexity(prompt)

    model = (
        "llama-3.1-70b-instruct"
        if complexity == "complex"
        else "llama-3.1-8b-instruct"
    )

    response = client.chat.completions.create(
        model=model,
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )

    return response.choices[0].message.content

For production use, replace the classifier with a lightweight local model to avoid the added API call overhead, or use rule-based routing based on prompt length and task structure. See our LLM inference overview for context on where routing fits in the full serving stack.

Technique 8 -- ASIC Hardware: Built for Inference, Not Adapted for It

Why GPUs Are Repurposed Training Hardware

GPUs were designed for training workloads: large matrix multiplications on large batches, sustained high-throughput compute. Inference has a different profile: smaller batch sizes, memory-bandwidth-limited token generation, and strict latency requirements. The H100 is excellent hardware, but it is architected primarily for training, with inference performance as a secondary consideration.

How ASICs Optimize for Token Generation

ASICs (application-specific integrated circuits) designed for inference can optimize directly for the decode bottleneck: memory bandwidth, token-level scheduling, and low-latency request handling. Without the architectural constraints of a general-purpose GPU (shared with gaming, scientific computing, and training), inference ASICs can allocate die area and power budget entirely to token generation efficiency.

GeneralCompute's ASIC Architecture

GeneralCompute runs on custom ASIC infrastructure designed from the ground up for LLM token generation. The result is faster TTFT and higher sustained tokens per second compared to equivalent GPU deployments, without requiring you to manage hardware. If you are currently using a GPU-based API provider and latency is a constraint, switching to an ASIC-based provider is one of the highest-leverage changes you can make without modifying your application code.

Combining Techniques: What to Stack and in What Order

The Diminishing Returns Problem

Each technique targets a specific bottleneck. Stacking techniques that target the same bottleneck provides less benefit than stacking techniques that address different ones. Quantization and Flash Attention both reduce memory pressure, so their combined speedup is less than the sum of their individual improvements. Understanding what each technique addresses is the prerequisite to building an effective stack.

Recommended Stacking Strategy by Use Case

Low-latency chat (TTFT-sensitive):

1. ASIC or bandwidth-optimized hardware
2. Prefix caching for shared system prompts
3. Disaggregated prefill (if self-hosting at scale)
4. FP8 quantization

High-throughput batch processing (TPS-sensitive):

1. Continuous batching (mandatory baseline)
2. FP8 or INT4 quantization
3. Flash Attention
4. Model routing for simpler requests

Agentic workloads (multi-step, mixed complexity):

1. Model routing to small models for simple steps
2. Speculative decoding on large model calls
3. Prefix caching for repeated system prompts
4. ASIC provider for lowest base latency per step

The general principle: identify the dominant bottleneck first, apply the technique that addresses it, measure the result, then add the next layer.

FAQ

What is the fastest way to run LLM inference?

For most teams, the answer is a managed inference provider running on purpose-built hardware (ASIC or LPU) with quantized models and prefix caching enabled. This requires no infrastructure management and typically outperforms self-hosted GPU deployments on both latency and cost. If you need more control, vLLM with FP8 quantization and continuous batching is the standard self-hosted baseline.

How much does speculative decoding actually speed things up?

Measured gains range from 1.5x to 3x in practice, depending on the task. Code generation sees larger gains than open-ended creative writing because repetitive, predictable patterns give the draft model a higher acceptance rate. The acceptance rate of the draft model -- typically 60-85% for well-matched model pairs -- is the primary driver of observed speedup.

Can I use all 8 techniques together?

Most combinations are additive. Flash Attention and continuous batching are on by default in all major serving frameworks. Quantization, speculative decoding, prefix caching, and model routing can all be layered. Disaggregated prefill/decode requires infrastructure-level control. ASIC hardware is a provider choice rather than a configuration option. The combination that works best depends on your specific latency, throughput, and cost constraints -- start with the speedup table above and measure each addition.

---

If you want to see what fast inference looks like in practice, try the GeneralCompute API with your existing OpenAI SDK -- just swap the base URL and run your current prompts to see the difference. The LLM inference overview is a useful companion read for understanding how these techniques fit into the full pipeline.