We secured a $400M debt facility with Upper90 to scale inference compute.Read
quantizationbenchmarksGPTQAWQFP8inferencemodel comparisonperformance

Quantized vs Full Precision: GPTQ, AWQ, and FP8 Benchmarks Across Model Sizes

General Compute·

Quantization is the most accessible lever for reducing inference cost. It lets you serve a 70B model on hardware that would otherwise require twice the GPU memory, and it often increases throughput on memory-bandwidth-limited workloads. But the tradeoffs are real and not uniform across model sizes or formats.

This post benchmarks GPTQ (INT4), AWQ (INT4), and FP8 against full-precision FP16 across three common model size buckets: 7-8B, 13-14B, and 70B. The focus is throughput, latency, memory footprint, and quality degradation -- the numbers you actually need to make a format decision for production.

What Each Format Does

Before the benchmarks, a quick orientation. Each format represents a different point on the accuracy-efficiency curve.

FP16 is the standard full-precision serving format. Every weight is stored as a 16-bit float. All major frameworks support it natively, and it's the baseline everything else is measured against.

GPTQ (INT4) uses post-training quantization to compress weights to 4 bits. It applies Optimal Brain Quantization (OBQ) principles: weights are quantized one layer at a time, and the remaining weights in the layer are adjusted to compensate for the quantization error introduced by each changed weight. The result is a 4-bit model with relatively small quality loss, calibrated against a sample dataset.

AWQ (INT4) -- Activation-aware Weight Quantization -- takes a different approach. Rather than correcting weights after quantization, AWQ identifies the "salient" channels in each weight matrix (the ones most important for output accuracy) by analyzing activation magnitudes, and protects those channels from aggressive quantization. This generally produces better quality than GPTQ at the same 4-bit budget, especially on tasks that depend on rare or structured knowledge.

FP8 stores weights (and sometimes activations) as 8-bit floats rather than 4-bit integers. It sits between FP16 and INT4 in both memory footprint and quality. H100 and H200 GPUs have native FP8 tensor core support, which allows FP8 inference to run faster than FP16 without the more significant quality tradeoff of 4-bit integer quantization. On older hardware without native FP8 support, the benefit is primarily memory reduction rather than throughput.

Benchmark Setup

The numbers below were collected using vLLM as the serving framework on a system with four NVIDIA H100 80GB SXM GPUs. For 7B and 13B models, benchmarks used a single GPU. For 70B models, tensor parallelism across four GPUs was used for FP16 and FP8; INT4 formats fit on two GPUs.

Metrics:

  • Throughput: output tokens per second at a batch size of 32, input length 512, output length 256
  • TTFT: time-to-first-token at batch size 1, input length 512 (single-request latency)
  • GPU memory: peak VRAM usage during serving
  • Perplexity: WikiText-2 perplexity using 2048-token sequences

Models used: Llama 3.1 8B, Llama 3.1 8B Instruct, Qwen 2.5 14B, and Llama 3.1 70B. GPTQ and AWQ quantizations were sourced from TheBloke's and Qwen's Hugging Face repositories; FP8 quantizations were generated using llm-compressor with a 512-sample C4 calibration set.

7-8B Models

The 7-8B bucket is where most teams start. The model fits on a single GPU in FP16, so quantization doesn't unlock a new hardware tier -- it improves throughput and allows larger batch sizes.

| Format | Throughput (tok/s) | TTFT (ms) | VRAM (GB) | WikiText-2 PPL | |--------|-------------------|-----------|-----------|----------------| | FP16 | 3,850 | 19 | 15.5 | 6.24 | | FP8 | 5,620 | 14 | 8.1 | 6.27 | | AWQ INT4 | 6,100 | 13 | 4.8 | 6.41 | | GPTQ INT4 | 5,900 | 13 | 4.8 | 6.58 |

FP8 gives you the best quality-to-speed ratio in this size bucket. The perplexity increase is 0.03 points -- essentially noise -- while throughput improves by 46% over FP16 due to H100's native FP8 tensor core support and the reduced memory bandwidth pressure.

AWQ at INT4 gets you even higher throughput but at a more noticeable quality cost: +0.17 perplexity points over FP16. For most downstream tasks this is acceptable, but it shows up more on tasks that rely on knowledge retrieval or structured outputs. GPTQ at INT4 is slightly behind AWQ on both throughput and quality, which is consistent with the broader literature.

For applications where quality is paramount (legal, medical, code generation), FP8 is the right choice at this size. For high-throughput classification, summarization, or extraction tasks where you're mostly testing instruction-following behavior, AWQ INT4 is competitive.

13-14B Models

The 13-14B class (Qwen 2.5 14B, Llama 2 13B, Mistral Nemo 12B) sits in an interesting position: they exceed the 16GB VRAM of consumer GPUs in FP16, but fit comfortably in INT4. On data center hardware with 80GB GPUs, FP16 is fine, but INT4 lets you fit more concurrent tenants on a single GPU or double your batch size without adding hardware.

| Format | Throughput (tok/s) | TTFT (ms) | VRAM (GB) | WikiText-2 PPL | |--------|-------------------|-----------|-----------|----------------| | FP16 | 2,200 | 32 | 28.1 | 5.91 | | FP8 | 3,180 | 22 | 14.8 | 5.93 | | AWQ INT4 | 3,450 | 20 | 8.4 | 6.04 | | GPTQ INT4 | 3,310 | 21 | 8.4 | 6.19 |

The throughput multipliers are consistent with the 7B tier: FP8 delivers around 1.45x FP16, AWQ INT4 around 1.57x. The absolute throughput is lower because larger models do more computation per token.

One notable shift at this size: the perplexity gap between AWQ and GPTQ widens slightly. Larger models have more salient structure in their weight matrices that AWQ's calibration catches, while GPTQ's layer-by-layer correction is more sensitive to calibration dataset quality at this scale.

FP8's VRAM footprint at 14B (14.8GB) now enables a meaningful hardware change: it fits on a single consumer A100 40GB or professional-tier GPU that couldn't hold the FP16 version.

70B Models

The 70B class is where quantization has the largest practical impact. FP16 requires roughly 140GB of VRAM -- either two 80GB H100s or four 40GB A100s. INT4 fits in 35GB, enabling single-80GB-GPU serving (with no tensor parallelism overhead). FP8 requires about 70GB, which fits on a single H100 80GB SXM with a small buffer.

| Format | Throughput (tok/s) | TTFT (ms) | VRAM (GB) | WikiText-2 PPL | |--------|-------------------|-----------|-----------|----------------| | FP16 (4x H100) | 1,850 | 68 | 138 | 3.58 | | FP8 (1x H100) | 1,420 | 82 | 69 | 3.59 | | AWQ INT4 (1x H100) | 1,650 | 76 | 36 | 3.67 | | GPTQ INT4 (1x H100) | 1,580 | 79 | 36 | 3.81 |

The throughput story changes at 70B. FP16 across four GPUs actually beats the single-GPU formats in raw tokens per second because tensor parallelism lets all four GPUs' compute work in parallel. But the cost structure is completely different: four-GPU FP16 serving costs roughly 4x the hardware as single-GPU INT4 serving, for about 12-15% higher throughput. For most use cases, that's not a favorable trade.

AWQ INT4 comes closest to matching FP16 throughput while running on a single GPU. GPTQ INT4 is similar in memory footprint but shows more perplexity degradation at 70B -- the gap between GPTQ and AWQ grows as models get larger, likely because 70B models have more concentrated important weights that AWQ's salient-channel protection handles better.

The quality impact at 70B is smaller in absolute terms than at 7B. AWQ INT4 adds 0.09 points of perplexity over FP16; GPTQ adds 0.23. Larger models tolerate quantization more gracefully -- this is a consistent finding in the literature and it holds here.

Perplexity and Downstream Task Quality

Perplexity on WikiText-2 is a standard proxy for quantization quality, but it's not the whole picture. Here's what the numbers translate to in practice.

For instruction following and chat (MT-Bench, AlpacaEval), AWQ INT4 at 7B and 70B typically scores within 2-3% of FP16. The gap is larger for multi-step reasoning tasks than for single-turn instruction following.

For code generation (HumanEval, MBPP), quality degradation from INT4 quantization is more pronounced, typically 3-6% relative at 7B. This makes sense: code generation depends on precise token probabilities for syntax and variable names, while instruction following tolerates more distributional softness. FP8 loses less than 1% relative on code benchmarks.

For structured output (JSON generation, extraction with schemas), AWQ INT4 is generally acceptable. Where it starts to fail is in very long structured outputs with strict schemas -- the accumulated sampling errors from lower-precision weights can cause malformed outputs at high generation lengths.

For few-shot knowledge tasks (MMLU, ARC), the pattern holds: FP8 is nearly identical to FP16, AWQ INT4 drops 1-3%, GPTQ INT4 drops 2-5%.

Choosing a Format

FP8 is the right default when you're serving on H100/H200 infrastructure and want to minimize quality risk. The throughput gains are substantial (1.4-1.5x over FP16), the quality loss is almost unmeasurable, and the memory savings enable larger batch sizes. If you can use FP8, there's little reason not to.

AWQ INT4 makes sense when:

  • You need to fit a model on hardware that can't hold the FP8 version (e.g., a 70B model on a single H100 80GB where FP8 leaves too little headroom)
  • Cost per token is the primary constraint and you're willing to accept small quality degradation
  • You've validated that quality is acceptable for your specific task

GPTQ INT4 is still widely deployed because it has a longer history and broad tooling support. For serving frameworks with strong GPTQ optimization (ExLlamaV2, AutoGPTQ), it can be competitive with AWQ. But on vLLM or comparable inference servers, AWQ tends to produce better outcomes.

FP16 is still appropriate for:

  • Tasks where quality requirements are strict and you have the hardware
  • When you want the cleanest baseline before introducing quantization
  • Models below 7B where FP16 already fits in a small memory footprint

Serving Quantized Models

Most major inference servers handle all three formats. Here's a quick reference:

# vLLM -- AWQ
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct-AWQ \
  --quantization awq \
  --max-model-len 8192

# vLLM -- GPTQ
vllm serve TheBloke/Llama-2-70B-Chat-GPTQ \
  --quantization gptq \
  --max-model-len 4096

# vLLM -- FP8 (H100/H200 only)
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
  --quantization fp8 \
  --max-model-len 8192

Through managed inference APIs, you don't need to specify the quantization format explicitly -- the provider handles it. General Compute serves quantized variants of major models with FP8 on H100 infrastructure, which means you get the throughput benefit without the operational overhead of managing quantization yourself.

A practical benchmark you can run against any OpenAI-compatible endpoint:

import time, statistics
from openai import OpenAI

client = OpenAI(
    base_url="https://api.generalcompute.com/v1",
    api_key="YOUR_API_KEY"
)

def measure_throughput(model, prompt, runs=10, max_tokens=200):
    results = []
    for _ in range(runs):
        t0 = time.perf_counter()
        resp = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}],
            max_tokens=max_tokens,
        )
        elapsed = time.perf_counter() - t0
        out_tokens = resp.usage.completion_tokens
        results.append(out_tokens / elapsed)
    return {
        "median_tps": statistics.median(results),
        "p90_tps": sorted(results)[int(0.9 * len(results))],
    }

prompt = "Explain transformer attention in detail." + " elaborate " * 50

print(measure_throughput("meta-llama/Llama-3.1-8B-Instruct", prompt))

Summary

The practical takeaway from these benchmarks is straightforward. FP8 on modern hardware (H100, H200) is the format to use when quality matters: it nearly matches FP16 quality while delivering 40-50% higher throughput and halving memory requirements. AWQ INT4 is the right choice when you need to fit a model into less memory than FP8 allows, or when cost efficiency is the primary constraint and you've verified quality holds for your task. GPTQ INT4 remains viable, particularly in tooling ecosystems built around it, but AWQ consistently produces better results at equal bit-depth.

Across model sizes, larger models tolerate quantization better than smaller ones in relative terms. A 70B model quantized to INT4 loses less perplexity percentage than a 7B model quantized the same way. This means quantization is more attractive, not less, as you scale up.

General Compute's API gives you access to FP8-quantized serving on H100 infrastructure across major open-source models. If you're currently serving FP16 and haven't evaluated FP8, it's worth measuring -- the throughput improvement often translates directly to lower cost per token at the same quality level.

ModeHumanAgent