Topic
Technical deep-dives on the building blocks of modern LLM inference: attention, quantization, decoding, and architectures.
17 posts
Flash Attention rewrites the attention computation to avoid moving a giant intermediate matrix in and out of GPU memory. Here is how the tiling and kernel fusion work, how v1, v2, and v3 evolved, and how to turn it on in PyTorch.
DPO aligns language models to human preferences without a separate reward model or reinforcement learning. Here is how it works, how it compares to RLHF, and when to reach for IPO, KTO, or ORPO instead.
A clear explanation of speculative decoding: how a small draft model proposes tokens that a large model verifies in parallel, why it preserves output quality, the Medusa and Eagle variants, and the real speedups you can expect in production.
A head-to-head look at LPUs, GPUs, and CPUs for AI inference, with the architecture reasons behind their performance, real-world latency and throughput trade-offs, cost-effectiveness, and which one fits each kind of workload.
A practical, end-to-end explanation of AI inference: what it is, how the pipeline works, the metrics that matter, the hardware that runs it, and the trade-offs you face when you put a model in production.
How attention concentrates on the first few tokens of every sequence, why naive sliding-window caching breaks long-context generation, and how StreamingLLM uses sink tokens to serve effectively unbounded streams.
How MoE routing actually works during serving, why sparse activation makes large models cheaper to run per token, and what changes for the inference stack.
A practical guide to knowledge distillation for production inference: what actually works, what to skip, and how to ship a smaller model without losing the behavior you cared about.
Why 8-bit floating point hits a different point on the accuracy/throughput curve than INT8, how E4M3 and E5M2 are used in practice, and what FP8 actually buys you in production serving.
A close look at how AWQ picks salient weight channels, applies per-channel scaling, and why it consistently beats round-to-nearest 4-bit quantization for LLM inference.
How structured state space models like Mamba achieve constant-time per-token inference, and why the selective scan changes the trade-off space for long-context serving.
How RWKV and linear attention architectures collapse the per-token cost of generation to O(1), and what that means for serving long-context workloads.
Attention cost grows with the square of sequence length. Token merging and token pruning shrink that sequence mid-network, trading a little accuracy for real speedups. Here is how ToMe works, how the idea extends to language models, and where it breaks down.
Ring Attention distributes the attention computation across devices in a ring topology, overlapping KV transfer with compute so context length scales linearly with the number of GPUs.
Quantization shrinks model weights from 16-bit to 4-bit or 8-bit, cutting memory usage and speeding up inference. Here's how the major techniques work and when to use each one.
The PagedAttention paper solved the biggest memory waste problem in LLM serving by borrowing an idea from operating systems. Here's how it works and why vLLM became the default serving framework.
FlashAttention rewrote the rules of transformer inference by treating attention as a memory problem, not a compute problem. Here's how it works and why it matters.