Blog directory

Entries

• How Transformer Architecture Determines Inference Speed and Memory Usage — A deep look at how the architectural decisions baked into a transformer model at training time -- attention variant, layer count, hidden size, and more -- directly determine how fast and how cheaply it can run at inference time.. View raw
• What Is an AI Inference Server? Cloud API vs Self-Hosted vs On-Premise Explained — An AI inference server accepts prompts and returns model completions. Here's how they work, the main deployment options, and how to choose between managed APIs and self-hosted solutions.. View raw
• Why TileRT Cannot Catch SambaNova SN50: The Limits of Software-Emulated Dataflow on the Wrong Hardware — TileRT is a clever software approach to dataflow execution, but it runs on hardware that was never designed for reconfigurable dataflow. Here's why purpose-built silicon like the SambaNova SN50 will always hold the edge.. View raw
• AI Inference Costs in 2025: How to Calculate and Reduce Your LLM API Spend — A practical guide to understanding LLM API pricing, calculating your monthly inference bill with real formulas, and eight proven strategies to cut costs without sacrificing quality.. View raw
• Fast AI Inference: 8 Proven Techniques That Deliver Real-World Speedups — A technical guide to the eight most effective LLM inference optimization techniques, with concrete speedup data, code examples, and stacking strategies for production deployments.. View raw
• What Is Tokens Per Second (TPS)? The LLM Throughput Metric That Actually Matters — Tokens per second (TPS) measures how fast a language model generates output. Here's what it means, how it differs from TTFT, what drives it, and how to benchmark it in your own stack.. View raw
• What Is Time to First Token (TTFT)? The LLM Latency Metric That Defines UX — Time to First Token (TTFT) measures how long it takes an LLM to generate its first output token after receiving your request. It's the metric that determines whether your AI app feels instant or sluggish.. View raw
• LLM Inference Explained: How It Works, What It Costs, and How to Optimize It — A complete technical guide to LLM inference: the prefill/decode pipeline, the metrics that matter, how providers price token generation, and five concrete ways to make it faster and cheaper.. View raw
• The 1B-and-Under Club: TinyLlama vs Phi-2 vs StableLM vs Qwen 0.5B — A benchmark comparison of the smallest capable language models -- TinyLlama 1.1B, Phi-2 2.7B, StableLM 2 1.6B, and Qwen 2.5 0.5B -- covering reasoning, coding, inference speed, memory footprint, and which to use for edge and mobile deployments.. View raw
• Time-to-First-Token Leaderboard: Every Major Open Model Ranked — TTFT benchmarks across every major open-source model at multiple prompt lengths. We measured prefill latency on identical hardware so you can pick the right model for latency-sensitive workloads.. View raw
• Batch Inference Throughput: Which Models Maximize Tokens-per-Dollar? — A throughput-focused benchmark comparing open-source models across batch sizes. We measured tokens-per-second, cost-per-million-tokens, and effective tokens-per-dollar so you can pick the right model for high-volume workloads.. View raw
• Long-Context Benchmarks: Who's Actually Fast at 128K Tokens? — We measured TTFT, decode speed, and effective throughput for major open-source models at 128K token context. Here's what the numbers show.. View raw
• MoE vs Dense at Equal Compute: Mixtral 8x7B vs Llama 70B — Mixtral 8x7B activates ~13B parameters per token while Llama 70B activates 70B. At equal active FLOPs, MoE architecture delivers dramatically better quality than dense. Here's why, with concrete benchmarks.. View raw
• Quantized vs Full Precision: GPTQ, AWQ, and FP8 Benchmarks Across Model Sizes — A practical comparison of GPTQ, AWQ, and FP8 quantization against FP16 full precision across 7B, 13B, and 70B model sizes, covering throughput, latency, memory, and quality impact.. View raw
• Instruction-Tuned vs Base Models: Does Fine-Tuning Cost You Speed? — A practical look at whether instruction tuning and RLHF actually affect inference speed, what the real sources of performance differences are, and how to benchmark fairly across checkpoints.. View raw
• Vision-Language Models: Llama 4 Scout vs Qwen2-VL vs InternVL at Inference — A practical comparison of Llama 4 Scout, Qwen2-VL, and InternVL2 on multimodal benchmarks and inference speed, with guidance on which model fits which deployment scenario.. View raw
• Embedding Model Benchmarks: Speed vs Quality for RAG Retrieval — A benchmark comparison of embedding models on retrieval quality (BEIR NDCG@10) and inference latency, with practical guidance on where the quality tradeoffs actually matter for RAG pipelines.. View raw
• Math and Reasoning Models: DeepSeek-R1 vs Qwen QwQ vs Llama Reasoning — A benchmark comparison of the leading open-source thinking/reasoning models -- DeepSeek-R1, QwQ-32B, and Llama-based reasoning distillations -- covering accuracy, token costs, and when to use each.. View raw
• Coding Model Benchmarks: Qwen 2.5 Coder vs DeepSeek Coder V2 vs CodeLlama vs StarCoder 2 — A technical benchmark comparison of the top open-source coding models -- Qwen 2.5 Coder, DeepSeek Coder V2, CodeLlama, and StarCoder 2 -- across HumanEval, MBPP, and real-world code tasks by size bucket.. View raw
• Frontier Open Models: Llama 4 Maverick vs DeepSeek V3 vs Qwen 2.5 110B — A technical comparison of the top frontier open-source models -- Llama 4 Maverick, DeepSeek V3, and Qwen 2.5 110B -- covering architecture, benchmarks, and inference trade-offs.. View raw
• Mid-Size Models: Llama 3.1 70B vs Qwen 2.5 72B vs Mistral Large vs DeepSeek-V2 Lite — A benchmark comparison of the 70B-class open models in 2025 -- covering reasoning, math, coding, context length, and inference speed across Llama 3.1 70B, Qwen 2.5 72B, Mistral Large 2, and DeepSeek-V2 Lite.. View raw
• The 7-8B Battle: Llama 3.1 8B vs Mistral 7B v0.3 vs Qwen 2.5 7B vs Gemma 2 9B — A benchmark comparison of the four most popular 7-9B parameter models in 2025 -- covering reasoning, math, coding, context length, and inference speed to help you pick the right model for your application.. View raw
• Small Models Showdown: Qwen 2.5 3B vs Llama 3.2 3B vs Phi-3.5 Mini vs Gemma 2 2B — A head-to-head benchmark comparison of the best sub-4B parameter models in 2025 -- covering reasoning, coding, math, context length, and inference speed to help you pick the right small model for your use case.. View raw
• Agentic RAG: When Retrieval and Generation Both Need to Be Fast — In agentic RAG, retrieve-then-generate runs in loops, not once. That changes everything about how latency compounds and where you need to optimize.. View raw
• Long-Running Agents: Managing Inference Costs Over Hours of Execution — When an agent runs for hours instead of seconds, inference costs and context management become the dominant engineering problems. Here's how to keep them under control.. View raw
• Evaluating Agent Performance: Latency as a First-Class Metric — Most agent benchmarks measure accuracy and task completion. Here's why latency belongs in your evaluation framework alongside accuracy, and how to instrument your agent to collect the data you need.. View raw
• Browser Agents: Inference Speed as the Gating Factor — Browser agents take dozens of LLM calls to complete a single web task. Here's why inference latency determines whether they're usable in practice.. View raw
• Voice Agents and the 500ms Window: An Inference Architecture Guide — Real-time voice conversations require end-to-end latency under 500ms. This guide breaks down the inference architecture, latency budgets, and streaming patterns needed to hit that target reliably.. View raw
• Planning and Search with LLMs: Tree-of-Thought at Inference Scale — Tree-of-Thought turns a single LLM call into a branching search process. Here's how it works, why inference speed determines how deep you can search, and how to build ToT agents that run fast enough to be useful.. View raw
• The Cost of Agent Retries: When Slow Inference Kills Reliability — Retries are how agents recover from failures, but when inference is slow, each retry multiplies latency. Here's how to think about retry budgets and why faster inference makes your agent more resilient.. View raw
• Whisper API for Real-Time Transcription: Alternatives to OpenAI Whisper — The OpenAI Whisper API works well for batch transcription but is not designed for real-time streaming. Here is how Faster-Whisper, Deepgram, and AssemblyAI compare on latency, accuracy, and pricing for live transcription use cases.. View raw
• Fireworks AI Alternative: Faster Inference at Lower Cost — Fireworks AI is a capable inference platform, but it has real limitations around latency, pricing at scale, and rate limits. Here's a direct comparison with alternatives and a step-by-step migration guide.. View raw
• DeepSeek API: How to Access DeepSeek V3 Without Rate Limits — DeepSeek V3 is one of the strongest open-source models available, but the official API has aggressive rate limits. Here's how to access it reliably at scale, with Python and Node.js integration examples.. View raw
• Inference API Pricing Guide: Groq vs Fireworks vs Together vs GeneralCompute — A practical breakdown of inference API pricing across Groq, Fireworks AI, Together AI, and GeneralCompute -- including per-token rates, hidden costs, and how to calculate your real monthly spend.. View raw
• vLLM vs Managed Inference APIs: When to Self-Host vs Pay-as-You-Go — A practical TCO comparison of self-hosting LLMs with vLLM versus using managed inference APIs. Includes break-even analysis and a decision framework for your team.. View raw
• Llama API: How to Run Llama 4 via API Without Managing Infrastructure — A practical guide to running Llama 4 through managed inference APIs. Compare providers, understand pricing and performance trade-offs, and see working integration code.. View raw
• Best OpenAI API Alternatives in 2025: Full Developer Comparison — A developer's guide to the top OpenAI API alternatives in 2025, covering GeneralCompute, Groq, Together AI, Fireworks AI, and Replicate with pricing, model selection, and migration code.. View raw
• GeneralCompute vs Groq: Speed, Pricing, and Model Selection Compared — A developer's comparison of GeneralCompute and Groq covering inference speed, model availability, API pricing, and when to choose each platform.. View raw
• Scaling Laws in LLMs: What They Mean for Inference Cost in Production — A practical breakdown of Chinchilla and Kaplan scaling laws, what they predict about model quality and size, and how to use that knowledge to make smarter inference cost decisions.. View raw
• Mixture of Experts (MoE) Models: Why They're Dominating 2025 — MoE models activate only a fraction of their parameters per token, giving you large-model quality at a fraction of the compute cost. Here's how the architecture works, why it won, and what the load balancing challenges actually look like.. View raw
• KV Cache in LLM Inference: How It Works and Why It Matters — The KV cache is one of the most important mechanisms in transformer inference. This post explains what it is, how prefill and decode phases use it, how paged attention manages it, and what MQA/GQA and quantization do to shrink it.. View raw
• Agentic AI in Production: Inference Requirements for Multi-Step Workflows — Building a single agentic AI demo is straightforward. Scaling it to production -- with observability, error handling, rate limits, and hundreds of concurrent agents -- requires thinking carefully about your inference layer from the start.. View raw
• How to Build a Real-Time Coding Assistant With Open-Source Models — A practical guide to building a coding assistant LLM using Qwen3-Coder or DeepSeek Coder, covering model selection, codebase indexing, and VS Code integration.. View raw
• Multi-Turn Conversations in LLM APIs: Best Practices for Agents — How to manage conversation history in LLM APIs without blowing up your context window or your bill. Covers sliding window, summarization, memory patterns, and cost optimization for production agents.. View raw
• Your AI Agent Is Only as Good as Its Inference Speed — Agent latency compounds across every sequential step. This post covers the multiplier effect, how model routing can cut costs without sacrificing quality, and why parallelizing calls is one of the highest-leverage improvements you can make to an agentic system.. View raw
• What Are Agentic AI Systems? How to Build Them With Fast Inference — Agentic AI systems chain LLM calls into autonomous loops that plan, act, and observe. This guide covers the core components, the main reasoning patterns (ReAct, Plan-and-Execute, Tree of Thoughts), and how inference speed shapes what you can actually build.. View raw
• SOC 2 Compliant AI Inference: What Enterprise Teams Need to Know — A practical guide to SOC 2 compliance for AI inference: Type I vs. Type II, data residency, audit logging, and a checklist for evaluating inference providers.. View raw
• Embedding Models in Production: Choosing the Right One for Your App — A practical guide to picking an embedding model for production: MTEB benchmarks, head-to-head comparisons of BGE, Nomic, E5, Cohere, and OpenAI models, multilingual considerations, and code to get started.. View raw
• How to Migrate From OpenAI to GeneralCompute in 10 Lines of Code — GeneralCompute's API is fully OpenAI-compatible. Here's exactly what to change in Python, Node.js, and LangChain, plus a validation checklist to make sure nothing breaks.. View raw
• Using Vercel AI SDK With GeneralCompute: Full Integration Guide — A complete guide to wiring the Vercel AI SDK's useChat and useCompletion hooks to GeneralCompute's OpenAI-compatible API, with streaming in Next.js App Router and edge runtime deployment.. View raw
• How to Build a RAG Pipeline Using Open-Source Models — A complete walkthrough of building a retrieval-augmented generation pipeline with open-source embedding and LLM models: ingestion, chunking, vector storage, retrieval, generation, and evaluation with RAGAS.. View raw
• JSON Mode in LLMs: How to Get Structured Outputs Every Time — A practical guide to JSON mode, structured output schemas, and Pydantic + instructor for getting reliable JSON from LLMs -- covering every major approach and when to use each one.. View raw
• OpenAI-Compatible APIs: How to Migrate Your App in 5 Minutes — A practical guide to migrating your OpenAI-powered app to a compatible API provider like GeneralCompute -- covering the base URL swap, supported endpoints, and integration code for LangChain, LlamaIndex, and Vercel AI SDK.. View raw
• Tool Calling With Open-Source LLMs: A Complete Guide — How function calling works in LLM APIs, which open-source models support it, and practical Python and Node.js examples for building tool-using applications.. View raw
• How to Build a Streaming Chat App With GeneralCompute + Node.js — A practical guide to building a streaming chat application using the OpenAI SDK for Node.js pointed at GeneralCompute's API, including SSE architecture, context management, and a full working example.. View raw
• GPU Cluster for LLM Inference: Build vs Buy Analysis for ML Teams — A rigorous TCO breakdown comparing building your own GPU cluster against managed inference APIs, with break-even analysis and hybrid approaches for ML teams at different scales.. View raw
• Real-Time AI Inference: How to Achieve <100ms Latency in Production — A practical guide to the full optimization stack for sub-100ms AI inference: model selection, quantization, infrastructure placement, and what to measure.. View raw
• Fine-Tuning vs RAG: Which Approach Is Right for Your Production App? — A practical decision framework for choosing between fine-tuning and retrieval-augmented generation, including cost comparison and when a hybrid approach makes sense.. View raw
• H100 vs H200 vs B200: Which GPU Is Best for LLM Inference? — A technical comparison of NVIDIA's H100, H200, and B200 GPUs for LLM inference workloads: specs, tokens/s benchmarks, cost per token, and multi-GPU scaling considerations.. View raw
• LLM Token Generation Speed: How Providers Compare in 2025 — A practical methodology for benchmarking LLM throughput across providers, a comparison of token generation speeds at different concurrency levels, and a guide to the throughput vs latency trade-off.. View raw
• AI Inference Latency Explained: TTFT, TPS, and How to Optimize Them — What time to first token and tokens per second actually measure, how to measure them correctly, and a layer-by-layer guide to reducing AI inference latency in production.. View raw
• GeneralCompute vs vLLM: Throughput, Latency, and Cost Benchmarks — A head-to-head comparison of vLLM self-hosted on H100s versus GeneralCompute's managed inference API: full methodology, throughput and latency numbers, and a total cost of operations breakdown.. View raw
• Open-Source LLM Landscape 2025: Top Models Compared — A practical map of the open-source LLM ecosystem in 2025: the leading model families, how they stack up by size and task, what the licenses actually let you do, and how to pick one for production.. View raw
• Faster-Whisper: Real-Time Speech-to-Text on GeneralCompute — Faster-Whisper reimplements OpenAI's Whisper on CTranslate2 with INT8 inference, running several times faster at the same accuracy. Here is how it works, how streaming differs from batch transcription, and how it fits into a real-time STT to LLM to TTS voice pipeline.. View raw
• QwQ-32B: The Reasoning Model That Rivals o1 — Complete Guide — QwQ-32B is a 32-billion-parameter open-weight reasoning model from the Qwen team that competes with much larger reasoning models. Here is how it works, how it compares to o1, o1-mini, and DeepSeek R1, and what its long reasoning traces mean when you serve it in production.. View raw
• How to Fine-Tune Llama 4: Step-by-Step Guide with Code — A practical walkthrough for fine-tuning Llama 4: when to do it, how to prepare data, and working LoRA, QLoRA, and full fine-tune code, plus evaluation and deployment.. View raw
• Qwen3-Coder: The Best Open-Source Coding Model? Benchmark + Guide — A close look at Qwen3-Coder: how it scores on HumanEval, MBPP, and SWE-bench, how it compares to Code Llama and DeepSeek Coder, and how to wire it into your editor and agents.. View raw
• Llama 4 on GeneralCompute: Getting Started Guide — A practical guide to running Llama 4 on GeneralCompute: the model variants, what hardware they need, how to make your first API call, and how to tune requests for speed and cost.. View raw
• DeepSeek R1: What It Is, How It Works, and Why It Matters — DeepSeek R1 is an open-weight reasoning model trained mostly through reinforcement learning. Here is how its architecture and training work, how it compares to GPT-4 class models, Claude, and Llama, and what its reasoning style means for inference.. View raw
• Flash Attention: Why Modern LLMs Run Faster With It — Flash Attention rewrites the attention computation to avoid moving a giant intermediate matrix in and out of GPU memory. Here is how the tiling and kernel fusion work, how v1, v2, and v3 evolved, and how to turn it on in PyTorch.. View raw
• What Is Direct Preference Optimization (DPO)? Explained Simply — DPO aligns language models to human preferences without a separate reward model or reinforcement learning. Here is how it works, how it compares to RLHF, and when to reach for IPO, KTO, or ORPO instead.. View raw
• Quantization Explained: INT4, GGUF, GPTQ and What They Mean for Your Model — A practical guide to LLM quantization: what INT4, GGUF, and GPTQ actually do, how much quality you lose, and how to quantize a model yourself with llama.cpp and AutoGPTQ.. View raw
• What Is Speculative Decoding? How It Makes LLMs 3x Faster — A clear explanation of speculative decoding: how a small draft model proposes tokens that a large model verifies in parallel, why it preserves output quality, the Medusa and Eagle variants, and the real speedups you can expect in production.. View raw
• LPU vs GPU vs CPU: Which Processor Wins for AI Inference? — A head-to-head look at LPUs, GPUs, and CPUs for AI inference, with the architecture reasons behind their performance, real-world latency and throughput trade-offs, cost-effectiveness, and which one fits each kind of workload.. View raw
• What Is AI Inference? A Developer's Complete Guide — A practical, end-to-end explanation of AI inference: what it is, how the pipeline works, the metrics that matter, the hardware that runs it, and the trade-offs you face when you put a model in production.. View raw
• Streaming for Agents: Why Partial Results Change the UX — Streaming in agentic pipelines is not the same as streaming chat tokens. Partial tool calls, pipelined steps, and early cancellation change what the user experiences.. View raw
• Parallel Tool Execution: How Fast Inference Enables Concurrent Agent Actions — Why running multiple tool calls in parallel changes the latency math of an agent, and how inference speed determines whether the parallelism is worth doing.. View raw
• Agent Memory Systems: Balancing Context Length vs Retrieval Latency — How agents reconstruct memory between turns, and the latency trade-offs between long context, RAG, summarization, and KV cache reuse.. View raw
• Building a Code Agent: Why Each Step Needs Sub-Second Inference — A practical breakdown of the latency budget inside a code agent, step by step, and why every link in the chain needs to land under a second to keep the loop usable.. View raw
• ReAct, Reflexion, and Chain-of-Thought: The Inference Cost of Reasoning Patterns — Popular agent reasoning patterns are described as prompt techniques, but they are inference cost multipliers. Here is how ReAct, Reflexion, and Chain-of-Thought actually shape the bill and the latency.. View raw
• Multi-Agent Architectures and the Inference Cost Explosion — Orchestrator and worker patterns make multi-agent systems easy to design and expensive to run. Here is where the inference cost actually goes, and what it means for the infrastructure underneath.. View raw
• Tool Calling Latency: The Bottleneck No One Talks About — Function calling looks simple on paper, but the latency budget of a tool-using LLM is dominated by short structured generations that most serving stacks are not optimized for. This is what actually makes tool calls feel slow.. View raw
• The Agentic Inference Tax: Why Agents Need 10x Faster Models — Agents make many sequential LLM calls per task, and each one pays the full latency of decoding. This post walks through how that compounds and why fast inference changes which agents are even viable.. View raw
• Compiler-Level Optimizations for Inference: TorchInductor, Triton, XLA — How modern ML compilers turn Python model code into fused, fast kernels. A practical look at TorchInductor, Triton, and XLA, and the tradeoffs each one makes for inference.. View raw
• Draft Model Selection for Speculative Decoding — Picking a draft model is the most consequential decision when deploying speculative decoding. A practical guide to acceptance rates, sizing, and the tradeoffs that decide whether you actually get a speedup.. View raw
• The Attention Sink Phenomenon: Why the First Token Matters — How attention concentrates on the first few tokens of every sequence, why naive sliding-window caching breaks long-context generation, and how StreamingLLM uses sink tokens to serve effectively unbounded streams.. View raw
• Mixture of Experts at Inference Time — How MoE routing actually works during serving, why sparse activation makes large models cheaper to run per token, and what changes for the inference stack.. View raw
• Tensor Parallelism vs Pipeline Parallelism for Model Serving — How tensor and pipeline parallelism actually differ in production inference, when to use each, and why most serving stacks end up combining them.. View raw
• Prefix Caching: Why Repeated Prompts Shouldn't Cost You Twice — How prefix caching works in modern LLM serving stacks, why it changes the economics of long system prompts and RAG, and what to watch out for in production.. View raw
• Distillation for Inference: How Smaller Models Learn From Larger Ones — A practical guide to knowledge distillation for production inference: what actually works, what to skip, and how to ship a smaller model without losing the behavior you cared about.. View raw
• FP8 Training and Inference: The Precision Sweet Spot — Why 8-bit floating point hits a different point on the accuracy/throughput curve than INT8, how E4M3 and E5M2 are used in practice, and what FP8 actually buys you in production serving.. View raw
• Activation-Aware Quantization (AWQ) Deep Dive — A close look at how AWQ picks salient weight channels, applies per-channel scaling, and why it consistently beats round-to-nearest 4-bit quantization for LLM inference.. View raw
• Mamba and State Space Models: Inference Without Attention — How structured state space models like Mamba achieve constant-time per-token inference, and why the selective scan changes the trade-off space for long-context serving.. View raw
• RWKV and Linear Attention: Recurrent Models as an Inference Shortcut — How RWKV and linear attention architectures collapse the per-token cost of generation to O(1), and what that means for serving long-context workloads.. View raw
• Dynamic Batching Strategies: From Naive to Continuous to Iteration-Level — Batching is the lever that turns idle GPU silicon into served tokens. This post walks through the evolution of batching for LLM serving, from one-at-a-time to static batches to request-level dynamic batching to iteration-level continuous batching, and shows where each strategy still leaves throughput on the floor.. View raw
• Token Merging and Token Pruning for Faster Transformers — Attention cost grows with the square of sequence length. Token merging and token pruning shrink that sequence mid-network, trading a little accuracy for real speedups. Here is how ToMe works, how the idea extends to language models, and where it breaks down.. View raw
• S3: Scheduling for Straggler Mitigation in LLM Serving — In LLM serving, a single long-running request can stall everyone else sharing the same batch. S3 attacks that by predicting output length and scheduling around it. Here is what stragglers actually cost you, and how output-length-aware scheduling helps.. View raw
• Chunked Prefill: Overlapping Compute and Communication — Prefill pins the compute units while decode starves for memory bandwidth. Sarathi-Serve splits prefill into chunks and piggybacks decodes on them, keeping both resources busy in the same batch. Here is how it works and where the limits are.. View raw
• Cascade Inference: Using Small Models to Route to Big Ones — FrugalGPT and its descendants show that most queries do not need the biggest model. We walk through the cascade pattern, routing classifiers, and the engineering trade-offs of sending easy work to cheap models and escalating only when needed.. View raw
• Lookahead Decoding: Parallel Token Generation Without Draft Models — Lookahead decoding from LMSYS speeds up autoregressive generation without requiring a draft model. We walk through the Jacobi iteration trick, the n-gram pool, and what the speedups actually look like in practice.. View raw
• Disaggregated Prefill and Decode (Splitwise / DistServe) — Prefill and decode have different compute profiles and clash when they share a GPU. Splitwise and DistServe separate them onto different hardware pools. We walk through why, how, and when it actually pays off.. View raw
• KV Cache Compression: MLA and Beyond — DeepSeek's Multi-Head Latent Attention cuts the KV cache by an order of magnitude without giving up quality. We walk through MLA, how it compares to MQA and GQA, and the other compression techniques worth knowing.. View raw
• Ring Attention: Scaling Context to Millions of Tokens — Ring Attention distributes the attention computation across devices in a ring topology, overlapping KV transfer with compute so context length scales linearly with the number of GPUs.. View raw
• Quantization for Inference: GPTQ, AWQ, SmoothQuant, and FP8 — Quantization shrinks model weights from 16-bit to 4-bit or 8-bit, cutting memory usage and speeding up inference. Here's how the major techniques work and when to use each one.. View raw
• Multi-Query and Grouped-Query Attention: Shrinking the KV Cache — MQA and GQA reduce the memory footprint of attention by sharing key-value heads across queries. A simple architectural change that makes inference dramatically faster.. View raw
• Continuous Batching: The Orca Paper That Changed LLM Serving — Before continuous batching, LLM servers wasted GPU cycles waiting for the slowest request in each batch. Orca's iteration-level scheduling fixed this with a 36x throughput improvement.. View raw
• Medusa, EAGLE, and Sequoia: The Next Generation of Speculative Decoding — The original speculative decoding papers needed a separate draft model. Medusa, EAGLE, and Sequoia found ways to speculate faster, smarter, and without the extra model.. View raw
• SGLang and RadixAttention: Smarter KV Cache Reuse — SGLang's RadixAttention stores KV cache in a radix tree, enabling automatic prefix sharing across requests. The result is up to 5x higher throughput for multi-turn and structured workloads.. View raw
• Speculative Decoding: Getting 3x Speedups Without Changing the Model — Speculative decoding uses a small draft model to predict multiple tokens ahead, then verifies them all at once. The result is mathematically identical output, 2-3x faster.. View raw
• PagedAttention and vLLM: Virtual Memory for LLM Serving — The PagedAttention paper solved the biggest memory waste problem in LLM serving by borrowing an idea from operating systems. Here's how it works and why vLLM became the default serving framework.. View raw
• FlashAttention: How Tri Dao Made Attention 4x Faster — FlashAttention rewrote the rules of transformer inference by treating attention as a memory problem, not a compute problem. Here's how it works and why it matters.. View raw
• Build a Real-Time Voice AI Agent with General Compute — A step-by-step tutorial for building a voice AI agent with sub-500ms response times. Plus: why General Compute is the only provider fast enough to use reasoning models in a voice pipeline.. View raw
• How Coding Agents Depend on Inference Speed — Coding agents make dozens of sequential LLM calls per task. Every millisecond of inference latency compounds across each step, making speed the single biggest infrastructure bottleneck for AI-powered developer tools.. View raw
• Why Inference Speed is the New Moat — Model quality has commoditized. The real competitive advantage in AI is how fast your infrastructure can deliver results. Inference speed is becoming the defining moat for AI-native products.. View raw