Notes from the
inference floor.
Posts on inference performance, model serving, and agent infrastructure.
We Signed a Multi-Year Agreement to Deploy Cerebras Wafer-Scale Inference
The CS-3 joins SambaNova in our fleet. It is the fastest per-user decode we can rack today, and it goes live in Q1 2027.
- Infrastructure
General Compute Raises $100M Series A
1 min - Infrastructure
General Compute Partners with Positron AI
1 min - Infrastructure
We Signed a Multi-Year Agreement to Deploy Cerebras Wafer-Scale Inference
4 min - Insights
How Transformer Architecture Determines Inference Speed and Memory Usage
12 min - Insights
What Is an AI Inference Server? Cloud API vs Self-Hosted vs On-Premise Explained
12 min - Insights
Why TileRT Cannot Catch SambaNova SN50: The Limits of Software-Emulated Dataflow on the Wrong Hardware
15 min - Insights
AI Inference Costs in 2025: How to Calculate and Reduce Your LLM API Spend
11 min - Insights
Fast AI Inference: 8 Proven Techniques That Deliver Real-World Speedups
12 min - Insights
What Is Tokens Per Second (TPS)? The LLM Throughput Metric That Actually Matters
13 min - Insights
What Is Time to First Token (TTFT)? The LLM Latency Metric That Defines UX
11 min - Insights
LLM Inference Explained: How It Works, What It Costs, and How to Optimize It
15 min - Insights
The 1B-and-Under Club: TinyLlama vs Phi-2 vs StableLM vs Qwen 0.5B
9 min - Insights
Time-to-First-Token Leaderboard: Every Major Open Model Ranked
10 min - Insights
Batch Inference Throughput: Which Models Maximize Tokens-per-Dollar?
9 min - Insights
Long-Context Benchmarks: Who's Actually Fast at 128K Tokens?
7 min - Insights
MoE vs Dense at Equal Compute: Mixtral 8x7B vs Llama 70B
8 min - Insights
Quantized vs Full Precision: GPTQ, AWQ, and FP8 Benchmarks Across Model Sizes
9 min - Insights
Instruction-Tuned vs Base Models: Does Fine-Tuning Cost You Speed?
8 min - Insights
Vision-Language Models: Llama 4 Scout vs Qwen2-VL vs InternVL at Inference
10 min - Insights
Embedding Model Benchmarks: Speed vs Quality for RAG Retrieval
8 min - Insights
Math and Reasoning Models: DeepSeek-R1 vs Qwen QwQ vs Llama Reasoning
7 min - Insights
Coding Model Benchmarks: Qwen 2.5 Coder vs DeepSeek Coder V2 vs CodeLlama vs StarCoder 2
9 min - Insights
Frontier Open Models: Llama 4 Maverick vs DeepSeek V3 vs Qwen 2.5 110B
8 min - Insights
Mid-Size Models: Llama 3.1 70B vs Qwen 2.5 72B vs Mistral Large vs DeepSeek-V2 Lite
12 min - Insights
The 7-8B Battle: Llama 3.1 8B vs Mistral 7B v0.3 vs Qwen 2.5 7B vs Gemma 2 9B
10 min - Insights
Small Models Showdown: Qwen 2.5 3B vs Llama 3.2 3B vs Phi-3.5 Mini vs Gemma 2 2B
8 min - Insights
Agentic RAG: When Retrieval and Generation Both Need to Be Fast
9 min - Insights
Long-Running Agents: Managing Inference Costs Over Hours of Execution
10 min - Insights
Evaluating Agent Performance: Latency as a First-Class Metric
9 min - Insights
Browser Agents: Inference Speed as the Gating Factor
8 min - Insights
Voice Agents and the 500ms Window: An Inference Architecture Guide
9 min - Insights
Planning and Search with LLMs: Tree-of-Thought at Inference Scale
9 min - Insights
The Cost of Agent Retries: When Slow Inference Kills Reliability
9 min - Insights
Whisper API for Real-Time Transcription: Alternatives to OpenAI Whisper
10 min - Insights
Fireworks AI Alternative: Faster Inference at Lower Cost
7 min - Insights
DeepSeek API: How to Access DeepSeek V3 Without Rate Limits
7 min - Insights
Inference API Pricing Guide: Groq vs Fireworks vs Together vs GeneralCompute
9 min - Insights
vLLM vs Managed Inference APIs: When to Self-Host vs Pay-as-You-Go
9 min - Insights
Llama API: How to Run Llama 4 via API Without Managing Infrastructure
8 min - Insights
Best OpenAI API Alternatives in 2025: Full Developer Comparison
8 min - Insights
GeneralCompute vs Groq: Speed, Pricing, and Model Selection Compared
8 min - Insights
Scaling Laws in LLMs: What They Mean for Inference Cost in Production
9 min - Insights
Mixture of Experts (MoE) Models: Why They're Dominating 2025
9 min - Insights
KV Cache in LLM Inference: How It Works and Why It Matters
9 min - Insights
Agentic AI in Production: Inference Requirements for Multi-Step Workflows
9 min - Insights
How to Build a Real-Time Coding Assistant With Open-Source Models
8 min - Insights
Multi-Turn Conversations in LLM APIs: Best Practices for Agents
10 min - Insights
Your AI Agent Is Only as Good as Its Inference Speed
8 min - Insights
What Are Agentic AI Systems? How to Build Them With Fast Inference
10 min - Insights
SOC 2 Compliant AI Inference: What Enterprise Teams Need to Know
9 min - Insights
Embedding Models in Production: Choosing the Right One for Your App
10 min - Insights
How to Migrate From OpenAI to GeneralCompute in 10 Lines of Code
6 min - Insights
Using Vercel AI SDK With GeneralCompute: Full Integration Guide
7 min - Insights
How to Build a RAG Pipeline Using Open-Source Models
10 min - Insights
JSON Mode in LLMs: How to Get Structured Outputs Every Time
8 min - Insights
OpenAI-Compatible APIs: How to Migrate Your App in 5 Minutes
6 min - Insights
Tool Calling With Open-Source LLMs: A Complete Guide
8 min - Insights
How to Build a Streaming Chat App With GeneralCompute + Node.js
7 min - Insights
GPU Cluster for LLM Inference: Build vs Buy Analysis for ML Teams
9 min - Insights
Real-Time AI Inference: How to Achieve <100ms Latency in Production
9 min - Insights
Fine-Tuning vs RAG: Which Approach Is Right for Your Production App?
8 min - Insights
H100 vs H200 vs B200: Which GPU Is Best for LLM Inference?
9 min - Insights
LLM Token Generation Speed: How Providers Compare in 2025
9 min - Performance
AI Inference Latency Explained: TTFT, TPS, and How to Optimize Them
9 min - Performance
GeneralCompute vs vLLM: Throughput, Latency, and Cost Benchmarks
9 min - Models
Open-Source LLM Landscape 2025: Top Models Compared
9 min - Models
Faster-Whisper: Real-Time Speech-to-Text on GeneralCompute
10 min - Models
QwQ-32B: The Reasoning Model That Rivals o1 — Complete Guide
10 min - Models
How to Fine-Tune Llama 4: Step-by-Step Guide with Code
9 min - Models
Qwen3-Coder: The Best Open-Source Coding Model? Benchmark + Guide
9 min - Models
Llama 4 on GeneralCompute: Getting Started Guide
9 min - Models
DeepSeek R1: What It Is, How It Works, and Why It Matters
9 min - Fundamentals
Flash Attention: Why Modern LLMs Run Faster With It
9 min - Fundamentals
What Is Direct Preference Optimization (DPO)? Explained Simply
10 min - Infrastructure
Quantization Explained: INT4, GGUF, GPTQ and What They Mean for Your Model
9 min - Fundamentals
What Is Speculative Decoding? How It Makes LLMs 3x Faster
11 min - Fundamentals
LPU vs GPU vs CPU: Which Processor Wins for AI Inference?
9 min - Fundamentals
What Is AI Inference? A Developer's Complete Guide
9 min - AI agents
Streaming for Agents: Why Partial Results Change the UX
10 min - AI agents
Parallel Tool Execution: How Fast Inference Enables Concurrent Agent Actions
11 min - AI agents
Agent Memory Systems: Balancing Context Length vs Retrieval Latency
10 min - AI agents
Building a Code Agent: Why Each Step Needs Sub-Second Inference
9 min - AI agents
ReAct, Reflexion, and Chain-of-Thought: The Inference Cost of Reasoning Patterns
9 min - AI agents
Multi-Agent Architectures and the Inference Cost Explosion
8 min - AI agents
Tool Calling Latency: The Bottleneck No One Talks About
9 min - AI agents
The Agentic Inference Tax: Why Agents Need 10x Faster Models
9 min - Infrastructure
Compiler-Level Optimizations for Inference: TorchInductor, Triton, XLA
8 min - Infrastructure
Draft Model Selection for Speculative Decoding
8 min - Fundamentals
The Attention Sink Phenomenon: Why the First Token Matters
9 min - Fundamentals
Mixture of Experts at Inference Time
9 min - Infrastructure
Tensor Parallelism vs Pipeline Parallelism for Model Serving
10 min - Infrastructure
Prefix Caching: Why Repeated Prompts Shouldn't Cost You Twice
9 min - Fundamentals
Distillation for Inference: How Smaller Models Learn From Larger Ones
10 min - Fundamentals
FP8 Training and Inference: The Precision Sweet Spot
9 min - Fundamentals
Activation-Aware Quantization (AWQ) Deep Dive
10 min - Fundamentals
Mamba and State Space Models: Inference Without Attention
9 min - Fundamentals
RWKV and Linear Attention: Recurrent Models as an Inference Shortcut
8 min - Infrastructure
Dynamic Batching Strategies: From Naive to Continuous to Iteration-Level
10 min - Fundamentals
Token Merging and Token Pruning for Faster Transformers
10 min - Infrastructure
S3: Scheduling for Straggler Mitigation in LLM Serving
10 min - Infrastructure
Chunked Prefill: Overlapping Compute and Communication
10 min - Infrastructure
Cascade Inference: Using Small Models to Route to Big Ones
10 min - Infrastructure
Lookahead Decoding: Parallel Token Generation Without Draft Models
10 min - Infrastructure
Disaggregated Prefill and Decode (Splitwise / DistServe)
9 min - Infrastructure
KV Cache Compression: MLA and Beyond
10 min - Fundamentals
Ring Attention: Scaling Context to Millions of Tokens
10 min - Fundamentals
Quantization for Inference: GPTQ, AWQ, SmoothQuant, and FP8
6 min - Infrastructure
Multi-Query and Grouped-Query Attention: Shrinking the KV Cache
5 min - Infrastructure
Continuous Batching: The Orca Paper That Changed LLM Serving
4 min - Infrastructure
Medusa, EAGLE, and Sequoia: The Next Generation of Speculative Decoding
7 min - Infrastructure
SGLang and RadixAttention: Smarter KV Cache Reuse
4 min - Infrastructure
Speculative Decoding: Getting 3x Speedups Without Changing the Model
5 min - Fundamentals
PagedAttention and vLLM: Virtual Memory for LLM Serving
5 min - Fundamentals
FlashAttention: How Tri Dao Made Attention 4x Faster
5 min - Performance
Build a Real-Time Voice AI Agent with General Compute
8 min - Performance
How Coding Agents Depend on Inference Speed
7 min - Performance
Why Inference Speed is the New Moat
6 min
Get the next post by email.
New benchmarks, hardware notes and company news. No more than twice a month.