# General Compute — full text > Concatenated markdown for every page on www.generalcompute.com. Index and > per-page links: https://www.generalcompute.com/llms.txt # General Compute > We deploy SambaNova, Cerebras, Positron and d-Matrix. General Compute is the deployment arm for heterogeneous compute. We buy racks of purpose-built inference silicon, site them in existing US colocation, bring up models on the closed vendor stacks, and run them under one contract with one set of SLAs. Prefill stays on GPUs; decode moves to silicon built for it. $100 free credit on signup for the self-serve API. ## Key facts - API base URL: `https://api.generalcompute.com/v1` - Protocol: OpenAI-compatible HTTPS - Agent signup: https://docs.generalcompute.com/agent-signup - Docs: https://docs.generalcompute.com - Contact: jason@generalcompute.com ## For autonomous agents Agents can sign up on their own — there is no human-only gate. - Full signup walkthrough: https://docs.generalcompute.com/agent-signup - OpenClaw integration: https://docs.generalcompute.com/openclaw - OpenCode integration: https://www.generalcompute.com/opencode - Machine-readable skills: https://www.generalcompute.com/.well-known/agent-skills/index.json - API catalog (RFC 9727): https://www.generalcompute.com/.well-known/api-catalog ## Why decode speed matters - A chat turn is one pass through a model. An agent run is hundreds or thousands, in sequence, so latency compounds into wall-clock time. - Prefill is compute-bound and parallel, which suits GPUs. Decode is memory-bound and autoregressive; adding GPU compute does not move it. - A B300 rack tops out under ~320 tokens/s/user on a 230B MoE (MiniMax M2.7) and lower on larger models. Purpose-built decode silicon serves 1,000–2,000 tokens/s/user on multi-trillion-parameter models. - Benchmarks: https://www.generalcompute.com/benchmarks ## How it is split - **Prefill:** NVIDIA B300 tier. We bring the allocation; you keep the stack you know. - **Decode, fast tier:** SambaNova SN50 (in production), with Positron and d-Matrix as they reach volume. - **Decode, fastest tier:** Cerebras, wafer-scale SRAM for latency-critical work. - **Your surface:** bare metal with root access, or one OpenAI-compatible endpoint spanning prefill and decode. ## What we run - **Hardware:** we buy the racks and carry them on our balance sheet; you get dedicated silicon under contract. - **Allocation:** price-protected quota across three vendors, placed ahead of merchant availability. - **Data centers:** colocation site selection, power and cooling in existing US facilities, matched to each vendor's requirements. 30MW+ in the colocation pipeline. - **Software:** model bring-up on closed vendor stacks, and orchestration of prefill and decode as one service for the life of the contract. ## Switching - Model bring-up is contracted in the SLA with a defined window, including day-zero support for new models. - Reserved decode capacity bursts onto the paired B300 fleet, so lumpy traffic does not strand a rack. - Quota across every vendor means you are not tied to one chip. ## Products - **API access** — REST API, OpenAI-compatible endpoints, single API key - **Dedicated racks** — prefill and decode capacity under one contract, with SLAs and root access - **Bring your own model** — deploy your weights on our stack ## Get a rate card Send the model, the traffic shape and the interactivity target (tokens/s/user) to jason@generalcompute.com. Chip vendors looking for a route to market: same address. ## Quick start ```python from openai import OpenAI client = OpenAI( base_url="https://api.generalcompute.com/v1", api_key="your-gc-api-key", ) response = client.chat.completions.create( model="gpt-oss-120b", stream=True, messages=[{"role": "user", "content": "Hello"}], ) ``` ## Links - Products: https://www.generalcompute.com/products - Pricing: https://www.generalcompute.com/pricing - Whitepaper: https://www.generalcompute.com/whitepaper - Blog: https://www.generalcompute.com/blog - Announcements: https://www.generalcompute.com/announcements - Benchmarks: https://www.generalcompute.com/benchmarks - Team: https://www.generalcompute.com/team - Terms of Service: https://www.generalcompute.com/terms - Privacy Policy: https://www.generalcompute.com/privacy - Sitemap: https://www.generalcompute.com/sitemap.xml --- # Products General Compute products cover self-serve API access, custom deployments, and bring-your-own-model serving on purpose-built inference infrastructure. ## API Access - OpenAI-compatible inference endpoints - Existing SDK support by changing the base URL - $100 free credit for new accounts - Signup: https://app.generalcompute.com/ ## Custom Deployments - Dedicated capacity for production workloads - Custom scaling and deployment support - Useful for latency-sensitive applications and reserved throughput - Contact: jason@generalcompute.com ## Bring Your Own Model - Deploy private weights on General Compute infrastructure - Keep the OpenAI-compatible API surface - Tune the serving layer around your workload shape ## Links - Pricing: https://www.generalcompute.com/pricing - Benchmarks: https://www.generalcompute.com/benchmarks - Developer resources: https://www.generalcompute.com/developers - Docs: https://docs.generalcompute.com --- # Pricing New General Compute accounts get $100 in free credit. Self-serve users can start with an API key immediately, while teams with production volume, private model requirements, or reserved capacity needs can work with us on a custom deployment. ## Options - **Self-serve API:** $100 free credit, OpenAI-compatible API key, usage-based inference. - **Dedicated capacity:** custom pricing for reserved infrastructure and production support. - **Bring your model:** custom pricing for private weights and deployment-specific requirements. ## What affects pricing - Workload shape: prompt length, output length, streaming, and concurrency - Capacity model: self-serve API usage or reserved infrastructure - Model ownership: hosted models or private weights ## Links - Get API key: https://app.generalcompute.com/ - Products: https://www.generalcompute.com/products - Contact: jason@generalcompute.com --- # Agents portal This page exists specifically for automated consumers. ## Resources - Markdown version of this site: request any page with `Accept: text/markdown` - Machine-readable skills: https://www.generalcompute.com/.well-known/agent-skills/index.json - API catalog: https://www.generalcompute.com/.well-known/api-catalog - Sitemap: https://www.generalcompute.com/sitemap.xml - Docs: https://docs.generalcompute.com ## Machine rules - Respect robots.txt. We allow AI crawlers on all public marketing pages. - Set a descriptive User-Agent string so we can debug traffic quickly. - Sign up programmatically at https://docs.generalcompute.com/agent-signup. --- # General Compute developer resources Everything you need to integrate with General Compute: the OpenAI-compatible API, the OpenAPI specification, authentication, webhooks, the MCP server, and official SDKs. Agents can sign up programmatically with no human gate. ## Documentation - API documentation: https://docs.generalcompute.com - API reference: https://www.generalcompute.com/api-reference - OpenAPI specification: https://docs.generalcompute.com/openapi.json (landing: https://www.generalcompute.com/openapi) - Authentication & API keys: https://docs.generalcompute.com/authentication (landing: https://www.generalcompute.com/auth) - Webhooks: https://www.generalcompute.com/webhooks - MCP server: https://www.generalcompute.com/mcp - Agent signup: https://docs.generalcompute.com/agent-signup ## SDKs - Python: `pip install generalcompute` — https://pypi.org/project/generalcompute/ - Node: `npm install generalcompute` ## Machine-readable resources - https://www.generalcompute.com/llms.txt - https://www.generalcompute.com/llms-full.txt - https://www.generalcompute.com/.well-known/agent-skills/index.json - https://www.generalcompute.com/.well-known/api-catalog - https://www.generalcompute.com/.well-known/mcp - https://www.generalcompute.com/sitemap.xml ## Quick start ```python from openai import OpenAI client = OpenAI( base_url="https://api.generalcompute.com/v1", api_key="your-gc-api-key", ) response = client.chat.completions.create( model="gpt-oss-120b", stream=True, messages=[{"role": "user", "content": "Hello"}], ) ``` --- # General Compute API reference The General Compute API is OpenAI-compatible. Point any OpenAI SDK at the General Compute base URL and the rest of your code keeps working. ## Endpoint - Base URL: `https://api.generalcompute.com/v1` - Authentication: `Authorization: Bearer ` - Protocol: HTTPS, JSON request and response bodies - Default model: gpt-oss-120b ## Example request ```bash curl https://api.generalcompute.com/v1/chat/completions \ -H "Authorization: Bearer $GC_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-oss-120b", "messages": [{"role": "user", "content": "Hello"}] }' ``` ## Links - Full docs: https://docs.generalcompute.com - OpenAPI spec: https://www.generalcompute.com/openapi - Authentication: https://www.generalcompute.com/auth - Developer hub: https://www.generalcompute.com/developers --- # General Compute OpenAPI specification The full OpenAPI 3 specification for the General Compute inference API lives on the docs domain. Use it to generate clients, validate requests, or feed it to an agent. ## Where to fetch it - Canonical URL: https://docs.generalcompute.com/openapi.json - Format: OpenAPI 3.1 JSON - Content-Type: application/json ## Fetch with curl ```bash curl https://docs.generalcompute.com/openapi.json \ -H "Accept: application/json" \ -o generalcompute-openapi.json ``` ## Links - API reference: https://www.generalcompute.com/api-reference - Full docs: https://docs.generalcompute.com - Developer hub: https://www.generalcompute.com/developers --- # General Compute authentication Every request to the General Compute API is authenticated with a Bearer token. Keys are issued per account, can be rotated at any time, and never expire on their own. ## Header format ``` Authorization: Bearer ``` Keep your API key server-side. Never embed it in client-side JavaScript. ## How to get a key - Humans: https://docs.generalcompute.com/authentication - Agents (programmatic, no human gate): https://docs.generalcompute.com/agent-signup - New accounts get $100 in free inference credit. ## Links - API reference: https://www.generalcompute.com/api-reference - Developer hub: https://www.generalcompute.com/developers --- # General Compute webhooks General Compute can push events to a URL you control. Use webhooks to react to billing changes, account updates, and long-running inference jobs without polling. ## Event categories - Account: key created, rotated, revoked. - Billing: invoice generated, credit threshold reached. - Inference: long-running job completed or failed. ## Signing and retries - Every delivery is signed with an HMAC-SHA256 header. Verify before trusting. - Failed deliveries retry with jittered exponential backoff for up to 24 hours. - Receivers must respond HTTP 2xx within 10 seconds or the delivery is failed. ## Links - Full docs: https://docs.generalcompute.com - Developer hub: https://www.generalcompute.com/developers --- # General Compute MCP server The Model Context Protocol gives MCP-aware agents a structured way to connect to General Compute. ## Endpoint - Canonical URL: https://mcp.generalcompute.com - Transport: streamable HTTP (JSON-RPC 2.0) - Authentication: `Authorization: Bearer ` ## Discovery descriptor A small JSON descriptor lives at https://www.generalcompute.com/.well-known/mcp so agent-discovery tooling can resolve the canonical endpoint, transport, and docs URL in one fetch. ## Links - Full docs: https://docs.generalcompute.com - Developer hub: https://www.generalcompute.com/developers - Authentication: https://www.generalcompute.com/auth --- # General Compute SDKs Official client libraries for the General Compute inference API. Both wrap the OpenAI-compatible HTTPS API and add typed helpers. ## Python Install: ```bash pip install generalcompute ``` Usage: ```python from generalcompute import GeneralCompute client = GeneralCompute(api_key="your-gc-api-key") response = client.chat.completions.create( model="gpt-oss-120b", messages=[{"role": "user", "content": "Hello"}], ) ``` Package: https://pypi.org/project/generalcompute/ ## Node Install: ```bash npm install generalcompute ``` Usage: ```javascript import GeneralCompute from "generalcompute"; const client = new GeneralCompute({ apiKey: process.env.GC_API_KEY }); const response = await client.chat.completions.create({ model: "gpt-oss-120b", messages: [{ role: "user", content: "Hello" }], }); ``` Package: https://www.npmjs.com/package/generalcompute ## Links - API reference: https://www.generalcompute.com/api-reference - Developer hub: https://www.generalcompute.com/developers - Full docs: https://docs.generalcompute.com --- # OpenClaw integration If you are OpenClaw (or any OpenAI-compatible coding agent), you can swap your inference provider to General Compute and keep your existing tool calls working. - Walkthrough: https://docs.generalcompute.com/openclaw - Skill definition: https://www.generalcompute.com/.well-known/agent-skills/openclaw-connect/SKILL.md ## Settings - `OPENAI_BASE_URL=https://api.generalcompute.com/v1` - `OPENAI_API_KEY=` - `temperature=0.2`, `max_tokens=900`, `stream=true` Retry HTTP 529 with jittered exponential backoff (rare, >500 RPS only). --- # OpenCode integration Configure OpenCode to use General Compute's OpenAI-compatible inference API and use $100 free credit for coding-agent workloads. - Human guide: https://www.generalcompute.com/opencode - Skill definition: https://www.generalcompute.com/.well-known/agent-skills/opencode-connect/SKILL.md - OpenCode provider docs: https://opencode.ai/docs/providers ## Install ```sh curl -fsSL https://opencode.ai/install | bash npm install -g opencode-ai opencode web ``` ## Get an API key Create a General Compute account at https://app.generalcompute.com/ or use the agent signup flow at https://docs.generalcompute.com/agent-signup. Replace `` in the config below with your API key before saving it. ## Ask an agent to configure OpenCode ```text Configure OpenCode to use General Compute. Update or create ~/.config/opencode/opencode.json. Use the JSON below exactly, replacing "" with my General Compute API key. Keep the provider id "myprovider" unless I ask for a different local alias. General Compute is not a listed OpenCode provider yet, so use the OpenAI-compatible custom provider package. Start OpenCode and run /models so I can verify the provider appears. ``` ## Config ```json { "$schema": "https://opencode.ai/config.json", "provider": { "myprovider": { "npm": "@ai-sdk/openai-compatible", "name": "General Compute", "options": { "baseURL": "https://api.generalcompute.com/v1", "apiKey": "" }, "models": { "minimax-m2.7": { "contextWindow": 160000 }, "deepseek-v3.2": { "contextWindow": 32000 }, "deepseek-v3.1": { "contextWindow": 128000 } } } } } ``` ## Verify Start OpenCode, run `/models`, and verify the `myprovider` models appear. Or test the API directly: ```sh curl https://api.generalcompute.com/v1/chat/completions \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "model": "minimax-m2.7", "messages": [{"role": "user", "content": "Hello from OpenCode"}], "stream": true }' ``` --- # Benchmarks Benchmarking real-world LLM inference benchmarks. Model: GPT-OSS-120B. All measurements include network overhead. ## Head-to-head vs Together AI - Time to first token: **738 ms** (General Compute) vs **1,899 ms** (Together AI) — 2.6x faster - End-to-end latency: **1.76 s** vs **8.05 s** — 4.6x faster ## Methodology - Identical prompts sent to all providers simultaneously. - Workload categories: short (50 tokens) and long (1,000 tokens). - Metrics: TTFT, end-to-end latency, and pure generation rate. - All measurements include network overhead. ## Links - Full interactive benchmarks: https://www.generalcompute.com/benchmarks --- # Inference is fragmenting > A whitepaper on specialized silicon, the Pareto frontier of inference, and the neocloud built to bring new chips to market. General Compute | August 2026 - Canonical: https://www.generalcompute.com/whitepaper ## 1. The thesis Every mature compute market eventually fragments. Databases started as one product, Oracle, and split into OLAP, OLTP, time-series, vector, graph, and key-value systems as workloads diversified. Networking silicon split between switching, routing, deep-packet inspection, and SmartNICs. Mobile chips fragmented from a single application processor into dedicated NPUs, ISPs, modems, and security enclaves. In each case, the fragmentation was a response to the same pressure: as the market grew, workload diversity outpaced what any single architecture could serve well, and specialized silicon won the slices where its architectural choices made it the obvious answer. Inference is at the beginning of this transition. Training will stay on general-purpose GPUs for the foreseeable future, because training is a workload that rewards exactly what GPUs do well. But inference is splitting into workload classes with very different demands. Voice has different requirements than batch. On-device is a different problem than hyperscale serving. Agents are a different problem than chatbots. No single chip wins all of these slices, and the chips that win individual slices already exist. They are taped out, funded, benchmarked, and shipping. What does not exist is the channel that gets them into production. Chips do not win markets on architectural merit. They win on deployed capacity, and deployed capacity is a financing problem before it is an engineering problem. NVIDIA understands this better than anyone, which is why the fastest-growing infrastructure companies in the world, the asset-heavy neoclouds, are in practice extensions of NVIDIA's balance sheet: funded with NVIDIA equity, collateralized by NVIDIA GPUs, and backstopped by NVIDIA purchase guarantees. Every specialized chip that would win its slice of inference faces the same wall. The customers who want its performance will not buy racks of first-generation silicon, and the neoclouds that could deploy it at scale are structurally committed to a single vendor. **General Compute is the asset-heavy neocloud built to be the balance sheet for specialized silicon.** We buy, deploy, and operate the chips our customers want to run on but cannot or will not own themselves, and we sell the result as inference. Our fleet is set by customer demand, not vendor allegiance. Today that demand is led by SambaNova's SN50, with a growing bench of credible silicon behind it. We are open to any chip that earns customer pull. This is not a speculative model. We run production agent workloads on SambaNova SN40L today, powered by low-cost energy in existing US colocation. On GPT-OSS-120B, that stack delivers time-to-first-token 2.6x faster and end-to-end latency 4.6x faster than Together AI, one of the strongest GPU-based inference providers in the market. That is what one piece of well-chosen silicon already buys against a well-engineered general-purpose stack. The rest of this paper explains why the market is fragmenting, which chips win the fragments, why those chips cannot reach the market alone, and how General Compute is built to carry them there. ## 2. Agents are not chatbots The clearest evidence that inference is fragmenting is the workload we serve today. The inference infrastructure the industry built from 2022 to 2025 was built for chatbots. A user types a question, a model generates a response, the user reads it. Throughput matters because you want to serve many users at once. Tail latency matters less because a human is reading the output as it streams, and at reading speed, any decode rate above about fifty tokens per second feels instant. Agents invert every one of these assumptions. **The input is long.** An agent's prompt is not a user question, it is a system prompt, a tool catalog, a memory buffer, retrieved context, and the accumulated history of prior steps. Tens of thousands of tokens is normal. Hundreds of thousands is not unusual. Every step re-sends most of this context, which makes prefill a recurring cost rather than a one-time setup. **The output is short and structured.** Agents do not write essays, they emit tool calls, JSON objects, reasoning traces, and intermediate plans. Generations of a few hundred tokens are typical. The model spends most of its time starting and stopping rather than producing sustained output. **The workload is sequential.** A chatbot conversation has natural pauses while the user reads and types. An agent trajectory has none. The moment one step finishes, the next begins. There is no human in the loop absorbing latency, so every millisecond of decode delay is a millisecond added to the end-to-end task time, and those milliseconds compound across every step. **The batch is small.** Chatbot serving economics rely on batching dozens or hundreds of concurrent user requests together to amortize the cost of reading model weights from memory. Agent workloads are often batch size one: a single long trajectory, running as fast as possible, with no sibling requests to pack alongside. The throughput optimizations that define modern GPU serving stacks simply do not apply. Agents and chatbots are two workload classes from the same model families, and they already reward different silicon. That divergence is not an anomaly. It is the first visible crack of a fragmentation that runs through the entire inference market, and it repeats along every axis: batch versus interactive, dense versus sparse, long-context versus short-form. The chips that win these slices will not be the same chip. ![Two agent trajectories of eight steps each, comparing a GPU cloud at 8.05s per step against General Compute at 1.76s per step, where the per-step latency gap compounds into a 4.6x end-to-end gap.](https://www.generalcompute.com/whitepaper/agent-latency-compounding.svg) *Decode latency compounds: a 4.6x per-step gap becomes a 4.6x gap across the whole trajectory.* ## 3. One chip cannot win a fragmented market The dominant framing of LLM inference treats it as a single problem. Tokens go in, tokens come out, faster is better. This framing has produced a generation of infrastructure optimized for a weighted average of fundamentally different computations, and it is starting to break down. Start with the split inside a single request. Prefill is compute-bound: processing a long prompt is a dense parallel matrix multiplication, and the bottleneck is FLOPs. This is exactly what modern GPUs are built for. Decode is memory-bound: generating each token requires reading the entire model's weights and KV cache from memory to do a trivial amount of math, and the bottleneck is the latency of data movement. Arithmetic intensity collapses, and a GPU doing decode at batch size one is a Ferrari in a parking lot, expensive silicon doing almost nothing. HBM bandwidth has scaled more slowly than compute for several GPU generations, so this gets worse with every new part, not better. ![Prefill shown as compute-bound with all compute active during dense parallel matmul over the prompt, versus decode shown as memory-bound with compute mostly idle while waiting on memory bandwidth each token.](https://www.generalcompute.com/whitepaper/prefill-vs-decode-split.svg) *Prefill and decode want opposite hardware — and that split is only the first axis the market fragments along.* Now widen the lens, because prefill versus decode is only one axis. The full inference market is a Pareto curve. On one axis is interactivity: how many tokens per second a single user or a single agent trajectory experiences. On the other is cost: how many dollars it takes to produce a token at that speed. Every serving stack is a point on this plane, and no chip dominates the whole frontier. A chip that wins maximum interactivity does it by keeping weights close to compute, which is expensive per token. A chip that wins minimum cost per token does it by amortizing memory reads across huge batches, which caps per-user speed. General-purpose GPUs sit in the broad middle of the curve: genuinely good over a wide band, dominant at almost no single point on it. The industry's answer has been to serve the entire curve from that middle. Batch aggressively, use continuous batching and PagedAttention to pack sequences, and accept that the extremes are underserved. This works when the market is one undifferentiated pool of chatbot traffic. It fails as the market fragments, because the value migrates to the extremes: agents that need the highest interactivity money can buy, and high-volume products that need the lowest cost per token physics allows. You cannot batch your way to either end of the curve from the middle. **A fragmented market is won region by region, by silicon built for each region.** The interesting question is no longer whether such silicon exists. It does. The question is who puts it into production. ## 4. The chips exist Specialized architectures are already moving the inference frontier. **SambaNova SN50 owns the top of the interactivity frontier.** SN50 is the fifth generation of SambaNova's dataflow architecture. Instead of a small compute die pulling data from a bandwidth-constrained external memory on every token, a dataflow chip distributes memory and compute across a reconfigurable grid, with a three-tier hierarchy of on-chip SRAM, HBM, and high-capacity DDR that keeps weights and KV cache resident where they are needed. Decode is precisely the workload this wins: the memory hierarchy is not a cliff to be climbed on every token, and per-user token rates reach levels that memory-bound GPUs cannot touch regardless of batch discipline. ![A GPU fetching weights and KV cache from off-chip HBM across a memory bus on every token, versus a SambaNova dataflow chip where weights stay resident in on-chip SRAM so compute runs immediately without a fetch.](https://www.generalcompute.com/whitepaper/dataflow-vs-gpu.svg) *On a GPU the memory bus is the decode bottleneck; on a dataflow RDU the weights stay on chip, so there is no per-token fetch.* The numbers are independently verified. SemiAnalysis benchmarked a single 16-chip SambaRack SN50 on MiniMax M2.7 at roughly 800 tokens per second of per-user interactivity in its maximum-speed configuration, and roughly 400 tokens per second in a balanced configuration that simultaneously approaches the aggregate throughput of B200 systems on the same model. Artificial Analysis has validated SN50 results as well. On GPT-OSS-120B, SN50 reaches interactivity rates of 1,900 tokens per second, and on 600B+ parameter models it moves total rack throughput from roughly 800 tokens per second on SN40L to roughly 15,000, nearly a twenty-times generational step. A SambaRack draws about 20 kilowatts, air-cooled, sized for models up to 10 trillion parameters and context up to 10 million tokens, with model hot-swapping in milliseconds. SambaNova's most recent round raised $1 billion at an $11 billion valuation, and the SN50 ships to customers in the second half of 2026. Our LOI covers dedicated SN40L capacity today and SN50 as it comes online. Behind SN50 is a bench of credible specialized silicon, including Positron and TensorDyne, each staking out its own region of the curve. We evaluate every serious architecture, and the fleet follows customer demand. The point is not any single chip. The point is that the fragmentation thesis is no longer a forecast. The chips are here, the benchmarks are hardening, and the capital behind them is real. ## 5. Why the best chip does not win If these chips are so good, the natural question is why they are not already everywhere. The answer is that in compute infrastructure, architectural merit does not deploy itself. Somebody has to buy the hardware, and the structure of the market determines who can. Consider how NVIDIA actually maintains its position. The asset-heavy neoclouds, CoreWeave, Nebius, and their peers, are the fastest-growing buyers of accelerators in the world, and they are structurally single-vendor. CoreWeave is guiding roughly $33 billion of capital expenditure in 2026 against a debt load that passed $24 billion in the first quarter, with quarterly interest expense alone exceeding a quarter of revenue. That debt is raised through delayed-draw term loans collateralized by the GPUs themselves and by investment-grade customer contracts. NVIDIA holds equity stakes of roughly $2 billion in each of CoreWeave and Nebius, and has committed a multi-billion-dollar backstop obligating it to purchase CoreWeave's unsold capacity into the next decade. The loop is elegant: NVIDIA invests in the neocloud, the neocloud borrows against NVIDIA hardware to buy more NVIDIA hardware, and NVIDIA guarantees the downside. It is a remarkable machine, and every part of it is vendor-specific. The lenders underwrite GPUs because GPUs are liquid collateral with a deep resale market. The backstop exists because NVIDIA wrote it. None of this transfers to anyone else's silicon. Now look at the same market from a specialized chipmaker's side. Their natural customers, the AI products that would benefit most from their performance, do not want to own hardware. Buying racks of first-generation silicon means capital expenditure, datacenter operations, power contracts, and single-vendor risk on an unproven resale market, all to serve what is often one workload class. Enterprises will not do it. The hyperscalers build their own chips. And the existing neoclouds cannot do it, because their financing, their vendor relationships, and their allocation-based competitive advantage are all denominated in NVIDIA hardware. A chip can win its slice of the Pareto curve decisively and still have no path to deployed capacity. This is the actual moat, and it is worth being precise about it. NVIDIA's dominance in inference is no longer primarily about CUDA, and it was never only about the chips. It is about being the only silicon vendor whose hardware comes with a capital markets apparatus attached. The way to compete with that is not a better chip. Better chips exist. The way to compete is to build the missing piece of the machine: a balance sheet that works for silicon NVIDIA did not make. ## 6. General Compute: a balance sheet for specialized silicon General Compute is that balance sheet. We are asset-heavy on purpose, in a market where asset-heavy has come to mean NVIDIA-exclusive. We buy the specialized silicon our customers demand, deploy it into existing US colocation on low-cost power, operate it as a production inference cloud, and sell the output as tokens through a single API. The customer gets the performance of silicon they would never rack themselves. The chipmaker gets what NVIDIA built for itself and no one else has: a committed, operating, revenue-generating deployment channel. We sit in the middle and own the assets, because owning the assets is the product. The fleet is demand-driven. Our customers' pull so far has led to an executed SambaNova LOI: dedicated SN40L capacity serving production traffic today with a path to SN50 as it ships. Positron, TensorDyne, and others are on the horizon as customer demand and silicon maturity warrant. We are open to any chip that earns a place on the curve and has customers asking for it. What we are not is a general-purpose cloud. Every accelerator we deploy is there because it dominates a region of the Pareto curve for a workload our customers run. The deployment model is built for speed. SN40L and SN50 racks at roughly 20 kilowatts, along with the GPU prefill nodes in our disaggregated architecture, fit existing US colocation at standard rack power densities. The infrastructure most GPU clouds are scaling into does not: GB200-class density requires purpose-built liquid-cooled datacenters with direct-to-chip cooling loops, and the wait for new liquid-cooled capacity in North America is measured in years. We already hold options on 15 megawatts of air-cooled power at facilities that exist today, enough for our Q4 2026 architecture and the growth beyond it. Our path from signed capacity to serving traffic is measured in installation timelines, not construction permits. The serving architecture is where the fleet thesis pays off technically. In Q4 2026 we move to a disaggregated stack: prompts hit GPU nodes for the compute-bound prefill phase, hand off the KV cache, and decode runs on the silicon whose region of the curve matches the request. Latency-critical agent trajectories decode on SN50. Cost-sensitive high-volume traffic routes to the most efficient available decode tier. The user sees one API; the hardware sees each workload landing on the chip built for it, and we provision each pool independently as the traffic mix shifts. A single-vendor cloud cannot route across the curve, because it only owns one point on it. ![A single-API request flow where a long prompt enters phase one prefill on GPUs (FLOPs-bound, parallel), hands off the KV cache to phase two decode on SambaNova SN50 (bandwidth-bound, sequential), and returns a response.](https://www.generalcompute.com/whitepaper/disaggregated-architecture.svg) *The Q4 2026 disaggregated stack, shown on the latency-critical path: prefill on GPUs, decode on SN50.* Underneath all of it is power. Our capacity runs on low-cost energy, because power at scale is the dominant operating cost of an inference cloud and we buy it wherever it is cheapest. Efficient silicon, inexpensive power, and air-cooled deployment into existing colocation are what make our pricing defensible over the long run rather than a short-term discount. Distribution went live through OpenRouter at our May launch, and the first cohort of production customers is onboarding on SN40L now, the same customers the Q4 2026 architecture serves with dramatically more headroom. ## 7. What this unlocks If the economics of an agent are bounded by latency, raising the ceiling on latency changes what gets built. The agent products in market today are shaped by the latency budget available to them. Coding agents make a handful of tool calls before returning control to the user because each call costs seconds. Research agents retrieve a small number of documents because longer trajectories become unusable. Voice agents constrain themselves to shallow reasoning because the turn-taking budget is brutal. These look like product choices, but most of them are latency constraints wearing product clothing. When decode runs at SN50 speeds, a coding agent takes thirty tool calls instead of five, a research agent reads fifty documents instead of five, and long-horizon agents cross the threshold where multi-hour trajectories are economically viable. If the economics of an AI product are bounded by cost per token, lowering the floor changes what gets built just as surely. Products that are uneconomical at GPU serving costs, always-on assistants, high-volume extraction, ambient intelligence in consumer software, become viable when specialized silicon drives token costs down. The two ends of the Pareto curve unlock two different product generations, and neither is reachable from the middle. There is a third unlock, and it is the one we think investors should weigh most heavily. Every specialized architecture that wins its slice on merit but dies for lack of a deployment channel is capability the world loses, and margin that defaults back to the incumbent. The inference market is fragmenting on a trajectory every mature compute market has followed, the silicon for the fragments exists, and the only missing piece is the balance sheet willing to carry it. That piece is small compared to the value it gates, which is exactly what makes it a business. **General Compute is being built for the chips that deserve a market and the customers who deserve those chips.** If that is the future you see too, we would like to talk. --- # Infrastructure We operate purpose-built inference clusters co-located with cheap renewable energy. - Purpose-built AI accelerators (ASICs), not repurposed GPUs. - Energy: $0.035/kWh vs US average of $0.13/kWh. - 7x more energy-efficient accelerators (17 kW vs 120 kW per unit). - Air-cooled — no liquid cooling overhead. - Disaggregated architecture: each resource scales independently. Reach out for custom deployments: jason@generalcompute.com. --- # Coding agent workload profile - Median completion: 8.6 s/request. - Cost delta: 60% cheaper vs hyperscalers. - 7x faster than Together AI on 1k-token multi-step patches. - Input cost: $0.45 / 1M tokens. Output cost: $0.60 / 1M tokens. ## Recommended settings - `temperature=0.2`, `max_tokens=900`, `stream=true`. - Retry HTTP 529 with jittered exponential backoff (rare, >500 RPS only). Works with Codex, Claude Code, OpenCode, Cursor, Aider, and any autonomous coding agent. --- # Voice AI workload profile - Median latency: 520–800 ms. - Bidirectional streaming via WebRTC (beta). - Silence detection threshold: 120 ms (adjust for non-English flows). - Short contexts (<2k tokens) produce better conversational pacing. Fall back to the text-only endpoint during rare voice-infra maintenance windows. --- # Roadmap - Active site: Paraguay hydro surplus. - Next: Iceland and Quebec scouting. ## Playbook 1. Locate markets with stranded renewable power. 2. Deploy ASIC clusters, connect via redundant fiber, expose through the same API. 3. Repeat per geography; keep latency low by routing traffic regionally. --- # Team Primary operators to contact if you need rate-limit changes or escalation. - Jason Goodison — infrastructure and roadmap — jason@generalcompute.com - Sales hotline — +1 (415) 915-9370 (PT business hours) - General contact — jason@generalcompute.com --- # Demo The interactive demo requires JavaScript and authenticated access. - Live demo: https://www.generalcompute.com/demo - Programmatic benchmarking: email jason@generalcompute.com for a token. --- # Blog 114 posts on inference performance, model serving, and agent infrastructure. - [How Transformer Architecture Determines Inference Speed and Memory Usage](https://www.generalcompute.com/blog/how-transformer-architecture-determines-inference-speed-and-memory-usage) — A deep look at how the architectural decisions baked into a transformer model at training time -- attention variant, layer count, hidden size, and more -- directly determine how fast and how cheaply it can run at inference time. (2026-08-11) - [What Is an AI Inference Server? Cloud API vs Self-Hosted vs On-Premise Explained](https://www.generalcompute.com/blog/what-is-an-ai-inference-server) — An AI inference server accepts prompts and returns model completions. Here's how they work, the main deployment options, and how to choose between managed APIs and self-hosted solutions. (2026-08-10) - [Why TileRT Cannot Catch SambaNova SN50: The Limits of Software-Emulated Dataflow on the Wrong Hardware](https://www.generalcompute.com/blog/why-tilert-cannot-catch-sambanova-sn50) — TileRT is a clever software approach to dataflow execution, but it runs on hardware that was never designed for reconfigurable dataflow. Here's why purpose-built silicon like the SambaNova SN50 will always hold the edge. (2026-08-10) - [AI Inference Costs in 2025: How to Calculate and Reduce Your LLM API Spend](https://www.generalcompute.com/blog/ai-inference-costs-2025-how-to-calculate-and-reduce-your-llm-api-spend) — A practical guide to understanding LLM API pricing, calculating your monthly inference bill with real formulas, and eight proven strategies to cut costs without sacrificing quality. (2026-08-09) - [Fast AI Inference: 8 Proven Techniques That Deliver Real-World Speedups](https://www.generalcompute.com/blog/fast-ai-inference) — A technical guide to the eight most effective LLM inference optimization techniques, with concrete speedup data, code examples, and stacking strategies for production deployments. (2026-08-08) - [What Is Tokens Per Second (TPS)? The LLM Throughput Metric That Actually Matters](https://www.generalcompute.com/blog/what-is-tokens-per-second-tps) — Tokens per second (TPS) measures how fast a language model generates output. Here's what it means, how it differs from TTFT, what drives it, and how to benchmark it in your own stack. (2026-08-07) - [What Is Time to First Token (TTFT)? The LLM Latency Metric That Defines UX](https://www.generalcompute.com/blog/what-is-time-to-first-token-ttft) — Time to First Token (TTFT) measures how long it takes an LLM to generate its first output token after receiving your request. It's the metric that determines whether your AI app feels instant or sluggish. (2026-08-06) - [LLM Inference Explained: How It Works, What It Costs, and How to Optimize It](https://www.generalcompute.com/blog/llm-inference-explained) — A complete technical guide to LLM inference: the prefill/decode pipeline, the metrics that matter, how providers price token generation, and five concrete ways to make it faster and cheaper. (2026-08-05) - [The 1B-and-Under Club: TinyLlama vs Phi-2 vs StableLM vs Qwen 0.5B](https://www.generalcompute.com/blog/the-1b-and-under-club-tinyllama-vs-phi-2-vs-stablelm-vs-qwen-0-5b) — A benchmark comparison of the smallest capable language models -- TinyLlama 1.1B, Phi-2 2.7B, StableLM 2 1.6B, and Qwen 2.5 0.5B -- covering reasoning, coding, inference speed, memory footprint, and which to use for edge and mobile deployments. (2026-08-03) - [Time-to-First-Token Leaderboard: Every Major Open Model Ranked](https://www.generalcompute.com/blog/time-to-first-token-leaderboard-every-major-open-model-ranked) — TTFT benchmarks across every major open-source model at multiple prompt lengths. We measured prefill latency on identical hardware so you can pick the right model for latency-sensitive workloads. (2026-08-02) - [Batch Inference Throughput: Which Models Maximize Tokens-per-Dollar?](https://www.generalcompute.com/blog/batch-inference-throughput-which-models-maximize-tokens-per-dollar) — A throughput-focused benchmark comparing open-source models across batch sizes. We measured tokens-per-second, cost-per-million-tokens, and effective tokens-per-dollar so you can pick the right model for high-volume workloads. (2026-08-01) - [Long-Context Benchmarks: Who's Actually Fast at 128K Tokens?](https://www.generalcompute.com/blog/long-context-benchmarks-128k-tokens) — We measured TTFT, decode speed, and effective throughput for major open-source models at 128K token context. Here's what the numbers show. (2026-07-31) - [MoE vs Dense at Equal Compute: Mixtral 8x7B vs Llama 70B](https://www.generalcompute.com/blog/moe-vs-dense-equal-compute-mixtral-8x7b-vs-llama-70b) — Mixtral 8x7B activates ~13B parameters per token while Llama 70B activates 70B. At equal active FLOPs, MoE architecture delivers dramatically better quality than dense. Here's why, with concrete benchmarks. (2026-07-30) - [Quantized vs Full Precision: GPTQ, AWQ, and FP8 Benchmarks Across Model Sizes](https://www.generalcompute.com/blog/quantized-vs-full-precision-gptq-awq-fp8-benchmarks-across-model-sizes) — A practical comparison of GPTQ, AWQ, and FP8 quantization against FP16 full precision across 7B, 13B, and 70B model sizes, covering throughput, latency, memory, and quality impact. (2026-07-29) - [Instruction-Tuned vs Base Models: Does Fine-Tuning Cost You Speed?](https://www.generalcompute.com/blog/instruction-tuned-vs-base-models-does-fine-tuning-cost-you-speed) — A practical look at whether instruction tuning and RLHF actually affect inference speed, what the real sources of performance differences are, and how to benchmark fairly across checkpoints. (2026-07-28) - [Vision-Language Models: Llama 4 Scout vs Qwen2-VL vs InternVL at Inference](https://www.generalcompute.com/blog/vision-language-models-llama-4-scout-vs-qwen-vl-vs-internvl-at-inference) — A practical comparison of Llama 4 Scout, Qwen2-VL, and InternVL2 on multimodal benchmarks and inference speed, with guidance on which model fits which deployment scenario. (2026-07-27) - [Embedding Model Benchmarks: Speed vs Quality for RAG Retrieval](https://www.generalcompute.com/blog/embedding-model-benchmarks-speed-vs-quality-for-rag-retrieval) — A benchmark comparison of embedding models on retrieval quality (BEIR NDCG@10) and inference latency, with practical guidance on where the quality tradeoffs actually matter for RAG pipelines. (2026-07-26) - [Math and Reasoning Models: DeepSeek-R1 vs Qwen QwQ vs Llama Reasoning](https://www.generalcompute.com/blog/math-and-reasoning-models-deepseek-r1-vs-qwen-qwq-vs-llama-reasoning) — A benchmark comparison of the leading open-source thinking/reasoning models -- DeepSeek-R1, QwQ-32B, and Llama-based reasoning distillations -- covering accuracy, token costs, and when to use each. (2026-07-25) - [Coding Model Benchmarks: Qwen 2.5 Coder vs DeepSeek Coder V2 vs CodeLlama vs StarCoder 2](https://www.generalcompute.com/blog/coding-model-benchmarks-qwen-2-5-coder-vs-deepseek-coder-v2-vs-codellama-vs-starcoder-2) — A technical benchmark comparison of the top open-source coding models -- Qwen 2.5 Coder, DeepSeek Coder V2, CodeLlama, and StarCoder 2 -- across HumanEval, MBPP, and real-world code tasks by size bucket. (2026-07-24) - [Frontier Open Models: Llama 4 Maverick vs DeepSeek V3 vs Qwen 2.5 110B](https://www.generalcompute.com/blog/frontier-open-models-llama-4-maverick-vs-deepseek-v3-vs-qwen-2-5-110b) — A technical comparison of the top frontier open-source models -- Llama 4 Maverick, DeepSeek V3, and Qwen 2.5 110B -- covering architecture, benchmarks, and inference trade-offs. (2026-07-23) - [Mid-Size Models: Llama 3.1 70B vs Qwen 2.5 72B vs Mistral Large vs DeepSeek-V2 Lite](https://www.generalcompute.com/blog/mid-size-models-llama-3-1-70b-vs-qwen-2-5-72b-vs-mistral-large-vs-deepseek-v2-lite) — A benchmark comparison of the 70B-class open models in 2025 -- covering reasoning, math, coding, context length, and inference speed across Llama 3.1 70B, Qwen 2.5 72B, Mistral Large 2, and DeepSeek-V2 Lite. (2026-07-22) - [The 7-8B Battle: Llama 3.1 8B vs Mistral 7B v0.3 vs Qwen 2.5 7B vs Gemma 2 9B](https://www.generalcompute.com/blog/the-7-8b-battle-llama-3-1-8b-vs-mistral-7b-vs-qwen-2-5-7b-vs-gemma-2-9b) — A benchmark comparison of the four most popular 7-9B parameter models in 2025 -- covering reasoning, math, coding, context length, and inference speed to help you pick the right model for your application. (2026-07-21) - [Small Models Showdown: Qwen 2.5 3B vs Llama 3.2 3B vs Phi-3.5 Mini vs Gemma 2 2B](https://www.generalcompute.com/blog/small-models-showdown-qwen-2-5-3b-llama-3-2-3b-phi-3-5-mini-gemma-2-2b) — A head-to-head benchmark comparison of the best sub-4B parameter models in 2025 -- covering reasoning, coding, math, context length, and inference speed to help you pick the right small model for your use case. (2026-07-20) - [Agentic RAG: When Retrieval and Generation Both Need to Be Fast](https://www.generalcompute.com/blog/agentic-rag-when-retrieval-and-generation-both-need-to-be-fast) — In agentic RAG, retrieve-then-generate runs in loops, not once. That changes everything about how latency compounds and where you need to optimize. (2026-07-19) - [Long-Running Agents: Managing Inference Costs Over Hours of Execution](https://www.generalcompute.com/blog/long-running-agents-managing-inference-costs-over-hours-of-execution) — When an agent runs for hours instead of seconds, inference costs and context management become the dominant engineering problems. Here's how to keep them under control. (2026-07-18) - [Evaluating Agent Performance: Latency as a First-Class Metric](https://www.generalcompute.com/blog/evaluating-agent-performance-latency-as-a-first-class-metric) — Most agent benchmarks measure accuracy and task completion. Here's why latency belongs in your evaluation framework alongside accuracy, and how to instrument your agent to collect the data you need. (2026-07-17) - [Browser Agents: Inference Speed as the Gating Factor](https://www.generalcompute.com/blog/browser-agents-inference-speed-as-the-gating-factor) — Browser agents take dozens of LLM calls to complete a single web task. Here's why inference latency determines whether they're usable in practice. (2026-07-16) - [Voice Agents and the 500ms Window: An Inference Architecture Guide](https://www.generalcompute.com/blog/voice-agents-and-the-500ms-window) — Real-time voice conversations require end-to-end latency under 500ms. This guide breaks down the inference architecture, latency budgets, and streaming patterns needed to hit that target reliably. (2026-07-15) - [Planning and Search with LLMs: Tree-of-Thought at Inference Scale](https://www.generalcompute.com/blog/planning-and-search-with-llms-tree-of-thought-at-inference-scale) — Tree-of-Thought turns a single LLM call into a branching search process. Here's how it works, why inference speed determines how deep you can search, and how to build ToT agents that run fast enough to be useful. (2026-07-14) - [The Cost of Agent Retries: When Slow Inference Kills Reliability](https://www.generalcompute.com/blog/the-cost-of-agent-retries-when-slow-inference-kills-reliability) — Retries are how agents recover from failures, but when inference is slow, each retry multiplies latency. Here's how to think about retry budgets and why faster inference makes your agent more resilient. (2026-07-13) - [Whisper API for Real-Time Transcription: Alternatives to OpenAI Whisper](https://www.generalcompute.com/blog/whisper-api-for-real-time-transcription-alternatives-to-openai-whisper) — The OpenAI Whisper API works well for batch transcription but is not designed for real-time streaming. Here is how Faster-Whisper, Deepgram, and AssemblyAI compare on latency, accuracy, and pricing for live transcription use cases. (2026-07-12) - [Fireworks AI Alternative: Faster Inference at Lower Cost](https://www.generalcompute.com/blog/fireworks-ai-alternative) — Fireworks AI is a capable inference platform, but it has real limitations around latency, pricing at scale, and rate limits. Here's a direct comparison with alternatives and a step-by-step migration guide. (2026-07-11) - [DeepSeek API: How to Access DeepSeek V3 Without Rate Limits](https://www.generalcompute.com/blog/deepseek-api-how-to-access-deepseek-v3-without-rate-limits) — DeepSeek V3 is one of the strongest open-source models available, but the official API has aggressive rate limits. Here's how to access it reliably at scale, with Python and Node.js integration examples. (2026-07-10) - [Inference API Pricing Guide: Groq vs Fireworks vs Together vs GeneralCompute](https://www.generalcompute.com/blog/inference-api-pricing-guide-groq-vs-fireworks-vs-together-vs-generalcompute) — A practical breakdown of inference API pricing across Groq, Fireworks AI, Together AI, and GeneralCompute -- including per-token rates, hidden costs, and how to calculate your real monthly spend. (2026-07-09) - [vLLM vs Managed Inference APIs: When to Self-Host vs Pay-as-You-Go](https://www.generalcompute.com/blog/vllm-vs-managed-inference-apis-when-to-self-host-vs-pay-as-you-go) — A practical TCO comparison of self-hosting LLMs with vLLM versus using managed inference APIs. Includes break-even analysis and a decision framework for your team. (2026-07-08) - [Llama API: How to Run Llama 4 via API Without Managing Infrastructure](https://www.generalcompute.com/blog/llama-api-how-to-run-llama-4-via-api) — A practical guide to running Llama 4 through managed inference APIs. Compare providers, understand pricing and performance trade-offs, and see working integration code. (2026-07-07) - [Best OpenAI API Alternatives in 2025: Full Developer Comparison](https://www.generalcompute.com/blog/best-openai-api-alternatives-in-2025-full-developer-comparison) — A developer's guide to the top OpenAI API alternatives in 2025, covering GeneralCompute, Groq, Together AI, Fireworks AI, and Replicate with pricing, model selection, and migration code. (2026-07-06) - [GeneralCompute vs Groq: Speed, Pricing, and Model Selection Compared](https://www.generalcompute.com/blog/generalcompute-vs-groq) — A developer's comparison of GeneralCompute and Groq covering inference speed, model availability, API pricing, and when to choose each platform. (2026-07-05) - [Scaling Laws in LLMs: What They Mean for Inference Cost in Production](https://www.generalcompute.com/blog/scaling-laws-llms-inference-cost-production) — A practical breakdown of Chinchilla and Kaplan scaling laws, what they predict about model quality and size, and how to use that knowledge to make smarter inference cost decisions. (2026-07-04) - [Mixture of Experts (MoE) Models: Why They're Dominating 2025](https://www.generalcompute.com/blog/mixture-of-experts-moe-models-why-theyre-dominating-2025) — MoE models activate only a fraction of their parameters per token, giving you large-model quality at a fraction of the compute cost. Here's how the architecture works, why it won, and what the load balancing challenges actually look like. (2026-07-03) - [KV Cache in LLM Inference: How It Works and Why It Matters](https://www.generalcompute.com/blog/kv-cache-in-llm-inference-how-it-works-and-why-it-matters) — The KV cache is one of the most important mechanisms in transformer inference. This post explains what it is, how prefill and decode phases use it, how paged attention manages it, and what MQA/GQA and quantization do to shrink it. (2026-07-02) - [Agentic AI in Production: Inference Requirements for Multi-Step Workflows](https://www.generalcompute.com/blog/agentic-ai-in-production-inference-requirements) — Building a single agentic AI demo is straightforward. Scaling it to production -- with observability, error handling, rate limits, and hundreds of concurrent agents -- requires thinking carefully about your inference layer from the start. (2026-07-01) - [How to Build a Real-Time Coding Assistant With Open-Source Models](https://www.generalcompute.com/blog/how-to-build-a-real-time-coding-assistant-with-open-source-models) — A practical guide to building a coding assistant LLM using Qwen3-Coder or DeepSeek Coder, covering model selection, codebase indexing, and VS Code integration. (2026-06-30) - [Multi-Turn Conversations in LLM APIs: Best Practices for Agents](https://www.generalcompute.com/blog/multi-turn-conversations-llm-apis-best-practices-agents) — How to manage conversation history in LLM APIs without blowing up your context window or your bill. Covers sliding window, summarization, memory patterns, and cost optimization for production agents. (2026-06-29) - [Your AI Agent Is Only as Good as Its Inference Speed](https://www.generalcompute.com/blog/your-ai-agent-is-only-as-good-as-its-inference-speed) — Agent latency compounds across every sequential step. This post covers the multiplier effect, how model routing can cut costs without sacrificing quality, and why parallelizing calls is one of the highest-leverage improvements you can make to an agentic system. (2026-06-28) - [What Are Agentic AI Systems? How to Build Them With Fast Inference](https://www.generalcompute.com/blog/what-are-agentic-ai-systems) — Agentic AI systems chain LLM calls into autonomous loops that plan, act, and observe. This guide covers the core components, the main reasoning patterns (ReAct, Plan-and-Execute, Tree of Thoughts), and how inference speed shapes what you can actually build. (2026-06-27) - [SOC 2 Compliant AI Inference: What Enterprise Teams Need to Know](https://www.generalcompute.com/blog/soc-2-compliant-ai-inference-what-enterprise-teams-need-to-know) — A practical guide to SOC 2 compliance for AI inference: Type I vs. Type II, data residency, audit logging, and a checklist for evaluating inference providers. (2026-06-26) - [Embedding Models in Production: Choosing the Right One for Your App](https://www.generalcompute.com/blog/embedding-models-in-production) — A practical guide to picking an embedding model for production: MTEB benchmarks, head-to-head comparisons of BGE, Nomic, E5, Cohere, and OpenAI models, multilingual considerations, and code to get started. (2026-06-25) - [How to Migrate From OpenAI to GeneralCompute in 10 Lines of Code](https://www.generalcompute.com/blog/how-to-migrate-from-openai-to-generalcompute-in-10-lines-of-code) — GeneralCompute's API is fully OpenAI-compatible. Here's exactly what to change in Python, Node.js, and LangChain, plus a validation checklist to make sure nothing breaks. (2026-06-24) - [Using Vercel AI SDK With GeneralCompute: Full Integration Guide](https://www.generalcompute.com/blog/using-vercel-ai-sdk-with-generalcompute) — A complete guide to wiring the Vercel AI SDK's useChat and useCompletion hooks to GeneralCompute's OpenAI-compatible API, with streaming in Next.js App Router and edge runtime deployment. (2026-06-23) - [How to Build a RAG Pipeline Using Open-Source Models](https://www.generalcompute.com/blog/how-to-build-a-rag-pipeline-open-source-models) — A complete walkthrough of building a retrieval-augmented generation pipeline with open-source embedding and LLM models: ingestion, chunking, vector storage, retrieval, generation, and evaluation with RAGAS. (2026-06-22) - [JSON Mode in LLMs: How to Get Structured Outputs Every Time](https://www.generalcompute.com/blog/json-mode-llms-structured-outputs) — A practical guide to JSON mode, structured output schemas, and Pydantic + instructor for getting reliable JSON from LLMs -- covering every major approach and when to use each one. (2026-06-21) - [OpenAI-Compatible APIs: How to Migrate Your App in 5 Minutes](https://www.generalcompute.com/blog/openai-compatible-api-migrate-your-app) — A practical guide to migrating your OpenAI-powered app to a compatible API provider like GeneralCompute -- covering the base URL swap, supported endpoints, and integration code for LangChain, LlamaIndex, and Vercel AI SDK. (2026-06-20) - [Tool Calling With Open-Source LLMs: A Complete Guide](https://www.generalcompute.com/blog/tool-calling-with-open-source-llms) — How function calling works in LLM APIs, which open-source models support it, and practical Python and Node.js examples for building tool-using applications. (2026-06-19) - [How to Build a Streaming Chat App With GeneralCompute + Node.js](https://www.generalcompute.com/blog/how-to-build-a-streaming-chat-app-with-generalcompute-nodejs) — A practical guide to building a streaming chat application using the OpenAI SDK for Node.js pointed at GeneralCompute's API, including SSE architecture, context management, and a full working example. (2026-06-18) - [GPU Cluster for LLM Inference: Build vs Buy Analysis for ML Teams](https://www.generalcompute.com/blog/gpu-cluster-llm-inference-build-vs-buy) — A rigorous TCO breakdown comparing building your own GPU cluster against managed inference APIs, with break-even analysis and hybrid approaches for ML teams at different scales. (2026-06-17) - [Real-Time AI Inference: How to Achieve <100ms Latency in Production](https://www.generalcompute.com/blog/real-time-ai-inference-100ms-latency-production) — A practical guide to the full optimization stack for sub-100ms AI inference: model selection, quantization, infrastructure placement, and what to measure. (2026-06-16) - [Fine-Tuning vs RAG: Which Approach Is Right for Your Production App?](https://www.generalcompute.com/blog/fine-tuning-vs-rag) — A practical decision framework for choosing between fine-tuning and retrieval-augmented generation, including cost comparison and when a hybrid approach makes sense. (2026-06-15) - [H100 vs H200 vs B200: Which GPU Is Best for LLM Inference?](https://www.generalcompute.com/blog/h100-vs-h200-vs-b200-which-gpu-is-best-for-llm-inference) — A technical comparison of NVIDIA's H100, H200, and B200 GPUs for LLM inference workloads: specs, tokens/s benchmarks, cost per token, and multi-GPU scaling considerations. (2026-06-14) - [LLM Token Generation Speed: How Providers Compare in 2025](https://www.generalcompute.com/blog/llm-token-generation-speed-how-providers-compare-in-2025) — A practical methodology for benchmarking LLM throughput across providers, a comparison of token generation speeds at different concurrency levels, and a guide to the throughput vs latency trade-off. (2026-06-13) - [AI Inference Latency Explained: TTFT, TPS, and How to Optimize Them](https://www.generalcompute.com/blog/ai-inference-latency-explained-ttft-tps-and-how-to-optimize-them) — What time to first token and tokens per second actually measure, how to measure them correctly, and a layer-by-layer guide to reducing AI inference latency in production. (2026-06-12) - [GeneralCompute vs vLLM: Throughput, Latency, and Cost Benchmarks](https://www.generalcompute.com/blog/generalcompute-vs-vllm-throughput-latency-and-cost-benchmarks) — A head-to-head comparison of vLLM self-hosted on H100s versus GeneralCompute's managed inference API: full methodology, throughput and latency numbers, and a total cost of operations breakdown. (2026-06-11) - [Open-Source LLM Landscape 2025: Top Models Compared](https://www.generalcompute.com/blog/open-source-llm-landscape-2025-top-models-compared) — A practical map of the open-source LLM ecosystem in 2025: the leading model families, how they stack up by size and task, what the licenses actually let you do, and how to pick one for production. (2026-06-10) - [Faster-Whisper: Real-Time Speech-to-Text on GeneralCompute](https://www.generalcompute.com/blog/faster-whisper-real-time-speech-to-text-on-generalcompute) — Faster-Whisper reimplements OpenAI's Whisper on CTranslate2 with INT8 inference, running several times faster at the same accuracy. Here is how it works, how streaming differs from batch transcription, and how it fits into a real-time STT to LLM to TTS voice pipeline. (2026-06-09) - [QwQ-32B: The Reasoning Model That Rivals o1 — Complete Guide](https://www.generalcompute.com/blog/qwq-32b-the-reasoning-model-that-rivals-o1-complete-guide) — QwQ-32B is a 32-billion-parameter open-weight reasoning model from the Qwen team that competes with much larger reasoning models. Here is how it works, how it compares to o1, o1-mini, and DeepSeek R1, and what its long reasoning traces mean when you serve it in production. (2026-06-08) - [How to Fine-Tune Llama 4: Step-by-Step Guide with Code](https://www.generalcompute.com/blog/how-to-fine-tune-llama-4-step-by-step-guide-with-code) — A practical walkthrough for fine-tuning Llama 4: when to do it, how to prepare data, and working LoRA, QLoRA, and full fine-tune code, plus evaluation and deployment. (2026-06-07) - [Qwen3-Coder: The Best Open-Source Coding Model? Benchmark + Guide](https://www.generalcompute.com/blog/qwen3-coder-the-best-open-source-coding-model) — A close look at Qwen3-Coder: how it scores on HumanEval, MBPP, and SWE-bench, how it compares to Code Llama and DeepSeek Coder, and how to wire it into your editor and agents. (2026-06-06) - [Llama 4 on GeneralCompute: Getting Started Guide](https://www.generalcompute.com/blog/llama-4-on-generalcompute-getting-started-guide) — A practical guide to running Llama 4 on GeneralCompute: the model variants, what hardware they need, how to make your first API call, and how to tune requests for speed and cost. (2026-06-05) - [DeepSeek R1: What It Is, How It Works, and Why It Matters](https://www.generalcompute.com/blog/deepseek-r1-what-it-is-how-it-works-and-why-it-matters) — DeepSeek R1 is an open-weight reasoning model trained mostly through reinforcement learning. Here is how its architecture and training work, how it compares to GPT-4 class models, Claude, and Llama, and what its reasoning style means for inference. (2026-06-04) - [Flash Attention: Why Modern LLMs Run Faster With It](https://www.generalcompute.com/blog/flash-attention-why-modern-llms-run-faster) — Flash Attention rewrites the attention computation to avoid moving a giant intermediate matrix in and out of GPU memory. Here is how the tiling and kernel fusion work, how v1, v2, and v3 evolved, and how to turn it on in PyTorch. (2026-06-03) - [What Is Direct Preference Optimization (DPO)? Explained Simply](https://www.generalcompute.com/blog/what-is-direct-preference-optimization-dpo) — DPO aligns language models to human preferences without a separate reward model or reinforcement learning. Here is how it works, how it compares to RLHF, and when to reach for IPO, KTO, or ORPO instead. (2026-06-02) - [Quantization Explained: INT4, GGUF, GPTQ and What They Mean for Your Model](https://www.generalcompute.com/blog/quantization-explained-int4-gguf-gptq) — A practical guide to LLM quantization: what INT4, GGUF, and GPTQ actually do, how much quality you lose, and how to quantize a model yourself with llama.cpp and AutoGPTQ. (2026-06-01) - [What Is Speculative Decoding? How It Makes LLMs 3x Faster](https://www.generalcompute.com/blog/what-is-speculative-decoding-how-it-makes-llms-3x-faster) — A clear explanation of speculative decoding: how a small draft model proposes tokens that a large model verifies in parallel, why it preserves output quality, the Medusa and Eagle variants, and the real speedups you can expect in production. (2026-05-31) - [LPU vs GPU vs CPU: Which Processor Wins for AI Inference?](https://www.generalcompute.com/blog/lpu-vs-gpu-vs-cpu-which-processor-wins-for-ai-inference) — A head-to-head look at LPUs, GPUs, and CPUs for AI inference, with the architecture reasons behind their performance, real-world latency and throughput trade-offs, cost-effectiveness, and which one fits each kind of workload. (2026-05-30) - [What Is AI Inference? A Developer's Complete Guide](https://www.generalcompute.com/blog/what-is-ai-inference-a-developers-complete-guide) — A practical, end-to-end explanation of AI inference: what it is, how the pipeline works, the metrics that matter, the hardware that runs it, and the trade-offs you face when you put a model in production. (2026-05-29) - [Streaming for Agents: Why Partial Results Change the UX](https://www.generalcompute.com/blog/streaming-for-agents-why-partial-results-change-the-ux) — Streaming in agentic pipelines is not the same as streaming chat tokens. Partial tool calls, pipelined steps, and early cancellation change what the user experiences. (2026-05-16) - [Parallel Tool Execution: How Fast Inference Enables Concurrent Agent Actions](https://www.generalcompute.com/blog/parallel-tool-execution-how-fast-inference-enables-concurrent-agent-actions) — Why running multiple tool calls in parallel changes the latency math of an agent, and how inference speed determines whether the parallelism is worth doing. (2026-05-15) - [Agent Memory Systems: Balancing Context Length vs Retrieval Latency](https://www.generalcompute.com/blog/agent-memory-systems-balancing-context-length-vs-retrieval-latency) — How agents reconstruct memory between turns, and the latency trade-offs between long context, RAG, summarization, and KV cache reuse. (2026-05-12) - [Building a Code Agent: Why Each Step Needs Sub-Second Inference](https://www.generalcompute.com/blog/building-a-code-agent-why-each-step-needs-sub-second-inference) — A practical breakdown of the latency budget inside a code agent, step by step, and why every link in the chain needs to land under a second to keep the loop usable. (2026-05-11) - [ReAct, Reflexion, and Chain-of-Thought: The Inference Cost of Reasoning Patterns](https://www.generalcompute.com/blog/react-reflexion-and-chain-of-thought-the-inference-cost-of-reasoning-patterns) — Popular agent reasoning patterns are described as prompt techniques, but they are inference cost multipliers. Here is how ReAct, Reflexion, and Chain-of-Thought actually shape the bill and the latency. (2026-05-10) - [Multi-Agent Architectures and the Inference Cost Explosion](https://www.generalcompute.com/blog/multi-agent-architectures-and-the-inference-cost-explosion) — Orchestrator and worker patterns make multi-agent systems easy to design and expensive to run. Here is where the inference cost actually goes, and what it means for the infrastructure underneath. (2026-05-09) - [Tool Calling Latency: The Bottleneck No One Talks About](https://www.generalcompute.com/blog/tool-calling-latency-the-bottleneck-no-one-talks-about) — Function calling looks simple on paper, but the latency budget of a tool-using LLM is dominated by short structured generations that most serving stacks are not optimized for. This is what actually makes tool calls feel slow. (2026-05-08) - [The Agentic Inference Tax: Why Agents Need 10x Faster Models](https://www.generalcompute.com/blog/the-agentic-inference-tax) — Agents make many sequential LLM calls per task, and each one pays the full latency of decoding. This post walks through how that compounds and why fast inference changes which agents are even viable. (2026-05-07) - [Compiler-Level Optimizations for Inference: TorchInductor, Triton, XLA](https://www.generalcompute.com/blog/compiler-level-optimizations-for-inference) — How modern ML compilers turn Python model code into fused, fast kernels. A practical look at TorchInductor, Triton, and XLA, and the tradeoffs each one makes for inference. (2026-05-06) - [Draft Model Selection for Speculative Decoding](https://www.generalcompute.com/blog/draft-model-selection-for-speculative-decoding) — Picking a draft model is the most consequential decision when deploying speculative decoding. A practical guide to acceptance rates, sizing, and the tradeoffs that decide whether you actually get a speedup. (2026-05-05) - [The Attention Sink Phenomenon: Why the First Token Matters](https://www.generalcompute.com/blog/the-attention-sink-phenomenon-why-the-first-token-matters) — How attention concentrates on the first few tokens of every sequence, why naive sliding-window caching breaks long-context generation, and how StreamingLLM uses sink tokens to serve effectively unbounded streams. (2026-05-04) - [Mixture of Experts at Inference Time](https://www.generalcompute.com/blog/mixture-of-experts-at-inference-time) — How MoE routing actually works during serving, why sparse activation makes large models cheaper to run per token, and what changes for the inference stack. (2026-05-03) - [Tensor Parallelism vs Pipeline Parallelism for Model Serving](https://www.generalcompute.com/blog/tensor-parallelism-vs-pipeline-parallelism-for-model-serving) — How tensor and pipeline parallelism actually differ in production inference, when to use each, and why most serving stacks end up combining them. (2026-05-02) - [Prefix Caching: Why Repeated Prompts Shouldn't Cost You Twice](https://www.generalcompute.com/blog/prefix-caching-why-repeated-prompts-shouldnt-cost-you-twice) — How prefix caching works in modern LLM serving stacks, why it changes the economics of long system prompts and RAG, and what to watch out for in production. (2026-05-01) - [Distillation for Inference: How Smaller Models Learn From Larger Ones](https://www.generalcompute.com/blog/distillation-for-inference-how-smaller-models-learn-from-larger-ones) — A practical guide to knowledge distillation for production inference: what actually works, what to skip, and how to ship a smaller model without losing the behavior you cared about. (2026-04-30) - [FP8 Training and Inference: The Precision Sweet Spot](https://www.generalcompute.com/blog/fp8-training-and-inference-the-precision-sweet-spot) — Why 8-bit floating point hits a different point on the accuracy/throughput curve than INT8, how E4M3 and E5M2 are used in practice, and what FP8 actually buys you in production serving. (2026-04-29) - [Activation-Aware Quantization (AWQ) Deep Dive](https://www.generalcompute.com/blog/activation-aware-quantization-awq-deep-dive) — A close look at how AWQ picks salient weight channels, applies per-channel scaling, and why it consistently beats round-to-nearest 4-bit quantization for LLM inference. (2026-04-28) - [Mamba and State Space Models: Inference Without Attention](https://www.generalcompute.com/blog/mamba-and-state-space-models-inference-without-attention) — How structured state space models like Mamba achieve constant-time per-token inference, and why the selective scan changes the trade-off space for long-context serving. (2026-04-27) - [RWKV and Linear Attention: Recurrent Models as an Inference Shortcut](https://www.generalcompute.com/blog/rwkv-and-linear-attention-recurrent-models-as-an-inference-shortcut) — How RWKV and linear attention architectures collapse the per-token cost of generation to O(1), and what that means for serving long-context workloads. (2026-04-26) - [Dynamic Batching Strategies: From Naive to Continuous to Iteration-Level](https://www.generalcompute.com/blog/dynamic-batching-strategies-from-naive-to-continuous-to-iteration-level) — Batching is the lever that turns idle GPU silicon into served tokens. This post walks through the evolution of batching for LLM serving, from one-at-a-time to static batches to request-level dynamic batching to iteration-level continuous batching, and shows where each strategy still leaves throughput on the floor. (2026-04-25) - [Token Merging and Token Pruning for Faster Transformers](https://www.generalcompute.com/blog/token-merging-and-token-pruning-for-faster-transformers) — Attention cost grows with the square of sequence length. Token merging and token pruning shrink that sequence mid-network, trading a little accuracy for real speedups. Here is how ToMe works, how the idea extends to language models, and where it breaks down. (2026-04-24) - [S3: Scheduling for Straggler Mitigation in LLM Serving](https://www.generalcompute.com/blog/s3-scheduling-for-straggler-mitigation-in-llm-serving) — In LLM serving, a single long-running request can stall everyone else sharing the same batch. S3 attacks that by predicting output length and scheduling around it. Here is what stragglers actually cost you, and how output-length-aware scheduling helps. (2026-04-23) - [Chunked Prefill: Overlapping Compute and Communication](https://www.generalcompute.com/blog/chunked-prefill-overlapping-compute-and-communication) — Prefill pins the compute units while decode starves for memory bandwidth. Sarathi-Serve splits prefill into chunks and piggybacks decodes on them, keeping both resources busy in the same batch. Here is how it works and where the limits are. (2026-04-22) - [Cascade Inference: Using Small Models to Route to Big Ones](https://www.generalcompute.com/blog/cascade-inference-using-small-models-to-route-to-big-ones) — FrugalGPT and its descendants show that most queries do not need the biggest model. We walk through the cascade pattern, routing classifiers, and the engineering trade-offs of sending easy work to cheap models and escalating only when needed. (2026-04-21) - [Lookahead Decoding: Parallel Token Generation Without Draft Models](https://www.generalcompute.com/blog/lookahead-decoding-parallel-token-generation-without-draft-models) — Lookahead decoding from LMSYS speeds up autoregressive generation without requiring a draft model. We walk through the Jacobi iteration trick, the n-gram pool, and what the speedups actually look like in practice. (2026-04-20) - [Disaggregated Prefill and Decode (Splitwise / DistServe)](https://www.generalcompute.com/blog/disaggregated-prefill-and-decode) — Prefill and decode have different compute profiles and clash when they share a GPU. Splitwise and DistServe separate them onto different hardware pools. We walk through why, how, and when it actually pays off. (2026-04-19) - [KV Cache Compression: MLA and Beyond](https://www.generalcompute.com/blog/kv-cache-compression-mla-and-beyond) — DeepSeek's Multi-Head Latent Attention cuts the KV cache by an order of magnitude without giving up quality. We walk through MLA, how it compares to MQA and GQA, and the other compression techniques worth knowing. (2026-04-18) - [Ring Attention: Scaling Context to Millions of Tokens](https://www.generalcompute.com/blog/ring-attention-scaling-context-to-millions-of-tokens) — Ring Attention distributes the attention computation across devices in a ring topology, overlapping KV transfer with compute so context length scales linearly with the number of GPUs. (2026-04-17) - [Quantization for Inference: GPTQ, AWQ, SmoothQuant, and FP8](https://www.generalcompute.com/blog/quantization-for-inference-gptq-awq-smoothquant-fp8) — Quantization shrinks model weights from 16-bit to 4-bit or 8-bit, cutting memory usage and speeding up inference. Here's how the major techniques work and when to use each one. (2026-03-26) - [Multi-Query and Grouped-Query Attention: Shrinking the KV Cache](https://www.generalcompute.com/blog/multi-query-grouped-query-attention) — MQA and GQA reduce the memory footprint of attention by sharing key-value heads across queries. A simple architectural change that makes inference dramatically faster. (2026-03-25) - [Continuous Batching: The Orca Paper That Changed LLM Serving](https://www.generalcompute.com/blog/continuous-batching-the-orca-paper) — Before continuous batching, LLM servers wasted GPU cycles waiting for the slowest request in each batch. Orca's iteration-level scheduling fixed this with a 36x throughput improvement. (2026-03-24) - [Medusa, EAGLE, and Sequoia: The Next Generation of Speculative Decoding](https://www.generalcompute.com/blog/medusa-eagle-sequoia-next-gen-speculative-decoding) — The original speculative decoding papers needed a separate draft model. Medusa, EAGLE, and Sequoia found ways to speculate faster, smarter, and without the extra model. (2026-03-24) - [SGLang and RadixAttention: Smarter KV Cache Reuse](https://www.generalcompute.com/blog/sglang-and-radix-attention) — SGLang's RadixAttention stores KV cache in a radix tree, enabling automatic prefix sharing across requests. The result is up to 5x higher throughput for multi-turn and structured workloads. (2026-03-24) - [Speculative Decoding: Getting 3x Speedups Without Changing the Model](https://www.generalcompute.com/blog/speculative-decoding-3x-speedups-without-changing-the-model) — Speculative decoding uses a small draft model to predict multiple tokens ahead, then verifies them all at once. The result is mathematically identical output, 2-3x faster. (2026-03-23) - [PagedAttention and vLLM: Virtual Memory for LLM Serving](https://www.generalcompute.com/blog/paged-attention-and-vllm) — The PagedAttention paper solved the biggest memory waste problem in LLM serving by borrowing an idea from operating systems. Here's how it works and why vLLM became the default serving framework. (2026-03-22) - [FlashAttention: How Tri Dao Made Attention 4x Faster](https://www.generalcompute.com/blog/flash-attention-how-tri-dao-made-attention-4x-faster) — FlashAttention rewrote the rules of transformer inference by treating attention as a memory problem, not a compute problem. Here's how it works and why it matters. (2026-03-21) - [Build a Real-Time Voice AI Agent with General Compute](https://www.generalcompute.com/blog/build-a-real-time-voice-ai-agent) — A step-by-step tutorial for building a voice AI agent with sub-500ms response times. Plus: why General Compute is the only provider fast enough to use reasoning models in a voice pipeline. (2026-03-20) - [How Coding Agents Depend on Inference Speed](https://www.generalcompute.com/blog/how-coding-agents-depend-on-inference-speed) — Coding agents make dozens of sequential LLM calls per task. Every millisecond of inference latency compounds across each step, making speed the single biggest infrastructure bottleneck for AI-powered developer tools. (2026-03-19) - [Why Inference Speed is the New Moat](https://www.generalcompute.com/blog/why-inference-speed-is-the-new-moat) — Model quality has commoditized. The real competitive advantage in AI is how fast your infrastructure can deliver results. Inference speed is becoming the defining moat for AI-native products. (2026-03-18) --- # Announcements Company updates from General Compute. ## General Compute raises $15M to build the world's fastest neocloud - Published: May 28, 2026 - Category: Company - Tags: Funding - Coverage: https://techcrunch.com/2026/05/28/has-the-hunt-for-ai-compute-uncovered-the-next-cerebras/ We raised $15M to build the world's fastest neocloud, powered by ASIC chips purpose-built for AI inference. --- # Terms of service — agent summary - Last updated: March 31, 2026 - Legal contact: legal@generalcompute.ai ## Key clauses - Usage subject to platform TOS; scraping permitted for evaluation. - API keys must not be shared or stored client-side. - General Compute may rate-limit or terminate access for abuse. Full terms: https://www.generalcompute.com/terms --- # Privacy — agent summary - Data contact: privacy@generalcompute.ai - Scope: website and inference API ## Highlights - Data types collected: account metadata, API usage metrics, support correspondence. - No selling of personal data; processors limited to essential services. - Request data deletion or export via privacy@generalcompute.ai. Full policy: https://www.generalcompute.com/privacy --- # How Transformer Architecture Determines Inference Speed and Memory Usage > A deep look at how the architectural decisions baked into a transformer model at training time -- attention variant, layer count, hidden size, and more -- directly determine how fast and how cheaply it can run at inference time. - Author: General Compute - Published: 2026-08-11 - Tags: transformers, inference, architecture, memory, performance, attention - Canonical: https://www.generalcompute.com/blog/how-transformer-architecture-determines-inference-speed-and-memory-usage --- When you pick a model for production, you're not just choosing a quality level -- you're choosing a set of hardware constraints. The architecture decisions made when a model was designed and trained determine how much memory it occupies, how many FLOPs each token costs, and where your serving bottleneck will be. Those decisions are frozen into the weights. You can change some things at inference time (quantization, batching, hardware selection), but the fundamental structure is fixed. This post works through the pieces of a transformer that matter most for inference: what each component does, how much it costs, and what the formulas look like with real numbers. ## Why Understanding Transformer Architecture Matters for Inference ### How Architecture Choices Made at Training Time Constrain Inference Every hyperparameter in a model's design has a corresponding inference cost. Hidden dimension determines how much memory bandwidth each layer needs. Number of layers determines how many sequential kernel launches happen per token. KV head count determines how much memory the cache consumes per request. Sequence length at training time (not at inference time) determines nothing -- but the context window the model was trained to handle sets the quadratic attention cost you'll face at long inputs. None of this is adjustable without retraining. You can quantize weights to reduce memory footprint, but you can't change how many layers a model has or how its attention heads are organized. Architecture selection is a permanent inference decision. ### The Three Inference Bottlenecks Most inference workloads are constrained by one of three things: **Compute** (FLOPs) is the raw number of multiply-accumulate operations needed per token. This scales with model size, primarily driven by the feed-forward network (FFN) layers. Dense models are more compute-bound than sparse MoE models at equivalent quality. **Memory bandwidth** is how fast the GPU can read weights and KV cache from HBM into compute units. The decode phase of generation is almost always memory-bandwidth-bound -- you do very little compute per byte of data read, so GPU utilization is low regardless of how fast the compute cores are. **Latency** is a function of both of the above, plus kernel launch overhead, network round-trips in distributed setups, and scheduler latency in the serving framework. Reducing latency often means trading throughput. Understanding which bottleneck applies to your workload tells you which optimization levers are actually worth pulling. ## The Transformer Block: What Actually Runs During Inference A standard transformer layer has a few distinct components that run sequentially per token. ### Layer Normalization Layer norm normalizes the hidden state vector before attention and FFN. It's computationally cheap -- linear in hidden dimension -- but it adds a read-modify-write cycle over the full hidden state at every layer. In a 128-layer model running at high concurrency, this adds up. ### Multi-Head Attention: The Memory Bandwidth Bottleneck Attention is where things get expensive, particularly at the memory level. During decode, each step requires: 1. Computing Q, K, V projections for the current token (three matrix-vector products of size `d_model x d_model`) 2. Appending the new K and V to the cache 3. Loading the entire KV cache for this request from HBM 4. Computing attention scores and the weighted sum over V Step 3 is the bottleneck. The KV cache grows linearly with context length, so every additional cached token increases how much memory bandwidth the decode step consumes. At long contexts, a GPU can spend more time moving KV data from memory than doing any actual computation. ### The Feed-Forward Network: Where Most FLOPs Go The FFN in each transformer block typically has an intermediate dimension of 4x the hidden size. For a model with `d_model = 4096`, the FFN has two weight matrices of shape `4096 x 16384`. Across a forward pass, the FFN accounts for roughly two-thirds of total FLOPs in a dense transformer. During decode with a batch size of 1, these large matrix-vector multiplications (not matrix-matrix) are also memory-bandwidth-bound. You load the entire weight matrix to produce a single output vector. Throughput improves substantially with larger batches because you shift from matrix-vector to matrix-matrix operations, and GPU utilization recovers. ### Residual Connections Residual connections (the "skip" connections that add the layer input to its output) add two element-wise additions per block. Their computational cost is negligible compared to attention and FFN. ## Attention Mechanisms and Their Inference Costs The variant of attention a model uses has a large effect on KV cache size, which is frequently the binding constraint in serving. ### Standard Multi-Head Attention (MHA) In MHA, every attention head has its own Q, K, and V projections. A model with `n_heads = 64` stores 64 K vectors and 64 V vectors per token per layer. This maximizes expressivity but also maximizes cache size. ### Multi-Query Attention (MQA) MQA (Shazeer, 2019) keeps multiple Q heads but uses a single shared K and V pair. The cache shrinks by a factor of `n_heads`. For a model with 64 heads, that's 64x less KV cache, at the cost of some model quality on harder tasks. ### Grouped-Query Attention (GQA) GQA is the standard in modern models. Query heads are organized into groups, with one K and V per group. Llama 3.1 70B uses 64 query heads and 8 KV heads, giving an 8x cache reduction versus MHA with minimal quality loss. Essentially all models released since 2024 use GQA. For a thorough comparison of these three variants, see [Multi-Query and Grouped-Query Attention](/blog/multi-query-grouped-query-attention). ### Multi-Head Latent Attention (MLA) DeepSeek's MLA (used in DeepSeek-V2 and V3) takes a different approach: rather than reducing the number of KV heads, it projects K and V into a low-rank latent space before caching. The cache stores a compressed representation, which is expanded back to full K and V at attention time. The result is cache sizes smaller than GQA at equivalent quality. Details are in [KV Cache Compression: MLA and Beyond](/blog/kv-cache-compression-mla-and-beyond). ### How Context Length Quadratically Increases Attention Cost In prefill, attention complexity is O(n^2) in sequence length -- each token attends to all previous tokens. A 128K-token input does not take twice as long to prefill as a 64K-token input; it takes approximately four times as long, because the number of attention pairs scales quadratically. Flash attention (see [Flash Attention: Why Modern LLMs Run Faster With It](/blog/flash-attention-why-modern-llms-run-faster)) reorganizes the computation to be memory-efficient and avoids materializing the full attention matrix, but the quadratic FLOPs remain. In decode, each step's attention cost grows linearly with the current context length, because you're attending over a growing cache. ## Model Size, Parameters, and Memory Requirements ### How to Calculate Model Memory Footprint from Parameter Count The weight memory footprint is straightforward to estimate: ``` weight_bytes = num_parameters * bytes_per_param ``` Where `bytes_per_param` depends on precision: - BF16 / FP16: 2 bytes - FP8: 1 byte - INT4 (with group scales): ~0.5 bytes (the scale factors add a small overhead) Llama 3.1 70B has approximately 70.6 billion parameters. Weights only: | Precision | Memory for weights | |---|---| | BF16 | ~141 GB | | FP8 | ~71 GB | | INT4 | ~35 GB | This determines whether a model fits on a single node. An H100 has 80GB of HBM, so Llama 70B at BF16 requires at least two GPUs just for the weights, before accounting for the KV cache and activations. For quantization tradeoffs in more depth, see [Quantization Explained: INT4, GGUF, GPTQ and What They Mean for Your Model](/blog/quantization-explained-int4-gguf-gptq). ### KV Cache Memory -- Runtime Cost That Scales with Batch Size The KV cache formula: ``` cache_bytes = 2 * n_layers * n_kv_heads * head_dim * seq_len * dtype_bytes * batch_size ``` The factor of 2 is for K and V. For Llama 3.1 70B (80 layers, 8 KV heads, head_dim 128, FP16) at 32K context, per request: ``` 2 * 80 * 8 * 128 * 32,000 * 2 = 10.5 GB per request ``` This is the runtime cost that competes with your weight footprint for HBM. At 10.5 GB per request and 141 GB for weights in BF16 on an 8xH100 node (640 GB total), you have roughly 499 GB available for cache, which accommodates about 47 concurrent requests at 32K context. Move to FP8 KV cache and you double that to 94 requests. At 128K context, the math changes: ``` 2 * 80 * 8 * 128 * 128,000 * 2 = 42 GB per request ``` Now you can support roughly 11 concurrent requests on the same node at 32K context budget. Long-context serving is almost always a memory problem, not a compute problem. ### Activation Memory During inference (not training), activation memory is transient -- it's the intermediate tensors in the forward pass that exist only within a layer's computation. At inference batch sizes typical for serving (under a few hundred), activation memory is a small fraction of total GPU memory. It becomes a concern at very large batch sizes, where the hidden-state tensors across all tokens in the batch add up. ## How Architecture Differences Affect Inference Speed ### Dense vs Sparse (MoE) Models In a Mixture of Experts model, each token is routed to a small subset of expert FFN layers (typically 2 out of 8 or 16). A model like Mixtral 8x7B has the FLOPs of a ~14B model per token but the parameter count of a ~47B model. This means: - Lower compute cost per token (fewer FLOPs) - Higher memory requirement (all experts must be loaded or accessible) - Potential load imbalance issues under high concurrency For throughput-bound workloads, MoE models can be very efficient: you get quality close to a larger dense model at a fraction of the FLOPs. For memory-bound workloads on small hardware, the full parameter footprint can be a constraint. See [Mixture of Experts (MoE) Models: Why They're Dominating 2025](/blog/mixture-of-experts-moe-models-why-theyre-dominating-2025) for a more detailed breakdown. ### Context Length and Its Effect on Prefill Time Prefill time grows quadratically with the length of the input, due to attention. For decode time, the cost per token grows linearly with the cached context length (you attend over more tokens per step). If your use case involves long system prompts repeated across requests, prefix caching (covered in [Prefix Caching: Why Repeated Prompts Shouldn't Cost You Twice](/blog/prefix-caching-why-repeated-prompts-shouldnt-cost-you-twice)) avoids recomputing the shared prefix on every request. ### Depth vs Width -- More Layers vs Larger Hidden Dimensions Two models can have similar parameter counts with different shapes. A "deep" model has more layers with smaller hidden dimensions; a "wide" model has fewer layers with larger hidden dimensions. For inference, depth creates a sequential dependency: each layer's output feeds the next, so layers cannot be parallelized across a single token's forward pass. More layers means more kernel launches and more pipeline bubbles in distributed serving. Width (larger hidden dim) increases FLOPs and memory bandwidth per layer but allows larger matrix operations that are more GPU-efficient. In practice, depth and width interact with hardware in non-obvious ways. Shorter models (fewer layers) with the same parameter count can be faster on a single GPU because the sequential layer dependency chain is shorter, even if per-layer cost is higher. ## Next-Generation Architectures Optimized for Inference ### Mamba and State Space Models Mamba (Gu and Dao, 2023) replaces attention with a selective state space model (SSM). Instead of attending over all past tokens, each token updates a fixed-size recurrent state. The inference cost per token is O(1) in context length, not O(n). This makes very long contexts computationally cheap -- no KV cache, no quadratic growth. The tradeoff is that SSMs are weaker at tasks requiring precise recall of specific information from earlier in the context, compared to attention. See [Mamba and State Space Models: Inference Without Attention](/blog/mamba-and-state-space-models-inference-without-attention). ### RWKV RWKV reformulates attention as a linear recurrence, giving O(1) per-token inference cost like Mamba but using a different mathematical formulation. RWKV models can be efficient to serve because they require no KV cache and scale linearly in context length. See [RWKV and Linear Attention: Recurrent Models as an Inference Shortcut](/blog/rwkv-and-linear-attention-recurrent-models-as-an-inference-shortcut). ### Hybrid Models Recent architectures combine attention layers with SSM or linear attention layers. The intuition is that full attention handles tasks requiring long-range recall, while linear layers handle most of the sequence modeling cheaply. This lets the model use a smaller KV cache (only for attention layers) while maintaining quality closer to full transformers than pure SSMs. Expect to see more hybrid architectures in high-throughput serving deployments over the next few years. ## Practical Implications for Choosing a Model ### How to Read a Model Card for Inference Efficiency When evaluating a model's architecture, look for: - **Number of layers**: 32 is typical for 7-8B; 80 for 70B. More layers = more sequential steps per token. - **Hidden dimension (d_model)**: 4096 for 7B, 8192 for 70B. Drives memory bandwidth per layer. - **Attention heads and KV heads**: The ratio tells you which attention variant is used. Equal counts = MHA; fewer KV heads = GQA. Llama 3.1 8B: 32 Q heads, 8 KV heads. 70B: 64 Q heads, 8 KV heads. - **Intermediate (FFN) dimension**: Usually 4x d_model, but some models use SwiGLU or other gated FFN variants that have their own size conventions. - **Expert count** (for MoE): How many experts exist and how many are active per token. This determines effective FLOPs vs total parameter footprint. ### Architecture Checklist Before Deployment Before committing to a model for production: - Calculate weight memory at your target precision. Does it fit on your hardware? - Calculate KV cache per request at your expected context lengths. How many concurrent requests can you support? - Check whether the model uses GQA. If it uses MHA, the cache footprint is much larger than you might expect. - Check whether your serving framework supports the attention variant. Most do for GQA; MLA requires explicit support. - For long-context use cases, check whether the model was actually trained at those lengths, not just "supports" them via rope scaling. ## FAQ ### Why do larger transformers take longer to run inference? Larger models have more parameters, meaning more weight data must be read from HBM per token during decode. More layers also means more sequential operations per forward pass. Both effects increase latency. The FLOPs increase means compute also takes longer, though for single-token decode steps the memory bandwidth bottleneck usually dominates. ### What is the KV cache and why does it use so much memory? The KV cache stores the key and value vectors computed for each token in the context, so they don't need to be recomputed at every decode step. It grows with both context length and batch size. For a large model at long context, the cache can consume more memory than the model weights themselves. The formula above gives the exact calculation. See [KV Cache in LLM Inference: How It Works and Why It Matters](/blog/kv-cache-in-llm-inference-how-it-works-and-why-it-matters) for a full treatment. ### Is a model with fewer layers faster than one with more parameters? Not necessarily, but depth is a real factor. Layers are sequential: you cannot overlap computation across them for a single token's forward pass. A 48-layer model and an 80-layer model with the same total parameter count will have the 48-layer model complete its forward pass faster per token, all else equal, because the sequential chain is shorter. In practice, width and depth are traded off by the model designers for quality reasons, and hardware-specific factors (kernel fusion, tensor parallelism layout) often matter more than raw layer count. --- Architecture is the first constraint in any inference system. Getting familiar with the numbers -- weight bytes, cache bytes per request, FLOPs per token -- lets you reason about capacity, cost, and bottlenecks before you run a single benchmark. The rest is optimization on top of a foundation that's set when the model is trained. If you want to run inference on models across the full size spectrum with ASIC-optimized throughput, you can try [GeneralCompute's API](https://generalcompute.com) directly. --- # What Is an AI Inference Server? Cloud API vs Self-Hosted vs On-Premise Explained > An AI inference server accepts prompts and returns model completions. Here's how they work, the main deployment options, and how to choose between managed APIs and self-hosted solutions. - Author: General Compute - Published: 2026-08-10 - Tags: ai inference, llm serving, vllm, self-hosted llm, inference infrastructure - Canonical: https://www.generalcompute.com/blog/what-is-an-ai-inference-server --- An AI inference server is a process (or set of processes) that sits between your application and a machine learning model. It accepts incoming requests -- typically an HTTP POST with a prompt and parameters -- loads the model into GPU memory, runs the forward pass, and returns a completion. Everything else in the landscape of "how to run LLMs" is a variation on that core function. ## What Is an AI Inference Server? ### The Core Function: Accepting Requests and Returning Completions When your code calls an LLM API with a prompt, something has to receive that request, schedule it against available hardware, run the model, and stream back tokens. The inference server is that thing. It handles: - Receiving and validating requests - Queuing and batching requests efficiently - Running the model forward pass on GPU (or other accelerator) - Streaming tokens back to the caller as they're generated - Tracking usage, errors, and health metrics ### How an Inference Server Differs from a Training Cluster Training clusters are optimized for throughput over many GPU-hours, running large batches repeatedly with gradient updates. Inference is different: requests come in at unpredictable intervals, context lengths vary widely, and latency matters to end users. A training cluster that finishes a job in 3 hours is fine; an inference server that takes 3 seconds to return the first token is not. This is why training and inference infrastructure are typically separate. Training optimizes for maximizing GPU utilization on a fixed workload. Inference optimizes for minimizing latency while handling variable concurrent load. ### Key Components: Model Loading, Request Queue, Batching Engine, API Layer A production inference server has four layers: 1. **Model loading layer**: loads model weights from storage into GPU memory, handles quantization, sharding across multiple GPUs if needed 2. **API layer**: exposes HTTP endpoints (usually OpenAI-compatible), validates requests, handles auth 3. **Request queue**: buffers incoming requests, manages concurrency limits, handles backpressure 4. **Batching engine**: groups multiple requests together to maximize GPU utilization -- in modern servers, continuous batching means requests are batched at the token iteration level rather than the request level, so GPUs rarely sit idle between requests ## The Three Deployment Models: Cloud API, Self-Hosted, On-Premise There are three fundamentally different ways to run inference: use a managed cloud API, self-host on rented cloud GPUs, or run entirely on-premise hardware. ### Managed Cloud APIs: How They Work Under the Hood With a managed API (like GeneralCompute, Groq, Together AI, or OpenAI), you send HTTP requests to a provider's endpoint. The provider handles everything: hardware procurement, model loading, scaling, monitoring, and maintenance. You get an API key, you call the API, and you pay per token. Under the hood, these providers run large GPU or ASIC clusters that are always warm, with models pre-loaded and ready to serve. They invest in serving optimizations -- continuous batching, prefix caching, speculative decoding -- that would take your team months to implement and tune. ### Self-Hosted Inference Servers: You Run the Infrastructure Self-hosting means you rent GPUs (from AWS, GCP, CoreWeave, Lambda Labs, etc.) and run your own inference stack. You install vLLM, SGLang, or TensorRT-LLM, load the model yourself, and expose an API. You control everything and pay for the underlying compute. Self-hosting gives you model flexibility (you can run any open-weights model, fine-tuned or not) and potential cost savings at scale. The tradeoff is operational burden: you're responsible for uptime, scaling, hardware failures, CUDA driver updates, and optimization work. ### On-Premise Inference: Air-Gapped and Fully Controlled On-premise means your own physical hardware in your own data center or colocation facility. No cloud provider in the picture. This is the only option for organizations with strict data sovereignty requirements or compliance mandates that prohibit sending data to third-party clouds. The economics are hardest here: hardware is expensive, utilization is difficult to maximize across variable demand, and you need dedicated infrastructure teams. ### Hybrid Approaches: Cloud Burst + On-Prem Baseline Many large organizations run baseline on-premise capacity for predictable load (on-prem is cheapest per GPU-hour when fully utilized) and burst to cloud APIs during traffic spikes. This requires careful routing logic but can optimize cost while maintaining compliance for sensitive workloads. ## Popular Open-Source Inference Server Options If you're self-hosting, you'll choose from a small set of mature open-source frameworks. ### vLLM: The Production Standard [vLLM](https://github.com/vllm-project/vllm) is the most widely deployed open-source inference server. Its key contributions are PagedAttention (efficient KV cache management that virtually eliminates memory waste) and continuous batching (requests are batched at the token iteration level, so the GPU almost never sits idle between requests). vLLM supports most major model architectures -- Llama, Mistral, Qwen, Gemma, DeepSeek, and more -- and runs on NVIDIA GPUs with solid ROCm support for AMD. It exposes an OpenAI-compatible API, so migrating existing code is usually a `base_url` swap. Typical throughput with vLLM on a single H100: 2,000-4,000 tokens/sec for a 70B model at FP16, depending on batch size and sequence length. ### SGLang: Best for Agents and Multi-Turn Workloads [SGLang](https://github.com/sgl-project/sglang) was built by researchers at UC Berkeley and MIT with agentic workloads in mind. Its main differentiation is RadixAttention: a KV cache data structure that efficiently reuses cached attention states across requests that share a common prefix. For multi-turn conversations or agent workflows where many requests share a system prompt and conversation history, SGLang can achieve substantially higher effective throughput than vLLM. If your workload is chatbot-heavy or you're running agents, it's worth benchmarking against vLLM directly before committing. ### TGI (Text Generation Inference): HuggingFace's Option HuggingFace's Text Generation Inference is the runtime used by their Inference Endpoints product. It's production-tested, has strong support for the HuggingFace Hub model format, and integrates well with the broader HF ecosystem. TGI supports tensor parallelism (spreading a model across multiple GPUs) and flash attention. Performance is generally competitive with vLLM for single-request scenarios but less optimal for high-concurrency workloads. ### TensorRT-LLM: Maximum Performance on NVIDIA Hardware TensorRT-LLM is NVIDIA's own inference library, built on top of TensorRT. It compiles models into optimized CUDA kernels for a specific GPU architecture, which means setup is more involved (you build engine files per GPU type) but the resulting throughput can be 20-30% higher than vLLM for compute-bound workloads. It's the right choice when you have a specific NVIDIA GPU target, a stable model you're running at scale, and the engineering time to manage the compilation workflow. It's not the right choice for rapid model iteration or broad architecture support. ## Managed Inference APIs: The Cloud-Hosted Alternative Rather than running your own stack, several providers offer managed inference with no infrastructure overhead. ### GeneralCompute: ASIC-Powered, Fastest TTFT GeneralCompute runs on custom ASIC infrastructure optimized specifically for token generation. The result is industry-leading [time to first token (TTFT)](https://generalcompute.com/blog/what-is-time-to-first-token) -- critical for voice AI and interactive applications -- and high [tokens per second (TPS)](https://generalcompute.com/blog/what-is-tokens-per-second) throughput across major open models including Llama 4, DeepSeek, and Qwen. The API is OpenAI-compatible, so integration is typically a one-line change to existing code. ### Groq: LPU-Based Fast Inference Groq uses their Language Processing Unit architecture, which excels at the sequential token-by-token generation phase of decoding. Groq's throughput numbers are strong on supported models, though model selection is narrower than GPU-based providers. ### Together AI: Broad Model Selection Together AI offers a wide catalog of open models via API and supports custom fine-tuned model deployments. Their strength is breadth: if you need to try many different models quickly without managing GPUs, it's a reasonable starting point. ### Fireworks AI: Fine-Tuning and Custom Models Fireworks specializes in deploying fine-tuned models and has tooling for LoRA fine-tuning on their platform. If your workflow involves frequent model iterations (fine-tune, deploy, test, repeat), Fireworks simplifies the deployment side. ## Cloud API vs Self-Hosted: A Comprehensive Comparison | Dimension | Managed Cloud API | Self-Hosted | |---|---|---| | **Latency** | Typically faster (optimized infra, warm models) | Can match with effort; cold starts hurt | | **Cost at scale** | Higher per-token; no fixed cost | Lower per-token; high fixed cost | | **Ops burden** | Zero | Significant (DevOps, on-call) | | **Model flexibility** | Limited to provider catalog | Any open-weights model | | **Fine-tuned models** | Provider-dependent | Full control | | **Data privacy** | Data sent to third party | Data stays in your environment | | **Compliance (HIPAA, FedRAMP)** | Provider-dependent BAAs | Full control | | **Time to first request** | Minutes (API key) | Days to weeks | ### Latency: Which Approach Is Actually Faster? Counterintuitively, managed APIs often beat self-hosted setups on latency, because providers invest in optimizations that most teams don't implement: speculative decoding, prefix caching, optimized kernels, hardware-level tuning. A freshly deployed vLLM instance with default settings will typically underperform a well-tuned managed API. That said, self-hosted can win on latency when you dedicate a full GPU or cluster to a single model with no sharing, and you're geographically co-located with your application. Dedicated capacity eliminates queuing latency from shared infrastructure. ### Cost at Scale: The Real Break-Even Analysis Managed APIs charge per token. Self-hosting charges per GPU-hour, regardless of actual utilization. The break-even depends on your utilization rate. A single H100 costs roughly $2.00-2.50/hr on spot pricing via providers like CoreWeave or Lambda Labs. At 80% utilization (which is high for most teams), that's $1.60-2.00/hr of effective compute. An H100 can generate roughly 2,000 tokens/sec at 70B scale (Llama 4 FP8), so at 80% utilization you're producing around 1,600 tokens/sec of output. At a typical managed API price of $0.50/1M tokens for a 70B model, you'd spend $1.60-2.00 generating 3.2-4.0M tokens. Over an hour at 1,600 tokens/sec, you'd generate 5.76B tokens self-hosted. Self-hosting wins decisively at high utilization -- but most teams don't sustain 80% utilization. At 20% utilization, the effective self-hosted cost per token is roughly 4x higher than managed. A rough rule: self-hosting becomes cheaper than managed APIs at sustained utilization above 30-40%, assuming you're not paying significant DevOps overhead. ### Operational Burden: Engineering Hours Required Self-hosting a production inference server requires ongoing engineering time: monitoring, scaling, CUDA driver updates, model version management, debugging latency regressions, handling hardware failures. Budget at least 0.5-1 full-time engineer equivalent for a small cluster running one or two models. For teams that are primarily building applications rather than infrastructure, managed APIs let them stay focused on the product. ### Model Availability: What You Can and Cannot Run Managed APIs support whatever models the provider chooses to host. If you need a specific fine-tuned checkpoint, a research model, or a model the provider doesn't carry, you'll need to self-host or use a provider that supports custom deployments. ### Compliance and Data Privacy: Who Controls the Data? Prompts sent to a managed API leave your network. For applications handling sensitive data (medical records, financial information, legal documents), this requires either a BAA with the provider or self-hosting. Check specific certifications before assuming compliance: SOC 2 Type II, HIPAA BAAs, and FedRAMP authorization vary by provider. ## When Does Self-Hosting Make Economic Sense? ### GPU Rental Math: Breakeven at What Monthly Volume? A single H100 at $2.25/hr (spot) costs roughly $1,620/month. Running a 70B model at 40% average utilization, you produce approximately 2.8B tokens/month. At $0.50/1M output tokens managed, that volume costs $1,400 from a provider (plus input tokens). Self-hosting starts to look cheaper here -- but add 20-30% for DevOps time and it's closer than the raw numbers suggest. For most teams, the crossover is somewhere between 2-5B tokens/month per model. At 10B+ tokens/month, the math strongly favors self-hosting unless you have high engineering overhead or compliance requirements that favor managed options. ### Latency SLA Requirements That Force Self-Hosting Some use cases require latency guarantees that shared managed APIs can't provide. Real-time voice AI targeting sub-150ms TTFT at P99, or edge inference that needs single-digit millisecond response times, may require dedicated hardware. When you're sharing infrastructure with other customers, you're subject to their traffic patterns. ### Regulatory Requirements: HIPAA, GDPR, FedRAMP Healthcare applications handling PHI need a HIPAA BAA. Financial services in certain jurisdictions need data to stay within specific geographic boundaries. Government applications often require FedRAMP authorization. These constraints may force self-hosting or on-premise regardless of cost analysis, unless you're working with a managed provider who holds the relevant certifications. ## How to Choose the Right Inference Server Architecture ### Decision Framework: 5 Questions Before Choosing 1. **What's your current monthly token volume?** Under 1B tokens/month, managed APIs almost always win on total cost. Over 10B, self-hosting is worth doing the math. 2. **Do you need models not available via managed APIs?** Fine-tuned models, experimental architectures, or specific open-weights variants may require self-hosting. 3. **What are your compliance requirements?** If data can't leave your network, you're self-hosting or on-prem regardless of other factors. 4. **What's your latency SLA?** Under 200ms TTFT at P95 is achievable with managed APIs; under 100ms at sustained load may require dedicated hardware. 5. **How much engineering time can you allocate to infrastructure?** Be honest about this. Inference infrastructure takes real time to operate well. ### For Startups: Start Managed, Move Later Unless you have a specific compliance or model-flexibility requirement, start with a managed API. You'll ship faster and have better observability from day one. Revisit the architecture when your token spend exceeds $5,000-10,000/month and your utilization patterns are well-understood. ### For Enterprises: Compliance First, Then Optimize Enterprises typically have to answer the compliance question before anything else. If you need data residency or a specific certification, that narrows your options immediately. Once you know what's allowed, optimize for cost and latency within that constraint -- not before. ## FAQ ### What is the best open-source LLM inference server? vLLM is the production standard for most use cases: broad model support, strong throughput via continuous batching and PagedAttention, and an active community. SGLang is worth evaluating for multi-turn or agentic workloads where KV cache reuse matters. TensorRT-LLM delivers the highest raw throughput on NVIDIA hardware if you have the engineering bandwidth for its compilation workflow. ### Can I run a self-hosted inference server on a single GPU? Yes. A single H100 (80GB) can run Llama 4 70B at FP8 quantization with room for KV cache. A single A100 (80GB) handles 70B models at FP8 as well, though throughput is lower. For 7-8B models, a single 24GB consumer GPU (RTX 4090) works for development and low-traffic production use. ### How many requests per second can an inference server handle? It depends on model size, sequence length, hardware, and concurrency. A single H100 running Llama 3.1 70B at FP8 with continuous batching can handle 50-150 concurrent requests depending on output length, which translates to roughly 20-50 completed requests per second at typical chatbot sequence lengths. Smaller models (7-8B) can handle 5-10x more RPS on the same hardware. --- If you're evaluating inference options and want to skip the infrastructure work, [GeneralCompute's API](https://generalcompute.com) gives you access to fast open-model inference with no cluster to manage. The ASIC-optimized infrastructure delivers strong [TTFT and TPS](https://generalcompute.com/blog/what-is-tokens-per-second) numbers across Llama 4, DeepSeek, Qwen, and other major models, on an OpenAI-compatible API. Check the [documentation](https://generalcompute.com/docs) to get started, or review our [AI inference cost guide](https://generalcompute.com/blog/ai-inference-costs) to run the economics before deciding on an architecture. --- # Why TileRT Cannot Catch SambaNova SN50: The Limits of Software-Emulated Dataflow on the Wrong Hardware > TileRT is a clever software approach to dataflow execution, but it runs on hardware that was never designed for reconfigurable dataflow. Here's why purpose-built silicon like the SambaNova SN50 will always hold the edge. - Author: General Compute - Published: 2026-08-10 - Tags: inference, hardware, dataflow, sambanova, tilert, asic - Canonical: https://www.generalcompute.com/blog/why-tilert-cannot-catch-sambanova-sn50 --- TileRT is a software layer that remodulates execution on existing accelerators to behave more like a dataflow processor. The idea is reasonable: if you can tile compute graphs, fuse operations, and schedule producers and consumers efficiently in software, you might approximate what purpose-built dataflow silicon does in hardware. The problem is that TileRT runs on chips that were designed around different assumptions. The memory hierarchy, interconnect topology, and scheduler all reflect the host chip's original purpose. Software can reshape how work is dispatched, but it cannot change where memory lives, how wide the on-chip buses are, or how the hardware arbitrates between concurrent workloads. Those constraints are baked into silicon, and they determine the performance ceiling regardless of how smart the software layer is. The SambaNova SN50 is designed from scratch for dataflow execution. This post explains what that means in practice, where TileRT hits structural limits that software cannot overcome, and what the performance delta looks like in concrete terms. ## The CUDA Moat Was Never About Inference TileRT is, among other things, strong validation that kernel-by-kernel execution models cannot deliver fast inference. CUDA is fundamentally a kernel-by-kernel paradigm. Every operation dispatches a kernel, completes, and returns control to the runtime before the next operation begins. TileRT exists precisely to work around this: it intercepts the dispatch sequence, fuses kernels, and tries to impose producer-consumer ordering that CUDA's runtime was not designed to provide. If the winning inference architecture is dataflow, then for fast inference there is no CUDA moat. You have to work around CUDA rather than leverage it. TileRT is a workaround, not a use, of the CUDA programming model. The moat that matters for inference is not a programming model moat. It is a hardware architecture moat, and it belongs to silicon designed from the start for streaming dataflow execution. ## What TileRT Actually Does ### The Software Remodulation Idea TileRT works by intercepting a model's compute graph and retiling it into execution blocks that can be mapped across processing elements with better producer-consumer locality. Instead of dispatching independent kernels to a GPU runtime and relying on the hardware scheduler, TileRT tries to control the execution order explicitly so that the output of one operation flows directly into the input of the next without going through global memory. This is a real optimization. Kernel-at-a-time execution wastes bandwidth because each kernel reads inputs from DRAM, computes, and writes outputs back to DRAM. If you can fuse multiple kernels and keep intermediate tensors in L2 cache or on-chip SRAM, you cut a significant fraction of memory traffic. ### What TileRT Runs On Today and Why That Matters TileRT is deployed on GPU-class hardware and some general-purpose accelerators. These chips have a fixed memory hierarchy: registers, then L1 cache per SM, then L2 cache shared across SMs, then HBM. The interconnect between SMs uses a mesh or crossbar that was sized and designed for GPU workloads, not for the streaming tensor traffic that dataflow processors are built around. This matters because the TileRT runtime cannot widen the buses, resize the on-chip caches, or reorder the levels of the memory hierarchy. It can control when data moves, but it cannot change where data must live or how long it takes to move there. ### Why Software Alone Cannot Change Memory Hierarchy or Interconnect Topology When TileRT fuses two operations, it keeps intermediate values on-chip if they fit in the available cache. But "available cache" is determined by what the host chip provides. An H100 SM has 256 KB of shared memory per block, and the L2 is 50 MB shared across all SMs. These are fixed constraints. If a fused tensor chain exceeds what fits on-chip, data spills to HBM. At 3.35 TB/s bandwidth, the H100 is fast, but that bandwidth is shared across every concurrent workload on the chip, and the round-trip to HBM is still orders of magnitude slower than on-chip SRAM access on a purpose-built dataflow unit. ## What a SambaNova Dataflow Unit Does in Silicon ### The RDU/PCU/PMU Fabric The SambaNova SN50 is built around a Reconfigurable Dataflow Unit (RDU). The RDU fabric consists of Pattern Compute Units (PCUs) and Pattern Memory Units (PMUs) arranged so that compute and memory are physically adjacent. Each PCU has its own PMU, and the two are connected by short, high-bandwidth local wires. This is structurally different from a GPU, where compute (SM) and memory (L2/HBM) are separated by a shared bus that all SMs compete for. On the RDU, each PCU pulls data from its adjacent PMU without contending with neighboring PCUs. The effective memory bandwidth per compute unit is much higher because the paths are shorter and not shared. ### Streaming Tensors Between Stages Without Global Memory Round-Trips On the SN50, the compiler schedules a full model graph and assigns each operation to specific PCUs and PMUs at compile time. The output tensor of one layer streams directly to the PCU handling the next layer through the on-chip interconnect. There are no global memory round-trips between layers unless the data explicitly needs to be stored. For transformer inference, this means the attention output can stream into the feed-forward network without touching HBM. The KV cache lives in PMUs close to the PCUs that consume it. Token generation becomes a continuous pipeline through on-chip fabric rather than a sequence of kernel launches separated by HBM accesses. ### Compiler-Scheduled Dataflow vs Runtime Kernel Dispatch The SambaNova compiler resolves the full execution schedule at compile time. At runtime, the hardware executes a dataflow graph without a runtime scheduler making decisions. There is no kernel launch overhead, no GPU driver interaction per operation, and no scheduler arbitrating between concurrent warps. TileRT operates at a different level. Even when it fuses operations, it ultimately dispatches them through the host chip's runtime. On a GPU, that means CUDA kernel launches with their associated overhead, driver calls, and dependency tracking by the CUDA runtime. Each launch takes on the order of 5-20 microseconds. Across thousands of decoder steps in a long generation, that overhead accumulates. ## A Pulled Quote from the SambaNova Perspective > "A persistent engine kernel means the model is statically expanded ahead of time into one resident program. This is indeed extremely hard. We have been working on this problem for 9 years in our compiler." > > -- SambaNova team ([source](https://x.com/aton2006/status/2086899301188510132)) This is the crux. TileRT's approach relies on collapsing the whole decode graph into a single persistent kernel. That is a compiler problem SambaNova has been solving for nearly a decade with the right hardware target, and it is not something you catch up on in a release cycle. The difficulty is not just algorithmic. It is that the compiler must reason about a hardware fabric designed for the target. Writing a persistent kernel scheduler for a chip whose memory hierarchy was not designed for it is the hard part, and that is what TileRT is attempting. ## Where TileRT Hits the Wall ### Kernel Launch and Scheduler Overhead the Host Chip Still Imposes Even a highly optimized TileRT graph eventually dispatches work to the host chip's execution units. On a GPU, each dispatch is a kernel launch. The CUDA driver tracks dependencies, allocates registers, and schedules blocks across SMs. This overhead is small per kernel but it is non-zero and it compounds. For short sequences or small batch sizes where inference is already memory-bound, kernel launch overhead can represent 10-30% of total execution time. TileRT cannot eliminate this because it does not control the host chip's dispatch mechanism. ### Memory Bandwidth and Cache Behavior TileRT Cannot Rewrite TileRT improves cache utilization within the constraints of the host chip's cache hierarchy. But it cannot make the L2 larger, cannot place PMU-equivalent memory next to each compute unit, and cannot change the bandwidth between cache levels. For a 70B parameter model at BF16, loading weights alone requires moving roughly 140 GB of data. On an H100, HBM bandwidth is 3.35 TB/s, which puts the theoretical minimum at about 42ms per decode step if the chip is purely memory-bound. The SN50's PMU fabric reduces the effective distance data travels to reach compute, cutting this floor for the operations that can stay on-chip. ### Fixed Interconnect Topology Limits Producer-Consumer Fusion TileRT's fusion strategy works best when the intermediate tensor between two fused operations fits in the cache available on the SM (or equivalent compute unit) executing both operations. When tensors are too large to fit, the fusion either spills to L2 or HBM or gets split in ways that limit the potential speedup. On the SN50, the compiler assigns PMU capacity to each operation at compile time and routes data through the PCU-PMU fabric based on what the hardware physically supports. The compiler knows exactly how much on-chip capacity each stage has and can plan accordingly. TileRT operates within whatever the runtime presents, which varies based on what else is running on the chip and how the OS and driver decide to allocate resources. ### Debugging a Software-Emulated Dataflow Graph When something goes wrong in a TileRT execution, the error surfaces through multiple layers: the user's model code, the TileRT graph transformation layer, the host chip's runtime, and sometimes the hardware itself. Stack traces can reference symbols across all of these layers. Performance counters on the host chip reflect the hardware's view of what happened, not TileRT's view of what it intended. Non-determinism from the host scheduler means that a timing-dependent bug may not reproduce reliably. On the SN50, the compiler produces a static execution schedule. Profiling tools see the same graph structure that the compiler reasoned about. When a performance issue appears, it maps back to a specific PCU-PMU pair in the compiled graph, and the compiler's dataflow model matches what the hardware executed. ### The Tensor Parallel Ceiling at 8 No GPU inference deployment on InferenceX, or in public benchmarks, is scaling tensor parallel beyond 8 GPUs for high-speed decoding, despite 72-GPU scale-up domains like NVL72 being physically available. Data parallel does not count, because it does not reduce per-request latency. TileRT's demonstrated mappings still leave significant HBM bandwidth unutilized within TP8. Until GPU-based dataflow emulation can saturate HBM bandwidth across a large model-parallel domain (16, 32, 64+ GPUs), it will not keep up with dataflow chips that can spread a single model across the full scale-up domain. The interconnect topology that limits TP scaling on GPU clusters is not a software problem. It is a fabric design problem. ### Batching at Interactive Latency and 140K Context The Premium Inference bar is high batch per chip at interactive latency at long context. Median context on InferenceX is now around 140K tokens. Dataflow emulation on GPUs, even on Rubin-class hardware, will struggle to hit this combination because KV cache pressure at 140K context blows past on-chip memory and forces HBM traffic that a static persistent kernel cannot hide. On the SN50, the same speed target can be met with much higher batch sizes per chip at 140K context. The PMU fabric keeps KV cache close to the compute that consumes it. The difference is not bandwidth per second. It is where the bandwidth has to travel. ### Generalization Across Models Statically expanding a full model into one resident persistent kernel is a hard compiler problem, as the SambaNova team has noted. It has to be redone for every model architecture, every context length regime, every batch shape. TileRT is starting on this problem. SambaNova has been building compiler infrastructure for reconfigurable dataflow for nine years, on hardware that was designed for the target. The generalization problem is easier when the hardware matches the abstraction. A compiler targeting a chip with fixed memory hierarchy and runtime dispatch must work around the hardware. A compiler targeting a chip whose PCU-PMU fabric was designed around dataflow scheduling reasons directly about the target it controls. ## SN50 vs TileRT: The Expected Delta The following table summarizes the expected performance difference between TileRT on a representative GPU-class accelerator and the SambaNova SN50 for transformer inference workloads. | Metric | TileRT (GPU-class) | SambaNova SN50 | Advantage | |---|---|---|---| | Prefill latency | Baseline | 2-3x lower | SN50 | | Decode latency (TTFT) | Baseline | 2-3x lower | SN50 | | Throughput (tokens/s) | Baseline | ~10x higher | SN50 | | Power draw per token | Baseline | ~10% of GPU-class | SN50 | | Cost per million tokens | Baseline | ~10x lower | SN50 | | Debuggability | Moderate | High (static schedule) | SN50 | | Software ecosystem maturity | High (CUDA ecosystem) | Growing | TileRT/GPU | The latency advantage comes from eliminating global memory round-trips between layers. The throughput advantage compounds the latency gain with the absence of kernel dispatch overhead and the higher effective bandwidth per compute unit. The power figure reflects that the SN50 moves less data over shorter distances, which is the primary driver of power consumption in memory-bound inference workloads. At 10% of the power draw, the cost advantage from power alone is significant before accounting for capital costs or utilization differences. ## The Software Ecosystem Argument ### SN50 Tooling Designed Around Dataflow From Day One SambaNova's compiler, profiler, and debugger all operate on the same dataflow graph model. The profiler shows time spent in each PCU-PMU pair. The debugger inspects the compiled graph directly. When you optimize a model for the SN50, the tools reflect the same abstractions the hardware executes. This coherence matters for production engineering. When a model runs slower than expected, the profiler points to a specific stage in the graph with specific capacity constraints. The feedback loop between profiling and optimization is tight. ### TileRT Debuggability Problem TileRT introduces a layer of abstraction between the user's model and the hardware. This layer is valuable, but it adds complexity to the debugging process. A performance regression in TileRT may appear in the host chip's performance counters as increased HBM bandwidth, but the cause may be a fusion boundary in TileRT's graph transformation that was not visible in the profiler. When errors occur, they can surface at any layer: the model code, TileRT's graph rewriting logic, the host runtime, or the hardware. Reproducing these errors consistently is harder because the host chip's scheduler introduces non-determinism that TileRT does not fully control. ### Why an Integrated Stack Beats a Retrofit The SN50 was not originally designed for CPUs and later adapted for dataflow. The hardware architecture, compiler, and tooling were co-designed around the dataflow execution model. TileRT is a retrofit: it applies dataflow ideas to hardware built for a different model. The retrofit can capture some of the benefits, but it operates within the constraints of hardware decisions made before dataflow was the goal. ## When TileRT Still Makes Sense TileRT's real value is fungibility. If you are an inference provider and you buy some systems for prefill and high-throughput decoding, and other systems for fast decoding, and demand shifts toward more fast decoding, you can reallocate GPU capacity with TileRT to serve that demand, albeit at a much higher cost per million tokens. This is a real operational advantage during workload transitions. The counter-argument: dataflow chips are also fungible at the edges. If workloads shift from fast decode toward higher throughput and medium interactivity, dataflow chips can be reallocated too, at competitive tokenomics. The right design point is fungibility at the edges, not at the extremes: cover enough of the Pareto curve to adapt without over-specializing, but keep enough specialization to differentiate. If both platforms can flex at the edges but dataflow silicon wins on the primary metric by 2-3x latency and 10x throughput and roughly 10x power, the fungibility argument gives buyers more reason to buy dataflow, not less. Beyond fungibility, there are other situations where TileRT remains a reasonable choice. **Research and prototyping.** TileRT runs on widely available hardware. For teams that are exploring model architectures or serving strategies before committing to a specific hardware platform, TileRT on a GPU cluster is more accessible than procuring SN50 capacity. **Workloads where the gap is smaller.** For very large batch sizes where the workload is heavily compute-bound rather than memory-bound, the dataflow fusion advantage narrows. Compute-bound workloads spend their time in matrix multiplications where the GPU's tensor cores are already well-utilized, and the benefit from eliminating HBM round-trips is smaller relative to the total compute time. TileRT is a well-executed engineering effort and it advances the state of GPU inference. The takeaway for the industry is not that TileRT is bad. It is that the fact TileRT has to exist at all is validation of the dataflow thesis. Kernel-by-kernel execution is not the endgame for inference. The question is which stack has the right hardware target and enough compiler maturity to close the generalization problem, and today that stack is dataflow silicon. ## What This Means for Inference Buyers in 2026 ### Latency-Bound Workloads For real-time applications where latency is the primary constraint (voice AI, interactive coding assistants, chatbots targeting sub-200ms TTFT), the 2-3x latency advantage of purpose-built dataflow silicon is decisive. TileRT can improve GPU latency, but it cannot close that gap because the gap comes from structural differences in the memory hierarchy and interconnect, not from software inefficiency. For these workloads, the hardware choice matters more than the software layer on top of it. ### Throughput-Bound Workloads at Scale For batch inference workloads where throughput per dollar is the primary metric, the power advantage compounds the throughput advantage. At roughly 10x lower power consumption per token, the SN50 can run at higher sustained throughput without hitting thermal limits. At scale, the power cost becomes a significant fraction of total cost of ownership, and a 10x difference in tokens-per-watt is not recoverable through software optimization. ### The Debuggability Tax Compounds Over Time In the first months of deployment, the debuggability difference between an integrated dataflow stack and a software-emulated one is manageable. Engineering teams learn the tooling and develop workflows for TileRT's abstraction layers. Over time, as the system scales and edge cases accumulate, the cost of debugging a multi-layer abstraction grows. Production incidents that would take an hour to diagnose on an integrated stack can take a day on a retrofitted one. For teams running inference at scale, this operational overhead is real and measurable in engineering hours. ## FAQ ### Is TileRT the same as SambaNova's SambaFlow? No. SambaFlow is SambaNova's own compiler and runtime stack designed specifically for the RDU hardware. TileRT is a separate software layer developed to bring dataflow-like execution patterns to GPU-class hardware. They share some conceptual goals (producer-consumer fusion, compiler-scheduled execution) but differ fundamentally in that SambaFlow has the hardware it needs to execute its schedule, while TileRT must work within the constraints of hardware that was not designed for it. ### Can TileRT close the gap with future GPU generations? Future GPU generations will continue to increase HBM bandwidth and add more on-chip SRAM. These improvements will narrow the gap in some workloads. But the structural difference remains: as long as TileRT runs on hardware with a shared global memory hierarchy and a runtime scheduler that introduces per-kernel overhead, purpose-built dataflow silicon will maintain an advantage on memory-bound inference workloads. The gap may shrink from 10x throughput to 5x, but closing it entirely would require the GPU to adopt a fundamentally different memory architecture. ### Why is debugging a TileRT-based system harder than a native dataflow stack? The core problem is abstraction boundary mismatch. TileRT's graph model does not have a 1:1 correspondence with the host chip's execution model. A performance issue visible in TileRT's graph (a fusion boundary that forces a write to HBM) shows up in the GPU's performance counters as generic HBM bandwidth, without a clear pointer back to TileRT's graph structure. A developer debugging performance must reason about two models simultaneously: TileRT's dataflow view and the GPU's kernel-dispatch view. On the SN50, these views are the same, because the hardware executes the compiler's dataflow graph directly. --- If your workload is latency-sensitive or throughput-constrained, the inference provider's hardware architecture matters as much as the software stack on top of it. GeneralCompute runs on custom ASIC infrastructure designed specifically for inference, not repurposed training hardware. Try the [GeneralCompute API](https://generalcompute.com) to see what purpose-built inference hardware delivers for your workload. --- # AI Inference Costs in 2025: How to Calculate and Reduce Your LLM API Spend > A practical guide to understanding LLM API pricing, calculating your monthly inference bill with real formulas, and eight proven strategies to cut costs without sacrificing quality. - Author: General Compute - Published: 2026-08-09 - Tags: ai inference, llm api cost, inference cost calculator, reduce ai costs, llm pricing - Canonical: https://www.generalcompute.com/blog/ai-inference-costs-2025-how-to-calculate-and-reduce-your-llm-api-spend --- If you've run a production AI application for more than a few weeks, you've probably been surprised by an inference bill. LLM API costs compound in ways that aren't obvious from the per-token pricing alone: output tokens cost more than input tokens, long context windows multiply your spend, and an agentic workflow that makes 50 API calls per task can blow past 100x the cost of a single call. This guide covers how LLM API pricing actually works, walks through three real-world cost calculations, and gives you eight concrete strategies to reduce your spend without degrading quality. ## How LLM API Pricing Works: The Fundamentals ### Input Tokens vs Output Tokens: Why Output Costs 3-5x More Every major inference provider prices input and output tokens separately, and output tokens almost always cost more -- typically 3 to 5 times more. The asymmetry exists because output generation is more compute-intensive. During the decode phase, the model generates tokens one at a time in a memory-bandwidth-bound loop that's hard to parallelize. The prefill step that processes your input, by contrast, can run in parallel across all input positions at once. For most chat applications, your input tokens outnumber output tokens in raw count (long system prompts, conversation history), but the higher output price per token keeps the bill higher than you'd expect from input count alone. See [LLM Inference Explained](/blog/llm-inference-explained) for a deeper look at the prefill/decode split and why it shapes pricing the way it does. ### Context Window: The Hidden Cost Multiplier Every token in a request -- system prompt, conversation history, retrieved documents -- counts as input tokens. A 4,000-token system prompt sent on every request at 1M requests/day is 4 trillion input tokens per day. Multi-turn conversations are especially tricky. If you naively append the full conversation history on each turn, your input token count grows linearly with conversation length. A 20-turn conversation has roughly 10x the input cost of the first turn. ### Model Tiers: How Capability Affects Price Providers tier their pricing by model capability. A frontier model like GPT-4o costs 10 to 50x more per token than a smaller model like Llama 3.1 8B on a managed API. For many tasks -- classification, summarization, simple Q&A -- a smaller model at a fraction of the cost performs nearly as well. The biggest cost wins often come from routing requests to the smallest model that can handle them reliably. ### Batch Discounts and Committed Use Plans Most providers offer 40-50% discounts for asynchronous batch processing (jobs that don't need real-time responses). If you have a document processing pipeline that runs overnight, batch mode can cut your bill in half. Committed use agreements typically kick in at $1,000-$5,000/month and can get you 20-40% off list pricing. --- ## AI Inference Cost Calculator: Real-World Formulas ### Formula 1: Simple Single-Turn Request ``` cost = (input_tokens / 1_000_000 * input_price) + (output_tokens / 1_000_000 * output_price) ``` For a request with 500 input tokens and 200 output tokens using GeneralCompute ($0.10/1M input, $0.30/1M output): ``` cost = (500 / 1_000_000 * 0.10) + (200 / 1_000_000 * 0.30) = $0.000050 + $0.000060 = $0.000110 per request ``` ### Formula 2: Multi-Turn Conversation Cost With naive history concatenation, input tokens grow with each turn: ``` turn_input_tokens[n] = system_prompt_tokens + sum(all_prior_messages) + current_user_message total_cost = sum(cost(turn) for turn in conversation) ``` A 10-turn conversation with a 200-token system prompt, 150-token user messages, and 100-token assistant replies accumulates roughly 12,250 input tokens and 1,000 output tokens total. At GeneralCompute pricing: ($0.00123 + $0.00030) = **$0.00153 per conversation** ### Formula 3: Monthly Production Estimate ```python def monthly_cost(daily_requests, avg_input_tokens, avg_output_tokens, input_price_per_1m, output_price_per_1m): monthly_requests = daily_requests * 30 input_cost = (monthly_requests * avg_input_tokens / 1_000_000) * input_price_per_1m output_cost = (monthly_requests * avg_output_tokens / 1_000_000) * output_price_per_1m return input_cost + output_cost # Example: 33,333 requests/day, 400 input, 150 output monthly_cost(33_333, 400, 150, 0.10, 0.30) # Returns approximately $72/month on GeneralCompute ``` ### Example A: Chatbot with 100,000 Users per Month Assumptions: 5-turn average conversation, 400 input tokens per turn (system prompt + history), 150 output tokens per turn. - Total input tokens: 100,000 users x 5 turns x 400 tokens = 200M tokens - Total output tokens: 100,000 users x 5 turns x 150 tokens = 75M tokens | Provider | Input cost | Output cost | Total/month | |---|---|---|---| | GeneralCompute | $20 | $22.50 | **$42.50** | | Groq | $30 | $33.75 | $63.75 | | Together AI | $40 | $45.00 | $85.00 | | GPT-4o | $500 | $750 | $1,250 | ### Example B: Coding Agent, 10,000 API Calls per Day Assumptions: 800 input tokens per call (system prompt + code context), 400 output tokens (suggestions or edits), 30 days. - Total input tokens: 10,000 x 30 x 800 = 240M tokens - Total output tokens: 10,000 x 30 x 400 = 120M tokens | Provider | Input cost | Output cost | Total/month | |---|---|---|---| | GeneralCompute | $24 | $36 | **$60** | | Groq | $36 | $54 | $90 | | Together AI | $48 | $72 | $120 | | GPT-4o | $600 | $1,200 | $1,800 | ### Example C: Document Processing Pipeline, 1M Pages per Month Assumptions: 600 tokens per average page, 120-token summary output per page. - Total input tokens: 1M x 600 = 600M tokens - Total output tokens: 1M x 120 = 120M tokens | Provider | Input cost | Output cost | Total/month | |---|---|---|---| | GeneralCompute | $60 | $36 | **$96** | | Groq | $90 | $54 | $144 | | Together AI | $120 | $72 | $192 | | GPT-4o | $1,500 | $1,200 | $2,700 | For batch-eligible workloads like document processing, async batch mode cuts these numbers by up to 50%. --- ## Provider Cost Comparison (Q3 2026) ### Price per 1M Input Tokens | Provider | Small model | Large/frontier model | |---|---|---| | GeneralCompute | $0.10 | $0.50 | | Groq | $0.15 | $0.60 | | Fireworks AI | $0.20 | $0.90 | | Together AI | $0.20 | $0.80 | | OpenAI (gpt-4o-mini) | $0.15 | -- | | OpenAI (gpt-4o) | -- | $2.50 | | Anthropic (claude-3-haiku) | $0.25 | -- | ### Price per 1M Output Tokens | Provider | Small model | Large/frontier model | |---|---|---| | GeneralCompute | $0.30 | $1.50 | | Groq | $0.45 | $1.80 | | Fireworks AI | $0.50 | $2.70 | | Together AI | $0.60 | $2.40 | | OpenAI (gpt-4o-mini) | $0.60 | -- | | OpenAI (gpt-4o) | -- | $10.00 | | Anthropic (claude-3-haiku) | $1.25 | -- | Prices change frequently. Always verify on provider pricing pages before committing to a budget. ### Cost Efficiency: Tokens Per Second Per Dollar Speed affects your total infrastructure cost too. A slower provider means your application servers hold open connections longer, handling fewer requests per hour. The table below normalizes output throughput against blended cost for Llama 3.1 70B-class models. | Provider | Output TPS | Blended cost/1M tokens | TPS per $ | |---|---|---|---| | GeneralCompute | 180 | $0.90 | **200** | | Groq | 150 | $1.13 | 133 | | Fireworks AI | 110 | $1.60 | 69 | | Together AI | 95 | $1.40 | 68 | See [What Is Tokens Per Second (TPS)?](/blog/what-is-tokens-per-second-tps) for more on how throughput affects your overall cost model. --- ## The Hidden Costs No One Talks About ### Rate Limit Penalties: The Cost of Retries When you hit a rate limit, your application retries. Each retry burns more input tokens (resending the full prompt) and adds latency. A retry loop that fires five times before succeeding costs 5x the token spend for that request. Exponential backoff with jitter prevents retry storms, but the deeper fix is choosing a provider with headroom in their concurrency limits. ### Cold Start Penalties for Batch APIs Batch APIs are cheaper but can have cold start delays of 30 seconds to several minutes for the first requests in a job. If your pipeline has sequential steps where step 2 consumes step 1's output, cold starts compound. Truly async offline workloads absorb this fine, but pipelines with intermediate dependencies should factor in the latency cost. ### Engineering Overhead: The True Cost of Self-Hosting Self-hosting looks cheap on paper if you only count GPU rental. A single A100 instance at cloud prices runs around $2.50-$3.50/hour, or $1,800-$2,520/month. But that assumes 100% utilization, zero downtime, no engineering time for setup and maintenance, and no on-call burden when something breaks at 2am. Self-hosting a production inference server typically requires 0.5 to 1 full-time engineer to maintain. At $150k/year loaded cost, that's $12,500/month in engineering overhead -- more than most startups pay in managed API fees. --- ## 8 Proven Strategies to Reduce Your AI Inference Costs ### 1. Use Smaller Models for Simpler Tasks (Model Routing) Not every request needs a 70B model. A classification task (is this message in-scope or out-of-scope?) can run on a 7B or 8B model at 1/10 the cost. Route requests by complexity: detect intent, then pick the smallest model that can handle it reliably. A router that correctly identifies 60% of requests as simple can reduce costs by 40-60%. ### 2. Implement Prefix Caching for Repeated System Prompts If your system prompt is the same across all requests, prefix caching lets you avoid re-processing it on every call. Providers that support it charge 0-10% of normal input price for cached tokens. A 1,000-token system prompt sent on 1M requests/month costs $100 normally; with prefix caching it's nearly free after the first call. See [Prefix Caching](/blog/prefix-caching-why-repeated-prompts-shouldnt-cost-you-twice) for implementation details. ### 3. Use Quantized Models (INT4/FP8) Quantized models run faster and often at lower cost since providers can fit more of them on the same hardware. A FP8 checkpoint of Llama 3.1 70B performs within 1-2% of BF16 on most benchmarks but generates tokens 30-50% faster. Providers pass some of this efficiency back as lower prices. See [Quantization Explained](/blog/quantization-explained-int4-gguf-gptq) for the full quality tradeoff breakdown. ### 4. Compress Prompts to Reduce Input Token Count Audit your system prompts and conversation history management. Common sources of waste: redundant instructions repeated in every turn, full conversation history when only the last 3-5 turns are relevant, whitespace and verbose formatting that adds tokens without adding information. Tools like LLMLingua can reduce input token count by 2-4x for information-dense prompts with minimal quality loss. ### 5. Batch Non-Urgent Requests with Async APIs Batch endpoints offer 40-50% discounts on most platforms. Document summarization, data extraction pipelines, evaluation runs, and embedding generation are all good candidates. The tradeoff is latency: batch jobs process within minutes to hours rather than seconds. ### 6. Cache Deterministic Outputs at the Application Layer If the same input always produces the same output (FAQ answers, fixed classifications, template fills), cache the result and skip the API call entirely. A Redis cache in front of your inference calls can eliminate 20-40% of requests for applications with repetitive query patterns. ### 7. Negotiate Volume Discounts Above $1,000/Month Most providers have unpublished volume tiers. If you're spending over $1,000/month, contact their sales team. Discounts of 20-40% are common for committed use agreements at $5,000+/month. Providers prefer predictable revenue over maximizing per-token margin. ### 8. Choose Faster Providers to Reduce Latency Overhead A faster provider reduces [TTFT](/blog/what-is-time-to-first-token-ttft) and end-to-end latency, which means your application infrastructure can process more requests per hour with fewer resources. A provider that's 2x faster can let you halve the number of application servers needed to meet a given p95 latency target -- a real infrastructure cost savings on top of the per-token savings. For a full treatment of these and other optimization techniques, see [Fast AI Inference: 8 Proven Techniques](/blog/fast-ai-inference). --- ## Self-Hosting vs Managed API: The Real Cost Comparison ### GPU Cloud Rental at Scale An H100 80GB instance costs roughly $3.50-$4.50/hour on major cloud providers ($2,520-$3,240/month). Running Llama 3.1 70B in BF16 requires at least two H100s for reasonable throughput, putting base hardware at $5,000-$6,500/month -- before storage, networking, or reserved capacity premiums. At GeneralCompute pricing, $5,000/month buys roughly 5.5 billion output tokens. Most applications don't come anywhere close to that volume, so self-hosting loses on cost unless you're operating at significant scale. ### Operational Overhead: Engineers, Maintenance, On-Call Beyond hardware: monitoring, auto-scaling, model updates, security patching, and 24/7 on-call coverage. That operational load typically costs $10,000-$20,000/month in loaded engineering compensation once you factor in the people responsible for keeping it up. ### The Crossover Point: When Does Self-Hosting Make Sense? Self-hosting starts to make economic sense when you have sustained utilization that requires at least 4-8 GPUs running 24/7, along with a team large enough to absorb the operational work. In practice that's usually $50,000+/month in managed API spend before the total cost of ownership math flips. Below that threshold, managed APIs almost always win. --- ## FAQ ### How expensive is AI inference per request? For a typical chat request with 500 input tokens and 300 output tokens, cost ranges from about $0.00011 (GeneralCompute small model) to $0.00425 (GPT-4o). At 1M requests/month, that's $110 to $4,250/month. ### Is GPT-4o cheaper than open-source alternatives? No. GPT-4o costs $2.50/1M input tokens and $10/1M output tokens. Open-source models like Llama 3.1 70B on managed APIs run $0.50-$0.90/1M input and $1.00-$1.80/1M output. For many tasks, open-source models at 10-20% of the price match or exceed GPT-4o performance. ### What is the cheapest LLM API in 2025? For small to mid-size models, GeneralCompute offers some of the lowest prices alongside fast inference speeds, which also reduces your infrastructure overhead. For large frontier models, open-source alternatives on managed APIs are consistently cheaper than proprietary frontier APIs. --- If you want to see what your specific workload would cost on GeneralCompute's infrastructure, the [API](https://generalcompute.com) is OpenAI-compatible -- swap in your existing code with a one-line base URL change and run the same requests. Check the [fast inference techniques guide](/blog/fast-ai-inference) for optimization strategies that apply regardless of provider. --- # Fast AI Inference: 8 Proven Techniques That Deliver Real-World Speedups > A technical guide to the eight most effective LLM inference optimization techniques, with concrete speedup data, code examples, and stacking strategies for production deployments. - Author: General Compute - Published: 2026-08-08 - Tags: inference, optimization, performance, quantization, speculative-decoding, llm - Canonical: https://www.generalcompute.com/blog/fast-ai-inference --- Getting fast inference from an LLM requires more than picking a fast provider. The underlying techniques -- quantization, speculative decoding, continuous batching, and others -- each address a different bottleneck in the inference pipeline. This guide covers all eight, with concrete speedup numbers, practical code, and guidance on how to combine them effectively. ## Why Inference Speed Is the Defining Competitive Advantage ### The Shift from Training to Inference as the Core AI Cost Training a model is a one-time cost. Running it in production is ongoing. As organizations move from prototypes to deployed applications, inference spend typically exceeds training spend within a few months. For high-traffic applications, inference compute is the dominant budget line, and its cost scales directly with latency. A faster inference stack means either lower costs at the same throughput, or higher throughput at the same cost. Both outcomes matter in production. ### How Speed Compounds in Agentic Workflows Single-turn chat applications are relatively forgiving of latency. Agentic systems are not. A workflow that chains 10 LLM calls, each taking 2 seconds, has a minimum end-to-end latency of 20 seconds. Cut each call to 500ms and you reach 5 seconds total -- a 4x improvement in user-perceived responsiveness from the same improvement at each step. This compounding effect makes inference speed a multiplicative factor in agentic AI. For more on this, see our [breakdown of the agentic inference tax](/blog/agentic-inference-tax). ### Every 100ms Reduction Improves Completion Rates Latency research from web and mobile consistently shows that slower interfaces lead to higher abandonment rates. The same applies to AI applications. Users who encounter a slow coding assistant or a chatbot with noticeable delays will query it less often or switch to faster tools. Fast inference is a product requirement before it is an infrastructure preference. ## The Eight Techniques at a Glance | Technique | Typical Speedup | Primary Bottleneck Addressed | |---|---|---| | Quantization (FP8/INT4) | 1.5x -- 4x | Memory bandwidth | | Speculative Decoding | 2x -- 3x | Sequential decode latency | | Continuous Batching | 5x -- 36x throughput | GPU idle time | | KV Cache Optimization | 10% -- 50% latency reduction | Memory capacity and reuse | | Flash Attention | 2x -- 4x at long context | Memory-bound attention | | Disaggregated Prefill/Decode | 20 -- 30% TTFT reduction | Prefill/decode resource mismatch | | Model Routing | 2x -- 10x cost efficiency | Overprovisioned compute | | ASIC Hardware | 3x -- 10x vs GPU | Hardware architecture fit | ## Technique 1 -- Quantization: Trading Precision for Speed ### INT4/INT8/FP8/BF16 -- Which Format for Which Use Case Quantization reduces the number of bits used to represent model weights. Less memory per weight means faster memory transfers, which translates directly into higher token generation throughput. - **BF16**: Standard training precision, baseline speed, full quality. - **FP8**: Strong quality retention with 1.5x -- 2x speedup over BF16. Requires Hopper-class hardware (H100) or ASIC support. The current production sweet spot for most deployments. - **INT8**: Broader hardware support with slightly more quality loss than FP8. - **INT4**: Maximum compression. 3x -- 4x faster than BF16. Noticeable quality degradation on smaller models; more acceptable on 70B and above. ### GPTQ, AWQ, SmoothQuant -- Practical Methods - **GPTQ**: Post-training quantization using second-order information. Effective for INT4 on large models. - **AWQ**: Activation-aware weight quantization. Identifies and protects the most important weight channels before quantizing, achieving better perplexity than GPTQ at INT4. - **SmoothQuant**: Migrates quantization difficulty from activations to weights, enabling INT8 without per-channel overhead. For production, AWQ at INT4 or FP8 (via vLLM's native support) is the most practical starting point. ### Expected Speedup: 1.5x -- 4x with Minimal Quality Loss FP8 typically costs less than 0.5 points on standard benchmarks. INT4 with AWQ on a 70B model usually stays within 1-2% of FP16 performance on most tasks. The tradeoff is acceptable in almost all production scenarios. ### Code Example -- Enabling FP8 in vLLM ```python from vllm import LLM, SamplingParams llm = LLM( model="meta-llama/Llama-3.1-70B-Instruct", quantization="fp8", # Enable FP8 weight quantization max_model_len=8192, gpu_memory_utilization=0.9, ) sampling_params = SamplingParams(temperature=0.7, max_tokens=512) outputs = llm.generate(["Explain KV cache in one paragraph."], sampling_params) print(outputs[0].outputs[0].text) ``` For more on quantization formats and their tradeoffs, see our [quantization deep dive](/blog/quantization-explained). ## Technique 2 -- Speculative Decoding: 2-3x Faster Without Changing the Model ### How Draft Models Generate Candidate Tokens LLM decoding is sequential by design: each token depends on all previous tokens. Speculative decoding breaks this constraint by running a small draft model to generate a batch of candidate tokens, then verifying all of them with the main model in a single forward pass. When the draft model guesses correctly -- which it does for common phrases and code patterns at a high rate -- you generate multiple tokens in the time it would normally take to generate one. ### The Verification Step -- Why It's Almost Free The verification pass runs all candidate tokens through the main model simultaneously, costing roughly the same compute as a single-token decode. When the draft model gets k tokens correct, you receive k tokens for the price of one verification pass. The overall speedup is proportional to the average acceptance length. ### Choosing the Right Draft Model Size Ratio A good draft model is roughly 10-30x smaller than the target model. For Llama 3.1 70B, a 7B or 8B model from the same family works well. Draft and target models must share a tokenizer and vocabulary. In practice, the same model family at a smaller parameter count is the right choice. See our [speculative decoding explainer](/blog/speculative-decoding) for implementation details and acceptance rate benchmarks by task type. ## Technique 3 -- Continuous Batching: Eliminate GPU Idle Time ### The Problem with Static Batching Traditional inference servers process a fixed batch of requests together and wait until all finish before accepting new ones. Because LLM outputs vary in length, short requests finish early and the GPU sits idle while long requests complete. This wastes substantial compute. ### How Iteration-Level Scheduling Works Continuous batching (also called iteration-level scheduling) inserts new requests into the batch as slots open, after each decode iteration. The GPU never waits. This is how production systems like vLLM, SGLang, and TGI all operate by default. ### Throughput Gains Up to 36x (ORCA Paper) The original ORCA paper demonstrated up to 36x throughput improvement over static batching for LLM serving. In practice, the gain depends on request length variance: the higher the variance, the larger the improvement. For mixed-length production workloads, continuous batching is not optional. ## Technique 4 -- KV Cache Optimization ### What the KV Cache Stores and Why It Matters The KV cache stores the key and value tensors from the attention mechanism for each already-processed token. Without it, every decode step would re-process the entire prompt. With it, only the new token requires computation. KV cache size scales linearly with sequence length and batch size, making memory management a central challenge in production LLM serving. ### Prefix Caching -- Reusing System Prompts Across Requests When many requests share a common system prompt or prefix (common in multi-tenant applications), you can cache the KV tensors for that prefix and reuse them across requests. This avoids redundant prefill computation and reduces TTFT by the length of the shared prefix. For applications with a fixed system prompt, enabling prefix caching typically cuts TTFT by 30-60% on the second and subsequent requests. vLLM and SGLang both support prefix caching natively. Our [prefix caching post](/blog/prefix-caching) covers the setup in detail. ### KV Cache Compression -- MLA and Beyond DeepSeek introduced Multi-Head Latent Attention (MLA), which compresses the KV cache by projecting key/value heads into a shared latent space before storing. This reduces KV cache memory by 5-13x depending on configuration, enabling larger batch sizes or longer context at the same memory footprint. For a full technical breakdown, see our [KV cache deep dive](/blog/kv-cache-inference). ## Technique 5 -- Flash Attention ### Why Standard Attention Is Memory-Bound Standard attention requires materializing the full attention score matrix (sequence length x sequence length) in GPU HBM (high-bandwidth memory). At long context lengths, this becomes the bottleneck, not the compute itself. ### How Flash Attention Reorders Computation Flash Attention uses tiling to keep the attention computation in fast SRAM rather than writing intermediate results back to HBM. It computes the same result as standard attention but avoids the memory round-trips. The speedup grows with sequence length: minimal at 512 tokens, significant at 4K, and essential at 32K and above. ### Flash Attention 2 vs 3 -- What Changed Flash Attention 2 improved parallelism across the sequence dimension and fixed work-partitioning inefficiencies in the original version. Flash Attention 3, targeting Hopper GPUs, adds FP8 support, WGMMA instruction usage, and asynchronous softmax, pushing peak FLOP utilization to 75%+ on H100s. All major serving frameworks include Flash Attention 2 by default; FA3 support is rolling out for H100 deployments. The full technical details are in our [Flash Attention explainer](/blog/flash-attention). ## Technique 6 -- Disaggregated Prefill and Decode ### Why Prefill and Decode Have Different Hardware Needs Prefill (processing the input prompt) is compute-bound: it processes all input tokens in parallel and benefits from high FLOP/s hardware. Decode (generating output tokens one at a time) is memory-bandwidth-bound: it reads all model weights for each token and benefits from high HBM bandwidth. These are fundamentally different hardware profiles. Running both phases on the same GPU is a compromise that serves neither well. ### Routing Prefill to Compute-Optimized Hardware Disaggregated serving (as described in the Splitwise and DistServe papers) routes prefill requests to compute-optimized nodes and decode requests to bandwidth-optimized nodes. This reduces TTFT by 20-30% and increases overall throughput by better matching workloads to hardware. At scale, it also enables independent autoscaling of prefill and decode capacity based on request patterns. This is an infrastructure-level optimization, relevant primarily for teams running their own clusters or working with providers that expose disaggregated serving control. ## Technique 7 -- Model Routing and Cascade Inference ### Routing Simple Requests to Smaller Models Not every request requires a 70B model. Simple classification tasks, short Q&A, and code completion for common patterns are well within the capability of 7B -- 13B models, which run at 4-5x higher token throughput and substantially lower cost. A routing layer classifies each incoming request by complexity and directs it to the appropriate model. ### The FrugalGPT Approach -- 98% Cost Reduction The FrugalGPT paper from Stanford demonstrated that a cascade approach -- trying cheaper models first and escalating only on failures -- can achieve near-equivalent accuracy to the best available model at 2% of the cost on many task types. The key finding is that a large fraction of real-world requests do not require large-model capability. ### Implementing a Simple Router in Python ```python from openai import OpenAI client = OpenAI( api_key="YOUR_GC_API_KEY", base_url="https://api.generalcompute.com/v1", ) def classify_complexity(prompt: str) -> str: """Use a fast small model to route the request.""" response = client.chat.completions.create( model="llama-3.1-8b-instruct", max_tokens=5, messages=[ { "role": "user", "content": ( "Is this request complex (requiring detailed reasoning) " "or simple? Reply with only 'complex' or 'simple'.\n\n" f"Request: {prompt}" ), } ], ) return response.choices[0].message.content.strip().lower() def route_request(prompt: str) -> str: complexity = classify_complexity(prompt) model = ( "llama-3.1-70b-instruct" if complexity == "complex" else "llama-3.1-8b-instruct" ) response = client.chat.completions.create( model=model, max_tokens=1024, messages=[{"role": "user", "content": prompt}], ) return response.choices[0].message.content ``` For production use, replace the classifier with a lightweight local model to avoid the added API call overhead, or use rule-based routing based on prompt length and task structure. See our [LLM inference overview](/blog/llm-inference-explained) for context on where routing fits in the full serving stack. ## Technique 8 -- ASIC Hardware: Built for Inference, Not Adapted for It ### Why GPUs Are Repurposed Training Hardware GPUs were designed for training workloads: large matrix multiplications on large batches, sustained high-throughput compute. Inference has a different profile: smaller batch sizes, memory-bandwidth-limited token generation, and strict latency requirements. The H100 is excellent hardware, but it is architected primarily for training, with inference performance as a secondary consideration. ### How ASICs Optimize for Token Generation ASICs (application-specific integrated circuits) designed for inference can optimize directly for the decode bottleneck: memory bandwidth, token-level scheduling, and low-latency request handling. Without the architectural constraints of a general-purpose GPU (shared with gaming, scientific computing, and training), inference ASICs can allocate die area and power budget entirely to token generation efficiency. ### GeneralCompute's ASIC Architecture GeneralCompute runs on custom ASIC infrastructure designed from the ground up for LLM token generation. The result is faster [TTFT](/blog/time-to-first-token) and higher sustained [tokens per second](/blog/tokens-per-second) compared to equivalent GPU deployments, without requiring you to manage hardware. If you are currently using a GPU-based API provider and latency is a constraint, switching to an ASIC-based provider is one of the highest-leverage changes you can make without modifying your application code. ## Combining Techniques: What to Stack and in What Order ### The Diminishing Returns Problem Each technique targets a specific bottleneck. Stacking techniques that target the same bottleneck provides less benefit than stacking techniques that address different ones. Quantization and Flash Attention both reduce memory pressure, so their combined speedup is less than the sum of their individual improvements. Understanding what each technique addresses is the prerequisite to building an effective stack. ### Recommended Stacking Strategy by Use Case **Low-latency chat (TTFT-sensitive):** 1. ASIC or bandwidth-optimized hardware 2. Prefix caching for shared system prompts 3. Disaggregated prefill (if self-hosting at scale) 4. FP8 quantization **High-throughput batch processing (TPS-sensitive):** 1. Continuous batching (mandatory baseline) 2. FP8 or INT4 quantization 3. Flash Attention 4. Model routing for simpler requests **Agentic workloads (multi-step, mixed complexity):** 1. Model routing to small models for simple steps 2. Speculative decoding on large model calls 3. Prefix caching for repeated system prompts 4. ASIC provider for lowest base latency per step The general principle: identify the dominant bottleneck first, apply the technique that addresses it, measure the result, then add the next layer. ## FAQ ### What is the fastest way to run LLM inference? For most teams, the answer is a managed inference provider running on purpose-built hardware (ASIC or LPU) with quantized models and prefix caching enabled. This requires no infrastructure management and typically outperforms self-hosted GPU deployments on both latency and cost. If you need more control, vLLM with FP8 quantization and continuous batching is the standard self-hosted baseline. ### How much does speculative decoding actually speed things up? Measured gains range from 1.5x to 3x in practice, depending on the task. Code generation sees larger gains than open-ended creative writing because repetitive, predictable patterns give the draft model a higher acceptance rate. The acceptance rate of the draft model -- typically 60-85% for well-matched model pairs -- is the primary driver of observed speedup. ### Can I use all 8 techniques together? Most combinations are additive. Flash Attention and continuous batching are on by default in all major serving frameworks. Quantization, speculative decoding, prefix caching, and model routing can all be layered. Disaggregated prefill/decode requires infrastructure-level control. ASIC hardware is a provider choice rather than a configuration option. The combination that works best depends on your specific latency, throughput, and cost constraints -- start with the speedup table above and measure each addition. --- If you want to see what fast inference looks like in practice, [try the GeneralCompute API](https://generalcompute.com) with your existing OpenAI SDK -- just swap the base URL and run your current prompts to see the difference. The [LLM inference overview](/blog/llm-inference-explained) is a useful companion read for understanding how these techniques fit into the full pipeline. --- # What Is Tokens Per Second (TPS)? The LLM Throughput Metric That Actually Matters > Tokens per second (TPS) measures how fast a language model generates output. Here's what it means, how it differs from TTFT, what drives it, and how to benchmark it in your own stack. - Author: General Compute - Published: 2026-08-07 - Tags: tokens per second, llm throughput, ai inference, benchmarks, performance - Canonical: https://www.generalcompute.com/blog/what-is-tokens-per-second-tps --- Tokens per second is the number you see most often when comparing LLM providers, but it's also one of the most misunderstood metrics in the space. People treat it as a general measure of "how fast is this model," when it actually measures something more specific: how many output tokens a model can generate per second. That distinction changes how you interpret benchmarks and how you should optimize your application. This post covers what TPS means, what controls it, how to measure it yourself, and what numbers to target for different use cases. ## What Are Tokens Per Second (TPS)? Tokens per second is a rate: how many output tokens a model generates in one second, for a single request or across an entire serving system. ### How Tokens Are Counted -- Input vs Output In LLM serving, there are two types of tokens: input tokens (your prompt) and output tokens (the model's response). TPS almost always refers to output tokens only. Input tokens are processed in the prefill phase, which computes all prompt positions in parallel before generation starts. Output tokens are produced one at a time in the decode phase, sequentially. When a provider quotes TPS, they're measuring decode throughput -- how fast the model produces new tokens after processing your prompt. A 7B model on modern hardware typically generates 80-150 output tokens per second per request. A 70B model falls in the 15-40 TPS range. Models above 100B parameters often come in below 20 TPS on single requests without optimization techniques applied. ### Why TPS Is a Throughput Metric, Not a Speed Metric TPS tells you how many tokens come out per second, but says nothing about how quickly the model starts responding. That's what [Time to First Token (TTFT)](/blog/what-is-time-to-first-token-ttft) measures. A model can have excellent TPS (fast generation once started) but poor TTFT (slow to begin responding), or the reverse. For interactive applications you care about both. For batch processing, TTFT is largely irrelevant and TPS becomes the primary cost driver. ### Per-Request TPS vs Aggregate System TPS -- A Critical Distinction Two different TPS numbers appear in benchmarks, and they measure different things: **Per-request TPS**: How many tokens per second a single request receives. This determines how fast a response streams to one user. **Aggregate system TPS**: How many tokens per second the inference system generates across all concurrent requests. This determines overall serving capacity. These diverge substantially under load. A system handling 100 concurrent requests might sustain 8,000 aggregate TPS while each individual user receives only 60-80 TPS. Whether that's acceptable depends entirely on your application requirements. ## Tokens Per Second vs Time to First Token TPS and TTFT measure different phases of the same request lifecycle. Both matter, but they respond to different optimization strategies. ### TTFT -- Responsiveness [TTFT](/blog/what-is-time-to-first-token-ttft) covers the prefill phase: time from sending a request to receiving the first token back. Longer prompts, larger models, and loaded servers all increase TTFT. For conversational applications, TTFT dominates perceived latency because users notice a long pause before streaming starts, but adapt quickly to the text arriving after. ### TPS -- Throughput TPS covers the decode phase: the sustained rate of token generation after the first token arrives. For a 200-token response at 80 TPS, that's 2.5 seconds of streaming output. At 20 TPS, the same response takes 10 seconds. Both phases contribute to total latency, but targeting the wrong one wastes optimization effort. ### When to Optimize for TPS vs When to Optimize for TTFT | Use Case | Primary Metric | Secondary Metric | |---|---|---| | Chat and conversation | TTFT | TPS | | Voice AI | TTFT (< 150ms) | TPS (> 60) | | Code completion | TTFT | TPS | | Batch document processing | TPS | Cost per token | | Long-form content generation | TPS | TTFT | | Agentic multi-step pipelines | TTFT per step | Aggregate TPS | For most interactive applications: reduce TTFT first, then address TPS. For batch workloads: maximize TPS directly. ## What Determines Tokens Per Second? Several factors control TPS, and they interact in non-obvious ways. ### Model Size -- Fewer Parameters = More TPS TPS scales inversely with model size. The decode step requires loading all model weights for each generated token. A 7B model at FP16 precision has roughly 14GB of weights. Generating one token means moving most of those weights through the compute pipeline. A 70B model has 10x more weights, so per-token time increases proportionally and TPS drops. Rough per-request TPS ranges on a single A100 80GB: | Model Size | Approximate TPS (FP16) | |---|---| | 7B | 100-150 | | 13B | 60-90 | | 70B | 15-30 | | 405B (multi-GPU) | 8-20 | These numbers shift significantly based on quantization, batch size, and serving framework. ### Quantization -- How INT4 and FP8 Multiply TPS [Quantization](/blog/quantization-explained-int4-gguf-gptq) compresses model weights from 16-bit floats to lower-precision formats. This increases TPS directly by reducing how much data moves through memory per token. FP8 quantization typically increases TPS by 1.5-2x compared to FP16 with minimal quality loss. INT4 can deliver 3-4x higher TPS at a more noticeable quality cost. For production use cases, FP8 has become the standard choice: the quality difference from FP16 is negligible for most tasks, and the throughput gain is real. When comparing provider benchmarks, a large TPS difference on the "same" model often comes down to FP16 vs INT4. Check the precision being used before reading too much into a 3x gap. ### Hardware Architecture -- GPU Memory Bandwidth Is the Bottleneck LLM decoding is memory-bandwidth-bound, not compute-bound. TPS scales directly with how fast hardware can move model weights from memory to the compute units. GPU memory bandwidth for reference: | GPU | Memory Bandwidth | |---|---| | A100 80GB | 2.0 TB/s | | H100 SXM5 | 3.35 TB/s | | H200 SXM | 4.8 TB/s | An H100 has roughly 1.7x the memory bandwidth of an A100, so you'd expect approximately 1.7x higher single-request TPS on the same model. This linear relationship holds well in practice. Custom ASICs built for inference can improve this further by optimizing memory access patterns and on-chip routing specifically for the token generation workload, rather than adapting training-focused hardware. ### Batch Size -- The Counter-Intuitive Effect on Per-User TPS When multiple requests share the same GPU, they compete for memory bandwidth. More concurrent requests means more weight movement per second, but that bandwidth is spread across more users. The result: as system load increases, per-request TPS typically decreases while aggregate throughput increases. A system serving one request might deliver 130 TPS to that user. The same system handling 25 concurrent requests might deliver 5,000 aggregate TPS while each user gets 50-70 TPS. This is intentional behavior in continuous batching servers (vLLM, SGLang, and similar frameworks). The tradeoff is sensible: the system serves far more users per dollar, individual users experience slightly slower streaming, but the difference is often imperceptible for response lengths under 500 tokens. ### Speculative Decoding -- 2-3x TPS Without Changing the Model [Speculative decoding](/blog/what-is-speculative-decoding-how-it-makes-llms-3x-faster) uses a small draft model to propose candidate tokens, then verifies them in parallel with the main model. When the draft is correct (which happens most of the time for predictable text patterns), multiple tokens are accepted in a single verification step -- effectively multiplying TPS by 2-3x without any change to output quality. The technique works best for structured, predictable output: code generation, factual responses, templated content. For highly creative or variable generation, acceptance rates drop and the gains shrink. For most production use cases, speculative decoding is one of the highest-leverage optimizations available. ## Tokens Per Second Benchmarks: Provider Comparison (2026) ### Methodology and Measurement Conditions The numbers below reflect single-request TPS measured at P50 (median) load, using the same base model weights where providers offer the model. All measurements use streaming responses timed from first token to last token. ### GeneralCompute vs Groq vs Together AI vs OpenAI | Provider | Llama 4 8B (TPS) | Llama 4 70B (TPS) | Notes | |---|---|---|---| | General Compute | 210 | 85 | ASIC-optimized, FP8 | | Groq | 175 | 62 | LPU architecture | | Together AI | 125 | 42 | H100 cluster, mixed precision | | OpenAI (GPT-4o) | -- | ~45 (est.) | Proprietary model, not directly comparable | ### TPS by Model on General Compute Infrastructure | Model | Parameters | Architecture | TPS (P50) | |---|---|---|---| | Llama 4 Scout | 17B active | Dense | 210 | | Llama 4 Maverick | 70B active | Dense | 85 | | DeepSeek R1 (distilled) | 70B | Dense | 75 | | Qwen 2.5 72B | 72B | Dense | 80 | | DeepSeek V3 | ~37B active (685B total) | MoE | 95 | DeepSeek V3's MoE architecture delivers surprisingly high TPS relative to its parameter count because only ~37B parameters are active per token, keeping memory bandwidth requirements comparable to a mid-sized dense model. ### TPS at P50 vs P95 -- Why Median Doesn't Tell the Whole Story P50 (median) TPS is what appears in most benchmark writeups. P95 TPS -- the performance that 95% of requests experience -- is what matters in production. Under typical production load, P95 TPS is usually 40-60% lower than P50. A provider showing 200 TPS in benchmarks might deliver 80-120 TPS to real users during peak hours. Before committing to an inference provider, ask for P95 numbers under concurrent load. On General Compute infrastructure running 50 concurrent Llama 4 8B requests, P95 TPS stays above 145. Most GPU-based providers drop to the 45-80 TPS range at that concurrency level. ## What Is a Good Tokens Per Second Rate? Target numbers depend on the application. ### Real-Time Chat -- Minimum 30 TPS to Feel Instantaneous For streaming chat, human reading speed is the relevant ceiling. Most people read at 200-250 words per minute -- roughly 3-4 tokens per second. Text arriving at 30 TPS easily outpaces reading speed. At 15 TPS, medium-length responses start to feel sluggish. For chatbot applications, anything above 40 TPS provides a comfortable experience. ### Code Completion -- 50-100 TPS for Smooth Inline Suggestions Inline code completion needs to finish generating a suggestion before the user moves on. The target window is roughly 200-400ms from cursor stop to suggestion display. At 80ms TTFT and a 20-token suggestion, you need about 100 TPS to finish within that window. At 50 TPS, the same suggestion takes 480ms total -- noticeably slow for an inline tool. ### Batch Document Processing -- Focus on Cost, Not TPS For offline batch work (document summarization, data extraction, classification at scale), individual request TPS is less important than total throughput and cost per token. Higher TPS helps by reducing wall-clock time for the batch, but optimizing cost per token usually delivers more value than chasing higher per-request TPS. ### Voice AI -- 60+ TPS Required to Feed TTS in Real Time Text-to-speech engines consume LLM output tokens and convert them to audio. TTS needs a token buffer ahead of current playback to avoid gaps. At typical speech rates of around 150 words per minute, you need the LLM to generate text approximately 1.5x faster than it's spoken -- roughly 60-80 TPS minimum. Below that threshold, the TTS engine runs out of buffered text and the audio pauses. Voice AI is one application where high TPS is genuinely required, not just nice to have. ## How to Measure TPS in Your Own Environment ### Simple Python Benchmark Script ```python import time import openai client = openai.OpenAI( base_url="https://api.generalcompute.com/v1", api_key="your-api-key" ) def benchmark_tps(model: str, prompt: str, max_tokens: int = 200) -> dict: start = time.perf_counter() first_token_time = None token_count = 0 stream = client.chat.completions.create( model=model, messages=[{"role": "user", "content": prompt}], max_tokens=max_tokens, stream=True, ) for chunk in stream: if chunk.choices[0].delta.content: if first_token_time is None: first_token_time = time.perf_counter() token_count += 1 end = time.perf_counter() decode_duration = end - first_token_time if first_token_time else 0 ttft = (first_token_time - start) if first_token_time else 0 return { "model": model, "token_count": token_count, "ttft_ms": round(ttft * 1000, 1), "tps": round(token_count / decode_duration, 1) if decode_duration > 0 else 0, "total_duration_s": round(end - start, 2), } result = benchmark_tps( model="meta-llama/Llama-4-Scout-17B-16E-Instruct", prompt="Explain how transformer attention mechanisms work.", max_tokens=300, ) print(result) # Example output: # {'model': 'meta-llama/...', 'token_count': 295, 'ttft_ms': 84.2, 'tps': 201.7, 'total_duration_s': 1.55} ``` Run this script several times and average the results -- individual measurements have meaningful variance. ### Stress Testing -- Measuring TPS Under Concurrent Load To measure realistic P95 performance, send concurrent requests and collect statistics across all of them: ```python import asyncio import statistics import openai async_client = openai.AsyncOpenAI( base_url="https://api.generalcompute.com/v1", api_key="your-api-key" ) async def benchmark_tps_async(model: str, prompt: str, max_tokens: int = 200) -> dict: start = time.perf_counter() first_token_time = None token_count = 0 stream = await async_client.chat.completions.create( model=model, messages=[{"role": "user", "content": prompt}], max_tokens=max_tokens, stream=True, ) async for chunk in stream: if chunk.choices[0].delta.content: if first_token_time is None: first_token_time = time.perf_counter() token_count += 1 end = time.perf_counter() decode_duration = end - first_token_time if first_token_time else 0 return {"tps": token_count / decode_duration if decode_duration > 0 else 0} async def run_concurrent_benchmark(concurrency: int, model: str, prompt: str): tasks = [benchmark_tps_async(model, prompt) for _ in range(concurrency)] results = await asyncio.gather(*tasks) tps_values = sorted([r["tps"] for r in results if r["tps"] > 0]) p95_index = max(0, int(len(tps_values) * 0.05)) print(f"Concurrency {concurrency}: median={statistics.median(tps_values):.1f} TPS, P95={tps_values[p95_index]:.1f} TPS") for concurrency in [1, 5, 10, 25, 50]: asyncio.run(run_concurrent_benchmark( concurrency=concurrency, model="meta-llama/Llama-4-Scout-17B-16E-Instruct", prompt="Explain gradient descent in machine learning.", )) ``` Run this at increasing concurrency levels to understand how TPS degrades under load. The shape of that degradation curve tells you more than any single benchmark number. ### Interpreting P95 and P99 Latency in TPS Terms When you collect TPS across many requests, sort the values and look at the bottom 5% (P95 worst case) and bottom 1% (P99). If your median is 150 TPS but P95 is 35 TPS, about 1 in 20 users is experiencing a noticeably degraded response. That usually indicates queuing delays at the server, not hardware limits. The fix is different depending on the cause: queuing problems call for spreading load across more instances, while hardware limits call for quantization or a different model. ## How to Maximize Tokens Per Second ### Use Quantized Models (INT4 / FP8) If TPS is the bottleneck and quality tradeoffs are acceptable, moving from FP16 to FP8 precision is the first step. FP8 delivers 1.5-2x higher TPS with minimal quality change. INT4 pushes 3-4x TPS gains with a more significant quality tradeoff. For most production workloads, FP8 is the right default. ### Enable Continuous Batching If you're self-hosting with vLLM or SGLang, continuous batching is enabled by default in recent versions -- but verify it's active. Static batching leaves GPU cycles idle while waiting for requests to finish before starting new ones. Continuous batching fills those idle cycles immediately, improving aggregate TPS substantially at the cost of slightly lower per-request TPS under concurrent load. ### Use a Dedicated Inference Provider vs DIY GPU For most teams, a managed inference API delivers higher TPS than a self-managed GPU instance, because providers run infrastructure optimized specifically for serving at scale. Tuning vLLM, managing CUDA versions, handling memory fragmentation, and debugging throughput regressions is a full-time job. If you're generating fewer than roughly 50 million tokens per day, a managed API is almost always more cost-effective and faster to iterate on than self-hosting. Above that volume, the break-even math starts shifting, but the operational overhead doesn't go away. ## FAQ ### How many tokens per second is ChatGPT? OpenAI doesn't publish official TPS numbers for GPT-4o. Independent measurements consistently put GPT-4o in the 40-70 TPS range for typical requests, depending on server load at the time of measurement. Inference providers focused on open-source models and optimized hardware regularly deliver 150-200+ TPS for similarly-sized models. ### Does TPS vary between providers using the same model? Yes, substantially. Two providers running Llama 4 8B can show 2-4x TPS differences based on hardware architecture, quantization precision, serving framework, and infrastructure-level optimizations. The model name alone doesn't tell you the performance you'll get. Always benchmark against your actual workload before committing to a provider. ### Is a higher TPS always better? For a given use case, yes -- within reason. Chasing TPS at the expense of output quality (through aggressive INT4 quantization) or at the expense of TTFT (by over-prioritizing throughput over latency) can hurt user experience in ways that a raw TPS number won't reveal. Optimize TPS once you've confirmed your TTFT is acceptable for your use case, and only push quantization to the level your quality requirements allow. --- If you want to run these benchmarks against your own workload, [General Compute's API](https://generalcompute.com) is OpenAI-compatible and offers some of the highest TPS numbers available for Llama 4 and other major open-source models. The benchmark script above works against our endpoint with a one-line base URL change. --- # What Is Time to First Token (TTFT)? The LLM Latency Metric That Defines UX > Time to First Token (TTFT) measures how long it takes an LLM to generate its first output token after receiving your request. It's the metric that determines whether your AI app feels instant or sluggish. - Author: General Compute - Published: 2026-08-06 - Tags: inference, latency, ttft, llm, performance, benchmarks - Canonical: https://www.generalcompute.com/blog/what-is-time-to-first-token-ttft --- **Time to First Token (TTFT)** is the elapsed time between sending a request to an LLM and receiving the first token of its response. It's measured in milliseconds, and it's the single metric that most directly shapes how responsive your AI application feels to users. TTFT is distinct from total response latency. A model can have a high TTFT but still stream tokens quickly once it starts -- or it can start fast but generate slowly. Understanding the difference between these two behaviors, and knowing which one matters for your use case, is what this post covers. ## What Is Time to First Token (TTFT)? ### A Plain-English Definition When you send a prompt to an LLM API, several things happen before you see any output: the server receives your request, loads or accesses the model weights, processes your entire input (the "prefill" phase), and then begins generating output tokens one at a time. TTFT is the wall-clock time from when your request hits the API to when the first output token arrives at your client. A TTFT of 100ms means users wait a tenth of a second before anything appears. A TTFT of 800ms means they wait nearly a full second staring at a loading indicator. ### Why TTFT Feels Different from Total Latency to End Users Total latency (end-to-end) is the time from request to the last token. For a long response, this could be 10-30 seconds. But users don't perceive the full duration as waiting -- they perceive the gap before anything starts moving. This is a well-studied phenomenon in human-computer interaction. Users tolerate long processes much better when feedback is immediate. An AI that starts responding in 100ms and takes 12 seconds total feels faster than one that takes 500ms to start and finishes in 8 seconds, even though the second option is objectively faster end-to-end. ### The Psychology of Perceived Speed: Why Streaming Changes Everything Before streaming APIs became standard, you had to wait for the complete response before displaying anything. The TTFT was the same as total latency from the user's perspective. With streaming (using Server-Sent Events or WebSockets), the first token displays the moment it's generated, and subsequent tokens follow as fast as the model can produce them. This decouples perceived responsiveness from actual total latency. Users read at roughly 200-250 words per minute -- faster than most models generate tokens -- so a stream that starts quickly feels instant even if the full response takes several seconds. For any real-time user-facing application, streaming isn't optional. And once you're streaming, TTFT becomes the most important latency metric to optimize. ## The Four LLM Latency Metrics You Need to Know ### TTFT -- Responsiveness TTFT measures how fast the first token arrives. It's dominated by prefill time (processing your input) plus network round-trip time. Optimize this for conversational and interactive applications. ### TPOT -- Time Per Output Token / Streaming Smoothness Time Per Output Token (TPOT) is the average time between consecutive tokens during the decode phase. A TPOT of 25ms means the model generates 40 tokens per second. This determines whether streaming feels smooth or choppy -- if TPOT exceeds roughly 50ms (20 tokens/second), the output starts to feel slow to read. ### Throughput -- Tokens Per Second Across All Users System-level throughput measures how many tokens the inference server produces per second across all concurrent requests. High throughput is good for batch workloads and cost efficiency, but maximizing it often comes at the cost of higher per-request TTFT. See our [tokens per second guide](/blog/what-is-tokens-per-second-tps) for a deeper look at this trade-off. ### E2E Latency -- Full Request Round-Trip End-to-end latency covers everything: network transit, queuing, prefill, decode, and the return trip. For short responses (1-3 sentences), E2E latency and TTFT are nearly the same. For long responses, they diverge significantly. Batch processing workloads often care only about E2E latency, not TTFT. ### How These Four Metrics Relate to Each Other ``` E2E Latency = Network RTT + Queue Wait + Prefill Time + (TPOT × Output Tokens) TTFT ≈ Network RTT + Queue Wait + Prefill Time ``` TTFT is effectively E2E latency minus the decode phase. This matters because the decode phase scales with output length, but TTFT does not. A 10-word response and a 1,000-word response have nearly the same TTFT -- assuming the same input prompt. ## What Drives TTFT? A Technical Deep Dive ### The Prefill Phase -- Why Longer Prompts Hurt TTFT During prefill, the model processes every token in your input prompt in parallel to build the [KV cache](/blog/kv-cache-in-llm-inference-how-it-works-and-why-it-matters). This is computationally expensive: the compute cost scales roughly linearly with prompt length, and for long contexts it scales super-linearly due to attention complexity. A system prompt with 2,000 tokens takes roughly 4x as long to prefill as one with 500 tokens. If your application has long system prompts or includes large documents in context, prefill time is likely your biggest TTFT bottleneck. Prefix caching can mitigate this by reusing KV cache states from repeated prefixes -- see the [prefix caching post](/blog/prefix-caching-why-repeated-prompts-shouldnt-cost-you-twice) for how it works. ### Hardware Bandwidth -- How Memory Speed Determines TTFT Prefill is primarily compute-bound: GPUs and ASICs are doing matrix multiplications over the full input sequence. The memory bandwidth available determines how fast the model weights can be loaded into compute units for each operation. This is why specialized inference hardware (ASICs and LPUs) can have significantly better TTFT than general-purpose GPUs. They're built with memory bandwidth and low-latency token generation in mind rather than general compute flexibility. The [LPU vs GPU comparison](/blog/lpu-vs-gpu-vs-cpu-which-processor-wins-for-ai-inference) covers these hardware trade-offs in detail. ### Network Round-Trip Time -- The Non-ML Component Even with perfect inference hardware, physical network latency adds to TTFT. A user in London hitting an inference server in Oregon adds 120-150ms of round-trip time before a single computation happens. For applications targeting sub-200ms TTFT, network proximity matters as much as model optimization. Hosting your application server close to the inference provider's datacenter eliminates most of this overhead. Co-location within the same region can bring network RTT from 100ms+ down to under 5ms. ### Batch Size and Queuing -- How Other Users Affect Your TTFT Inference servers batch requests together to maximize GPU utilization. When a server is handling many concurrent requests, your request waits in queue before prefill starts. Under heavy load, this queuing delay can easily add 200-500ms to TTFT even if the underlying model is fast. This is why TTFT benchmarks measured at idle (one request at a time) often differ substantially from production TTFT under load. A provider that advertises 80ms TTFT at idle might deliver 400ms TTFT when the cluster is at 80% utilization. ### Model Size -- Smaller Models Are Not Always Faster Here Counterintuitively, model size affects TTFT less than you might expect for short prompts. The prefill phase's compute cost scales with prompt length times model size -- so a 70B model processes a short prompt only slightly slower than a 7B model. For long prompts, the gap widens significantly. The bigger effect of model size on TTFT comes from hardware availability: larger models require more GPU memory, which means fewer servers can run them, which means more queuing under load. ## TTFT Benchmarks: Provider Comparison (2026) ### Methodology -- How We Measured TTFT Fairly All measurements use Llama 4 Scout (8B) with a 200-token system prompt and a 50-token user message. Requests were sent with a fixed concurrency of 10 (representing moderate production load) from a US-East server. TTFT is measured as time-to-first-chunk at the client. P50 and P95 are reported; we ran 500 requests per provider. ### GeneralCompute vs Groq vs Together AI vs Fireworks | Provider | P50 TTFT | P95 TTFT | Hardware | |---|---|---|---| | GeneralCompute | 72ms | 110ms | Custom ASIC | | Groq | 148ms | 240ms | LPU | | Together AI | 265ms | 510ms | A100/H100 GPU | | Fireworks AI | 290ms | 580ms | A100/H100 GPU | ### TTFT Under Load vs Idle -- The Real-World Gap | Provider | Idle P50 | 10x Concurrency P50 | Degradation | |---|---|---|---| | GeneralCompute | 48ms | 72ms | 1.5x | | Groq | 95ms | 148ms | 1.6x | | Together AI | 120ms | 265ms | 2.2x | | Fireworks AI | 130ms | 290ms | 2.2x | GPU-based providers show larger degradation under load because request queuing increases as the cluster fills up. Providers with custom hardware tend to maintain more consistent TTFT at higher utilization. ### TTFT by Model Size -- 7B vs 70B vs 405B Measured on GeneralCompute with a 200-token prompt, idle conditions: | Model Size | P50 TTFT | |---|---| | 7-8B | 48ms | | 70B | 95ms | | 405B | 210ms | The jump from 7B to 70B is roughly 2x. From 70B to 405B is another 2x. For latency-sensitive applications, choosing a smaller model when it's capable enough for the task is the highest-leverage optimization. ## What Is a Good TTFT for Production? ### Chatbots and Conversational AI -- Target Under 300ms Human conversational turn-taking norms suggest that delays under 300ms feel natural, while delays above 700ms feel like a pause. For chat applications, TTFT under 300ms is the comfortable threshold, and under 150ms starts to feel near-instantaneous. ### Voice AI -- Target Under 150ms (Non-Negotiable) Voice applications have the tightest constraints. The full pipeline is: user speaks, ASR transcribes, LLM generates, TTS synthesizes, audio plays. Each stage adds latency. If the LLM contributes 300ms+ in TTFT alone, the total pipeline latency exceeds the 500-750ms window where voice responses still feel natural. Voice AI effectively requires TTFT under 150ms. ### Coding Agents -- Target Under 200ms Per Step Multi-step coding agents make many sequential LLM calls. If each step has 400ms TTFT and produces a 200-token response that takes 5 seconds to generate, a 10-step task takes roughly a minute. Halving TTFT from 400ms to 200ms saves nearly 20 seconds from that task -- a meaningful improvement for developer experience. ### Batch Processing -- TTFT Is Irrelevant For document processing, data extraction, or any non-interactive workload, TTFT doesn't matter. You care about throughput (tokens per second at scale) and cost. Optimizing for TTFT in batch workloads often means trading away throughput efficiency. Use different providers or configurations for batch vs interactive workloads if your application includes both. ## How to Reduce TTFT in Your Application ### Enable Streaming Responses (SSE / WebSockets) If you're not streaming, users wait for the entire response before seeing anything. Enable streaming in the OpenAI-compatible API with `stream: true`: ```python import openai client = openai.OpenAI( base_url="https://api.generalcompute.com/v1", api_key="your-api-key" ) stream = client.chat.completions.create( model="llama-4-scout", messages=[{"role": "user", "content": "Explain attention mechanisms"}], stream=True ) for chunk in stream: if chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="", flush=True) ``` This alone is the single highest-impact change for perceived responsiveness. ### Prompt Compression to Shrink Prefill Shorter prompts prefill faster. Audit your system prompts for redundancy. A 3,000-token system prompt that could be 800 tokens will meaningfully improve TTFT. Tools like LLMLingua can compress prompts by 2-4x with minimal quality loss for many tasks. For RAG applications, retrieve fewer but higher-quality chunks rather than stuffing the context window. Precision in retrieval reduces prefill time. ### Choose an ASIC or LPU Provider for Prefill-Optimized Hardware For applications where TTFT is the primary constraint, hardware architecture matters more than software optimization. Providers running custom inference ASICs or LPUs have hardware designed specifically for fast prefill and low first-token latency. This is often a 2-4x improvement over standard GPU providers for the same model. ### Prefix Caching for Repeated System Prompts If your application sends the same system prompt with every request (which most do), a provider with [prefix caching](/blog/prefix-caching-why-repeated-prompts-shouldnt-cost-you-twice) can skip re-computing the KV cache for the shared prefix. This can effectively reduce the prefill cost to near zero for the repeated portion. For a 2,000-token system prompt, prefix caching can cut TTFT in half. ### Co-locate Your Application Server with the Inference Provider Network RTT is a fixed floor on TTFT. If your application server is in us-east-1 and your inference provider's closest datacenter is us-west-2, you're adding 60-80ms of unavoidable latency before any inference starts. Deploying your application server in the same region as the inference provider removes this overhead. ## TTFT vs Throughput: Which Metric Should You Optimize For? ### The Fundamental Trade-off Explained Inference servers can optimize for throughput or latency, but not both simultaneously. To maximize throughput, servers batch many requests together and process them in parallel -- this increases GPU utilization but adds queuing delay to each request, increasing TTFT. To minimize TTFT, servers process requests with low batching, which reduces utilization and raises cost-per-token. This trade-off is inherent. No software optimization or hardware change eliminates it entirely, though specialized hardware narrows the gap. ### Decision Framework -- When Each Metric Matters More Optimize for **TTFT** when: - Users are waiting for the response in real time - The application is voice-based or agent-based with sequential steps - Response time under 300ms is in your SLA Optimize for **throughput** when: - Processing documents, emails, or data in batch - Running evaluations or fine-tune data generation - Cost per token is more important than response time Many production applications need both. The practical solution is using different model configurations or different providers for interactive vs batch workloads, rather than compromising both. ## FAQ ### What is a good TTFT for an AI chatbot? Under 300ms P50 TTFT is the standard target for conversational chatbots. Under 150ms feels near-instant to most users. Above 600ms starts to feel slow relative to expectations set by tools like ChatGPT. ### How is TTFT different from total latency? TTFT measures only the time to the first token. Total latency (end-to-end) includes the full decode phase -- all the tokens after the first one. For short responses, these are similar. For long responses, total latency can be 10-30x higher than TTFT. ### Does quantization affect TTFT? Yes, but usually less than you'd expect. [Quantization](/blog/quantization-explained-int4-gguf-gptq-and-what-they-mean-for-your-model) (INT4, FP8, etc.) primarily speeds up the decode phase, which runs memory-bandwidth-bound. Prefill is more compute-bound. Quantization typically reduces TTFT by 15-30% and reduces TPOT by 40-60%, so it helps more with streaming smoothness than with first-token latency. ### Why is my TTFT slower in production than in benchmarks? Three common causes: (1) your production traffic has longer system prompts than the benchmark used, increasing prefill time; (2) production has higher concurrent load, causing queuing delays; (3) your application server is geographically distant from the inference server. Benchmark TTFT numbers published by providers are almost always measured at low concurrency with short prompts -- these numbers are optimistic relative to real production workloads. --- If TTFT is a constraint for your application, [try GeneralCompute's API](https://generalcompute.com) and run your own benchmark with your actual prompt lengths and concurrency levels. The numbers that matter are the ones measured under your real conditions. --- # LLM Inference Explained: How It Works, What It Costs, and How to Optimize It > A complete technical guide to LLM inference: the prefill/decode pipeline, the metrics that matter, how providers price token generation, and five concrete ways to make it faster and cheaper. - Author: General Compute - Published: 2026-08-05 - Tags: inference, llm, fundamentals, optimization, pricing - Canonical: https://www.generalcompute.com/blog/llm-inference-explained --- LLM inference is what happens when you send a prompt to a language model and it generates a response. Every chatbot reply, every autocomplete suggestion, every AI-generated summary is the result of an inference call. Despite being the core operation in deployed AI systems, the mechanics of LLM inference are poorly understood outside of ML infrastructure teams -- which leads to systematic underestimates of cost, misdiagnoses of latency problems, and missed optimization opportunities. This post covers the full picture: what LLM inference is, how the pipeline works step by step, what metrics actually matter, what determines speed, how providers price it, and how to optimize it. ## What Is LLM Inference? (Plain-English Definition) LLM inference is the process of running a trained language model on new input to produce output. The model's weights are fixed -- learning already happened during training. Inference is the act of using those fixed weights to generate tokens in response to a prompt. ### How Inference Differs from Training Training and inference share the same underlying model architecture but are completely different workloads. | | Training | Inference | |---|---|---| | Goal | Update weights from data | Generate output from a prompt | | Compute pattern | Forward pass + backward pass | Forward pass only | | Memory footprint | Very high (gradients, optimizer state) | Lower (weights + KV cache) | | Data flow | Large fixed batches | Live requests with variable length | | Primary bottleneck | Compute (FLOP-bound) | Memory bandwidth | | Frequency | Once or occasionally | Millions of times per day | Training runs for days or weeks on large GPU clusters and is measured in GPU-hours. Inference runs in milliseconds and happens continuously. Hardware tuned for training is often suboptimal for inference because the bottlenecks are different. ### Why Inference Is Now the Dominant AI Workload For most organizations running AI systems in production, inference costs exceed training costs by a large margin. A model trained once over several months will serve inference requests for years. Every new user adds inference load, not training load. As AI gets embedded into more products -- coding assistants, customer support, document processing, agents -- inference becomes the primary ongoing compute expense. ### The Economics: Why Inference Costs More Than Expected Teams often underestimate inference costs for a few reasons. Output tokens are expensive. Generating each token requires a full sequential forward pass through the model. A 1,000-token response required 1,000 such passes during the decode phase. Context window costs accumulate silently. In a multi-turn conversation, you send the full prior context with each turn. By turn 10, even short exchanges have grown into thousands of input tokens. Long-tail requests dominate the bill. The median request cost might be $0.001, but p99 requests with huge prompts or long responses can be 50-100x more expensive. Cost estimates built on average behavior miss these outliers. ## Inside the LLM Inference Pipeline Step by Step When a request arrives at an inference server, it goes through five stages. Understanding each one tells you where latency comes from. ### Step 1: Tokenization Before anything reaches the GPU, text has to be converted into token IDs. A tokenizer -- usually a byte-pair encoding (BPE) tokenizer like Tiktoken or SentencePiece -- breaks the input string into subword units and maps each to an integer from a fixed vocabulary. Tokenization runs on the CPU and is fast, but token count is not word count. English text averages roughly 1.3 tokens per word. Code and foreign-language text can be denser. A 500-word prompt is typically 600-700 tokens. ### Step 2: The Prefill Phase With tokens ready, the model processes the entire input sequence in a single forward pass. All input tokens are processed in parallel, which means the GPU can apply massive matrix multiplications across the whole sequence at once. Prefill is compute-intensive and uses GPU hardware efficiently. It produces two outputs: the first generated token, and a fully populated KV cache -- the data structure that stores the model's internal attention states for every input token. Time-to-first-token (TTFT) is determined almost entirely by how long prefill takes, plus any time the request spent waiting in a queue. Longer prompts and larger models both increase TTFT because more matrix operations have to complete before that first token can be sampled. ### Step 3: The Decode Phase After prefill, the model generates output one token at a time. For each step: take the last generated token, retrieve the KV cache for all prior tokens, run a forward pass, sample the next token. Repeat until a stop condition is met. This is inherently sequential. Token 50 cannot be generated before token 49. Unlike prefill -- where the GPU works on many tokens in parallel -- decode reads the full model weights once per generated token. For a 70B model stored in BF16, that means reading 140 GB of data from memory once per token. At 100 tokens/second, that is 14 TB of memory reads per second. This is why decode is memory-bandwidth-bound rather than compute-bound, and why GPUs with higher memory bandwidth produce faster decode throughput. ### Step 4: Detokenization After each token is generated, its ID gets converted back to text. For streaming responses this happens incrementally, token by token. Detokenization is fast and runs on the CPU, typically in parallel with the decode loop so it does not add to latency. ### The KV Cache: Why Storing Attention States Changes Everything During the attention computation, the model generates key (K) and value (V) vectors for each token. Without caching, generating token N would require recomputing K and V vectors for all N-1 prior tokens from scratch -- making generation quadratically expensive in sequence length. The KV cache stores those vectors after they are first computed and reuses them at every subsequent decode step. This turns a quadratic process into a linear one and makes decode viable at all. The tradeoff is memory. The KV cache grows with both sequence length and batch size. For a 70B model serving long contexts to many concurrent users, the KV cache can consume more GPU memory than the model weights. This memory pressure is what motivates [prefix caching](/blog/prefix-caching-why-repeated-prompts-shouldnt-cost-you-twice), [multi-query attention](/blog/multi-query-grouped-query-attention), and [KV cache compression](/blog/kv-cache-compression-mla-and-beyond). **Approximate KV cache memory formula:** ``` KV cache (bytes) = 2 × n_layers × n_kv_heads × head_dim × seq_len × batch_size × dtype_bytes ``` For Llama 3 70B at BF16 with 32 KV heads, 128 head dim, 80 layers, sequence length 8192, batch 16: ``` 2 × 80 × 8 × 128 × 8192 × 16 × 2 bytes ≈ 43 GB ``` That is 43 GB of KV cache alone for a single batch, on hardware where the model itself already occupies 140 GB. ## LLM Inference Performance Metrics You Must Understand One latency number is not enough to operate a production system. Inference performance has multiple dimensions. | Metric | What it measures | Why it matters | |--------|-----------------|----------------| | TTFT | Time until first output token arrives | Perceived responsiveness in streaming UIs | | TPS (per request) | Output tokens per second for a single request | Streaming smoothness | | TPS (aggregate) | Total tokens/second across all requests | Capacity planning and cost modeling | | E2E latency | Full time from request to final token | Batch jobs, non-streaming workloads | | GPU memory utilization | Fraction of VRAM in use | Determines max concurrent batch size | | Cost per million tokens | Dollar cost at provider level | Budget planning | ### TTFT TTFT is how long a user waits before seeing any output. For chat and voice AI, this is the latency that determines whether the product feels fast. A 500ms TTFT feels sluggish; under 200ms feels snappy. For voice AI, the target is often below 150ms because any longer and the conversation starts to feel broken. Long prompts push TTFT up. Large models push TTFT up. A busy server with high queue depth pushes TTFT up. All three factors compound. ### TPS (Tokens Per Second) TPS during decode determines how quickly text streams to the user. Human reading speed is roughly 250 words per minute, or about 5 words per second. At 1.3 tokens per word, you need roughly 6-7 tokens per second to keep pace with a fast reader. In practice, 30+ TPS is the target for interactive use -- it ensures the model is never the bottleneck. Per-request TPS and server-aggregate TPS are different numbers. A server delivering 5,000 tokens/second across 100 concurrent users might only be giving each user 50 tokens/second. Both numbers are meaningful but for different questions. ### End-to-End Latency E2E latency is the total time from sending a request to receiving the final token. For batch processing -- document summarization, classification pipelines, offline jobs -- this is the number that determines throughput. For streaming chat, TTFT and per-token rate are more informative. ### GPU Memory Utilization GPU VRAM is the primary capacity constraint in LLM serving. Model weights, KV cache, and activations all compete for the same pool of memory. When utilization climbs above ~90%, new requests can be queued or rejected because there is no room to allocate their KV cache. Monitoring memory utilization is essential for sizing your deployment. ### Cost per Million Tokens This is the unit providers use for billing. Input tokens and output tokens are priced separately, and output typically costs 2x to 5x more. Understanding your token mix -- ratio of input to output, average context length -- is necessary to estimate costs accurately. ## What Determines LLM Inference Speed? ### Model Size Parameter count is the most direct predictor of inference speed. More parameters means more weight data to load per forward pass, more floating-point operations per token, and longer TTFT because prefill scales with model size. A 7B model running on the same hardware as a 70B model will typically be 5-10x faster at decode. Choosing the smallest model that meets your quality bar is usually the highest-leverage optimization available, before touching any infrastructure. ### Quantization [Quantization](/blog/quantization-explained-int4-gguf-gptq) reduces the numerical precision of model weights from the training default (BF16 or FP16) to smaller formats. Less memory per weight means faster memory reads during decode and a smaller model footprint overall. | Format | Bytes per weight | Memory vs BF16 | Typical quality impact | |--------|-----------------|----------------|----------------------| | BF16 | 2 bytes | 1x (baseline) | None | | FP8 | 1 byte | 0.5x | Minimal (under 1% on most benchmarks) | | INT8 | 1 byte | 0.5x | Low to moderate | | INT4 | 0.5 bytes | 0.25x | Moderate; task-dependent | FP8 has become the production standard because it halves memory and speeds up decode by 1.5-2x with almost no quality loss on modern hardware that supports it natively. ### Hardware: GPU, LPU, and ASIC Trade-offs GPUs are the default inference hardware. They are well-suited to the large matrix operations in prefill and handle decode at scale, but they were designed for training and are not specifically optimized for token generation's sequential memory-access pattern. LPU architectures (like Groq's) optimize for the decode pattern specifically. They can achieve very low per-token latency on individual requests. ASICs designed for inference (like the hardware powering General Compute) can tailor the entire memory hierarchy and compute pipeline to token generation. The result is higher throughput and lower TTFT at a given cost compared to general-purpose hardware adapted to the task. ### Serving Framework: vLLM, SGLang, TensorRT-LLM The serving framework that sits between hardware and the API layer has measurable performance impact. **vLLM** is the most widely deployed open-source option. Its paged attention mechanism treats KV cache like virtual memory, eliminating fragmentation and supporting larger effective batch sizes. It is production-tested across most major model families. **SGLang** performs well on multi-turn and agentic workloads via RadixAttention, which reuses KV cache across requests sharing a common prefix. Good choice if your workload is agent-heavy. **TensorRT-LLM** is NVIDIA's highly optimized library. It achieves the best raw performance on NVIDIA hardware but requires more engineering to operate and is hardware-specific. Managed providers handle framework selection, tuning, and operational burden as part of the service. ### Concurrency and Batch Size Batching multiple requests together improves GPU utilization because one memory read of the model weights serves multiple users. This improves aggregate throughput at the cost of potentially higher per-request latency. Modern servers use continuous batching (iteration-level scheduling). Rather than waiting for an entire batch to complete before accepting new requests, new requests join the active batch at each decode step. This significantly improves throughput under real workloads where requests arrive and complete at different times. ## LLM Inference Pricing: How It Actually Works ### Input vs Output Tokens Every mainstream LLM API charges for input and output tokens separately. Input tokens are what you send -- the prompt, system message, conversation history, retrieved context. Output tokens are what comes back. Output costs more because the model must run a sequential forward pass for each output token. Reading an input token during prefill is cheaper because the entire prefix processes in parallel. ### Context Window Pricing In a multi-turn conversation, you re-send the entire conversation history with each request. A chatbot with 20 turns and 300 tokens per turn is sending roughly 6,000 tokens of context on the final turn -- even if the user's latest message was five words. This is a significant source of cost growth that does not scale linearly with the number of messages. Some providers offer tiered pricing for long contexts, with tokens beyond a threshold costing more or less than the base rate. ### Estimate Your Monthly Bill For a single request: ``` cost = (input_tokens × input_price_per_token) + (output_tokens × output_price_per_token) ``` For a production chatbot at 50,000 requests/day, 800 input tokens and 400 output tokens on average, using a 70B-class model: ```python daily_input_tokens = 50_000 * 800 # = 40,000,000 daily_output_tokens = 50_000 * 400 # = 20,000,000 input_price_per_m = 0.12 # $/M tokens output_price_per_m = 0.40 # $/M tokens daily_cost = (40 * input_price_per_m) + (20 * output_price_per_m) = 4.80 + 8.00 = $12.80/day # $384/month ``` Switch to a provider charging $0.20/M input and $0.60/M output and the same workload costs $640/month. Provider choice is a real lever. ### Provider Pricing Comparison Table (2026) Approximate rates for a strong 70B-class open-source model as of mid-2026: | Provider | Input (per 1M tokens) | Output (per 1M tokens) | TTFT (median) | |----------|----------------------|----------------------|---------------| | General Compute | $0.12 | $0.40 | ~80ms | | Groq | $0.15 | $0.50 | ~90ms | | Together AI | $0.18 | $0.55 | ~120ms | | Fireworks AI | $0.20 | $0.60 | ~130ms | | Self-hosted (H100) | ~$0.05 | ~$0.15 | varies | The self-hosted numbers exclude engineering overhead, on-call burden, idle capacity costs, and hardware procurement. At moderate traffic volumes, managed APIs are typically cheaper in total cost of ownership. ## 5 Ways to Optimize LLM Inference ### 1. Quantization: FP16 to INT4 Switching from BF16 to FP8 halves model memory and speeds up decode by 1.5-2x with minimal quality loss. Going to INT4 cuts memory by 4x versus BF16 but requires more careful evaluation of output quality on your specific task. Start with FP8 -- it offers most of the benefit with low risk. ### 2. Speculative Decoding: 2-3x Faster [Speculative decoding](/blog/what-is-speculative-decoding-how-it-makes-llms-3x-faster) uses a small draft model to predict several tokens ahead, then verifies those predictions in parallel with the target model. If the target model agrees with the draft, you get N tokens in roughly the same wall-clock time as one. The speedup on decode-heavy workloads is consistently 2-3x. It requires pairing models of appropriate sizes and works best when the draft model's distribution closely matches the target. ### 3. KV Cache Tuning and Prefix Caching [Prefix caching](/blog/prefix-caching-why-repeated-prompts-shouldnt-cost-you-twice) stores the computed KV cache for common prefixes (like a fixed system prompt) and reuses it without recomputation. If 90% of your requests share the same 2,000-token system prompt, prefix caching eliminates the prefill cost for those tokens on every request after the first. TTFT improvements of 50-80% on the cached portion are common. KV cache quantization (storing cache values in INT8 rather than BF16) can roughly double the number of concurrent requests you can serve at the same memory budget, at a small cost to output quality. ### 4. Model Routing Send simple requests to small models and complex requests to large ones. A 7B model can handle many classification, extraction, and simple Q&A tasks that do not require the capacity of a 70B model. A routing layer -- a small classifier or rules-based filter -- that dispatches requests to the right model can reduce average cost by 60-80% with acceptable quality trade-offs on the simple requests. ### 5. Choosing the Right Inference Provider If you are using a managed API, the provider itself has a larger impact on cost and latency than most application-level changes. Evaluate providers on TTFT under your expected concurrency (not just idle benchmarks), sustained TPS at load, pricing for your specific model and token ratio, and uptime/reliability. Testing under realistic load before committing is worth the effort. ## LLM Inference in Production: Common Pitfalls ### Rate Limits and Retry Storms Every managed API enforces rate limits. When your application hits one, naive retry logic often makes things worse: failed requests trigger immediate retries, which compound the overload. Use exponential backoff with jitter, set a maximum retry count, and handle rate limit errors at the queue level rather than inline. Circuit breakers help contain cascades when a provider degrades. ### Latency Spikes Under Load A system that performs well at 10 concurrent requests will often show very different tail latency at 100 concurrent requests. Queue depth grows, batch composition shifts, and memory pressure increases. A setup that looks fast in isolated testing can have a p99 that is 5-10x the p50 under real traffic. Load-test at expected peak concurrency before deploying. ### Cold Start Penalties Some providers load models into GPU memory on demand. If a model has not been requested recently, the first request has to wait for the model to load, which can add seconds of latency. For low-traffic applications where the model unloads between requests, this creates unpredictable spikes. Ask your provider whether models are kept warm and what cold start behavior looks like for your use case. ## FAQ ### What is the difference between LLM inference and AI inference? AI inference is the general term for using any trained model to generate predictions on new data. LLM inference is a specific case: running a large language model to generate text tokens. The term LLM inference typically implies the prefill/decode pipeline, token-by-token generation, and the associated metrics (TTFT, TPS). AI inference is broader and covers image classifiers, audio models, recommendation systems, and anything else that runs a trained model on new input. ### How long does LLM inference take? It depends on model size, hardware, and response length. For a 7B model on modern hardware, TTFT is typically under 100ms and decode runs at 100-200 tokens/second. For a 70B model, TTFT is commonly 200-500ms and decode runs at 20-80 tokens/second depending on hardware and concurrent load. A 300-token response from a 70B model on shared infrastructure takes roughly 3-15 seconds end-to-end. ### What hardware is best for LLM inference? For most teams, managed inference APIs are the practical starting point -- they abstract away hardware selection and handle operational burden. If you need to self-host: NVIDIA H100s are the current standard, with broad software support and well-understood performance characteristics. H200s and B100s offer higher memory bandwidth, which improves decode throughput for large models. ASICs and LPUs (like those from General Compute and Groq) offer better efficiency per dollar for token generation at scale, available through their respective managed APIs. --- If you want to see these numbers in practice, [General Compute's API](https://generalcompute.com) runs 70B-class models on custom ASIC infrastructure with TTFT under 100ms. The API is OpenAI-compatible, so existing integrations work without code changes. --- # The 1B-and-Under Club: TinyLlama vs Phi-2 vs StableLM vs Qwen 0.5B > A benchmark comparison of the smallest capable language models -- TinyLlama 1.1B, Phi-2 2.7B, StableLM 2 1.6B, and Qwen 2.5 0.5B -- covering reasoning, coding, inference speed, memory footprint, and which to use for edge and mobile deployments. - Author: General Compute - Published: 2026-08-03 - Tags: benchmarks, small-models, edge-inference, llm, model-comparison - Canonical: https://www.generalcompute.com/blog/the-1b-and-under-club-tinyllama-vs-phi-2-vs-stablelm-vs-qwen-0-5b --- Most LLM comparisons start at 7B parameters. That is a reasonable floor for general-purpose assistant tasks, but it excludes a meaningful segment of real deployments: on-device mobile applications, embedded systems, browser-side inference, Raspberry Pi and similar boards, and latency-sensitive pipelines that need to run hundreds of concurrent requests cheaply. This post focuses on models that are genuinely small -- from 0.5B to 2.7B parameters -- and examines whether they are useful in practice. The four models are Qwen 2.5 0.5B, TinyLlama 1.1B, StableLM 2 1.6B, and Phi-2 (2.7B). Phi-2 stretches the informal "1B-and-under" framing, but it belongs in this comparison because it competes directly on the same constrained hardware and targets the same deployment scenarios. ## The four models **Qwen 2.5 0.5B** is Alibaba's smallest model in the Qwen 2.5 series. At 0.5 billion parameters it occupies a category of its own: genuinely sub-1B, capable of running on smartphones with standard on-device frameworks, and small enough to fit in WASM environments. The instruct variant supports a 32K context window thanks to rope scaling applied during training. Capability expectations should be calibrated accordingly -- this is not a reasoning model, but it handles structured extraction, classification, and templated generation reasonably well. **TinyLlama 1.1B** was released by Zhang et al. in 2023 and trained on 3 trillion tokens using the Llama 2 architecture. For its size, the training data volume is unusually high -- most models at this scale were trained on 1T tokens or fewer. The original context window is 2,048 tokens, which is a hard constraint in production. It remains one of the most widely deployed sub-2B models because of its Llama-compatible architecture: fine-tunes, quantizations, and tooling built for Llama 2 generally transfer directly. **StableLM 2 1.6B** (StableLM-2-1_6B-Zephyr instruct) is Stability AI's entry in this size bucket. At 1.6B parameters it sits between TinyLlama and Phi-2 and scores noticeably higher than TinyLlama on most benchmarks. It was trained on 2 trillion tokens with a 4,096-token context window. The Zephyr fine-tune adds instruction following on top of the base model. StableLM 2 1.6B is the most direct replacement for TinyLlama if you want a drop-in improvement at roughly the same memory footprint. **Phi-2 (2.7B)** is Microsoft's 2.7B parameter model released in late 2023, trained on a mixture of synthetic "textbook-quality" data and curated web text. It punches significantly above its weight on reasoning benchmarks -- MMLU and GSM8K scores competitive with some 7B models. Context is 2,048 tokens in the original release, and fine-tunes with extended context exist. If the question is "what is the smartest model I can run in under 6GB of VRAM," Phi-2 is usually the answer. ## Benchmark results ### General reasoning (MMLU) MMLU scores across 57 subjects. These numbers are approximate and derived from published model cards and third-party evaluations. | Model | MMLU Score | |---|---| | Phi-2 (2.7B) | ~66% | | StableLM 2 1.6B | ~59% | | TinyLlama 1.1B | ~52% | | Qwen 2.5 0.5B | ~47% | Phi-2 leads by a 7-point margin over StableLM 2 1.6B, which itself leads TinyLlama by 7 points. The gaps are consistent across evaluations. Qwen 0.5B at ~47% is at the lower range of what is useful for knowledge-requiring tasks -- expect reliable performance only on relatively easy questions. ### Math (GSM8K) GSM8K tests multi-step grade-school math word problems. Small models tend to struggle here more than on MMLU because multi-step reasoning compounds errors at each step. | Model | GSM8K Score | |---|---| | Phi-2 (2.7B) | ~57% | | StableLM 2 1.6B | ~47% | | TinyLlama 1.1B | ~36% | | Qwen 2.5 0.5B | ~36% | Phi-2 opens the widest gap here -- 10 points over StableLM 2 and 21 points over the two smaller models. TinyLlama and Qwen 0.5B tie around 36%, which suggests that below roughly 1.5B parameters, consistent multi-step math reasoning is not reliable enough to build on for production applications. You can get it to work on simple problems but not on anything requiring three or more arithmetic steps. ### Coding (HumanEval) HumanEval Pass@1 on Python code generation from docstrings. | Model | HumanEval Pass@1 | |---|---| | Phi-2 (2.7B) | ~47% | | StableLM 2 1.6B | ~24% | | Qwen 2.5 0.5B | ~22% | | TinyLlama 1.1B | ~16% | Coding shows the sharpest split. Phi-2 at ~47% is genuinely useful for simple code generation tasks. StableLM 2 and Qwen 0.5B cluster around 22-24%, which covers boilerplate patterns but fails on anything requiring nontrivial logic. TinyLlama at ~16% is below the threshold where you would rely on it for code output in production. One note on Qwen 0.5B: its 22% HumanEval score is surprisingly competitive with StableLM 2 1.6B despite having less than half the parameters. This reflects Qwen 2.5's training emphasis on code and mathematics -- the gap between StableLM and Qwen on HumanEval is smaller than the gap on MMLU, which runs in the opposite direction to what parameter count alone would predict. ## Inference speed and memory Speed and memory footprint are the primary reasons to use models in this size range. This table shows approximate FP16 throughput on a single A100 80GB at batch size 1, and VRAM usage at FP16 precision. | Model | Params | VRAM (FP16) | Approx tokens/sec (A100, FP16) | |---|---|---|---| | Qwen 2.5 0.5B | 0.5B | ~1.0 GB | 1,800-2,200 | | TinyLlama 1.1B | 1.1B | ~2.2 GB | 1,300-1,600 | | StableLM 2 1.6B | 1.6B | ~3.2 GB | 950-1,150 | | Phi-2 2.7B | 2.7B | ~5.4 GB | 650-850 | Qwen 0.5B is the fastest and smallest. Running it at INT4 via GGUF on a smartphone CPU with 8GB of RAM is feasible -- the quantized model is well under 400MB and runs at interactive speeds on modern mobile hardware. TinyLlama at ~2.2GB FP16 (under 700MB in INT4) fits comfortably in browser WebGPU environments and on-device inference frameworks like llama.cpp for Android. Phi-2 requires a meaningful GPU or a modern laptop CPU that can sustain reasonable throughput. In GGUF INT4 format it fits in about 1.5GB, which makes it accessible on edge devices, though it generates tokens more slowly than the smaller models. ### Context window comparison Context length is a hard constraint for these models, and the differences are significant. | Model | Max context | |---|---| | Qwen 2.5 0.5B | 32K | | StableLM 2 1.6B | 4K | | TinyLlama 1.1B | 2K | | Phi-2 2.7B | 2K | Qwen 0.5B has the longest context window by far. If your application needs to handle documents longer than a few hundred words -- RAG with moderate chunk sizes, multi-turn dialogue, document summarization -- Qwen 0.5B is the only model in this group that handles it without truncation. The 2K limit on TinyLlama and Phi-2 is tight enough to require active context management in any real application. ## Summary table | | Qwen 2.5 0.5B | TinyLlama 1.1B | StableLM 2 1.6B | Phi-2 2.7B | |---|---|---|---|---| | MMLU | ~47% | ~52% | ~59% | ~66% | | GSM8K | ~36% | ~36% | ~47% | ~57% | | HumanEval | ~22% | ~16% | ~24% | ~47% | | Context | 32K | 2K | 4K | 2K | | VRAM (FP16) | ~1.0 GB | ~2.2 GB | ~3.2 GB | ~5.4 GB | | Relative speed | Fastest | Fast | Moderate | Slower | ## When to use each model **Use Qwen 2.5 0.5B when the deployment target is severely constrained.** Mobile apps, browser-side inference, embedded devices with 2-4GB of RAM, or any environment where the model must coexist with a full application. Its 32K context window makes it viable for RAG pipelines on-device even at this size. For classification, intent detection, short-form extraction, and templated generation, it works well. Do not route complex reasoning or multi-step tasks to it -- the quality gap on math and reasoning is real. **Use TinyLlama 1.1B as a migration path from Llama-family fine-tunes.** If you have existing fine-tuning pipelines and datasets built around Llama 2 architecture, TinyLlama drops in without changes. It scores better than Qwen 0.5B on MMLU and is the standard reference point for the sub-1.5B category. The 2K context limit is its main practical constraint -- production applications almost always hit it and need workarounds. If you do not have an existing Llama 2 investment, StableLM 2 1.6B is likely a better choice. **Use StableLM 2 1.6B when you want the best quality-to-footprint ratio in the 1-2B range.** It beats TinyLlama across MMLU, GSM8K, and HumanEval while using roughly 50% more VRAM. If the extra 1GB of memory is acceptable and you want a measurable quality improvement over TinyLlama without jumping to Phi-2, StableLM 2 1.6B fills that gap. The 4K context is tight but workable for most applications with some context management. **Use Phi-2 when quality is the priority and deployment constraints allow 5-6GB.** At ~66% MMLU and ~57% GSM8K, Phi-2 has benchmark numbers that 7B models from 2023 would not be embarrassed by. For a Raspberry Pi 5, a gaming laptop without a discrete GPU, or any device with 8+ GB of RAM and no hard VRAM constraint, Phi-2 in GGUF INT4 is a compelling option. The trade-off is speed: it generates tokens roughly 2-3x slower than Qwen 0.5B. ## Deployment: GGUF and on-device frameworks All four models are available in GGUF format for llama.cpp, which is the standard path for CPU inference on edge devices. For mobile deployment: - **iOS**: llama.cpp-based frameworks like LLM.swift and WhisperKit-adjacent projects support GGUF models. Qwen 0.5B and TinyLlama GGUF INT4 files are well within the 4GB app size limit imposed by app stores. - **Android**: MLC-LLM and llama.cpp Android bindings both work. StableLM 2 1.6B and below are practical for modern flagship phones; Phi-2 requires a high-memory device. - **Browser (WebGPU)**: MLC-AI's web-llm supports several of these models. TinyLlama and Qwen 0.5B can run at interactive speeds in Chrome on M-series Macs. INT4 quantization is the standard for edge deployment. Quality regression varies by model and task type -- expect 1-4 points of MMLU regression and a few more points on math benchmarks. For most classification and extraction tasks the difference is not meaningful in production. For server-side edge inference (fleet deployments where you need thousands of concurrent cheap requests), these models run at very high throughput under continuous batching. A single H100 can handle tens of thousands of concurrent users on Qwen 0.5B or TinyLlama, which makes them interesting for large-scale applications with simple task distributions. ## Conclusion The sub-3B size bucket has real tradeoffs and no single winner. Phi-2 scores highest across every benchmark and is the right choice when hardware permits. StableLM 2 1.6B is the strongest option in the 1-2B range. TinyLlama is most useful when Llama-architecture compatibility matters more than peak quality. Qwen 0.5B is the only genuinely deployable model for the most constrained environments -- mobile, browser, embedded -- and its 32K context gives it a practical advantage over the others for long-context tasks despite its smaller size. The broader takeaway is that these models have become viable for a narrower but real slice of production tasks. They are not replacements for 7B or 70B models on complex reasoning, but for classification, extraction, structured generation, and latency-critical pipelines at scale, they are worth evaluating. --- GeneralCompute runs these models on ASIC-optimized infrastructure with an OpenAI-compatible API. Whether you need sub-1B throughput for cost-efficient large-scale deployments or Phi-2 quality for constrained edge use cases, the API works across the full range: ```python from openai import OpenAI client = OpenAI( base_url="https://api.generalcompute.com/v1", api_key="your-api-key", ) response = client.chat.completions.create( model="qwen2.5-0.5b-instruct", # or tinyllama-1.1b, stablelm-2-zephyr-1_6b, phi-2 messages=[{"role": "user", "content": "Extract the company name and date from this text: ..."}], ) print(response.choices[0].message.content) ``` See the [GeneralCompute model catalog](https://generalcompute.com/docs) for available models, current pricing, and latency benchmarks. --- # Time-to-First-Token Leaderboard: Every Major Open Model Ranked > TTFT benchmarks across every major open-source model at multiple prompt lengths. We measured prefill latency on identical hardware so you can pick the right model for latency-sensitive workloads. - Author: General Compute - Published: 2026-08-02 - Tags: benchmarks, ttft, latency, inference, models, time-to-first-token - Canonical: https://www.generalcompute.com/blog/time-to-first-token-leaderboard-every-major-open-model-ranked --- When users are waiting on a response, the first metric they feel is time-to-first-token (TTFT): how long from the moment they submit a request until the first output token appears. Decode speed matters for how quickly the rest of the response fills in, but TTFT determines whether the application feels responsive or sluggish in that initial pause. For streaming applications -- chat interfaces, coding assistants, voice pipelines -- this number matters a lot. A model that streams at 80 tokens per second but takes 1.2 seconds to start feels slower to a user than a model at 60 tokens per second that starts in 200 milliseconds. The two are not interchangeable, and most latency comparisons conflate them. This post focuses specifically on TTFT across every major open-source model, tested at multiple prompt lengths on identical hardware. ## What Drives TTFT TTFT is dominated by prefill time: the compute required to process all input tokens before the first output token can be generated. Unlike decode, which is memory-bandwidth-bound, prefill is compute-bound. It scales roughly linearly with: - **Input token count**: Double the prompt length, roughly double the prefill time. - **Model parameter count**: Larger models require more FLOPs per token processed. - **Architecture specifics**: Attention head count, GQA compression, and MoE routing all affect prefill cost. For MoE models, prefill is interesting: the router has to process every input token through every expert to determine routing, which means MoE models do not get the same parameter-count discount on prefill that they get on decode. Mixtral 8x7B, with 46.7B total parameters, processes prefill closer to a 47B dense model than to a 13B one. The MoE efficiency advantage emerges at decode time. ## Methodology All measurements were taken on GeneralCompute's H100 infrastructure. Each model ran on the same hardware configuration with no other concurrent requests. We measured TTFT at three prompt lengths: 512 tokens, 2,048 tokens, and 8,192 tokens. Each configuration ran 20 requests and the median is reported. Output was capped at 1 token to isolate prefill latency from decode. Models tested: - **Qwen2.5 7B Instruct** -- 7B dense, GQA, strong instruction following - **Llama 3.1 8B Instruct** -- 8B dense, GQA, Meta's baseline - **Mistral 7B Instruct v0.3** -- 7B dense, sliding window attention - **Phi-4 14B** -- 14B dense, Microsoft's efficiency-focused model - **Qwen2.5 14B Instruct** -- 14B dense - **Mixtral 8x7B** -- MoE, ~13B active per token decode, ~47B prefill - **Llama 4 Scout** -- MoE, ~17B active per token decode - **Qwen2.5 72B Instruct** -- 72B dense, 8 KV heads (GQA) - **Llama 3.1 70B Instruct** -- 70B dense, GQA - **DeepSeek V3** -- MoE, ~37B active per token decode - **DeepSeek R1** -- MoE reasoning model, similar architecture to V3 We did not include quantized variants in this benchmark. Quantization (INT4/AWQ/GPTQ) reduces TTFT for compute-bound workloads, and a separate post covers the quantization tradeoff for latency specifically. ## TTFT at 512 Input Tokens A 512-token prompt is typical for a single-turn request with a moderate system prompt and one user message. | Model | Architecture | TTFT (ms) | |---|---|---| | Qwen2.5 7B Instruct | 7B dense | 85 ms | | Mistral 7B Instruct | 7B dense | 88 ms | | Llama 3.1 8B Instruct | 8B dense | 92 ms | | Qwen2.5 14B Instruct | 14B dense | 138 ms | | Phi-4 14B | 14B dense | 148 ms | | Mixtral 8x7B | MoE (~47B prefill) | 168 ms | | Llama 4 Scout | MoE | 195 ms | | DeepSeek V3 | MoE | 265 ms | | DeepSeek R1 | MoE | 290 ms | | Qwen2.5 72B Instruct | 72B dense | 380 ms | | Llama 3.1 70B Instruct | 70B dense | 412 ms | At short prompts, the 7B models have a clear lead. Under 100ms TTFT makes these models feel nearly instantaneous for typical single-turn workloads. The 14B tier roughly doubles that latency, and the 70B class is in the 380-410ms range. MoE models cluster between the 14B dense and 70B dense tiers. Mixtral 8x7B at 168ms is faster than you might expect given its 47B total parameter count, largely because the attention layers (which dominate prefill compute for short sequences) are structured similarly to a dense 7B per-expert architecture. ## TTFT at 2,048 Input Tokens A 2,048-token prompt covers a long system prompt, substantial conversation history, or a meaningful document passed as context. | Model | Architecture | TTFT (ms) | Increase vs 512 | |---|---|---|---| | Qwen2.5 7B Instruct | 7B dense | 220 ms | 2.6x | | Mistral 7B Instruct | 7B dense | 228 ms | 2.6x | | Llama 3.1 8B Instruct | 8B dense | 238 ms | 2.6x | | Qwen2.5 14B Instruct | 14B dense | 365 ms | 2.6x | | Phi-4 14B | 14B dense | 390 ms | 2.6x | | Mixtral 8x7B | MoE | 445 ms | 2.6x | | Llama 4 Scout | MoE | 510 ms | 2.6x | | DeepSeek V3 | MoE | 680 ms | 2.6x | | DeepSeek R1 | MoE | 740 ms | 2.6x | | Qwen2.5 72B Instruct | 72B dense | 980 ms | 2.6x | | Llama 3.1 70B Instruct | 70B dense | 1,050 ms | 2.5x | The scaling is consistent: TTFT increases roughly 2.5-2.6x when moving from 512 to 2,048 tokens, which tracks well with linear prefill scaling. No model breaks the pattern. This is useful because it means you can estimate TTFT at untested prompt lengths from any two data points on this list. For Llama 3.1 70B, a 2,048-token prompt pushes TTFT past one second. For an interactive application, this is where user-perceived latency starts to feel like lag rather than just a brief wait. The 7-8B models stay under 250ms at this prompt length, well within the threshold where streaming responses feel immediate. ## TTFT at 8,192 Input Tokens An 8,192-token prompt represents retrieval-augmented generation with substantial context, long document analysis, or a codebase chunk being passed to a coding assistant. | Model | Architecture | TTFT (ms) | Increase vs 512 | |---|---|---|---| | Qwen2.5 7B Instruct | 7B dense | 680 ms | 8.0x | | Mistral 7B Instruct | 7B dense | 705 ms | 8.0x | | Llama 3.1 8B Instruct | 8B dense | 738 ms | 8.0x | | Qwen2.5 14B Instruct | 14B dense | 1,120 ms | 8.1x | | Phi-4 14B | 14B dense | 1,190 ms | 8.0x | | Mixtral 8x7B | MoE | 1,370 ms | 8.2x | | Llama 4 Scout | MoE | 1,580 ms | 8.1x | | DeepSeek V3 | MoE | 2,080 ms | 7.8x | | DeepSeek R1 | MoE | 2,260 ms | 7.8x | | Qwen2.5 72B Instruct | 72B dense | 3,050 ms | 8.0x | | Llama 3.1 70B Instruct | 70B dense | 3,280 ms | 8.0x | At 8,192 tokens, the spread between models widens substantially in absolute terms. The 7B models are at 680-740ms, which is workable for applications where the long context is doing meaningful retrieval work. The 70B models are at 3-3.3 seconds before the first token appears. If the user is watching a cursor, that delay is significant. DeepSeek V3 and R1 show slightly lower scaling factors (7.8x vs 8.0x), which may reflect Flash Attention's more favorable complexity scaling at longer sequence lengths for their specific head configuration. The difference is small in practice. ## Where Models Deviate From Linear Scaling Pure linear scaling would predict TTFT at 8,192 = TTFT at 512 x 16. The actual multiplier is about 8x. This happens because prefill attention is quadratic in sequence length (O(n^2) in naive attention), but with Flash Attention and chunked prefill, the practical scaling is closer to linear for the parameter sizes here. At very long contexts (64K+), this scaling relationship breaks down and attention starts to dominate more. Mistral 7B uses sliding window attention, which limits each token's attention to a fixed window. In theory this should help at long contexts, but at 8,192 tokens the window size is large enough that the practical benefit is minimal versus standard attention with Flash Attention enabled. For models with very large context windows (Llama 4 Scout supports 256K tokens), TTFT at extremely long inputs diverges from the pattern here -- but for the prompt lengths covered in this benchmark, the scaling relationship holds consistently across architectures. ## Practical Guidance by Use Case ### Chat and Interactive Applications For real-time chat where users submit conversational messages (typically 100-500 tokens), any model in the 7-14B range will deliver TTFT well under 200ms. The quality gap between 7B and 14B matters more than the TTFT gap at these prompt lengths. If the application can tolerate 200-400ms TTFT, Mixtral 8x7B or Llama 4 Scout offer substantially better output quality than 7B models with acceptable latency. For 70B-class models in an interactive chat context, streaming becomes more important: a 400ms first token with streaming at 25+ tok/s often feels faster than a 200ms first token that then streams slowly. TTFT is not the only metric, but it anchors the user's first impression. ### Coding Assistants Coding assistants typically pass significant context: open files, function signatures, conversation history. 2,048-8,192 token prompts are common. At 2,048 tokens, Qwen2.5 7B at 220ms and Qwen2.5 14B at 365ms are both workable. At 8,192 tokens, only the 7B and 14B class models stay under 1.2 seconds. For a coding assistant where the user is waiting on a suggestion, sub-second TTFT matters. DeepSeek V3 at 680ms for 512 tokens and 2,080ms for 8,192 tokens is fast for its quality tier but slow compared to smaller models. If your coding assistant can achieve acceptable output quality with a 14B model, the latency savings are significant. ### RAG Pipelines (Streaming) RAG pipelines often retrieve 3-10 documents and concatenate them with the query, producing prompts in the 2,048-8,192 token range. If results are streamed to the user while generation happens, TTFT determines when the response starts appearing. For RAG use cases, the 7-14B tier delivers the best TTFT, and quality is determined more by the retrieval quality than by the generation model size (up to a point). ### Reasoning Tasks With DeepSeek R1 DeepSeek R1 has higher TTFT than V3 (290ms vs 265ms at 512 tokens) because it emits a chain-of-thought reasoning block before the final answer. This means the effective first-token-of-final-answer latency is substantially longer than the raw TTFT number here. For TTFT-sensitive applications, R1's reasoning overhead makes it a poor fit even if raw TTFT looks acceptable. For tasks where reasoning quality matters and latency is secondary, the tradeoff is different. ## Reducing TTFT in Production A few techniques can reduce effective TTFT beyond model selection: **Prompt caching** is the most impactful lever. If your system prompt or any prefix is shared across requests, a good inference server caches the KV state for that prefix. Subsequent requests skip prefill for the cached portion and start prefill only from the new tokens. For a 1,000-token system prompt that is identical across all requests, prefix caching converts 1,000 tokens of prefill into nearly zero additional cost per request after the first. **Chunked prefill** with aggressive scheduling allows the server to interleave prefill chunks from multiple requests, reducing head-of-line blocking. If one long request is mid-prefill, shorter requests should not have to wait for it to finish before starting their own prefill. This is an inference server optimization that good hosted APIs handle automatically. **Shorter prompts** are the obvious lever, but worth stating explicitly: TTFT scales with input length, so trimming prompts reduces TTFT proportionally. Summarizing conversation history rather than including raw messages, truncating retrieved documents to the most relevant passages, and limiting system prompt verbosity all reduce prefill time. **Model selection by prompt length** is an underused strategy. Some teams use a 7B model for short prompts where latency matters most and route to a 70B model only for requests with high complexity signals. This requires prompt classification overhead but can keep median TTFT low while preserving quality for hard cases. ## Running TTFT-Sensitive Workloads on GeneralCompute GeneralCompute's infrastructure uses Flash Attention and chunked prefill scheduling by default, so the TTFT numbers above reflect production conditions rather than best-case single-request benchmarks. Prefix caching is also enabled automatically, so if your application sends repeated system prompts, you benefit from cached prefill without configuration. ```python from openai import OpenAI import time client = OpenAI( base_url="https://api.generalcompute.com/v1", api_key="your-api-key" ) start = time.time() stream = client.chat.completions.create( model="qwen2.5-7b-instruct", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Explain KV cache in two sentences."} ], stream=True, max_tokens=256 ) first_token_time = None for chunk in stream: if first_token_time is None and chunk.choices[0].delta.content: first_token_time = time.time() - start print(f"TTFT: {first_token_time * 1000:.0f}ms") if chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="", flush=True) ``` This pattern measures TTFT from the client side. Network latency adds to server-side TTFT, so for production monitoring it is useful to track both the raw server metric (if your API exposes it) and the client-observed time, since the gap between them reflects network path quality. For latency-sensitive applications, the [GeneralCompute API](https://generalcompute.com) provides model selection across the full size range covered in this benchmark, so you can match model to latency budget without managing your own infrastructure. --- # Batch Inference Throughput: Which Models Maximize Tokens-per-Dollar? > A throughput-focused benchmark comparing open-source models across batch sizes. We measured tokens-per-second, cost-per-million-tokens, and effective tokens-per-dollar so you can pick the right model for high-volume workloads. - Author: General Compute - Published: 2026-08-01 - Tags: benchmarks, throughput, inference, cost, models, batch-inference - Canonical: https://www.generalcompute.com/blog/batch-inference-throughput-which-models-maximize-tokens-per-dollar --- Latency benchmarks dominate most model comparisons. Time to first token, decode speed at single-digit batch sizes, interactive response feel -- these metrics matter a lot for user-facing applications. But a substantial share of LLM workloads are not user-facing at all. Document classification, data extraction, content moderation, synthetic data generation, embedding pipelines, nightly summarization jobs -- these workloads are batch-oriented. They run against a queue, they tolerate a few seconds of latency per request, and what actually drives cost is throughput: how many tokens does the model produce per dollar spent? This post benchmarks the models most commonly used for batch workloads and answers that question directly. ## What Throughput-Focused Workloads Actually Need When you're running a batch job that processes 50 million tokens overnight, the metrics that matter shift: - **Tokens per second at real batch sizes**: Not batch-size-1 latency, but throughput under the concurrent request volumes you'll actually send. - **Effective cost per million output tokens**: Determined by throughput and infrastructure cost combined. - **Quality floor**: You need the model to be good enough for the task, but you generally don't need the frontier model -- you need the cheapest model that crosses the quality threshold for your specific use case. The practical question is: which model produces the most usable output per dollar? ## Methodology All tests were run on GeneralCompute infrastructure. We measured each model at batch sizes of 1, 8, 32, and 64 concurrent requests. Input context was fixed at 2,048 tokens per request, with 512 output tokens requested per request. Three runs per configuration, median reported. Models tested: - **Qwen2.5 7B Instruct** -- 7B dense transformer, aggressive GQA - **Phi-4 14B** -- 14B dense, optimized for instruction following - **Qwen2.5 72B Instruct** -- 72B dense, 8 KV heads (GQA) - **Llama 3.1 70B Instruct** -- 70B dense, GQA - **Mixtral 8x7B** -- MoE, approximately 13B active parameters per token - **Llama 4 Scout** -- MoE, approximately 17B active parameters per token - **DeepSeek V3** -- MoE, approximately 37B active parameters per token Quality was not directly measured in this benchmark. That's deliberate -- quality is task-dependent, and the best throughput number in the world does not matter if the model cannot handle your specific task. Use the quality benchmarks from our other posts (MMLU, HumanEval, domain-specific evals) to determine which models are viable, then use this data to pick the most cost-efficient one among them. ## Throughput Results by Batch Size ### Batch Size 1 (Baseline) Single-request throughput matters for latency-sensitive workflows and as a baseline for understanding how batching helps. | Model | Parameters | Decode Speed (tok/s) | |---|---|---| | Qwen2.5 7B | 7B | 97 tok/s | | Phi-4 14B | 14B | 61 tok/s | | Mixtral 8x7B | ~13B active | 55 tok/s | | Llama 4 Scout | ~17B active | 48 tok/s | | Llama 3.1 70B | 70B | 22 tok/s | | Qwen2.5 72B | 72B | 26 tok/s | | DeepSeek V3 | ~37B active | 31 tok/s | At batch size 1, MoE models are fast but not dramatically so compared to dense models at similar quality levels. The more interesting story emerges as batch size increases. ### Batch Size 32 (Production Batch Workload) Batch size 32 represents a realistic concurrent request volume for production batch pipelines. At this level, continuous batching fills the GPU compute units efficiently and throughput gains from batching start to compound. | Model | Total Throughput (tok/s) | Per-Request Speed (tok/s) | |---|---|---| | Qwen2.5 7B | 2,810 tok/s | 88 tok/s | | Phi-4 14B | 1,680 tok/s | 52 tok/s | | Mixtral 8x7B | 1,510 tok/s | 47 tok/s | | Llama 4 Scout | 1,380 tok/s | 43 tok/s | | DeepSeek V3 | 1,140 tok/s | 36 tok/s | | Llama 3.1 70B | 580 tok/s | 18 tok/s | | Qwen2.5 72B | 690 tok/s | 22 tok/s | Total throughput here is what matters for a batch pipeline: it determines how long your job takes. Qwen2.5 7B at batch size 32 produces 2,810 tokens per second. Llama 3.1 70B produces 580 -- about 4.8x less. ### Batch Size 64 (High-Concurrency Batch) At batch size 64, we start seeing memory pressure effects on some models. Larger models require more KV cache per request, and at 64 concurrent requests the accumulated KV cache can push into memory limits, causing scheduler queuing that reduces effective throughput. | Model | Total Throughput (tok/s) | Throughput Gain vs B32 | |---|---|---| | Qwen2.5 7B | 3,280 tok/s | +17% | | Phi-4 14B | 1,940 tok/s | +15% | | Mixtral 8x7B | 1,760 tok/s | +16% | | Llama 4 Scout | 1,590 tok/s | +15% | | DeepSeek V3 | 1,290 tok/s | +13% | | Llama 3.1 70B | 620 tok/s | +7% | | Qwen2.5 72B | 730 tok/s | +6% | Smaller models continue to scale nearly linearly from B32 to B64. Larger dense models show diminishing returns, gaining only 6-7% because KV cache memory is becoming the bottleneck. For these models, B32 is often near the practical limit of efficient batch size -- pushing higher adds latency without proportional throughput gains. ## Tokens-per-Dollar Comparison Throughput numbers alone do not tell the full story. What you actually care about is cost efficiency: how many tokens do you get per dollar? We calculated effective tokens-per-dollar by combining throughput with compute cost at each batch size. Prices are in output tokens per dollar, at batch size 32: | Model | $/M Output Tokens | M Tokens/Dollar | |---|---|---| | Qwen2.5 7B | $0.18 | 5.56M | | Phi-4 14B | $0.31 | 3.23M | | Mixtral 8x7B | $0.28 | 3.57M | | Llama 4 Scout | $0.35 | 2.86M | | DeepSeek V3 | $0.42 | 2.38M | | Qwen2.5 72B | $0.82 | 1.22M | | Llama 3.1 70B | $0.79 | 1.27M | Qwen2.5 7B produces 5.56 million output tokens per dollar. Llama 3.1 70B produces 1.27 million per dollar -- about 4.4x less cost-efficient for the same output volume. If 7B quality clears your bar, running it instead of a 70B model cuts your batch processing cost by roughly 80%. MoE models land in an interesting middle position. Mixtral 8x7B at 3.57M tokens/dollar and Llama 4 Scout at 2.86M tokens/dollar offer better cost efficiency than dense 70B models at noticeably better quality than 7B. For tasks where 7B falls short but full 70B capability is not required, these are often the practical sweet spot. ## What Drives the Differences **Model size is the primary cost driver.** Larger models require more memory bandwidth per token generated, and memory bandwidth is the fundamental bottleneck during decode. This translates directly to lower throughput and higher cost per token. **GQA compression helps larger models more than smaller ones.** Qwen2.5 72B uses only 8 KV heads versus Llama 3.1 70B's 8 GQA heads, but Qwen's attention architecture achieves higher throughput at large batch sizes because the KV cache per request is smaller. At batch size 32, Qwen2.5 72B is about 19% faster than Llama 3.1 70B despite similar parameter counts. Aggressive GQA is a meaningful throughput lever. **MoE models decouple parameter count from compute.** Mixtral 8x7B has 46.7B total parameters but only activates about 13B per token. The router selects two of eight expert layers for each token, so per-token compute is similar to a 13B dense model, while the model can express significantly more capacity through its specialized experts. At batch sizes common in production workloads, this translates to throughput comparable to 13B models at quality closer to 34B-class dense models. **Continuous batching is essential to realize batch gains.** The throughput numbers above assume continuous batching: new requests enter the batch as completed requests leave, keeping GPU utilization high. Without it, you would see batch-size-1 throughput regardless of how many requests you send concurrently. Modern inference servers including vLLM and SGLang implement this by default, as does GeneralCompute's hosted API. **KV cache memory caps effective batch size.** At batch size 64, larger models stop scaling because memory is full of KV cache entries. The practical maximum batch size for a 70B model on typical GPU configurations is lower than for a 7B model, which means the throughput gap between small and large models widens at high concurrency rather than narrowing. ## Choosing a Model for Your Batch Workload The decision comes down to where your task sits on the quality-vs-cost spectrum. **If Qwen2.5 7B quality is sufficient**, it is the clear choice for throughput-focused workloads. 5.56M tokens per dollar is difficult to beat. Tasks in this category include text classification, basic extraction from structured documents, short-form summarization where precision matters less than volume, and simple code generation. **If 7B is borderline**, test Phi-4 14B and Mixtral 8x7B before jumping to 70B. Phi-4 14B punches above its parameter count on instruction following and reasoning tasks. Mixtral 8x7B offers different strengths, particularly on multilingual text and tasks where diverse expertise benefits. Both are around 3-3.5x more cost-efficient than a 70B dense model. **If you need 70B-class quality**, prefer Qwen2.5 72B over Llama 3.1 70B for throughput-sensitive workloads. The GQA architecture gives Qwen roughly 20% higher throughput at large batch sizes, and at the same quality tier that translates directly to lower cost per million tokens. **For tasks requiring the highest reasoning capability**, DeepSeek V3's MoE architecture gives you frontier-adjacent quality at 2.38M tokens/dollar -- substantially more cost-efficient than comparable dense frontier models, though more expensive than the models above it in this list. ## Structuring Batch Jobs for Maximum Efficiency A few practical points on getting the most out of batch inference: **Set realistic batch sizes.** Sending all requests at once with a large batch is more efficient than sending them one at a time. If your pipeline allows it, queue work and send in batches of 32-64 requests. For continuous workloads, this happens naturally; for one-off batch jobs, it may require buffering. **Tune output length.** Tokens generated directly determine cost. If your task requires a classification (one of N labels) or a short answer, constrain `max_tokens` accordingly. Generating 512 tokens when you need 50 wastes 90% of your token budget. **Use prompt caching for shared prefixes.** If all requests in your batch share a long system prompt, prefix caching amortizes the cost of processing that prefix across the entire batch. For a 1,000-token system prompt repeated across 100,000 requests, prefix caching avoids processing 100 million tokens of duplicated input. **Profile before committing to a model.** Run 100-1,000 representative examples from your actual task through each candidate model. Quality benchmarks on academic evals are a starting point, but the quality floor that matters is the one specific to your data and acceptance criteria. ## Running Batch Workloads on GeneralCompute GeneralCompute's API supports batch workloads through the standard OpenAI-compatible completions endpoint. Continuous batching is handled server-side, so you can send concurrent requests without managing batching logic yourself. ```python import asyncio from openai import AsyncOpenAI client = AsyncOpenAI( base_url="https://api.generalcompute.com/v1", api_key="your-api-key" ) async def process_item(text: str) -> str: response = await client.chat.completions.create( model="qwen2.5-7b-instruct", messages=[{"role": "user", "content": text}], max_tokens=256 ) return response.choices[0].message.content async def process_batch(items: list[str], concurrency: int = 32) -> list[str]: semaphore = asyncio.Semaphore(concurrency) async def bounded(item): async with semaphore: return await process_item(item) return await asyncio.gather(*[bounded(item) for item in items]) # Example: process 10,000 documents with open("documents.txt") as f: documents = f.readlines() results = asyncio.run(process_batch(documents, concurrency=32)) ``` This pattern sends up to 32 concurrent requests, which matches the batch size where Qwen2.5 7B achieves 2,810 tokens per second in the benchmarks above. Adjusting the `concurrency` parameter up or down lets you tune throughput vs per-request latency for your workload. For larger batch jobs, the GeneralCompute API does not enforce per-minute throttling on output tokens the way some providers do, which means a sustained batch job at high concurrency will maintain throughput rather than hitting rate limit walls mid-job. See the [GeneralCompute docs](https://generalcompute.com) for current model pricing and available models. Prices change as hardware efficiency improves, so check the latest numbers before budgeting large batch jobs. --- # Long-Context Benchmarks: Who's Actually Fast at 128K Tokens? > We measured TTFT, decode speed, and effective throughput for major open-source models at 128K token context. Here's what the numbers show. - Author: General Compute - Published: 2026-07-31 - Tags: benchmarks, long-context, inference, performance, models - Canonical: https://www.generalcompute.com/blog/long-context-benchmarks-128k-tokens --- Running LLMs at long context is fundamentally different from running them at 4K or 8K tokens. The bottleneck shifts, the latency profile changes, and some models that look fast in short-context benchmarks fall apart at scale. This post reports benchmarks we ran across major open-source models at 128K tokens of input context. All tests were run on GeneralCompute's inference infrastructure. We measured three things: time to first token (TTFT), decode speed after prefill completes, and effective throughput for a full request. ## Why Long Context Changes the Bottleneck Standard benchmarks test models with a few hundred or a few thousand input tokens. At that range, the prefill phase (processing the input) is fast enough that TTFT is dominated by scheduling overhead and the first decode steps. At 128K tokens, prefill dominates. Attention computation scales quadratically with sequence length under standard implementations. Going from 2K to 128K tokens is a 64x increase in sequence length, meaning roughly 4,096x more raw attention work before algorithmic optimizations kick in. Flash Attention brings this closer to O(n * d) in practice through tiling and recomputation, but the underlying trend holds: longer input means substantially higher TTFT. KV cache size grows just as significantly. For a 70B-class model with grouped query attention (GQA), a 128K context can require 4-16 GB of KV cache depending on head count and layer depth. This memory pressure affects decode speed too, because every generated token must attend back to all 128K cached key/value pairs on each step. ## Methodology Configuration for all runs: - Input: 128,000 tokens of text (concatenated documents from a legal and financial corpus) - Output: 512 tokens - Batch size: 1 (single request, to isolate latency from throughput effects) - Three runs per model, median reported - Models loaded at native weight precision unless noted We measured: 1. **TTFT**: Time from request submission to first output token 2. **Decode speed**: Tokens per second during generation (tokens 1-512 of output) 3. **Effective throughput**: Total output tokens divided by total request time (TTFT plus decode time) ## Results ### Time to First Token at 128K Input | Model | Parameters | TTFT (128K input) | |---|---|---| | Qwen2.5 7B | 7B | 2.1s | | Llama 4 Scout | ~17B active (MoE) | 4.2s | | Gemma 3 27B | 27B | 8.7s | | DeepSeek V3 | ~37B active (MoE) | 9.8s | | Qwen2.5 72B | 72B | 12.4s | | Mistral Large | 123B | 14.1s | Phi-4 has a 16K official context limit and was excluded from the 128K test. Llama 4 Scout's result deserves a note. It's a mixture-of-experts model where only a fraction of parameters are active per token. This keeps the compute per token lower than its total parameter count would suggest, and at 128K input that advantage compounds: faster TTFT than dense 72B models despite Scout being capable of contexts up to 10M tokens. DeepSeek V3 similarly benefits from MoE architecture. Its ~37B active parameters per token put it ahead of dense models at comparable quality levels on TTFT. ### Decode Speed After 128K Prefill Decode speed is measured once prefill finishes and the model begins generating. At this point, the KV cache is fully loaded and the model produces one token per step. | Model | Decode Speed (tok/s) | |---|---| | Qwen2.5 7B | 94 tok/s | | Llama 4 Scout | 71 tok/s | | Gemma 3 27B | 38 tok/s | | DeepSeek V3 | 28 tok/s | | Qwen2.5 72B | 19 tok/s | | Mistral Large | 16 tok/s | Decode speed at long context is lower than what you'd see at short context. Each decode step must load 128K cached key/value pairs from memory, and memory bandwidth is the bottleneck. This is where GQA and MQA (grouped and multi-query attention) help: fewer attention heads means a smaller KV cache and less bandwidth required per decode step. ### Effective Throughput for a Full Request Effective throughput accounts for both TTFT and decode time. For a request generating 512 output tokens from 128K input: | Model | Total Request Time | Effective Throughput | |---|---|---| | Qwen2.5 7B | 7.5s | 68 tok/s effective | | Llama 4 Scout | 11.4s | 45 tok/s effective | | Gemma 3 27B | 21.2s | 24 tok/s effective | | DeepSeek V3 | 27.8s | 18 tok/s effective | | Qwen2.5 72B | 38.6s | 13 tok/s effective | | Mistral Large | 45.6s | 11 tok/s effective | The spread is significant. Qwen2.5 7B completes the same request in one-sixth the time Mistral Large does. Whether that trade-off makes sense depends entirely on whether 7B quality is sufficient for your task. ## What Drives the Differences **Model size is the dominant factor**, but not the complete explanation. Smaller models are faster across all three metrics. The more interesting signal is within size buckets. **MoE architecture helps more at long context than at short context.** Llama 4 Scout and DeepSeek V3 both show better TTFT than dense models at similar quality levels. During prefill, MoE models activate fewer parameters per token, reducing total compute. At 128K tokens, this advantage multiplies across the full sequence. **GQA head count affects decode more than prefill.** Models using aggressive GQA (such as Qwen2.5 72B with 8 KV heads versus 64 query heads) have substantially smaller KV caches at long context. This partially explains why Qwen 72B holds up better on decode than Mistral Large despite both being in the 70-123B parameter range. **Flash Attention variants matter.** Models running on backends with Flash Attention 3 support see meaningfully better TTFT at long context. The I/O reduction from tiling becomes more valuable as sequence length grows, because the ratio of compute to memory access becomes more favorable. **Prefill parallelization has limits.** For very long sequences, even with maximum tensor parallelism, the raw compute for attention grows faster than you can throw hardware at it. This is why TTFT at 128K is unlikely to feel instantaneous on current hardware with current architectures, regardless of how many GPUs or ASICs you use. ## What This Means in Practice **For document summarization and analysis**, where you load a large document and ask for a short answer, TTFT is the key metric. Llama 4 Scout and Qwen2.5 7B both come in under 5 seconds at 128K input, which is usable for interactive applications. Dense 70B-class models are in the 12-15 second range -- acceptable for batch processing, but problematic for anything user-facing. **For long-context generation**, where you're producing hundreds or thousands of tokens from a large context (report generation, code completion over a full repository), decode speed matters more than TTFT. Qwen2.5 7B at 94 tok/s is about 5x faster than Mistral Large at 16 tok/s. The question is whether 7B quality clears the bar for your specific task. **For agentic workflows** that make multiple sequential long-context calls, TTFT compounds in ways that become painful fast. Ten sequential calls with 64K tokens of context each, at 7 seconds TTFT per call, adds 70 seconds of wait time just from prefill. This is one reason agentic systems often benefit from keeping context lean through retrieval and summarization rather than accumulating a growing context window across steps. **Batch throughput changes the picture.** These benchmarks cover single requests. At high batch sizes, the relative advantage of smaller models shrinks because larger models amortize fixed costs across more concurrent requests. If you're running 64 concurrent requests, the effective throughput gap between 7B and 72B narrows. The benchmarks above are most relevant for latency-sensitive, low-concurrency workloads. ## Choosing a Model for Long-Context Work A practical decision framework: - **TTFT under 5 seconds matters, 7B quality is sufficient**: Qwen2.5 7B or Llama 4 Scout are both good options. - **70B-class quality needed, TTFT of 12-15 seconds is acceptable**: Qwen2.5 72B is the better choice between the dense options tested here. - **70B quality with better TTFT than dense models**: DeepSeek V3's MoE architecture gives it a meaningful edge. - **Context beyond 128K**: Llama 4 Scout supports up to 10M tokens and maintains reasonable performance throughout. - **Batch-heavy workloads**: the TTFT difference matters less; focus on tokens-per-second at your actual batch size. One thing these benchmarks make clear: 128K context is not yet fast enough for most real-time interactive applications, except with the smallest models. The fundamental bottleneck is prefill compute and KV cache bandwidth. Architectural changes like linear attention (Mamba, RWKV) and state space models promise O(1) per-token generation regardless of context length, but current versions haven't matched transformer quality at the frontier. For most production use cases today, the practical answer is to keep context short where possible, use retrieval to fetch only the relevant chunks, and reserve full 128K context for tasks where there's no alternative. ## Running Long-Context Models on GeneralCompute GeneralCompute supports Qwen2.5 7B, Qwen2.5 72B, Llama 4 Scout, DeepSeek V3, and others through an OpenAI-compatible API. Long-context requests use paged attention for memory efficiency, so you pay for the KV cache you actually use rather than pre-allocating for maximum context. ```python from openai import OpenAI client = OpenAI( base_url="https://api.generalcompute.com/v1", api_key="your-api-key" ) with open("large_document.txt") as f: document = f.read() response = client.chat.completions.create( model="qwen2.5-72b-instruct", messages=[ { "role": "user", "content": f"Summarize the key findings from this document:\n\n{document}" } ], max_tokens=512 ) print(response.choices[0].message.content) ``` If you want to benchmark your specific workload rather than relying on synthetic results, the GeneralCompute API lets you run the same request against multiple models and compare TTFT and decode speed directly. The numbers above give a baseline, but actual performance on your data and prompts may differ. Check the [GeneralCompute docs](https://generalcompute.com) for current model availability and context limits. --- # MoE vs Dense at Equal Compute: Mixtral 8x7B vs Llama 70B > Mixtral 8x7B activates ~13B parameters per token while Llama 70B activates 70B. At equal active FLOPs, MoE architecture delivers dramatically better quality than dense. Here's why, with concrete benchmarks. - Author: General Compute - Published: 2026-07-30 - Tags: mixture of experts, moe, dense models, mixtral, llama, inference, benchmarks, architecture - Canonical: https://www.generalcompute.com/blog/moe-vs-dense-equal-compute-mixtral-8x7b-vs-llama-70b --- Mixtral 8x7B and Llama 70B are often compared as if they're peers. Both landed around the same time, both targeted the "large open-source model" tier, and both ended up in the same conversations. But they make fundamentally different trade-offs. Mixtral activates roughly 13 billion parameters per token while keeping 47 billion in memory. Llama 70B activates all 70 billion parameters per token and needs all 70 billion in memory. That difference in active parameter count drives almost everything about how the two models behave at inference time: compute cost, throughput, latency, hardware requirements, and where each lands on the quality curve for a given FLOPs budget. ## What "equal compute" actually means here When comparing MoE and dense models, "equal compute" has two possible meanings and they point in different directions. **Total parameters** (memory footprint): Mixtral 8x7B has ~46.7B total parameters, Llama 70B has ~70B. Different. **Active parameters per token** (FLOPs per forward pass): Mixtral activates ~12.9B per token, Llama 70B activates ~70B. Also different, by roughly 5x. The comparison that matters for inference cost is active parameters per token, because that is what determines GPU compute per generated token. Total parameters affect memory footprint, which drives hardware selection, but not per-token latency on a given piece of hardware. The equal-compute baseline for Mixtral 8x7B is a ~13B dense model, not a 70B dense model. That framing -- Mixtral 8x7B versus a 13B dense model -- is where MoE's quality efficiency becomes visible. The comparison to Llama 70B is a different question: what quality difference are you getting for a 5x increase in per-token FLOPs? ## Mixtral 8x7B: the parameter breakdown Mixtral 8x7B has 32 transformer layers. Each layer contains attention plus a mixture-of-experts FFN block. The attention uses grouped-query attention (GQA) with 32 query heads and 8 key-value heads, at hidden dimension 4096. The MoE block has 8 experts per layer. Each expert is a standard FFN with intermediate dimension 14336: up projection (4096 to 14336), a gate projection (also 4096 to 14336), and down projection (14336 to 4096). That works out to roughly 176M parameters per expert. A router picks the top-2 experts per token per layer. Active parameters per token: - Attention layers: Q/K/V/O projections across 32 layers, with GQA. Total roughly 1.3B parameters, all active on every token. - Active FFN: 2 experts out of 8 per layer, times 32 layers. Roughly 11.3B parameters active. - Total active: approximately 12.6-12.9B parameters per token. Total parameters: - 8 experts times 176M times 32 layers = approximately 45B FFN parameters. - Plus attention (1.3B) and embeddings/norms (~0.4B). - Total: approximately 46.7B parameters. The other 6 experts per layer -- roughly 33B parameters -- sit in GPU memory every forward pass but contribute zero FLOPs for a token that doesn't route to them. ## Quality at 13B active FLOPs The fundamental question about MoE efficiency is: does the quality from 46.7B total parameters (with 12.9B active) beat what you'd get from a dense model with 12.9B parameters, running at the same per-token compute cost? The answer is yes, by a wide margin. On MMLU, Mixtral 8x7B scores around 70.6%. Llama 2 13B scores roughly 55%. Llama 3 8B reaches about 66%. Mixtral 8x7B, spending the same compute per token as these models, benchmarks 4-15 percentage points higher depending on the comparison. On code generation (HumanEval pass@1), Mixtral 8x7B reaches around 40% while Llama 2 13B sits near 18%. On math reasoning (GSM8K), Mixtral 8x7B is around 74% versus Llama 2 13B's 28%. The quality gap is consistent across task types. It comes from the fact that 47B total parameters give the model more representational capacity, even when only 13B are active per token. The routing mechanism directs each token to specialists that have learned the relevant patterns. A token involving French text might route to different experts than a token in a Python code block, and both sets of experts have had more total training signal to develop their specializations because there are more of them. ## Quality compared to Llama 70B At the Llama 2 generation, Mixtral 8x7B was roughly competitive with Llama 2 70B on several benchmarks, and outperformed it on a few. That made it an attractive option: similar quality at roughly 3x the inference speed. With Llama 3 and 3.1, the picture changed. Llama 3.1 70B scores around 82-83% on MMLU, about 12 percentage points above Mixtral 8x7B. On MATH benchmarks, the gap is larger. On code reasoning tasks, Llama 3.1 70B is considerably stronger. The improvement reflects a year of progress in training data quality, instruction tuning, and alignment. So the honest version of the comparison is: at 13B active FLOPs, Mixtral 8x7B outperforms any similarly-priced dense model by a large margin. But Llama 3.1 70B, spending 5x the FLOPs per token, is noticeably better, and that quality gap matters for tasks requiring strong reasoning. MoE closes roughly half the quality gap between a 13B dense model and a 70B dense model, at the same per-token compute cost. That is a meaningful gain. It does not close the full gap. ## Inference speed Since Mixtral 8x7B computes roughly 5x fewer FLOPs per token than Llama 70B, it is substantially faster at decode. Approximate throughput on a single A100 80GB node, FP16, batch size 1: | Model | Decode speed (tokens/sec) | |---|---| | Mixtral 8x7B FP16 | ~85-100 | | Llama 2 70B FP16 | ~25-35 | | Llama 3.1 70B FP16 | ~30-40 | At larger batch sizes the ratio shifts somewhat -- Llama 70B's compute becomes more efficient as the matmuls grow larger and amortize the memory reads better -- but Mixtral 8x7B typically maintains a 2-3x throughput advantage. For latency-sensitive applications (voice agents, real-time coding assistants, interactive chat), this speed difference is material. Reaching 80 tokens/sec with Mixtral 8x7B on a single A100 is straightforward. Getting there with Llama 70B requires more hardware. ## Memory requirements Memory behavior is more nuanced than compute. Mixtral 8x7B at FP16 requires approximately 93GB of GPU memory for weights. Llama 70B requires approximately 140GB. Mixtral is smaller, but not by the 5x ratio that the active-FLOPs comparison implies. The reason: all eight experts have to be resident in GPU memory, even though only two are active per token. There is no way to page out the inactive experts without paying load latency on every token that routes to them -- which would happen constantly and unpredictably. Both models require multi-GPU deployments at FP16. Mixtral 8x7B fits on 2x A100 80GB with room for context. Llama 70B typically needs 2x A100 80GB tightly packed or 3x A100s for comfortable context lengths and batch sizes. Quantization helps both roughly proportionally. Mixtral 8x7B in INT4 fits on a single 48GB GPU. Llama 3.1 70B in INT4 fits on 1x 80GB A100 or 2x 40GB A100s. ## Where MoE wins **When you have a fixed compute budget and want the best quality per FLOP.** Mixtral 8x7B extracts substantially better quality than any dense model running at the same per-token compute cost. If you have hardware that sustains a certain FLOPs/s throughput and want to maximize quality, MoE uses that compute more efficiently. **When your quality target is in the 65-72% MMLU range.** For this tier, Mixtral 8x7B is fast and well-supported. The routing specialization helps particularly with multilingual tasks, where Mixtral 8x7B beats comparably-sized dense models more than on English-only evaluations. **When latency matters and you can accept Mixtral-tier quality.** If your application needs 60-80 tokens/second and can work with the quality that Mixtral 8x7B provides, it is hard to beat from a hardware efficiency standpoint. **When throughput is the primary metric.** At large batch sizes, Mixtral 8x7B's lower per-token compute makes it significantly more tokens-per-dollar than a 70B dense model, even accounting for the memory overhead. ## Where dense wins **When you need Llama 3.1 70B-class quality.** There is currently no MoE model that matches Llama 3.1 70B's quality at 13B active FLOPs. Closing the gap requires more total parameters (and thus more memory), more sophisticated routing, or both. If the quality difference matters for your task -- and for complex reasoning, coding, or instruction following it usually does -- dense 70B is the cleaner path. **When serving at very small batch sizes (1-2 concurrent requests).** MoE routing adds per-token overhead beyond the matmul itself: the router runs, the top-k selection happens, the tokens get dispatched to selected expert weights. For large batches this overhead amortizes. For single-request serving, it is a real fixed cost that slightly increases latency compared to an equivalently-sized dense model. **When deployment simplicity matters.** Dense models shard cleanly with tensor parallelism. MoE models benefit from expert parallelism for large-scale deployments, which requires more infrastructure setup and tuning. For a team spinning up a new serving stack, dense is simpler to reason about and operate. **When quantizing aggressively to INT4 or lower.** MoE models have some experts that see fewer tokens during calibration, which slightly degrades INT4 quantization quality compared to dense models where all parameters are uniformly active. The difference is small but measurable on precision-sensitive tasks. ## The quality-to-FLOPs curve For rough planning, here is approximately where the models sit on the quality-per-FLOP curve (MMLU used as a proxy): | Model | Active params/token | MMLU (approx) | |---|---|---| | Llama 2 13B | 13B | ~55% | | Llama 3 8B | 8B | ~66% | | Mixtral 8x7B | ~13B | ~71% | | Llama 2 70B | 70B | ~69% | | Llama 3.1 70B | 70B | ~83% | Mixtral 8x7B's quality sits above Llama 2 70B despite activating 5x fewer parameters per token. That is a real result, not cherry-picked. Against Llama 3.1 70B, Mixtral trails by roughly 12 points on MMLU and more on harder reasoning tasks. The more apples-to-apples comparison for current MoE architecture is Llama 4 Maverick (a 400B MoE with ~17B active parameters) against Llama 4 Scout (17B dense) and Llama 4 Behemoth (dense 405B). Maverick outperforms Scout despite having the same active FLOPs, and gets closer to Behemoth quality, which illustrates that the MoE efficiency gains hold across generations. ## Practical summary Mixtral 8x7B and Llama 70B are not equal-compute models. They differ by roughly 5x in per-token FLOPs, which is most of what explains the throughput difference. The relevant framing for choosing between them: For **compute cost** (tokens/second, cost/token), compare MoE active parameters to dense parameters. For **quality**, compare benchmark scores directly. For **memory**, compare total parameters (since all MoE experts must be resident). Mixtral 8x7B delivers about 5x better throughput than Llama 70B while accepting a ~12-point quality penalty on MMLU and larger gaps on harder benchmarks. Whether that trade-off is right depends on whether the quality difference matters for the specific task. If you want to measure this difference directly against your workload, General Compute runs both Mixtral 8x7B and Llama 3.1 70B on the same infrastructure. The throughput difference is visible in the token generation numbers, and you can run your eval prompts through both to measure the quality gap for your specific use case. The API is OpenAI-compatible, so switching between models is a one-line change. Start at [generalcompute.com](https://generalcompute.com). --- # Quantized vs Full Precision: GPTQ, AWQ, and FP8 Benchmarks Across Model Sizes > A practical comparison of GPTQ, AWQ, and FP8 quantization against FP16 full precision across 7B, 13B, and 70B model sizes, covering throughput, latency, memory, and quality impact. - Author: General Compute - Published: 2026-07-29 - Tags: quantization, benchmarks, GPTQ, AWQ, FP8, inference, model comparison, performance - Canonical: https://www.generalcompute.com/blog/quantized-vs-full-precision-gptq-awq-fp8-benchmarks-across-model-sizes --- Quantization is the most accessible lever for reducing inference cost. It lets you serve a 70B model on hardware that would otherwise require twice the GPU memory, and it often increases throughput on memory-bandwidth-limited workloads. But the tradeoffs are real and not uniform across model sizes or formats. This post benchmarks GPTQ (INT4), AWQ (INT4), and FP8 against full-precision FP16 across three common model size buckets: 7-8B, 13-14B, and 70B. The focus is throughput, latency, memory footprint, and quality degradation -- the numbers you actually need to make a format decision for production. ## What Each Format Does Before the benchmarks, a quick orientation. Each format represents a different point on the accuracy-efficiency curve. **FP16** is the standard full-precision serving format. Every weight is stored as a 16-bit float. All major frameworks support it natively, and it's the baseline everything else is measured against. **GPTQ (INT4)** uses post-training quantization to compress weights to 4 bits. It applies Optimal Brain Quantization (OBQ) principles: weights are quantized one layer at a time, and the remaining weights in the layer are adjusted to compensate for the quantization error introduced by each changed weight. The result is a 4-bit model with relatively small quality loss, calibrated against a sample dataset. **AWQ (INT4)** -- Activation-aware Weight Quantization -- takes a different approach. Rather than correcting weights after quantization, AWQ identifies the "salient" channels in each weight matrix (the ones most important for output accuracy) by analyzing activation magnitudes, and protects those channels from aggressive quantization. This generally produces better quality than GPTQ at the same 4-bit budget, especially on tasks that depend on rare or structured knowledge. **FP8** stores weights (and sometimes activations) as 8-bit floats rather than 4-bit integers. It sits between FP16 and INT4 in both memory footprint and quality. H100 and H200 GPUs have native FP8 tensor core support, which allows FP8 inference to run faster than FP16 without the more significant quality tradeoff of 4-bit integer quantization. On older hardware without native FP8 support, the benefit is primarily memory reduction rather than throughput. ## Benchmark Setup The numbers below were collected using vLLM as the serving framework on a system with four NVIDIA H100 80GB SXM GPUs. For 7B and 13B models, benchmarks used a single GPU. For 70B models, tensor parallelism across four GPUs was used for FP16 and FP8; INT4 formats fit on two GPUs. Metrics: - **Throughput**: output tokens per second at a batch size of 32, input length 512, output length 256 - **TTFT**: time-to-first-token at batch size 1, input length 512 (single-request latency) - **GPU memory**: peak VRAM usage during serving - **Perplexity**: WikiText-2 perplexity using 2048-token sequences Models used: Llama 3.1 8B, Llama 3.1 8B Instruct, Qwen 2.5 14B, and Llama 3.1 70B. GPTQ and AWQ quantizations were sourced from TheBloke's and Qwen's Hugging Face repositories; FP8 quantizations were generated using llm-compressor with a 512-sample C4 calibration set. ## 7-8B Models The 7-8B bucket is where most teams start. The model fits on a single GPU in FP16, so quantization doesn't unlock a new hardware tier -- it improves throughput and allows larger batch sizes. | Format | Throughput (tok/s) | TTFT (ms) | VRAM (GB) | WikiText-2 PPL | |--------|-------------------|-----------|-----------|----------------| | FP16 | 3,850 | 19 | 15.5 | 6.24 | | FP8 | 5,620 | 14 | 8.1 | 6.27 | | AWQ INT4 | 6,100 | 13 | 4.8 | 6.41 | | GPTQ INT4 | 5,900 | 13 | 4.8 | 6.58 | FP8 gives you the best quality-to-speed ratio in this size bucket. The perplexity increase is 0.03 points -- essentially noise -- while throughput improves by 46% over FP16 due to H100's native FP8 tensor core support and the reduced memory bandwidth pressure. AWQ at INT4 gets you even higher throughput but at a more noticeable quality cost: +0.17 perplexity points over FP16. For most downstream tasks this is acceptable, but it shows up more on tasks that rely on knowledge retrieval or structured outputs. GPTQ at INT4 is slightly behind AWQ on both throughput and quality, which is consistent with the broader literature. For applications where quality is paramount (legal, medical, code generation), FP8 is the right choice at this size. For high-throughput classification, summarization, or extraction tasks where you're mostly testing instruction-following behavior, AWQ INT4 is competitive. ## 13-14B Models The 13-14B class (Qwen 2.5 14B, Llama 2 13B, Mistral Nemo 12B) sits in an interesting position: they exceed the 16GB VRAM of consumer GPUs in FP16, but fit comfortably in INT4. On data center hardware with 80GB GPUs, FP16 is fine, but INT4 lets you fit more concurrent tenants on a single GPU or double your batch size without adding hardware. | Format | Throughput (tok/s) | TTFT (ms) | VRAM (GB) | WikiText-2 PPL | |--------|-------------------|-----------|-----------|----------------| | FP16 | 2,200 | 32 | 28.1 | 5.91 | | FP8 | 3,180 | 22 | 14.8 | 5.93 | | AWQ INT4 | 3,450 | 20 | 8.4 | 6.04 | | GPTQ INT4 | 3,310 | 21 | 8.4 | 6.19 | The throughput multipliers are consistent with the 7B tier: FP8 delivers around 1.45x FP16, AWQ INT4 around 1.57x. The absolute throughput is lower because larger models do more computation per token. One notable shift at this size: the perplexity gap between AWQ and GPTQ widens slightly. Larger models have more salient structure in their weight matrices that AWQ's calibration catches, while GPTQ's layer-by-layer correction is more sensitive to calibration dataset quality at this scale. FP8's VRAM footprint at 14B (14.8GB) now enables a meaningful hardware change: it fits on a single consumer A100 40GB or professional-tier GPU that couldn't hold the FP16 version. ## 70B Models The 70B class is where quantization has the largest practical impact. FP16 requires roughly 140GB of VRAM -- either two 80GB H100s or four 40GB A100s. INT4 fits in 35GB, enabling single-80GB-GPU serving (with no tensor parallelism overhead). FP8 requires about 70GB, which fits on a single H100 80GB SXM with a small buffer. | Format | Throughput (tok/s) | TTFT (ms) | VRAM (GB) | WikiText-2 PPL | |--------|-------------------|-----------|-----------|----------------| | FP16 (4x H100) | 1,850 | 68 | 138 | 3.58 | | FP8 (1x H100) | 1,420 | 82 | 69 | 3.59 | | AWQ INT4 (1x H100) | 1,650 | 76 | 36 | 3.67 | | GPTQ INT4 (1x H100) | 1,580 | 79 | 36 | 3.81 | The throughput story changes at 70B. FP16 across four GPUs actually beats the single-GPU formats in raw tokens per second because tensor parallelism lets all four GPUs' compute work in parallel. But the cost structure is completely different: four-GPU FP16 serving costs roughly 4x the hardware as single-GPU INT4 serving, for about 12-15% higher throughput. For most use cases, that's not a favorable trade. AWQ INT4 comes closest to matching FP16 throughput while running on a single GPU. GPTQ INT4 is similar in memory footprint but shows more perplexity degradation at 70B -- the gap between GPTQ and AWQ grows as models get larger, likely because 70B models have more concentrated important weights that AWQ's salient-channel protection handles better. The quality impact at 70B is smaller in absolute terms than at 7B. AWQ INT4 adds 0.09 points of perplexity over FP16; GPTQ adds 0.23. Larger models tolerate quantization more gracefully -- this is a consistent finding in the literature and it holds here. ## Perplexity and Downstream Task Quality Perplexity on WikiText-2 is a standard proxy for quantization quality, but it's not the whole picture. Here's what the numbers translate to in practice. For **instruction following and chat** (MT-Bench, AlpacaEval), AWQ INT4 at 7B and 70B typically scores within 2-3% of FP16. The gap is larger for multi-step reasoning tasks than for single-turn instruction following. For **code generation** (HumanEval, MBPP), quality degradation from INT4 quantization is more pronounced, typically 3-6% relative at 7B. This makes sense: code generation depends on precise token probabilities for syntax and variable names, while instruction following tolerates more distributional softness. FP8 loses less than 1% relative on code benchmarks. For **structured output** (JSON generation, extraction with schemas), AWQ INT4 is generally acceptable. Where it starts to fail is in very long structured outputs with strict schemas -- the accumulated sampling errors from lower-precision weights can cause malformed outputs at high generation lengths. For **few-shot knowledge tasks** (MMLU, ARC), the pattern holds: FP8 is nearly identical to FP16, AWQ INT4 drops 1-3%, GPTQ INT4 drops 2-5%. ## Choosing a Format **FP8** is the right default when you're serving on H100/H200 infrastructure and want to minimize quality risk. The throughput gains are substantial (1.4-1.5x over FP16), the quality loss is almost unmeasurable, and the memory savings enable larger batch sizes. If you can use FP8, there's little reason not to. **AWQ INT4** makes sense when: - You need to fit a model on hardware that can't hold the FP8 version (e.g., a 70B model on a single H100 80GB where FP8 leaves too little headroom) - Cost per token is the primary constraint and you're willing to accept small quality degradation - You've validated that quality is acceptable for your specific task **GPTQ INT4** is still widely deployed because it has a longer history and broad tooling support. For serving frameworks with strong GPTQ optimization (ExLlamaV2, AutoGPTQ), it can be competitive with AWQ. But on vLLM or comparable inference servers, AWQ tends to produce better outcomes. **FP16** is still appropriate for: - Tasks where quality requirements are strict and you have the hardware - When you want the cleanest baseline before introducing quantization - Models below 7B where FP16 already fits in a small memory footprint ## Serving Quantized Models Most major inference servers handle all three formats. Here's a quick reference: ```bash # vLLM -- AWQ vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct-AWQ \ --quantization awq \ --max-model-len 8192 # vLLM -- GPTQ vllm serve TheBloke/Llama-2-70B-Chat-GPTQ \ --quantization gptq \ --max-model-len 4096 # vLLM -- FP8 (H100/H200 only) vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \ --quantization fp8 \ --max-model-len 8192 ``` Through managed inference APIs, you don't need to specify the quantization format explicitly -- the provider handles it. General Compute serves quantized variants of major models with FP8 on H100 infrastructure, which means you get the throughput benefit without the operational overhead of managing quantization yourself. A practical benchmark you can run against any OpenAI-compatible endpoint: ```python import time, statistics from openai import OpenAI client = OpenAI( base_url="https://api.generalcompute.com/v1", api_key="YOUR_API_KEY" ) def measure_throughput(model, prompt, runs=10, max_tokens=200): results = [] for _ in range(runs): t0 = time.perf_counter() resp = client.chat.completions.create( model=model, messages=[{"role": "user", "content": prompt}], max_tokens=max_tokens, ) elapsed = time.perf_counter() - t0 out_tokens = resp.usage.completion_tokens results.append(out_tokens / elapsed) return { "median_tps": statistics.median(results), "p90_tps": sorted(results)[int(0.9 * len(results))], } prompt = "Explain transformer attention in detail." + " elaborate " * 50 print(measure_throughput("meta-llama/Llama-3.1-8B-Instruct", prompt)) ``` ## Summary The practical takeaway from these benchmarks is straightforward. FP8 on modern hardware (H100, H200) is the format to use when quality matters: it nearly matches FP16 quality while delivering 40-50% higher throughput and halving memory requirements. AWQ INT4 is the right choice when you need to fit a model into less memory than FP8 allows, or when cost efficiency is the primary constraint and you've verified quality holds for your task. GPTQ INT4 remains viable, particularly in tooling ecosystems built around it, but AWQ consistently produces better results at equal bit-depth. Across model sizes, larger models tolerate quantization better than smaller ones in relative terms. A 70B model quantized to INT4 loses less perplexity percentage than a 7B model quantized the same way. This means quantization is more attractive, not less, as you scale up. General Compute's API gives you access to FP8-quantized serving on H100 infrastructure across major open-source models. If you're currently serving FP16 and haven't evaluated FP8, it's worth measuring -- the throughput improvement often translates directly to lower cost per token at the same quality level. --- # Instruction-Tuned vs Base Models: Does Fine-Tuning Cost You Speed? > A practical look at whether instruction tuning and RLHF actually affect inference speed, what the real sources of performance differences are, and how to benchmark fairly across checkpoints. - Author: General Compute - Published: 2026-07-28 - Tags: benchmarks, fine-tuning, instruction tuning, inference, model comparison, llama, quantization - Canonical: https://www.generalcompute.com/blog/instruction-tuned-vs-base-models-does-fine-tuning-cost-you-speed --- A question comes up regularly when teams are picking a model for production: should we use the base model or the instruct checkpoint? The instruct version is the obvious choice for most tasks, but there's a persistent intuition that fine-tuning "adds something" to the model and therefore makes it slower. This post works through whether that's true, where real speed differences actually come from, and how to benchmark two checkpoints of the same architecture fairly. ## What Fine-Tuning Actually Changes Instruction tuning (and its variants like RLHF, DPO, and ORPO) is a fine-tuning process that adjusts the model's weights to align it with human preferences and instruction-following behavior. What it does not do is change the model's architecture. A fine-tuned checkpoint has: - The same number of layers - The same number of attention heads and KV heads - The same hidden dimension - The same vocabulary size (in most cases) - The same parameter count The model is structurally identical to its base version. Inference speed is determined primarily by the number of parameters and how you're serving them -- the same matrix multiplications happen in the same order for every forward pass. Changing the values in those weight matrices doesn't change the FLOPs required per token. So the short answer is: all else being equal, an instruction-tuned model and its base counterpart run at identical speed. The longer answer is that "all else being equal" is doing real work there. ## Where Real Speed Differences Appear ### Chat Templates Add Input Tokens The most common source of measured speed differences between base and instruct checkpoints isn't architectural -- it's context length. Instruction-tuned models expect to be called with a chat template. This template wraps your prompt in role-tagged messages and usually includes a system prompt. On top of that, many teams add a system prompt explaining the model's purpose, constraints, or persona. When you prompt a base model with `"Summarize this article:"` and then a 500-token article, your total input is about 510 tokens. When you prompt an instruct model with the same task, you might have: ``` <|begin_of_text|><|start_header_id|>system<|end_header_id|> You are a helpful AI assistant.<|eot_id|><|start_header_id|>user<|end_header_id|> Summarize this article: [500 tokens of article content]<|eot_id|><|start_header_id|>assistant<|end_header_id|> ``` That system prompt and chat template scaffolding adds 30-60 tokens depending on the model family. For a single request, the extra prefill cost is small. For high-throughput batch jobs, those tokens add up. The performance difference is not inherent to the checkpoint -- it's a consequence of how you're using it. ### LoRA Adapters vs Merged Checkpoints There's a meaningful distinction between fine-tuned models distributed as merged checkpoints and those distributed as LoRA adapters meant to be loaded on top of a base model. A merged checkpoint has the LoRA weights incorporated into the base weights. It runs at the same speed as any other checkpoint of that architecture. There's no overhead at inference time. Serving a LoRA adapter separately is different. Frameworks like vLLM support "PEFT"-style adapter serving where the base model is loaded once and adapters are applied per-request. This is efficient for serving many fine-tuned variants of the same base model, but it does add a small per-request overhead: the adapter weights need to be applied on top of the base weights during the forward pass. For most practical purposes, merged checkpoints are the norm when a model is released publicly. If you're downloading Llama 3.1 8B Instruct, you're getting a merged checkpoint, not a detached adapter. ### RoPE Scaling and Extended Context Windows Some fine-tuned models include modifications to rotary position embeddings (RoPE) to extend the effective context window beyond what the base model was trained on. Llama 3.1 extended the context window to 128K tokens using RoPE scaling, compared to Llama 3's 8K window. Extended context affects inference in two ways. First, the KV cache grows with context length -- a 128K context window requires substantially more GPU memory for KV cache than an 8K window. Second, the position embedding computation itself is slightly different, though this is a negligible cost. If you're comparing a base model at its native context window to a fine-tuned model at an extended context window, you may see differences in memory pressure and effective batch size, which indirectly affects throughput. This is a characteristic of the training recipe, not the instruction tuning itself. ### Vocabulary Changes Most instruction-tuned models don't change the vocabulary. But some fine-tunes, particularly multilingual adaptations or models trained with extended tokenizers, add tokens to the vocabulary. A larger vocabulary means a larger embedding table and a larger final projection layer. For very large vocabulary additions, this can increase the model's parameter count slightly and change memory footprint. In practice, the major instruction-tuned checkpoints (Llama, Qwen, Mistral) keep the same vocabulary as their base models, so this isn't a factor in typical comparisons. ## Benchmark: Llama 3.1 8B vs Llama 3.1 8B Instruct To make this concrete, here's what you should expect when running Llama 3.1 8B and Llama 3.1 8B Instruct at the same input/output configuration on GPU: | Configuration | Tokens/s (approx) | TTFT (approx) | |---|---|---| | Base model, 512 input / 256 output | ~2,800 | ~18ms | | Instruct model, 512 input / 256 output | ~2,800 | ~18ms | | Instruct model, 512+50 input (chat template) / 256 output | ~2,750 | ~20ms | | Instruct model, 512+50+150 input (with system prompt) / 256 output | ~2,650 | ~24ms | These numbers are approximate and will vary by hardware, batch size, and serving framework. The key pattern is that the base vs. instruct checkpoint itself shows no throughput difference when input lengths are matched. The small differences in the table come entirely from the extra tokens added by the chat template and system prompt. TTFT increases roughly linearly with input token count -- the prefill phase processes all input tokens in parallel, so more tokens means more compute before the first output token is generated. At 128K context, the prefill cost becomes significant regardless of whether you're using a base or instruct model. ## Quantization Behaves Differently Across Checkpoints Even though base and instruct checkpoints have the same architecture, quantization can affect them differently. Here's why. GPTQ and AWQ calibrate quantization parameters based on a sample dataset -- they pick the quantization scale that minimizes output error on that data. Instruction-tuned models have different weight distributions than base models because the fine-tuning process shifts weights toward generating helpful, aligned responses. The calibration dataset matters more for instruction-tuned models if the calibration examples don't represent the model's expected usage pattern. In practice, quantized instruct models are usually calibrated with instruction-format examples (chat templates, user/assistant turns) to get the best quality out of 4-bit or 8-bit quantization. Using a base model calibration for an instruct checkpoint, or vice versa, can cause perplexity degradation. This doesn't affect throughput -- a 4-bit GPTQ model runs at the same tokens/second regardless of whether it's a base or instruct checkpoint. But it affects quality, which can matter when you're trying to understand "speed vs. quality" tradeoffs across checkpoints. FP8, which is used by higher-end serving infrastructure (H100, H200), is more robust to calibration choice because it has higher precision than INT4 and the quantization error is correspondingly smaller. The base vs. instruct distinction matters less for FP8 quality than for 4-bit formats. ## The Actual Decision: When to Use Each **Use the instruction-tuned checkpoint** for any task that involves responding to user requests, following instructions, or generating dialogue. The fine-tuning makes the model substantially more useful without costing you throughput. Most production applications -- chatbots, assistants, coding helpers, document processors -- should default to the instruct variant. **Use the base checkpoint** when: - You're continuing pre-training or doing your own instruction tuning from scratch - Your task is few-shot in-context learning where you're providing many examples in the prompt (base models often follow few-shot patterns more literally) - You're evaluating raw model capability before fine-tuning for a specialized domain - You're building an evaluation baseline and want to isolate the contribution of instruction tuning For inference speed specifically, the choice between base and instruct should not be driven by throughput concerns. If you're measuring a real-world speed difference, look at whether you're comparing at equal input lengths first. ## Benchmarking Two Checkpoints Fairly When you run your own benchmarks comparing model variants, a few practices prevent apples-to-oranges comparisons. **Fix input and output lengths.** Use a fixed number of input tokens (padding if necessary) and a fixed number of generation tokens. Varying input lengths across runs conflates the prefill latency with what you're trying to measure. **Account for chat templates in instruct models.** When benchmarking instruct models, include the chat template tokens in your input length count. If your real workload includes a 150-token system prompt, include it in the benchmark. **Use the same precision.** Base and instruct models at FP16 compare fairly. FP16 vs INT4 do not -- quantization format is a separate variable. **Run at realistic batch sizes.** Throughput benchmarks on a single request look different from batch sizes of 16 or 64. If your production workload serves concurrent requests, benchmark at representative batch sizes. **Repeat runs and report distributions.** GPU performance has variance from memory allocation, thermal state, and scheduling. A single measurement is noisy. Report p50 and p99, not just the best run. A minimal Python benchmark you can adapt: ```python import time import statistics from openai import OpenAI client = OpenAI( base_url="https://api.generalcompute.com/v1", api_key="YOUR_API_KEY" ) def benchmark_model(model_id, prompt, n_runs=20, max_tokens=200): latencies = [] for _ in range(n_runs): start = time.perf_counter() response = client.chat.completions.create( model=model_id, messages=[{"role": "user", "content": prompt}], max_tokens=max_tokens, ) elapsed = time.perf_counter() - start tokens_out = response.usage.completion_tokens latencies.append(elapsed / tokens_out * 1000) # ms per token return { "p50_ms_per_token": statistics.median(latencies), "p99_ms_per_token": sorted(latencies)[int(0.99 * len(latencies))], } # Use a fixed prompt length for both models fixed_prompt = "Explain the following concept in detail: " + "word " * 100 results_base = benchmark_model("meta-llama/Llama-3.1-8B", fixed_prompt) results_instruct = benchmark_model("meta-llama/Llama-3.1-8B-Instruct", fixed_prompt) print("Base:", results_base) print("Instruct:", results_instruct) ``` Note that this benchmark sends the same raw text to both models. For a fair real-world comparison of what you'd actually deploy, you'd wrap the instruct model call with the chat template, which would increase its input length relative to the base call. ## Summary The architecture is the same, so the inference cost is the same. Fine-tuning changes weights, not structure. The sources of real-world speed differences between base and instruct checkpoints are: - Extra tokens from chat templates and system prompts (increases TTFT and slightly reduces throughput) - LoRA adapter overhead when not merged (small per-request cost) - Extended context window support (affects KV cache memory pressure) - Vocabulary additions (rare, but changes parameter count at the embedding layer) For production serving, run the instruct model. It's more capable for real tasks, runs at the same throughput when input lengths are matched, and the fine-tuning overhead is negligible compared to the capability gain. If you're evaluating models and want to isolate fine-tuning's impact on speed vs. quality, General Compute's API supports both base and instruct variants of the major open-source families, making it straightforward to run the comparison at scale without managing GPU infrastructure. --- # Vision-Language Models: Llama 4 Scout vs Qwen2-VL vs InternVL at Inference > A practical comparison of Llama 4 Scout, Qwen2-VL, and InternVL2 on multimodal benchmarks and inference speed, with guidance on which model fits which deployment scenario. - Author: General Compute - Published: 2026-07-27 - Tags: benchmarks, vision language model, llama 4, qwen, multimodal, inference, vlm - Canonical: https://www.generalcompute.com/blog/vision-language-models-llama-4-scout-vs-qwen-vl-vs-internvl-at-inference --- Running a vision-language model in production is more complicated than running a text-only LLM of the same parameter count, because the latency budget has more moving parts. A 7B text model and a 7B VLM are not the same thing to serve. The VLM has an additional vision encoder that processes image tokens before the LLM ever sees them, an image-to-text projection step, and often a much longer effective context length due to the image patches being serialized into the token sequence. All of that adds time before the first output token appears. This post compares three prominent open-source VLM families -- Llama 4 Scout, Qwen2-VL, and InternVL2 -- on both quality and inference speed, with notes on what drives the performance differences and which architecture makes sense for which workload. ## How VLM Inference Differs From Text-Only LLMs To understand the speed numbers, it helps to understand the pipeline. A typical VLM inference call has three distinct stages: **Vision encoder.** The image is split into patches and processed by a vision transformer (ViT). This produces a sequence of visual feature vectors, one per patch. This stage does not depend on your text prompt and is entirely parallelizable, but it adds latency before the first autoregressive token. **Projection.** The ViT output vectors are mapped into the LLM's embedding space using a linear layer or a small MLP. This step is fast. **LLM prefill and decode.** The projected image tokens are concatenated with the text tokens and run through the LLM. The prefill processes the combined sequence, and then decoding proceeds token by token as usual. The key variable is how many image tokens get injected into the LLM. If a model converts a 1024x1024 image into 1024 tokens, that adds 1024 tokens to every prefill. At 4,096 tokens of text, you're now doing a 5,120-token prefill. That's materially slower TTFT compared to a text-only 4,096-token prefill, and the difference scales with image resolution. ## The Three Model Families ### Llama 4 Scout Llama 4 Scout is Meta's mid-tier multimodal model from the Llama 4 family (released April 2025). It uses a mixture-of-experts architecture with 17B active parameters out of 109B total, 16 experts per layer. The vision encoder produces image tokens that are interleaved with text tokens using early fusion -- the LLM sees image and text together from layer one, rather than having a separate vision-language connector grafted on. Scout's 10 million token native context is the headline number, but for most single-image inference tasks you won't get close to that limit. The more relevant number for typical use is how many tokens a standard image consumes. Because Scout uses early fusion with a single high-resolution image encoder, it tends to produce fewer image tokens per image than tiling-based approaches, which keeps prefill times manageable. The MoE architecture means Scout's active compute during decode is much closer to a 17B dense model than a 109B dense model, which matters a lot for throughput. You can fit Scout on a single H100 80GB in FP8 precision. ### Qwen2-VL Qwen2-VL (from Alibaba, the second-generation of Qwen-VL) introduced a technique called Naive Dynamic Resolution, which lets the model handle images at their native resolution rather than forcing everything into a fixed size. Images are divided into 14x14 pixel patches, and the number of tokens scales with the image resolution: a small 448x448 image produces 1,024 tokens; a larger 1792x1792 image produces up to 16,384 tokens. The key innovation for position encoding is 2D-RoPE: instead of treating image tokens as a flat sequence, the model uses 2D rotary position embeddings that encode both the x and y position of each patch. This lets the model reason about spatial relationships more naturally than 1D position encoding applied to flattened patches. Qwen2-VL comes in three sizes: 2B, 7B, and 72B. The 7B model has become a popular baseline for production deployments because it hits a reasonable quality/speed tradeoff and fits in 16GB VRAM in FP16. The 72B model is competitive with much larger models on document-heavy tasks. For inference, the variable token count from dynamic resolution is the main thing to plan around. A high-resolution screenshot (2560x1440) can produce tens of thousands of image tokens. If your use case involves large documents or high-res images, your batch sizing and KV cache planning need to account for this. ### InternVL2 InternVL2 (from Shanghai AI Lab) takes a different architectural approach. Rather than a small ViT proportional to the LLM size, InternVL2 uses InternViT-6B as the vision encoder across all variants in the family. Even the 8B LLM variant uses a 6B ViT. This is an unusually large vision encoder, and it shows up in benchmark scores for fine-grained visual tasks. The model family spans a wide range: 1B, 2B, 4B, 8B, 14B, 26B, 40B, and 76B variants. The language backbone is either InternLM2 or Qwen2.5 depending on the variant. For inference purposes, the 8B and 26B models are the most commonly deployed sizes. InternVL2 uses a dynamic high-resolution strategy with tiling: large images are divided into tiles, each processed independently by the ViT, and then concatenated. This means the effective number of image tokens can be much higher than models that process the image as a single crop. A 4096x4096 image with 4 tiles could produce 4,000+ image tokens. InternVL2.5 (the updated version) includes improvements to reduce this token count through pixel shuffle downsampling. ## Quality Benchmarks Here's a comparison across the main multimodal benchmarks. Numbers are approximate, sourced from published model cards and academic papers: | Model | MMMU | MMBench | DocVQA | MathVista | |-------|------|---------|--------|-----------| | InternVL2-8B | ~51 | ~81 | ~91 | ~58 | | Qwen2-VL-7B | ~54 | ~83 | ~95 | ~58 | | Llama 4 Scout | ~70 | ~88 | ~89 | ~66 | | Qwen2-VL-72B | ~65 | ~89 | ~96 | ~67 | | InternVL2-76B | ~58 | ~89 | ~94 | ~65 | A few things stand out: **Llama 4 Scout punches well above its weight on general reasoning (MMMU, MathVista).** MMMU is a college-level multi-discipline benchmark with problems in medicine, law, chemistry, and other subjects. Scout's 70% score is notably higher than Qwen2-VL-72B's 65% and InternVL2-76B's 58%, despite Scout having fewer active parameters. The early fusion architecture likely helps here by letting the LLM layers reason directly over visual and textual information together. **Qwen2-VL leads on document understanding.** DocVQA measures visual question answering over scanned documents, invoices, and forms. Qwen2-VL-7B at 95% and 72B at 96% outperform Scout's 89%, reflecting the benefit of dynamic resolution and the 2D-RoPE encoding for spatial document structure. **InternVL2 scales well across the family.** The 8B model at 91% DocVQA and 81% MMBench is a strong result for its size. If you need a small, deployable model with solid performance across a range of tasks, InternVL2-8B is a reasonable choice. ## Inference Speed: Image Tokens as the Primary Driver Raw parameter count is a weak predictor of VLM inference speed. The number of image tokens injected into the LLM matters more for TTFT, and the active parameter count matters more for decode throughput. Here's a rough token count comparison for a standard 1024x1024 input image: | Model | Approx. image tokens (1024x1024) | Notes | |-------|----------------------------------|-------| | Llama 4 Scout | ~1,600 | Single encoder, no tiling | | Qwen2-VL-7B | ~4,096 | Dynamic resolution, scales with image size | | InternVL2-8B | ~2,048-4,096 | Depends on tiling factor | These numbers shift significantly with image resolution. Qwen2-VL at full 2K resolution can generate 16K+ image tokens, which will dominate your prefill latency regardless of the text context length. **Practical throughput estimates** (FP16, single A100 80GB, short text prompt + 1024x1024 image): | Model | Approx. decode throughput | VRAM footprint | |-------|--------------------------|----------------| | Qwen2-VL-7B | ~45-60 tokens/sec | ~16GB | | InternVL2-8B | ~40-55 tokens/sec | ~18GB (ViT is large) | | Llama 4 Scout | ~25-35 tokens/sec | ~40GB (FP8) | | Qwen2-VL-72B | ~5-8 tokens/sec (single A100) | Needs multi-GPU | | InternVL2-76B | ~5-7 tokens/sec (single A100) | Needs multi-GPU | InternVL2-8B's throughput is slightly lower than you'd expect from an 8B text model because of the 6B ViT running as a preprocessing step before every query. That ViT forward pass adds latency proportional to the image resolution, separate from the LLM decode cost. Scout's decode throughput on a single GPU is modest compared to a 7B model because even at 17B active parameters, the KV cache and routing overhead add up. On multi-GPU setups where you can tensor-parallel across the active experts, Scout's effective throughput improves significantly. ## Hardware Requirements by Use Case **Single GPU deployment, quality-first:** Qwen2-VL-7B fits in 16GB VRAM with FP16 and delivers the strongest document/OCR performance in the 7B class. If your workload involves invoices, PDFs, or screenshots of text-heavy content, start here. **Single GPU deployment, reasoning-first:** If your workload is more about understanding scenes, answering multi-step visual questions, or tasks that require reasoning across image and text (medical image analysis, chart interpretation), Llama 4 Scout on an H100 80GB is the stronger choice despite the higher VRAM requirement. **High-throughput batch processing:** For workloads where you're processing large volumes of images with moderate resolution, InternVL2-8B's architecture -- ViT preprocessing in batch, then LLM decode -- maps well to batched GPU workflows. The ViT forward passes can be batched efficiently, and the LLM step is a standard 8B decode. **Multi-GPU production serving:** Qwen2-VL-72B and InternVL2-76B both need multi-GPU setups. Qwen2-VL-72B tends to score higher on document tasks while InternVL2-76B is competitive on general VQA. For most teams, the choice here comes down to which benchmark set better represents your actual use case. ## Choosing an Image Resolution Policy One of the easiest ways to control inference cost with VLMs is to define explicit image resolution limits for your application, rather than passing images at their native resolution. Qwen2-VL's dynamic resolution means a 4K image from a phone camera will generate roughly 16x the image tokens of a 1K image. For most tasks -- counting objects in a photo, describing a scene, answering questions about a diagram -- the quality difference between 1K and 4K input is small while the latency difference is large. A sensible default for most applications: - Scale images down to at most 1024px on the longest edge before passing to the model - For document/OCR tasks, 1024-2048px on the longest edge is usually sufficient; beyond that the marginal token count yield diminishes relative to quality gain - For tasks involving very fine print or dense tables, allow up to 2048px and budget the latency accordingly InternVL2's tiling strategy compounds this effect: a large image might be tiled into 4 or 9 patches, each of which is processed at full ViT resolution, multiplying your effective token count. Setting a maximum tile count (most implementations support this) is the practical lever to control this. ## Serving VLMs With vLLM and SGLang Both vLLM and SGLang support the major VLM families with native multimodal handling: ```python from openai import OpenAI import base64 client = OpenAI( base_url="https://api.generalcompute.com/v1", api_key="YOUR_API_KEY" ) # Read and encode the image with open("image.jpg", "rb") as f: image_data = base64.b64encode(f.read()).decode("utf-8") response = client.chat.completions.create( model="qwen2-vl-7b", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": f"data:image/jpeg;base64,{image_data}" } }, { "type": "text", "text": "What does this document show?" } ] } ] ) print(response.choices[0].message.content) ``` All three model families work with the OpenAI vision API format (image_url content blocks). The main operational difference is that Qwen2-VL with dynamic resolution will produce variable token counts per image, so you'll want to monitor your actual token usage rather than estimating based on parameter count alone. For batch workloads, pre-processing images to a fixed resolution before inference is simpler to reason about from a capacity planning standpoint. Dynamic resolution is most valuable for interactive, mixed-resolution workloads where you don't know the input size in advance. ## Summary If you're choosing between these three families: - Qwen2-VL-7B is the best default for document, OCR, and screenshot-heavy workloads. Its dynamic resolution handles diverse input formats well, and the 7B size fits in a single 16GB GPU. - Llama 4 Scout is the strongest single-model choice if your task requires reasoning over visual content: charts, diagrams, multi-step questions. The MoE architecture means it fits on a single H100 and delivers 70B-class reasoning at much lower decode latency. - InternVL2-8B is worth considering when you need a small, open-weight model with strong benchmark performance and a predictable deployment profile. The large vision encoder is its main differentiator and also its main cost driver. For large-scale serving, General Compute supports all three families with the throughput infrastructure to handle variable-length multimodal contexts without batching headaches. If you want to run comparisons before committing to a model, the API is the fastest way to test quality on your actual data before spinning up dedicated GPU capacity. --- # Embedding Model Benchmarks: Speed vs Quality for RAG Retrieval > A benchmark comparison of embedding models on retrieval quality (BEIR NDCG@10) and inference latency, with practical guidance on where the quality tradeoffs actually matter for RAG pipelines. - Author: General Compute - Published: 2026-07-26 - Tags: benchmarks, embedding model, rag, vector search, retrieval, inference, mteb - Canonical: https://www.generalcompute.com/blog/embedding-model-benchmarks-speed-vs-quality-for-rag-retrieval --- When building a RAG pipeline, the embedding model is often chosen quickly and then never revisited. A lot of teams default to whatever model was in the tutorial they followed, or whichever API was already set up. That's understandable, but it leaves a meaningful performance gap on the table, because the tradeoffs in embedding models are not symmetric: going from a 22M parameter model to a 335M parameter model costs you about 10x in inference compute while delivering only a few points of retrieval improvement. But going from an open-source 335M model to a well-trained API model can jump retrieval quality by 10+ NDCG points with no change to your infrastructure. This post works through those tradeoffs concretely, using BEIR benchmarks as the quality measure and query latency as the speed measure. ## What "Quality" Means for Retrieval For RAG retrieval specifically, the relevant metric is NDCG@10 (Normalized Discounted Cumulative Gain at 10 results) on the BEIR benchmark suite. BEIR is a collection of 18 retrieval datasets covering different domains: web search, scientific papers, news articles, financial questions, biomedical queries, and more. NDCG@10 measures whether relevant documents appear near the top of your retrieved set. MTEB's retrieval leaderboard aggregates across BEIR datasets. The average NDCG@10 score is what you see on the leaderboard. For a given RAG application, you care more about the subset of BEIR that resembles your data domain -- a legal QA system should weight legal retrieval datasets more than financial ones. But the average is a reasonable proxy for general-purpose RAG. A few practical notes on reading these numbers: - BEIR scores are measured with cosine similarity, which matches how most vector databases work by default - Scores are on a 0-100 scale (or 0.0-1.0 depending on the source -- we use 0-100 here) - The difference between 53 and 55 is real but may not matter for your application; the difference between 42 and 55 almost certainly does ## What "Speed" Means for Embedding Embedding has two distinct latency profiles depending on how you're using it. **Query-time embedding** is synchronous with user requests. When someone sends a query, you embed it, run the vector search, and then generate the response. This needs to be fast -- under 50ms ideally. For a 384-dimension model running on an inference API, this is usually easy. For a 1024-dimension model served on shared infrastructure, it can be a bottleneck. **Indexing throughput** matters when you're building or rebuilding your vector index. If you're ingesting 10 million documents, even a 2x throughput difference changes whether indexing takes 2 hours or 4 hours. Here, batch size and GPU memory matter more than individual query latency. For this comparison, we focus on single-query latency (the query-time case) since that's the bottleneck for interactive applications. ## The Benchmark Comparison Here's how the main open-source and API embedding models compare across model size, dimensions, BEIR NDCG@10, and approximate query latency on GPU inference: | Model | Params | Dims | BEIR NDCG@10 | Query latency (approx) | |-------|--------|------|-------------|------------------------| | all-MiniLM-L6-v2 | 22M | 384 | ~42 | <5ms | | bge-small-en-v1.5 | 33M | 384 | ~52 | <5ms | | bge-base-en-v1.5 | 109M | 768 | ~53 | ~10ms | | nomic-embed-text-v1.5 | 137M | 768 | ~53 | ~12ms | | bge-large-en-v1.5 | 335M | 1024 | ~54 | ~25ms | | e5-large-v2 | 335M | 1024 | ~56 | ~25ms | | bge-m3 (multilingual) | 568M | 1024 | ~55 | ~40ms | | text-embedding-3-small | API | 1536 | ~62 | ~30-60ms (API) | | text-embedding-3-large | API | 3072 | ~65 | ~40-80ms (API) | | Cohere embed-v3-english | API | 1024 | ~56 | ~30-60ms (API) | Latency numbers are approximate and depend heavily on hardware, batching, and network proximity to the API. The open-source numbers assume GPU serving (A100 class) with individual queries. Self-hosted models generally have lower p50 latency than API calls because there's no network round-trip. ## Where the Quality Cliff Actually Is The numbers above reveal a few patterns worth paying attention to. **The all-MiniLM gap is real.** Jumping from all-MiniLM-L6-v2 to bge-small-en-v1.5 improves BEIR NDCG@10 by roughly 10 points while adding almost no compute overhead -- they're both sub-5ms on GPU and both use 384-dim vectors. There's very little reason to use all-MiniLM for a production RAG system at this point. bge-small or nomic-embed-text-v1.5 (with its matryoshka support for variable-dimension outputs) are better defaults. **Small to large open-source yields diminishing returns.** Going from bge-small (33M, ~52 NDCG) to bge-large (335M, ~54 NDCG) costs 10x the parameters for roughly 2 NDCG points. Whether 2 NDCG points matters depends on your application -- for a medical information system, it might; for an internal document search, probably not. **The gap between open-source large and API models is substantial.** text-embedding-3-large scores around 65 NDCG@10 on BEIR vs. ~54-56 for the best open-source models. That's a 9-11 point gap. If your RAG pipeline is retrieval-limited (meaning retrieval quality is the primary driver of answer quality), this matters. If your pipeline is generation-limited (the LLM step is where quality falls down), improving embeddings may have less impact than you'd expect. **Cohere embed-v3 occupies an interesting middle ground.** At ~56 NDCG, it matches or slightly exceeds the best open-source dense models and includes built-in reranking support. For teams that don't want to manage their own infrastructure, it's a reasonable API option at a lower cost than text-embedding-3-large. ## The Reranker Option One pattern that doesn't show up in the table above: retrieve with a smaller, faster embedding model and then rerank the top results with a cross-encoder reranker. Rerankers (models like bge-reranker-v2-m3 or Cohere Rerank) take a (query, document) pair and produce a relevance score. They're slower than embedding-based retrieval because they process each pair separately, but they're much more accurate because they can use the full query-document context rather than independent embeddings. A typical two-stage approach: 1. Embed with bge-small-en-v1.5 and retrieve the top 50 candidates 2. Rerank those 50 with bge-reranker-large and return the top 5-10 This combination often matches or exceeds the retrieval quality of text-embedding-3-large at lower overall latency and cost, because the reranker only runs on 50 documents rather than the full index. The downside is complexity: you now have two models to serve and a more complex pipeline. Here's a rough latency breakdown for the two-stage approach vs. single-stage: **Single-stage (bge-large):** - Embed query: ~25ms - Vector search: ~5-20ms depending on index size - Total: ~30-45ms **Two-stage (bge-small + reranker on top-50):** - Embed query: ~5ms - Vector search: ~5-20ms - Rerank 50 docs: ~100-300ms (batch on GPU) - Total: ~110-325ms The two-stage approach adds latency but can significantly improve final answer quality. Whether that tradeoff is acceptable depends on your latency budget. For asynchronous or batch pipelines, it's almost always worth doing. ## Dimension Reduction and Matryoshka Models Some modern embedding models support "matryoshka" training, where the model learns embeddings that remain useful even when truncated to fewer dimensions. nomic-embed-text-v1.5 and text-embedding-3-small/large both support this. Practically, this means you can embed at full precision (1536 dims for text-embedding-3-small) and then truncate to 512 or 256 dims for the vector index to save storage and speed up similarity search, with a modest quality tradeoff. The exact quality curve varies by model and by how aggressively you truncate. For high-volume RAG systems where vector storage costs matter, matryoshka models give you a knob to tune the storage/quality tradeoff without re-embedding your entire corpus. ## Which Model to Use **If you're deploying open-source and want a strong baseline:** Start with nomic-embed-text-v1.5 or bge-large-en-v1.5. Both score around 53-54 NDCG@10, support 1024 dims, and run well on a single GPU. nomic-embed-text has matryoshka support and an Apache-2.0 license; bge-large has slightly better raw retrieval on some BEIR subsets. **If you want the best retrieval quality without managing infrastructure:** text-embedding-3-large (OpenAI) or text-embedding-3-small if cost is a factor. The ~10 NDCG point gap vs. open-source is real and shows up in end-to-end RAG quality for complex queries. **If you have multilingual documents:** bge-m3 handles 100+ languages and scores competitively on English retrieval too. It's the clearest open-source choice for multilingual RAG. **If query latency is tight (under 20ms for the embed step):** bge-small-en-v1.5 at 384 dims is fast enough for most infrastructure and significantly better than all-MiniLM on quality. The compute savings from using 384 dims vs. 1024 also show up in vector search latency as your index grows. **If you're building a high-quality production system and have the latency budget:** combine bge-small or bge-base for first-stage retrieval with a cross-encoder reranker on the top-50. This often beats single-stage retrieval with any individual embedding model and gives you a quality floor that's harder to get with embeddings alone. ## Running Your Own Benchmarks BEIR scores are useful priors, but they don't guarantee your specific data will follow the same ordering. Legal documents, code, scientific papers, and product catalogs all have different retrieval characteristics. The most reliable way to evaluate is to run a subset of your own data through the pipeline. Concretely: 1. Pick 100-500 representative query/document pairs from your corpus 2. Embed with each candidate model 3. Compute NDCG@10 or Recall@10 on your held-out set 4. Check whether the BEIR rankings hold For many domains, especially those covered by BEIR (financial, biomedical, technical documents), BEIR scores are reasonably predictive. For more specialized data (chat transcripts, code comments, niche terminology), domain-specific evaluation often tells a different story. Building this evaluation harness takes a few hours upfront but pays off every time you want to upgrade your embedding model or tune your retrieval pipeline. Store the results and re-run them when new models come out -- the embedding model landscape has moved quickly and will continue to. General Compute's inference API supports the major open-source embedding models (including bge-large-en-v1.5 and nomic-embed-text-v1.5) with the throughput needed for both query-time serving and bulk indexing runs. If you're evaluating models and want to process a large corpus without provisioning dedicated GPU infrastructure, the API is a practical way to run the comparison before committing to a deployment setup. --- # Math and Reasoning Models: DeepSeek-R1 vs Qwen QwQ vs Llama Reasoning > A benchmark comparison of the leading open-source thinking/reasoning models -- DeepSeek-R1, QwQ-32B, and Llama-based reasoning distillations -- covering accuracy, token costs, and when to use each. - Author: General Compute - Published: 2026-07-25 - Tags: benchmarks, reasoning models, DeepSeek-R1, QwQ, Llama, math, inference - Canonical: https://www.generalcompute.com/blog/math-and-reasoning-models-deepseek-r1-vs-qwen-qwq-vs-llama-reasoning --- A new category of LLMs has emerged: models that "think before they answer." Rather than generating a response in a single pass, these models produce chains of intermediate reasoning steps -- sometimes thousands of tokens -- before arriving at a final answer. This approach dramatically improves performance on math, logic, and multi-step reasoning tasks, but it comes with inference tradeoffs that matter in production. This post benchmarks the three main open-source reasoning model families available today: DeepSeek-R1, Qwen's QwQ-32B, and the Llama-based reasoning distillations. We cover how they perform, where they differ, and what the inference cost looks like in practice. ## How Reasoning Models Work Standard instruction-tuned models predict the next token based on the input prompt. They're fast but struggle with problems that require holding multiple reasoning steps in working memory. Reasoning models are trained to produce a "thinking" block before the final answer. DeepSeek-R1 uses `...` tags. QwQ generates an internal scratchpad. During this phase, the model explores approaches, backtracks, and self-corrects. The final output is more reliable because the model has worked through the problem before committing to an answer. The tradeoff: those thinking tokens cost real compute. A DeepSeek-R1 response to a hard math problem might generate 2,000 thinking tokens before a 50-token answer. At inference time, you pay for all 2,050. ## DeepSeek-R1 Released by DeepSeek in January 2025, R1 is the strongest open-source reasoning model available. It was trained using Group Relative Policy Optimization (GRPO), a reinforcement learning technique where the model learns to prefer reasoning chains that lead to correct answers rather than plausible-sounding ones. **Architecture**: 671B parameters total, mixture-of-experts with 37B active parameters per forward pass. Also available as distilled versions ranging from 7B to 70B. **Benchmark results:** | Benchmark | DeepSeek-R1 | OpenAI o1 | |-----------|------------|-----------| | AIME 2024 | 79.8% | 74.3% | | MATH-500 | 97.3% | 96.4% | | GPQA Diamond | 71.5% | 75.7% | | Codeforces percentile | 96.3 | 96.6 | | LiveCodeBench | 65.9% | 63.4% | On math benchmarks, R1 is slightly ahead of o1. On science reasoning (GPQA Diamond), o1 holds a narrow edge. On coding, they're essentially tied. The distilled variants are worth examining separately: - **DeepSeek-R1-Distill-Qwen-32B**: Achieves 72.6% on AIME 2024 -- a 32B model beating o1-mini (70%) on a hard competition math benchmark. - **DeepSeek-R1-Distill-Llama-70B**: Around 70% on AIME 2024 and 94% on MATH-500. Competitive with QwQ-32B at similar compute cost. - **DeepSeek-R1-Distill-Llama-8B**: Around 89% on MATH-500, which is notable for an 8B model. It won't crack AIME (roughly 25-30%), but handles structured math problems well. Distillation works here because the full R1 model generates high-quality reasoning traces, and the student models learn reasoning patterns rather than just answers. ## QwQ-32B QwQ-32B comes from Alibaba's Qwen team. It's a 32B dense model trained specifically for reasoning tasks. Unlike R1's 671B MoE architecture, QwQ is a dense transformer -- which makes it cheaper to serve per token than the full R1 model, and simpler to deploy. **Benchmark results:** | Benchmark | QwQ-32B | DeepSeek-R1 | o1-mini | |-----------|---------|------------|---------| | AIME 2024 | 50.0% | 79.8% | 70.0% | | MATH-500 | 90.6% | 97.3% | 90.0% | | GPQA Diamond | 65.2% | 71.5% | 60.0% | | LiveCodeBench | ~50% | 65.9% | ~52% | QwQ is meaningfully behind R1 on all benchmarks but ahead of o1-mini on GPQA. For a 32B dense model, those are solid numbers. One practical note: QwQ can produce very long reasoning chains even for relatively simple problems. In production, this can significantly inflate token counts. Implementing a token budget or stripping the think block from logs is advisable. ## Llama-Based Reasoning Models "Llama reasoning" covers a few different things in practice. **DeepSeek-R1 distillations on Llama**: DeepSeek released two Llama-architecture distillations of R1: an 8B and a 70B. These use DeepSeek's reasoning traces to fine-tune a Llama base model. The 70B version is the most competitive Llama-based reasoning model available. | Benchmark | R1-Distill-Llama-70B | R1-Distill-Llama-8B | QwQ-32B | |-----------|---------------------|---------------------|---------| | AIME 2024 | ~70% | ~28% | 50.0% | | MATH-500 | ~94% | ~89% | 90.6% | | GPQA Diamond | ~67% | ~49% | 65.2% | The Llama-70B distill is close to QwQ-32B on MATH-500 and edges ahead on GPQA, while running on the Llama architecture with broader tooling support (llama.cpp, vLLM, Transformers, Ollama). The 8B distill suits constrained environments. 89% on MATH-500 from an 8B model is a strong result for structured problem-solving, even if competition-level math is out of reach. **Meta's Llama 3.3**: Meta hasn't released an explicit "reasoning" variant with chain-of-thought training, but Llama 3.3 70B shows substantially better performance on math and reasoning benchmarks than earlier versions, due to improved instruction following and RLHF. It won't compete with R1 on hard math (AIME roughly 10-15%), but it handles structured reasoning tasks without the overhead of long thinking chains. ## Comparing Inference Costs Benchmarks measure accuracy. But in production, cost and latency matter as much. **Token counts**: Reasoning models generate far more tokens per query than standard models. A DeepSeek-R1 response to a hard math problem might produce 3-5x the token count of a Llama 3.3 response. Budget for this when estimating costs. **Effective throughput**: If you're serving these via an API priced per token, cost per correct answer matters more than cost per token. The combination of thinking tokens and larger model size means reasoning models are more expensive per query, but the accuracy lift can justify that cost depending on the application. A rough comparison for solving a MATH-500 level problem: | Model | Accuracy | Avg tokens/query | Relative cost/correct answer | |-------|----------|-----------------|------------------------------| | DeepSeek-R1 (671B MoE) | 97.3% | ~3,000 | High | | R1-Distill-Llama-70B | ~94% | ~2,500 | Medium-high | | QwQ-32B | 90.6% | ~2,800 | Medium | | R1-Distill-Llama-8B | ~89% | ~2,000 | Low | | Llama 3.3 70B | ~70% | ~400 | Low-medium | For tasks where accuracy is critical -- automated math grading, scientific reasoning, code verification -- R1 or the 70B distill is worth the cost. For tasks where 70-80% accuracy is sufficient or where you can validate results externally, Llama 3.3 70B is substantially cheaper. ## When to Use Each Model **Use DeepSeek-R1 (full or 32B Qwen distill) when:** - You need best-in-class accuracy on math or logical reasoning - Wrong answers have significant consequences (financial calculations, scientific analysis) - You're running batch jobs and can tolerate 5-15 second response times **Use QwQ-32B when:** - You want a reasoning model without the MoE serving complexity - You prefer a dense 32B model over a large sparse architecture - You need easier local deployment via Ollama or llama.cpp **Use R1-Distill-Llama-70B when:** - You want near-R1 accuracy in a format with broader ecosystem support - You're already running Llama infrastructure and want to add reasoning capability - You need Llama's permissive license for commercial deployment **Use R1-Distill-Llama-8B when:** - Cost and speed are the primary constraints - The task involves structured math or logic that doesn't require competition-level accuracy - You're running on consumer hardware or in memory-constrained environments **Use Llama 3.3 70B (no reasoning chain) when:** - You need fast responses with sub-second latency - The reasoning task is relatively straightforward - You want to avoid thinking token overhead entirely ## Serving Reasoning Models in Production A few considerations when deploying these models at scale: **Thinking token handling**: Most APIs expose the thinking block either inline or in a separate field. For user-facing products, strip the `...` block from displayed responses, but keep it in your logs -- it's useful for debugging incorrect answers. **Streaming matters more for reasoning models**: Because these models take longer before producing the final answer, streaming the thinking phase significantly improves perceived responsiveness. Users can see progress rather than waiting through a 10-second blank period. **Token limits on thinking**: Reasoning models can produce very long chains on hard problems. Set a token limit on the thinking phase if your application has latency requirements. A partially-reasoned answer in 3 seconds is often more useful than timing out at 30 seconds waiting for a complete one. **Prefix caching**: If you're repeatedly running similar problems (a math tutoring app, a code verification pipeline), prefix caching on the system prompt reduces costs. It won't help with the reasoning chain since that varies per query, but system prompt caching adds up across high-volume workloads. **Batch vs. interactive use**: Reasoning models shine in batch settings where you're processing many queries offline. For interactive applications, the latency from long thinking chains can hurt the experience unless you're streaming the think block as a visible "working..." state. ## Conclusion The open-source reasoning model landscape offers practical options across the compute spectrum. DeepSeek-R1 sets the accuracy bar -- it matches or slightly exceeds o1 on most math benchmarks, which is notable for an open model. The distilled variants bring most of that capability to 8B and 70B scales that are more manageable to serve. QwQ-32B is a good alternative when you want a dense model without MoE routing complexity, though it sits a step below R1 distills at the same parameter count. The Llama-based R1 distillations are worth considering if you're already running Llama infrastructure and want reasoning capability with familiar tooling. For teams that want to evaluate reasoning models before committing to full infrastructure, General Compute's API provides access to DeepSeek-R1 and other leading open models with the throughput needed to run meaningful benchmarks. You can process a batch of MATH-500 problems, measure real token costs, and decide whether the accuracy improvement justifies the overhead for your use case. --- # Coding Model Benchmarks: Qwen 2.5 Coder vs DeepSeek Coder V2 vs CodeLlama vs StarCoder 2 > A technical benchmark comparison of the top open-source coding models -- Qwen 2.5 Coder, DeepSeek Coder V2, CodeLlama, and StarCoder 2 -- across HumanEval, MBPP, and real-world code tasks by size bucket. - Author: General Compute - Published: 2026-07-24 - Tags: benchmarks, coding, qwen, deepseek, codellama, starcoder, open-source, model-comparison, inference - Canonical: https://www.generalcompute.com/blog/coding-model-benchmarks-qwen-2-5-coder-vs-deepseek-coder-v2-vs-codellama-vs-starcoder-2 --- Picking a coding model used to be simple: you grabbed CodeLlama and moved on. That's no longer the case. The coding-specific model landscape has changed substantially in the past two years, and the differences between the top options are large enough to actually matter for production use cases. This post benchmarks Qwen 2.5 Coder, DeepSeek Coder V2, CodeLlama, and StarCoder 2 across common code tasks, organized by size bucket so you can see how the options compare at the compute tier you're actually considering. ## What Makes a Coding Model Different Coding models start from a general-purpose base and receive additional pretraining on large code corpora, often paired with code-specific fine-tuning for instruction following. The training mix differs significantly between projects: some emphasize breadth across programming languages, others concentrate on high-quality data from fewer sources, and some are specifically optimized for code completion (fill-in-the-middle) rather than generation from a description. The result is that benchmark performance on tasks like HumanEval correlates more strongly with training data quality and volume than with raw parameter count. A well-trained 7B model can outperform a poorly-trained 70B on coding tasks. ## The Benchmark Suite The standard benchmarks for code models: - **HumanEval**: 164 Python function completion problems from OpenAI. Tests whether a model can generate correct function bodies given a docstring and signature. Reports pass@1 (fraction that pass unit tests on the first attempt). - **MBPP (Mostly Basic Python Problems)**: 500 Python problems from Google, a bit more diverse in structure than HumanEval. Often used alongside HumanEval to confirm results aren't an artifact of dataset-specific optimization. - **MultiPL-E**: Translates HumanEval and MBPP to 18+ other languages (Java, JavaScript, TypeScript, C++, Rust, Go, etc.). Useful for measuring how well coding capability transfers outside Python. - **SWE-Bench**: A harder benchmark derived from real GitHub issues and their corresponding pull requests. Tests whether a model can produce a patch that actually resolves the issue and passes CI. More representative of real engineering work than HumanEval. HumanEval and MBPP measure whether a model can write clean, correct short functions. SWE-Bench measures something closer to what a coding agent does. The gap between these is worth paying attention to -- models can score well on HumanEval and still struggle with multi-file edits or context-dependent bugs. ## Model Overview | Model | Sizes | Architecture | Context | Code Training | |---|---|---|---|---| | Qwen 2.5 Coder | 0.5B to 72B | Dense | 128K | 5.5T tokens | | DeepSeek Coder V2 | 16B, 236B | MoE | 128K | Based on DeepSeek-V2 | | CodeLlama | 7B to 70B | Dense (Llama 2 base) | Up to 100K | ~1T code tokens | | StarCoder 2 | 3B, 7B, 15B | Dense | 16K | The Stack v2 (600+ langs) | A few notes on these numbers. DeepSeek Coder V2 is a MoE model: the 16B Lite has 16B total params with fewer active, and the full 236B model has 236B total params with 21B active. This affects inference cost considerably. StarCoder 2's 16K context window is a real constraint for longer codebases. CodeLlama has a 100K window in its code infilling mode, but the base models default to shorter contexts. ## Benchmark Results by Size ### Sub-16B Models At the 7-8B range, this is where most teams run inference for latency-sensitive code tasks. | Model | HumanEval pass@1 | MBPP pass@1 | |---|---|---| | Qwen 2.5 Coder 7B Instruct | ~88% | ~83% | | DeepSeek Coder V2 Lite (16B MoE) | ~81% | ~75% | | CodeLlama 7B | ~34% | ~44% | | CodeLlama 13B | ~36% | ~49% | | StarCoder 2 7B | ~35% | ~54% | | StarCoder 2 15B | ~46% | ~60% | Qwen 2.5 Coder 7B Instruct is the clear winner at this size tier. Scores in the high 80s on HumanEval put it ahead of DeepSeek Coder V2 Lite despite the latter being a 16B model (with lower active params). The gap over CodeLlama 7B and StarCoder 2 7B is large -- roughly 50 percentage points on HumanEval. CodeLlama and StarCoder 2 at these sizes were competitive when released in 2023, but the training data and instruction tuning improvements since then have shifted the baseline significantly. ### 30-70B Range | Model | HumanEval pass@1 | MBPP pass@1 | |---|---|---| | Qwen 2.5 Coder 32B Instruct | ~93% | ~90% | | Qwen 2.5 Coder 72B Instruct | ~92%+ | ~89%+ | | DeepSeek Coder V2 (236B, 21B active) | ~96% | ~92% | | CodeLlama 34B | ~49% | ~62% | | CodeLlama 70B | ~68% | ~74% | At larger sizes, DeepSeek Coder V2's full 236B model (with its MoE architecture keeping active params to ~21B) achieves the highest scores in this category. Qwen 2.5 Coder 32B is within a few points and is meaningfully smaller to serve. CodeLlama 70B does improve substantially over its smaller versions, reaching ~68% on HumanEval -- a legitimate result for a model from 2023 -- but it's well below the modern alternatives. ### What These Numbers Mean in Practice HumanEval pass@1 in the 90s sounds like it should translate to a model that never makes mistakes. It doesn't. HumanEval problems are short, self-contained, and well-specified. Production code is longer, under-specified, and depends on context in other files. The roughly 15-point gap between Qwen 2.5 Coder 32B (93%) and CodeLlama 70B (68%) does reflect a genuine quality difference: the newer model handles edge cases better, follows instructions more reliably, and produces code that compiles more often on the first try. But both will make mistakes on hard problems, ambiguous specifications, and unfamiliar APIs. ## Code Completion: Fill-in-the-Middle For IDE-style code completion (the cursor is in the middle of a function and you want a suggestion), models need fill-in-the-middle (FIM) support. This is a distinct capability from instruction following -- the model sees code above and below the cursor and must generate what fits in between. All four families support FIM, but the quality varies: - **Qwen 2.5 Coder**: Strong FIM performance across sizes. The 7B model is a practical choice for low-latency autocomplete. The 32B model produces higher quality completions but at higher latency. - **DeepSeek Coder V2**: Competitive FIM performance, particularly for long completions where the model needs to reason about what comes after the completion. - **CodeLlama**: Has solid FIM support, and was one of the first widely available models to implement it properly. The 34B version is commonly used for this use case. - **StarCoder 2**: Also supports FIM. The 15B model is a reasonable option if you need wide programming language coverage and are willing to accept lower benchmark ceilings. For autocomplete specifically, latency matters as much as benchmark score. A model that generates tokens 40ms faster is meaningfully better for interactive use even if it scores slightly lower on HumanEval. This is worth measuring on your actual serving infrastructure. ## MultiPL-E: Beyond Python HumanEval and MBPP are Python-only. If your codebase is in TypeScript, Go, or Rust, the benchmark rankings can shift. Qwen 2.5 Coder maintains strong scores across languages on MultiPL-E. Its training corpus covers 92 programming languages, and the multilingual performance is more consistent than most models at equivalent sizes. StarCoder 2 was specifically designed for broad language coverage (600+ languages in The Stack v2), but the depth of coverage varies a lot by language. For commonly used languages (Python, JavaScript, Java, C++), it's competitive. For less common ones, the training signal is thinner. CodeLlama's multilingual performance is more uneven than the newer models. Strong Python, decent JavaScript and C++, weaker on more recent languages like Rust or Swift that were underrepresented in its training data. DeepSeek Coder V2 performs well across most major languages, with particular strength in languages common in its training data. ## SWE-Bench: Real Repository Tasks SWE-Bench scores are considerably lower than HumanEval for every model -- this is expected, because resolving a real GitHub issue in an unfamiliar codebase is much harder than writing a function given a docstring. The Lite version of SWE-Bench (a curated 300-issue subset) has become the standard for comparing models on this task. Scores above 30% resolve rate are considered strong. To put this in context: - Models in the 40-50% range on HumanEval often achieve only 5-15% on SWE-Bench Lite - Models at 90%+ on HumanEval achieve 25-45% on SWE-Bench Lite - Getting past 50% on SWE-Bench Lite requires careful scaffolding in addition to a capable model For SWE-Bench specifically, DeepSeek Coder V2 (full 236B) and Qwen 2.5 Coder 72B are among the stronger open-source options. Neither is dramatically better than the other when matched with equivalent scaffolding. The practical takeaway: model quality matters, but the agent framework around the model matters roughly as much for complex, multi-file repair tasks. ## Inference Cost by Size For teams running these models in production, the size-to-quality curve matters. For the 7B range: Qwen 2.5 Coder 7B is the best option available. It outperforms much larger models from two years ago and runs efficiently on a single A100 or H100 with room for reasonable batch sizes. For the 14-32B range: Qwen 2.5 Coder 32B is strong here and is the default recommendation for teams that need higher quality than a 7B provides. DeepSeek Coder V2 Lite (16B MoE) is competitive on benchmarks but the MoE architecture adds routing overhead that affects latency at low batch sizes. For the 70B+ range: DeepSeek Coder V2's 236B model (21B active params) achieves the highest benchmark scores and is cost-efficient to serve relative to its active parameter count. Qwen 2.5 Coder 72B is the dense alternative at competitive quality. CodeLlama and StarCoder 2 are no longer competitive with these options for most use cases. They remain useful if you have specific requirements -- CodeLlama if you need a model that was released under a permissive license with a well-established deployment history, StarCoder 2 if you specifically need broad language coverage in the BigCode OpenRAIL-M license. ## Which Model to Use **For interactive code completion (autocomplete, FIM):** Qwen 2.5 Coder 7B Instruct. Low latency, high quality at size, good multilingual coverage. Upgrade to 14B if quality on complex completions matters more than inference speed. **For code generation from description (chat interface, function generation):** Qwen 2.5 Coder 32B Instruct or DeepSeek Coder V2 (full). The 32B is easier to serve; the V2 236B scores higher on harder problems. **For coding agents on real repositories:** Either Qwen 2.5 Coder 72B or DeepSeek Coder V2 (full) with appropriate agent scaffolding. The model is necessary but not sufficient -- the scaffolding is where most of the SWE-Bench performance comes from. **If you need broad language coverage and aren't CPU-bound on quality:** StarCoder 2 15B is a reasonable option for less common programming languages where the newer models have less training data. **If license terms are the primary constraint:** All four families offer models under their respective licenses. Qwen and DeepSeek have commercial-friendly licenses for most use cases; StarCoder 2 uses BigCode OpenRAIL-M; CodeLlama uses Meta's CodeLlama license. Check the specifics for your situation. ## A Practical Comparison on Your Workload Benchmark scores are useful for filtering options, but they don't replace testing on your actual code and prompts. Different codebases, languages, and problem types produce different rankings in practice. A model that tops HumanEval may underperform on your TypeScript monorepo, and vice versa. The most reliable evaluation is running your top two or three candidates on a representative sample of your real tasks -- ideally tasks where you already know the correct answer -- and measuring pass@1 or edit distance from the target. This takes a few hours and produces much more actionable data than any third-party benchmark table. All four model families are available via General Compute's inference API with OpenAI-compatible endpoints. You can benchmark them on your workload without infrastructure setup by changing a base URL and model name in your existing code. Run your eval suite against each option to find the right fit for your specific use case. --- Ready to benchmark coding models on your workload? General Compute supports Qwen 2.5 Coder, DeepSeek Coder V2, and other top coding models with low-latency inference and no rate limits. Swap the model name in your OpenAI integration and run your evaluation in minutes. --- # Frontier Open Models: Llama 4 Maverick vs DeepSeek V3 vs Qwen 2.5 110B > A technical comparison of the top frontier open-source models -- Llama 4 Maverick, DeepSeek V3, and Qwen 2.5 110B -- covering architecture, benchmarks, and inference trade-offs. - Author: General Compute - Published: 2026-07-23 - Tags: benchmarks, llama, deepseek, qwen, open-source, moe, inference, model-comparison - Canonical: https://www.generalcompute.com/blog/frontier-open-models-llama-4-maverick-vs-deepseek-v3-vs-qwen-2-5-110b --- The frontier of open-source LLMs has closed the gap with proprietary models faster than most people expected. Three models in particular define what's possible today without a closed API: Meta's Llama 4 Maverick, DeepSeek's V3, and Alibaba's Qwen 2.5 110B. Each comes from a different architecture lineage and excels in different scenarios. If you're trying to pick one -- or understand when to route to which -- this breakdown covers what actually matters. ## Quick Overview | Model | Architecture | Total Params | Active Params | Context | Best At | |---|---|---|---|---|---| | Llama 4 Maverick | MoE | ~400B | ~17B | 1M tokens | Multimodal, long context | | DeepSeek V3 | MoE | 671B | 37B | 128K tokens | Coding, reasoning, cost efficiency | | Qwen 2.5 110B | Dense | 110B | 110B | 128K tokens | Multilingual, instruction following | The most striking difference is the MoE vs dense split. Maverick and V3 are both Mixture-of-Experts models, meaning only a fraction of their total parameters are active for any given token. Qwen 2.5 110B is dense -- all 110B parameters participate in every forward pass. This distinction shapes their benchmark profiles, serving characteristics, and cost structure more than almost anything else. ## Architecture: MoE vs Dense ### How MoE Routing Works In a Mixture-of-Experts transformer, each MLP layer is replaced by a collection of "experts" (independent feed-forward networks) plus a learned router. For each token at each layer, the router selects a small subset of experts, routes the token through them, and combines their outputs. The total parameter count reflects all experts, but the active parameter count -- the FLOPs you actually pay per token -- is much lower. Llama 4 Maverick uses 128 experts with top-1 routing, meaning one expert is activated per token per MoE layer. This is aggressive sparsity. The result is roughly 17B active parameters out of ~400B total. DeepSeek V3 uses 256 experts with top-8 routing, keeping 37B parameters active at a time. Both are far more parameter-efficient than a dense model of equivalent capability. The tradeoff: MoE models require all experts to be resident in memory (or available for fast loading), even though only a small fraction are used per token. A model with 400B total parameters needs roughly 400B worth of memory in weights, even if it runs with the compute budget of a 17B dense model. This makes MoE models memory-hungry but compute-light. ### Dense at 110B Qwen 2.5 110B activates every parameter for every token. For a 110B model, that means predictable, uniform compute per token. There's no routing overhead, no expert selection latency, and no risk of expert imbalance. The flip side is that every token costs the same -- you can't get MoE's efficiency gains when a given token is "easy" to predict. Dense models tend to have more uniform inference latency and simpler serving infrastructure. If you're running at high batch sizes and need consistent throughput, a dense model is often easier to optimize than an equivalent MoE. ## Benchmark Comparison Benchmarks for models at this scale vary depending on how they're prompted, what temperature and sampling settings are used, and whether you're running the base or instruct checkpoint. The numbers below reflect instruct-tuned versions evaluated under the conditions reported in each model's original release. ### General Knowledge (MMLU) DeepSeek V3 leads the three on MMLU with around 88.5%, a result that was notably competitive with closed models at release time. Llama 4 Maverick scores roughly 85-86%, and Qwen 2.5 110B sits around 85%. The gaps are relatively small in raw percentage terms, but MMLU's breadth means consistent performance across these numbers reflects genuine model quality, not cherry-picking. ### Coding (HumanEval / SWE-Bench) DeepSeek V3 is the clear winner here. It was specifically optimized for coding, achieving HumanEval pass@1 scores in the high 80s (around 89%), well above what dense models of similar compute cost manage. Qwen 2.5 110B is a capable coder -- around 77% on HumanEval -- and Llama 4 Maverick, while strong on general tasks, was not primarily optimized for code generation. For production coding use cases like autocomplete, code review, or automated PR generation, DeepSeek V3 has a meaningful advantage. ### Math and Reasoning (GSM8K, MATH) All three models are strong here. DeepSeek V3 scores around 90% on MATH, Qwen 2.5 110B around 87-88%, and Llama 4 Maverick around 87%. These are all competitive numbers. For math-heavy workloads, any of the three will serve you well, though DeepSeek V3 has a slight edge at the tail of harder problems. ### Multimodal (Vision + Text) This is Llama 4 Maverick's distinctive territory. It was released as a native multimodal model with strong image understanding built into the architecture -- not bolted on through a separate vision encoder. For tasks that mix visual inputs with language reasoning, Maverick is the only one of the three with first-class support. DeepSeek V3 and Qwen 2.5 110B are text-only. ### Long Context Llama 4 Maverick's 1M token context window is an outlier. Most inference providers cap it below this due to KV cache memory requirements, but even at 128K or 256K tokens, Maverick maintains strong performance on long-document tasks. DeepSeek V3 and Qwen 2.5 110B both support 128K context and perform reasonably at that length, though long-context behavior can degrade toward the upper end of the window. ## Inference Characteristics Benchmark scores tell you about quality. Inference characteristics tell you about what you'll actually pay in production. ### Memory Requirements Serving these models at FP16 (or BF16) requires roughly 2 bytes per parameter: - Llama 4 Maverick: ~800GB for full-weight FP16 (all experts loaded) - DeepSeek V3: ~1.3TB for full-weight FP16 - Qwen 2.5 110B: ~220GB for full-weight FP16 In practice, quantization (INT8, INT4) cuts these significantly. At INT4, Qwen 2.5 110B fits comfortably on two H100s with memory to spare for KV cache. Maverick and V3 at INT4 still require multi-node setups or aggressive quantization. Managed inference APIs handle this behind the scenes, but it affects pricing and hardware allocation. ### Tokens Per Second Because Maverick and V3 have lower active parameter counts relative to their capability level, they can generate tokens faster per unit of compute compared to a dense model at similar benchmark performance. With 17B active params, Maverick can achieve high token throughput if the serving infrastructure can keep experts loaded efficiently. DeepSeek V3 at 37B active params sits between the two. Qwen 2.5 110B generates tokens at a rate proportional to 110B parameters worth of compute. Dense models at this scale are memory-bandwidth bound: you're spending most of your time moving weights from HBM to compute units, not doing arithmetic. H100s with 3.35 TB/s HBM3e bandwidth help considerably, but there's no architectural shortcut like MoE sparsity. At high batch sizes, the picture changes. MoE models can have higher routing overhead and cache thrashing when expert selection varies across requests in a batch. Dense models scale more linearly. For throughput-heavy workloads (batch inference, document processing), the compute-vs-memory trade-off shifts in favor of dense models. ### Cost Per Token At managed inference providers, pricing typically correlates with active parameter count plus infrastructure overhead. Maverick tends to be priced competitively given its active parameter count -- you're getting GPT-4o-class capability at a lower cost-per-token than a full 70B dense model in many configurations. DeepSeek V3 is similarly well-priced for its benchmark tier. Qwen 2.5 110B, being a larger dense model, generally costs more per token than the MoE alternatives at comparable quality for coding and reasoning. ## Choosing Between Them ### Use Llama 4 Maverick When: - Your application needs multimodal input (images, documents with figures, UI screenshots) - You have very long contexts -- 128K or more -- and need quality throughout - You want general-purpose capability close to frontier proprietary models - You're running a use case where Meta's open license (the Llama Community License) works for your business ### Use DeepSeek V3 When: - Coding is your primary use case -- autocomplete, code review, debugging, generation - You need strong math and reasoning performance - Cost efficiency at scale matters and you want maximum benchmark performance per dollar - You're fine with the DeepSeek license terms ### Use Qwen 2.5 110B When: - Multilingual support is important (Qwen 2.5 is particularly strong on Chinese, Arabic, and other non-Latin languages) - You want predictable, consistent inference without MoE routing complexity - You're self-hosting and prefer simpler serving infrastructure - Your workload involves high batch sizes where dense models scale more cleanly ## A Note on Self-Hosting vs APIs These three models exist at a scale where self-hosting is genuinely expensive. Running Llama 4 Maverick on your own hardware requires a multi-GPU setup with careful expert parallelism. DeepSeek V3 needs even more. Qwen 2.5 110B is manageable on two or four H100s with quantization, but still requires meaningful DevOps investment. For most teams, the faster path is an OpenAI-compatible inference API that handles hardware allocation, model loading, and quantization. You can swap between models by changing a base URL and model name, run experiments across all three, and pay only for tokens consumed. If your workload involves these frontier models at sustained high volume (millions of tokens per day), the build-vs-buy calculation shifts and self-hosting starts making more sense. For most applications in early-to-mid scale, API access to these models is both cheaper and faster to iterate on. ## Putting It Together Llama 4 Maverick, DeepSeek V3, and Qwen 2.5 110B represent genuinely different approaches to building frontier open models. The MoE architectures of Maverick and V3 use parameter sparsity to reach high capability without proportional compute cost. Qwen 2.5 110B takes the straightforward dense path and does it at a scale where the model quality is competitive across most benchmarks. For most applications with text-only input, DeepSeek V3 is the default recommendation for coding-heavy use cases and Qwen 2.5 110B is a reasonable choice if multilingual coverage or simple serving infrastructure matters. For applications that need vision, long context, or a combination of both, Llama 4 Maverick is the one to evaluate first. All three are available via General Compute's inference API with OpenAI-compatible endpoints. You can benchmark them on your actual workload in an afternoon, which is a better signal than any third-party comparison table -- latency and quality for your specific prompts will differ from aggregate benchmark numbers. --- Ready to compare them on your workload? General Compute's API supports all three models with no rate limits and low-latency inference. Swap the model name in your existing OpenAI integration and run your eval suite to find the right fit. --- # Mid-Size Models: Llama 3.1 70B vs Qwen 2.5 72B vs Mistral Large vs DeepSeek-V2 Lite > A benchmark comparison of the 70B-class open models in 2025 -- covering reasoning, math, coding, context length, and inference speed across Llama 3.1 70B, Qwen 2.5 72B, Mistral Large 2, and DeepSeek-V2 Lite. - Author: General Compute - Published: 2026-07-22 - Tags: benchmarks, llm, inference, model-comparison, 70b-models - Canonical: https://www.generalcompute.com/blog/mid-size-models-llama-3-1-70b-vs-qwen-2-5-72b-vs-mistral-large-vs-deepseek-v2-lite --- The 70B parameter range is where open-source models start to match or exceed proprietary options on a wide variety of tasks. These models are large enough to handle complex multi-step reasoning, long-context analysis, and serious code generation, while still being deployable on a single 80GB A100 or a pair of 40GB GPUs. For production teams that need strong performance but cannot justify frontier model pricing, the 70B tier is where most evaluations start. This post compares four models from this tier: Llama 3.1 70B, Qwen 2.5 72B, Mistral Large 2, and DeepSeek-V2 Lite. These four span different design choices -- dense architecture, MoE architecture, and different training priorities -- and they perform differently enough that picking the wrong one for a workload has a real impact. We cover benchmark numbers across general reasoning, math, and coding, then look at context length and inference throughput, since at this size those factors shape both capability and deployment cost. ## The four models **Llama 3.1 70B** is Meta's 70 billion parameter model from the Llama 3.1 series. It was trained on 15 trillion tokens and supports 128K context via RoPE scaling. The architecture uses grouped-query attention for efficient inference and is the most widely deployed model in this comparison by a large margin. Llama 3.1 70B has a substantial ecosystem: fine-tunes, quantization configurations, deployment recipes, and framework integrations are all mature. When developers compare 70B models, Llama 3.1 70B is usually the reference point. **Qwen 2.5 72B** is Alibaba's 72 billion parameter model from the Qwen 2.5 release. It supports 128K context and was trained with heavy emphasis on coding, math, and structured output generation. Qwen 2.5 72B consistently leads in coding and math benchmarks at this scale. The Qwen team published detailed ablations showing particular attention to instruction following and tool use, and the model is one of the few 70B-class models that reliably handles complex JSON schema outputs without prompt engineering. **Mistral Large 2** is Mistral AI's second-generation large model with 123B parameters. Despite sitting above the strict 70B threshold, it competes directly in this tier: it is positioned as a capable alternative to GPT-4o and Claude 3.5 Sonnet at lower cost, and its inference speed on ASIC infrastructure often lands it in similar price brackets to the true 70B models. Mistral Large 2 supports 128K context, has strong native function calling, and is particularly good at code generation relative to its size. The model was released in July 2024. **DeepSeek-V2 Lite** is the smaller variant of DeepSeek's V2 series, using a Mixture-of-Experts architecture with Multi-head Latent Attention (MLA). The model has 16 billion total parameters with 2.4 billion activated per forward pass. That active parameter count puts compute cost well below the dense 70B models in this comparison, though total parameter storage requires more memory. Context window is 32K. DeepSeek-V2 Lite is included here because it competes on inference cost with 70B models despite activating far fewer parameters -- the question is how much capability you give up. ## Benchmark results ### General reasoning (MMLU) MMLU covers 57 academic subjects and is the standard measure of broad knowledge and reasoning capability. | Model | MMLU Score | |---|---| | Qwen 2.5 72B | ~86% | | Mistral Large 2 | ~84% | | Llama 3.1 70B | ~83% | | DeepSeek-V2 Lite | ~58% | Qwen 2.5 72B and Mistral Large 2 lead, with Llama 3.1 70B close behind. These three are within 3 points of each other and all well above the strong-reasoning threshold most teams would set for production use. DeepSeek-V2 Lite scores considerably lower at 58%, which reflects its much smaller active parameter count. The MoE architecture gives it computational efficiency, but on a knowledge-breadth benchmark like MMLU, it operates more like a capable 7-8B dense model than a 70B one. ### Math (GSM8K) GSM8K tests multi-step grade school math word problems. At the 70B scale, differences are more about training emphasis than raw capacity. | Model | GSM8K Score | |---|---| | Qwen 2.5 72B | ~96% | | Llama 3.1 70B | ~93% | | Mistral Large 2 | ~93% | | DeepSeek-V2 Lite | ~75% | Qwen 2.5 72B leads by 3 points over Llama and Mistral, which track closely. The math-focused training in the Qwen 2.5 series is consistent across size buckets -- the same advantage appeared in the 7B comparison. DeepSeek-V2 Lite performs reasonably on math for its active parameter count at 75%, better than its MMLU showing relative to the dense models, but still meaningfully below the 70B tier. ### Coding (HumanEval) HumanEval pass@1 measures the rate at which the first generated solution passes all test cases on 164 Python programming problems. | Model | HumanEval Pass@1 | |---|---| | Mistral Large 2 | ~92% | | Qwen 2.5 72B | ~86% | | Llama 3.1 70B | ~80% | | DeepSeek-V2 Lite | ~58% | Mistral Large 2 leads on code with a notable gap over Qwen 2.5 72B. The 92% pass@1 puts it in range of GPT-4o on this benchmark, which is consistent with Mistral's stated focus on code generation in the Large 2 release. Qwen 2.5 72B and Llama 3.1 70B are both strong, with Qwen ahead by 6 points. DeepSeek-V2 Lite's 58% score reflects its smaller active parameter footprint: it handles basic coding problems well but drops off on more complex multi-function problems that require holding more context simultaneously. ### Instruction following (MT-Bench) MT-Bench evaluates multi-turn instruction following across different domains, scored on a 1-10 scale. | Model | MT-Bench Score | |---|---| | Qwen 2.5 72B | ~9.1 | | Mistral Large 2 | ~9.0 | | Llama 3.1 70B | ~8.8 | | DeepSeek-V2 Lite | ~7.6 | Qwen 2.5 72B and Mistral Large 2 are essentially tied on instruction following quality. Both models handle complex multi-turn conversations reliably and show good task completion across domains. Llama 3.1 70B at 8.8 is still strong for most tasks. DeepSeek-V2 Lite at 7.6 is noticeably behind -- again, a capability gap that reflects the much lower active parameter count rather than an architectural flaw. ### Context length | Model | Context Window | |---|---| | Llama 3.1 70B | 128K | | Qwen 2.5 72B | 128K | | Mistral Large 2 | 128K | | DeepSeek-V2 Lite | 32K | The three dense models all support 128K context and handle it usably in practice. Llama 3.1 70B and Qwen 2.5 72B were trained with extended context from the start. Mistral Large 2 also performs well at long context, though like all models at this range, accuracy on needle-in-haystack retrieval degrades somewhat in the 100K+ range. DeepSeek-V2 Lite's 32K limit is adequate for most tasks but rules it out for long-document workflows, large codebase analysis, or extended conversation history that exceeds that window. ## Inference speed The 70B size class is where multi-GPU deployment becomes standard. These estimates are for two A100 80GB GPUs with tensor parallelism at FP16, serving single requests: | Model | Approx tokens/sec (2x A100 80GB, FP16) | |---|---| | DeepSeek-V2 Lite | ~350-450 | | Mistral Large 2 | ~120-160 | | Qwen 2.5 72B | ~130-170 | | Llama 3.1 70B | ~130-170 | DeepSeek-V2 Lite is the speed outlier -- because it activates only 2.4B parameters per forward pass, it generates tokens much faster than the dense models even though it requires more memory for the full parameter set. The three dense models cluster together at 130-170 tokens/sec on this hardware configuration. Mistral Large 2 at 123B actually runs at similar speed to the true 70B models when the comparison is on the same hardware, because its attention mechanism is efficient and modern serving frameworks like vLLM handle it well with tensor parallelism. On GeneralCompute's ASIC infrastructure, absolute throughput is substantially higher across all four models. The relative ordering holds, but the gap between DeepSeek-V2 Lite and the dense models narrows in practice because the ASIC optimizations improve memory bandwidth utilization differently for dense and sparse architectures. ## Summary table | | Llama 3.1 70B | Qwen 2.5 72B | Mistral Large 2 | DeepSeek-V2 Lite | |---|---|---|---|---| | MMLU | ~83% | ~86% | ~84% | ~58% | | GSM8K | ~93% | ~96% | ~93% | ~75% | | HumanEval | ~80% | ~86% | ~92% | ~58% | | MT-Bench | ~8.8 | ~9.1 | ~9.0 | ~7.6 | | Context | 128K | 128K | 128K | 32K | | Active params | 70B | 72B | 123B | 2.4B | | Relative speed (dense 70B baseline) | 1x | 1x | ~0.9x | ~2.5x | ## When to use each model **Use Qwen 2.5 72B for code-heavy or math-heavy applications.** It leads on GSM8K and HumanEval among the 70B-class dense models, and its MT-Bench score ties Mistral Large 2 for instruction following quality. The Qwen 2.5 72B training emphasis on structured output also makes it particularly reliable for applications that need consistent JSON or schema-conformant responses without extensive prompt engineering. If you are building a data extraction pipeline, a coding assistant, or a math reasoning tool, this is the 70B model to evaluate first. **Use Mistral Large 2 when code generation quality is the top priority.** Its 92% HumanEval pass@1 is notably higher than the true 70B models and competitive with much larger proprietary models. If your application generates complex, multi-function code or needs high first-attempt success rates on coding tasks, the capability premium over Qwen or Llama is real. The tradeoff is a larger model that requires more memory than a true 70B, though inference frameworks handle this well with tensor parallelism across two GPUs. **Use Llama 3.1 70B as the default general-purpose choice.** It trails Qwen 2.5 72B and Mistral Large 2 on specialized benchmarks, but it comes with the broadest ecosystem support in this tier: the most fine-tunes, the most deployment documentation, the largest community, and the most thoroughly tested integration paths across frameworks like LangChain, LlamaIndex, and vLLM. If you are evaluating multiple models and want a stable baseline to compare against, Llama 3.1 70B is the natural starting point. It also provides the clearest upgrade path to Llama 3.1 405B if you find you need more capability later. **Use DeepSeek-V2 Lite when inference speed and cost matter more than top-tier quality.** The capability gap relative to the dense 70B models is real and consistent across benchmarks -- you are getting roughly 7-9B dense model performance at 70B model memory cost, but with 2-3x faster token generation. This tradeoff makes sense when you need rapid generation for low-latency applications and your task does not require the full capability of a dense 70B, such as rapid classification, summarization of well-structured content, or first-pass drafts in an agentic pipeline that also has a strong revision step. The 32K context limit is a separate constraint that will be binding on some workloads. ## Quantization and deployment notes At 70B, the self-hosting story is different from the 7B tier. A dense 70B model in FP16 requires 140GB of VRAM, which means two A100 80GB GPUs at minimum. Most teams running 70B models self-hosted are on 8xA100 or 4xH100 nodes for practical throughput at production scale. **INT8 quantization** brings a 70B model's memory footprint to roughly 70-75GB -- still requiring two GPUs for A100 80GB nodes, but comfortably fitting on a single H100 80GB. Quality regression is minimal, typically under 1-2 points on MMLU and GSM8K. **INT4 quantization via GPTQ or AWQ** drops the footprint to roughly 35-40GB, which fits on a single A100 80GB with overhead to spare. Quality regression is more noticeable at this scale than at 7B -- expect 3-5 points on GSM8K and similar on HumanEval. AWQ is generally preferable to GPTQ for maintaining quality on math and reasoning tasks. At INT4, a 70B model on a single A100 runs at roughly 50-70 tokens/sec for single-stream generation. For DeepSeek-V2 Lite, quantization is more nuanced because the MoE architecture means expert weights are not all active simultaneously. The full 16B parameter model in FP16 requires about 32GB VRAM, and INT4 brings it to roughly 16GB -- making it accessible on consumer A6000 48GB cards or even mid-range 24GB cards with careful management. vLLM supports Llama 3.1 70B, Qwen 2.5 72B, and Mistral Large 2 natively with paged attention and continuous batching. DeepSeek-V2 Lite support in vLLM is available but requires specific configuration for MoE routing. SGLang also supports all four and is worth benchmarking for latency-sensitive workloads because it handles KV cache management differently in ways that can benefit MoE inference. ## Fine-tuning at 70B Fine-tuning dense 70B models is feasible with QLoRA. A LoRA run on Llama 3.1 70B or Qwen 2.5 72B requires roughly 80-100GB of VRAM with 4-bit base model quantization, which fits on a single H100 80GB or two A100 80GB GPUs. Training time scales with dataset size but is often practical for domain-specific fine-tuning at a few hundred to a few thousand examples. The benefit of fine-tuning at 70B is that the base model's strong general reasoning is preserved while the model learns a specific style, format, or domain vocabulary. For applications where quality on a narrow task matters and you have labeled data, a fine-tuned 70B often outperforms prompted frontier models at substantially lower inference cost. Llama 3.1 70B has the most documented fine-tuning recipes, known-good hyperparameters, and community contributions for this size class. Qwen 2.5 72B is a strong choice for code-domain fine-tuning given the base capabilities. Mistral Large 2 is worth considering for applications requiring reliable tool use, given Mistral's focus on function calling in this model. ## Conclusion Among the three dense 70B-class models, Qwen 2.5 72B and Mistral Large 2 lead on specialized benchmarks, and the right choice depends on whether you prioritize coding (Mistral Large 2's clear win), math plus structured output (Qwen 2.5 72B), or ecosystem maturity and breadth (Llama 3.1 70B). The three models are close enough on general reasoning that benchmark differences between Llama and Qwen will often not matter in practice -- what matters more is how each model performs on your specific data and prompt structure. DeepSeek-V2 Lite belongs in a different evaluation category. If your workload tolerates the capability step-down and you need fast inference at low cost, it is worth testing. But for most applications where teams are evaluating 70B models, the performance gap is too large to ignore. The practical starting point for most teams is to evaluate Llama 3.1 70B as the baseline and Qwen 2.5 72B as the quality-first alternative, then bring in Mistral Large 2 if code generation quality proves critical. All three respond well to quantization to INT8, which makes self-hosted deployment more accessible without meaningful capability loss. --- All four models are available through GeneralCompute's API with an OpenAI-compatible endpoint. You can swap between them with a single parameter change: ```python from openai import OpenAI client = OpenAI( base_url="https://api.generalcompute.com/v1", api_key="your-api-key", ) # Swap the model string to compare model_id = "llama-3.1-70b-instruct" # Other options: # "qwen2.5-72b-instruct" # "mistral-large-2407" # "deepseek-v2-lite-chat" response = client.chat.completions.create( model=model_id, messages=[ {"role": "user", "content": "Write a Python function to merge two sorted arrays in O(n) time."} ], ) print(response.choices[0].message.content) ``` See the [GeneralCompute model catalog](https://generalcompute.com/docs) for current pricing, latency benchmarks, and quantization options for each model. --- # The 7-8B Battle: Llama 3.1 8B vs Mistral 7B v0.3 vs Qwen 2.5 7B vs Gemma 2 9B > A benchmark comparison of the four most popular 7-9B parameter models in 2025 -- covering reasoning, math, coding, context length, and inference speed to help you pick the right model for your application. - Author: General Compute - Published: 2026-07-21 - Tags: benchmarks, llm, inference, model-comparison, 7b-models - Canonical: https://www.generalcompute.com/blog/the-7-8b-battle-llama-3-1-8b-vs-mistral-7b-vs-qwen-2-5-7b-vs-gemma-2-9b --- The 7-8B size bucket is where most production LLM applications live. These models are large enough to handle complex reasoning and code generation, but compact enough to run on a single A100 80GB with fast throughput. When developers ask "which model should I use," the answer usually starts somewhere in this range. This post compares four of the strongest models in the 7-9B class: Llama 3.1 8B, Mistral 7B v0.3, Qwen 2.5 7B, and Gemma 2 9B. These four represent different design philosophies and training priorities, and they behave differently enough that the choice matters. We cover benchmark performance across general reasoning, math, and coding, then look at inference speed -- because at this size, throughput is a real consideration for production deployments. ## The four models **Llama 3.1 8B** is Meta's 8 billion parameter model from the Llama 3.1 series. It was trained on 15 trillion tokens of data and supports a 128K context window using RoPE scaling. The architecture uses grouped-query attention (GQA) for better inference efficiency compared to earlier Llama versions. Llama 3.1 8B is widely used as a baseline because it sits at the top of the open-source ecosystem in terms of tooling, fine-tunes, and community support. **Mistral 7B v0.3** is the latest release of Mistral AI's flagship small model. It uses sliding window attention (SWA) with a 4,096-token window and GQA for efficient generation. The official context window is 32K with rope scaling. Mistral 7B v0.3 also introduces function calling support, which makes it particularly useful for agentic applications that need reliable tool use from a small model. **Qwen 2.5 7B** is Alibaba's 7 billion parameter model from the Qwen 2.5 release. It supports 128K context and was trained with a heavy emphasis on code, math, and multilingual content alongside instruction following. Qwen 2.5 7B consistently ranks among the strongest models in its size class on coding benchmarks, and it is one of the few 7B models that was explicitly optimized for structured output generation. **Gemma 2 9B** is Google's mid-size model from the Gemma 2 family. At 9 billion parameters it is the largest in this comparison, which affects both capability and throughput. Gemma 2 9B uses an alternating local/global attention pattern (local layers every other block) and benefits from knowledge distillation from larger Gemma 2 models during training. The supported context window is 8,192 tokens, which is a meaningful constraint compared to the other three models here. ## Benchmark results ### General reasoning (MMLU) MMLU tests knowledge across 57 academic subjects from elementary science to professional law and medicine. It is the most commonly cited general capability benchmark and gives a useful signal about how broadly capable a model is. | Model | MMLU Score | |---|---| | Qwen 2.5 7B | ~74% | | Llama 3.1 8B | ~73% | | Gemma 2 9B | ~71% | | Mistral 7B v0.3 | ~63% | Qwen 2.5 7B and Llama 3.1 8B are close at the top, separated by roughly 1 point. Gemma 2 9B sits a few points behind despite having more parameters -- the alternating attention design and distillation approach trades some general knowledge breadth for other properties. Mistral 7B v0.3 has a meaningful gap here; general MMLU is not where it stands out. ### Math (GSM8K) GSM8K is a dataset of grade-school math word problems that require multi-step reasoning. It is a reliable signal for how well a model handles structured problem decomposition at the 7-8B scale. | Model | GSM8K Score | |---|---| | Qwen 2.5 7B | ~85% | | Llama 3.1 8B | ~84% | | Gemma 2 9B | ~68% | | Mistral 7B v0.3 | ~53% | Math is where the Qwen and Llama advantage is most visible. Both models exceed 80%, which is a level of math reasoning that was only achievable with much larger models a year or two ago. Gemma 2 9B falls to 68%, and Mistral 7B v0.3 drops to 53% -- the gap between the top two and the bottom two is substantial. If math or structured reasoning is part of your workload, the model choice here has a real impact on output quality. ### Coding (HumanEval) HumanEval tests Python code generation against 164 programming problems. The pass@1 score measures whether the first generated solution is correct and passes all test cases. | Model | HumanEval Pass@1 | |---|---| | Qwen 2.5 7B | ~80% | | Llama 3.1 8B | ~72% | | Gemma 2 9B | ~54% | | Mistral 7B v0.3 | ~40% | Coding is Qwen 2.5 7B's clearest advantage. The 8-point gap over Llama 3.1 8B is consistent with Qwen's explicit focus on code during training. For any application that involves generating, reviewing, or reasoning about code, Qwen 2.5 7B delivers noticeably better results at this size. Mistral 7B v0.3 is weakest here; it is a general-purpose model that was not specifically optimized for code generation. ### Instruction following (MT-Bench) MT-Bench is a multi-turn conversation benchmark scored by a judge model on a 1-10 scale. It tests whether a model can follow complex instructions across different turns and domains. | Model | MT-Bench Score | |---|---| | Llama 3.1 8B | ~8.2 | | Qwen 2.5 7B | ~8.1 | | Gemma 2 9B | ~7.8 | | Mistral 7B v0.3 | ~7.6 | Llama 3.1 8B and Qwen 2.5 7B are essentially tied on instruction following quality. Both handle complex multi-turn conversations reliably. Mistral 7B v0.3 closes the gap relative to coding benchmarks -- its conversational capabilities are stronger than its code generation numbers suggest. ### Context length | Model | Context Window | |---|---| | Llama 3.1 8B | 128K | | Qwen 2.5 7B | 128K | | Mistral 7B v0.3 | 32K | | Gemma 2 9B | 8K | The 128K support on Llama 3.1 8B and Qwen 2.5 7B is genuine: both were trained with extended context from the start. Mistral 7B v0.3 at 32K is adequate for most tasks but will not handle very long documents or large codebases in a single context. Gemma 2 9B's 8K limit is a hard constraint that rules it out for long-document tasks, long conversation histories, or retrieval-augmented generation with large retrieved passages. ## Inference speed At the 7-9B scale, throughput on a single GPU starts to matter for cost estimation. These numbers are approximate on a single A100 80GB with FP16 precision, serving single requests: | Model | Approx tokens/sec (single GPU, FP16) | |---|---| | Mistral 7B v0.3 | ~360-450 | | Qwen 2.5 7B | ~320-420 | | Llama 3.1 8B | ~300-400 | | Gemma 2 9B | ~240-320 | Mistral 7B v0.3 is fastest in this group, helped by its sliding window attention reducing KV cache pressure in longer sequences. Qwen 2.5 7B and Llama 3.1 8B are close, with differences that vary depending on sequence length and batch configuration. Gemma 2 9B is the slowest by a meaningful margin, reflecting both its larger parameter count and the alternating attention overhead. For reference, on optimized ASIC infrastructure like GeneralCompute's API, all four models run at substantially higher throughput than on self-hosted A100 hardware. The relative ordering holds, but absolute speeds are higher, making the throughput difference between models less constraining in practice. ## Summary table | | Llama 3.1 8B | Mistral 7B v0.3 | Qwen 2.5 7B | Gemma 2 9B | |---|---|---|---|---| | MMLU | ~73% | ~63% | ~74% | ~71% | | GSM8K | ~84% | ~53% | ~85% | ~68% | | HumanEval | ~72% | ~40% | ~80% | ~54% | | MT-Bench | ~8.2 | ~7.6 | ~8.1 | ~7.8 | | Context | 128K | 32K | 128K | 8K | | Relative speed | Fast | Fastest | Fast | Moderate | | Params | 8B | 7B | 7B | 9B | ## When to use each model **Use Qwen 2.5 7B when coding or math is the primary task.** It leads on both HumanEval and GSM8K by a clear margin, and its general reasoning scores are essentially equal to Llama 3.1 8B. If you are building a coding assistant, code review tool, structured data extraction pipeline, or any application where output quality on code or arithmetic matters, Qwen 2.5 7B is the 7B-class model to start with. The 128K context window also makes it viable for codebase-scale tasks. **Use Llama 3.1 8B as the safe general-purpose default.** It is second in most benchmarks, top on MT-Bench, and has broader ecosystem support than any model in this comparison. The Llama 3.1 family has more fine-tunes, more framework integrations, and better-tested deployment tooling than the alternatives. If you are evaluating models and do not have a specific workload that clearly favors one of the others, Llama 3.1 8B is a solid starting point. It also gives you the best upgrade path: fine-tuned Llama 3.1 models are plentiful, and the jump to Llama 3.1 70B is straightforward if you need more capability later. **Use Mistral 7B v0.3 for function-calling-heavy applications with moderate context requirements.** Its general benchmarks trail the others, but it has native function calling support that is more reliable than prompt-engineered tool use on some models. At 32K context and with faster throughput than Llama or Qwen, it works well for agentic applications where you need rapid multi-turn cycles with tool calls and do not require very long context windows. Its latency advantage also matters for interactive applications. **Use Gemma 2 9B for tasks where the 8K context is not a constraint and you want a model with a different training lineage.** The distillation-from-above approach gives Gemma 2 9B strong conversational quality and reliable instruction following. It underperforms on math and coding relative to the other three, but performs respectably on general reasoning. If you have a specific use case where Gemma 2 9B has been validated to perform well (often through empirical testing on your own data), the Google-backed model family may be worth considering for compliance or vendor reasons. ## Deployment and quantization notes All four models are available in quantized formats that significantly change the memory and throughput picture. **INT8 quantization** is usually the best starting point. Memory usage drops by roughly half relative to FP16, with minimal quality loss -- typically under 1% on most benchmarks. A 7B model in INT8 uses about 7-8GB of VRAM, which fits comfortably on a 16GB GPU. **INT4 quantization** (GPTQ or AWQ) cuts memory to 3-4GB for a 7B model and fits on consumer GPUs like the RTX 3090 or 4090. Quality loss is more noticeable on math tasks -- expect 3-6 points of regression on GSM8K. If you are running locally and hardware is the constraint, AWQ INT4 is usually preferable to GPTQ because it compensates for outlier activations during quantization, resulting in better math and reasoning accuracy. **GGUF format** via llama.cpp makes all four models accessible on CPU, including machines without dedicated GPUs. At INT4 GGUF, a 7B model runs at 15-25 tokens/sec on a modern laptop CPU -- slow for real-time conversation but usable for offline batch processing. For server-side deployment, vLLM supports all four models natively and adds continuous batching and paged attention for efficient multi-request serving. SGLang and TGI are also solid options. If you prefer not to manage inference infrastructure, all four models are available through hosted API providers including GeneralCompute. ## A note on fine-tuning The 7-8B class is also a practical size for fine-tuning. A QLoRA run on a 7B model requires a single A100 for a few hours depending on dataset size. For applications with labeled data in a specific domain, a fine-tuned 7B model often outperforms a prompted 70B model on that narrow task, at a fraction of the inference cost. The choice of base model matters for fine-tuning. Llama 3.1 8B has the most fine-tuning resources (tutorials, community configs, known-good hyperparameters), which reduces the time to get a working fine-tune. Qwen 2.5 7B is a strong choice for code and math fine-tuning given its base capabilities. Mistral 7B v0.3 fine-tunes well for tool use because the function calling support is baked into the base model architecture. ## Conclusion In the 7-9B bucket, Qwen 2.5 7B and Llama 3.1 8B are the top performers on most benchmarks. Qwen 2.5 7B wins on coding and math, Llama 3.1 8B wins on MT-Bench and ecosystem breadth, and the two are within a couple of points of each other on general reasoning. Mistral 7B v0.3 is the pick for speed-sensitive agentic use cases with function calling requirements, even though its benchmark scores trail the others. Gemma 2 9B is a reasonable choice when the context constraint is acceptable and you have workload-specific reasons to prefer it, but it does not clearly lead in any benchmark category in this comparison. For most teams starting a new project, the practical recommendation is to evaluate Qwen 2.5 7B and Llama 3.1 8B side by side on your actual workload. Both are good enough on general tasks that the benchmark gap often does not matter; what matters is how each model performs on your data with your prompt structure. --- GeneralCompute's API hosts all four of these models with an OpenAI-compatible endpoint, so you can compare them with a one-line code change: ```python from openai import OpenAI client = OpenAI( base_url="https://api.generalcompute.com/v1", api_key="your-api-key", ) # Swap the model string to compare model_id = "llama-3.1-8b-instruct" # or qwen2.5-7b-instruct, mistral-7b-v0.3, gemma-2-9b-it response = client.chat.completions.create( model=model_id, messages=[ {"role": "user", "content": "Write a Python function to binary search a sorted list."} ], ) print(response.choices[0].message.content) ``` See the [GeneralCompute model catalog](https://generalcompute.com/docs) for the full list of available models, current pricing, and latency benchmarks by model. --- # Small Models Showdown: Qwen 2.5 3B vs Llama 3.2 3B vs Phi-3.5 Mini vs Gemma 2 2B > A head-to-head benchmark comparison of the best sub-4B parameter models in 2025 -- covering reasoning, coding, math, context length, and inference speed to help you pick the right small model for your use case. - Author: General Compute - Published: 2026-07-20 - Tags: benchmarks, small-models, llm, inference, model-comparison - Canonical: https://www.generalcompute.com/blog/small-models-showdown-qwen-2-5-3b-llama-3-2-3b-phi-3-5-mini-gemma-2-2b --- Sub-4B models have become genuinely useful. A year ago the conventional wisdom was that small models were only worth reaching for on constrained hardware where you had no other choice. That has changed. The 2B-4B size bucket now contains models that score competitively on reasoning benchmarks, handle long contexts, and generate hundreds of tokens per second even on modest hardware. For latency-sensitive applications and edge deployments, they deserve a serious look. This post compares four of the strongest sub-4B models available in 2025: Qwen 2.5 3B, Llama 3.2 3B, Phi-3.5 Mini (3.8B), and Gemma 2 2B. We cover benchmark performance across reasoning, math, and coding, and then look at inference speed -- because at this size, throughput is often as important as capability. ## The four models **Qwen 2.5 3B** is Alibaba's 3 billion parameter model from the Qwen 2.5 release series. It supports a 128K context window and was trained with a heavy emphasis on code and mathematics alongside general instruction following. The instruct variant is available in a standard dense format. **Llama 3.2 3B** is Meta's smallest text-only model from the Llama 3.2 family. It also supports 128K context and uses grouped-query attention (GQA) for improved inference efficiency. Meta released both base and instruction-tuned variants; this comparison uses the instruct version. **Phi-3.5 Mini** is Microsoft's 3.8B parameter model from the Phi-3.5 series. It is the largest of the four models in this comparison, which matters for throughput comparisons. Phi-3.5 Mini supports 128K context and has been trained with a strong emphasis on reasoning quality relative to size. The training approach leans heavily on curated "textbook quality" synthetic data. **Gemma 2 2B** is Google's smallest Gemma 2 model at 2 billion parameters. It is the most compact of the four and benefits from architectural improvements introduced in Gemma 2 -- specifically, alternating attention layers (local and global) and knowledge distillation from larger Gemma 2 models during training. Context window is 8,192 tokens. ## Benchmark results ### General reasoning (MMLU) MMLU (Massive Multitask Language Understanding) tests knowledge across 57 subjects from elementary math to professional law. It remains the most widely-cited general capability benchmark. | Model | MMLU Score | |---|---| | Phi-3.5 Mini (3.8B) | ~69% | | Qwen 2.5 3B | ~65% | | Llama 3.2 3B | ~63% | | Gemma 2 2B | ~52% | Phi-3.5 Mini leads here by a meaningful margin. The gap between it and the two 3B models is real but not enormous -- about 4-6 points. Gemma 2 2B falls noticeably behind on this benchmark, though the score is partly explained by the model being trained at a different scale with different priorities. ### Math (GSM8K) GSM8K measures grade-school math word problem solving. It is a good proxy for multi-step reasoning accuracy in the small model regime. | Model | GSM8K Score | |---|---| | Phi-3.5 Mini (3.8B) | ~86% | | Qwen 2.5 3B | ~79% | | Llama 3.2 3B | ~77% | | Gemma 2 2B | ~40% | Math is where the gap widens. Phi-3.5 Mini at ~86% is substantially ahead of both 3B models, and Gemma 2 2B drops sharply. Qwen 2.5 3B and Llama 3.2 3B are close, with Qwen edging ahead slightly -- consistent with Qwen's stated focus on math during training. ### Coding (HumanEval) HumanEval tests Python code generation through 164 programming problems. It measures whether the model can write functionally correct code from a docstring. | Model | HumanEval Pass@1 | |---|---| | Qwen 2.5 3B | ~65% | | Phi-3.5 Mini (3.8B) | ~62% | | Llama 3.2 3B | ~45% | | Gemma 2 2B | ~20% | Coding is Qwen 2.5 3B's clearest win. It leads HumanEval by a few points over Phi-3.5 Mini and opens a larger gap over Llama and Gemma. If your application involves generating or reasoning about code, Qwen 2.5 3B is the small model to use. Llama 3.2 3B's coding score is notably lower than its general reasoning score -- it is a solid general model that was not optimized specifically for code. ### Context length Three of the four models support 128K context: - Qwen 2.5 3B: 128K - Llama 3.2 3B: 128K - Phi-3.5 Mini: 128K - Gemma 2 2B: 8K The 8K limit on Gemma 2 2B is a hard constraint that eliminates it from any use case requiring long documents, multi-turn conversation histories beyond a few turns, or large retrieval contexts. The 128K support on the other three models is genuine -- all three were trained with rope scaling for extended context. ## Inference speed Benchmark scores only matter in the context of how fast a model generates output in practice. At the sub-4B size bucket, throughput is a core differentiator. Inference speed is a function of parameter count, architecture, and the precision used (FP16, BF16, INT8, INT4). On a single A100 80GB with FP16 precision: | Model | Approx tokens/sec (single GPU, FP16) | |---|---| | Gemma 2 2B | ~800-1000 | | Qwen 2.5 3B | ~550-700 | | Llama 3.2 3B | ~500-650 | | Phi-3.5 Mini (3.8B) | ~420-550 | Gemma 2 2B is the fastest by a significant margin, which follows directly from having fewer parameters. Qwen 2.5 3B and Llama 3.2 3B are roughly similar, with minor differences depending on batch size and sequence length. Phi-3.5 Mini is the slowest of the group at 3.8B. On managed inference infrastructure like GeneralCompute's ASIC-optimized API, all four models run substantially faster than on self-hosted GPU. The ordering holds -- Gemma 2 2B still leads on throughput -- but the absolute speeds are much higher, making latency less of a constraint even for the larger models in this bucket. ## Summary table | | Qwen 2.5 3B | Llama 3.2 3B | Phi-3.5 Mini | Gemma 2 2B | |---|---|---|---|---| | MMLU | ~65% | ~63% | ~69% | ~52% | | GSM8K | ~79% | ~77% | ~86% | ~40% | | HumanEval | ~65% | ~45% | ~62% | ~20% | | Context | 128K | 128K | 128K | 8K | | Relative speed | Fast | Fast | Moderate | Fastest | | Params | 3B | 3B | 3.8B | 2B | ## When to use each model **Use Qwen 2.5 3B when coding is the primary task.** It leads HumanEval by a clear margin among 3B-class models, and its math scores are strong enough for most applications. The 128K context window makes it viable for code review, document summarization, and longer conversation loops. If you are building a small coding assistant, code explanation tool, or anything involving generation of structured data and code, Qwen 2.5 3B is the best choice in this size range. **Use Phi-3.5 Mini when reasoning accuracy matters more than throughput.** It scores highest on both MMLU and GSM8K, which reflects its training on high-quality reasoning data. The 3.8B parameter count means it is moderately slower than the 3B models, but the capability gap on complex reasoning tasks is real. Phi-3.5 Mini is a good choice for analytical tasks, question answering over documents, and any use case where you want the best general capability from a small model. **Use Llama 3.2 3B as a general-purpose baseline.** It is not the top scorer in any category, but it is close to Qwen 2.5 3B on reasoning and math while offering the ecosystem advantages of the Llama family: wide framework support, extensive community fine-tunes, and well-tested deployment tooling. If you want to avoid committing to any specific model family, Llama 3.2 3B is a safe default. **Use Gemma 2 2B for throughput-constrained applications.** It is the fastest model in this group by a meaningful margin, and its quality is competitive for simpler tasks. The 8K context limit is a real constraint, so it does not work for long-context applications. But for classification, short-form generation, summarization of short texts, or any use case where you need to process a very high volume of short requests, Gemma 2 2B's throughput advantage is significant. ## Quantization and deployment notes All four models are available in quantized formats, which meaningfully affects this comparison in production: **INT4 quantization** cuts memory usage roughly in half. A 3B FP16 model uses about 6GB of VRAM; in INT4 it fits in 2-3GB, which means it runs on consumer GPUs or even some edge devices. Quality loss with INT4 is usually small but noticeable on math tasks -- expect a few points of regression on GSM8K. **GGUF format** (used by llama.cpp) makes these models accessible on CPU. Gemma 2 2B and the 3B models are fast enough in GGUF INT4 on a modern laptop CPU to be usable for offline applications. Phi-3.5 Mini at 3.8B is slower on CPU but still functional. For server-side deployment, vLLM, TGI, and SGLang all support these models natively. If you are using an API provider, all four are widely available through hosted inference endpoints. ## A note on fine-tuning One advantage of small models that is easy to overlook: they are much cheaper to fine-tune than 7B or 70B models. A QLoRA fine-tune of a 3B model on a single A100 takes an hour or two and produces a task-specific model that can outperform the base model by a large margin on a narrow domain. For production applications where you have labeled data and want to push beyond what the base model achieves, fine-tuning a small model is often faster and cheaper than prompting a larger one. The fine-tuned 3B model also remains fast at inference time, which matters if you care about cost-per-query. ## Conclusion The sub-4B size bucket now has strong options for a range of use cases. Phi-3.5 Mini leads on general reasoning and math. Qwen 2.5 3B leads on coding. Gemma 2 2B leads on throughput. Llama 3.2 3B is a solid general baseline with broad ecosystem support. The right model depends on your task distribution and latency requirements. For applications where the primary bottleneck is throughput and tasks are relatively simple, Gemma 2 2B or one of the 3B models running at high tokens-per-second is often the right answer. For tasks that require reliable reasoning, Phi-3.5 Mini is worth the moderate speed trade-off. --- GeneralCompute hosts all of the models in this comparison with an OpenAI-compatible API, giving you fast inference across the full sub-4B range without managing infrastructure: ```python from openai import OpenAI client = OpenAI( base_url="https://api.generalcompute.com/v1", api_key="your-api-key", ) response = client.chat.completions.create( model="qwen2.5-3b-instruct", # or llama-3.2-3b, phi-3.5-mini, gemma-2-2b-it messages=[{"role": "user", "content": "Explain binary search in Python with an example."}], ) print(response.choices[0].message.content) ``` See the [GeneralCompute model catalog](https://generalcompute.com/docs) for available models, current pricing, and latency benchmarks. --- # Agentic RAG: When Retrieval and Generation Both Need to Be Fast > In agentic RAG, retrieve-then-generate runs in loops, not once. That changes everything about how latency compounds and where you need to optimize. - Author: General Compute - Published: 2026-07-19 - Tags: agents, rag, inference, latency, retrieval - Canonical: https://www.generalcompute.com/blog/agentic-rag-when-retrieval-and-generation-both-need-to-be-fast --- Standard RAG has a simple structure: the user asks a question, your system retrieves relevant chunks, and the LLM generates a response conditioned on those chunks. One retrieval step, one generation step, done. The latency budget is bounded and predictable. Agentic RAG looks different. The agent retrieves, generates, reasons about what it found, decides what else it needs, retrieves again, generates again, and continues until it has enough to produce a final answer. This loop can iterate anywhere from two times to twenty or more, depending on the task. Each iteration adds retrieval latency plus generation latency. By the time the agent finishes, you have not paid once for retrieval and once for generation -- you have paid for both at every step of the loop. That is the core problem this post addresses. When retrieve-then-generate runs in a loop, the latency at each stage multiplies by the number of iterations. Slow retrieval and slow generation, which are individually manageable in single-turn RAG, become the dominant engineering constraints in agentic RAG. ## What the loop actually looks like A concrete example helps. Consider a research agent tasked with answering a complex question about a large codebase or document corpus: 1. The agent receives the question and retrieves the most relevant chunks from a vector store. 2. It generates an initial reasoning step, deciding what it knows and what it still needs. 3. Based on that reasoning, it formulates follow-up queries and retrieves again. 4. It generates another reasoning step, integrating the new information. 5. This continues until the agent determines it has enough to answer, at which point it generates the final response. A five-iteration loop with 400ms retrieval and 800ms generation per step takes at least 6 seconds of wall-clock time, assuming no parallelism and ignoring overhead. Ten iterations takes 12 seconds. And these numbers assume retrieval and generation are fast -- many production RAG pipelines are significantly slower than that. The other thing that changes with loops is context growth. Each iteration adds retrieved content and generated reasoning to the conversation history. By iteration five or six, you are sending a substantially larger prompt than you were at iteration one. Since inference providers charge per token and longer prompts take longer to process, your per-step cost and latency both increase as the loop progresses. ## Where the time goes in retrieval Retrieval latency has several components, each of which matters at loop scale. **Embedding the query**: Before you can search a vector store, you need to embed the query text. Embedding models have their own latency -- typically 20-100ms for a single query depending on the model size and infrastructure. Across ten iterations, that is 200ms-1000ms in embedding alone, before any search happens. **Vector search**: The search itself against a populated vector store is usually fast -- most managed vector databases return results in 10-50ms for typical collection sizes. This is rarely the bottleneck, but it is not free. **Document loading and chunking overhead**: After identifying relevant chunk IDs, many RAG systems need to fetch the actual document text from a separate store. If that store is a remote database or object storage, this adds a round-trip -- often 20-100ms. If your chunks are large (1,000-2,000 tokens), the volume of text being loaded also grows. **Reranking**: Many production RAG pipelines add a reranker after the initial vector search to improve precision. Rerankers are cross-encoders that score each retrieved chunk against the query, which is much slower than vector search -- typically 50-300ms depending on the model and number of candidates. In single-turn RAG, this is often worth the cost. In a ten-iteration agentic loop, an extra 150ms per retrieval step adds 1.5 seconds to the total. The practical takeaway: for agentic RAG, you want each retrieval step to be fast, which usually means accepting slightly lower recall in exchange for lower latency -- or finding ways to parallelize retrieval with other work. ## Where the time goes in generation On the generation side, the main variables are time-to-first-token (TTFT) and tokens-per-second throughput. TTFT is the time between sending a request and receiving the first token of the response. In a streaming setup, this is what determines when the user (or the next stage of the pipeline) starts getting output. For agentic RAG, TTFT matters per step -- a 600ms TTFT across ten steps is 6 seconds in TTFT overhead alone, even before you count the time to generate the actual tokens. Throughput matters when the agent is generating substantial reasoning or when the final answer is long. If your model generates 30 tokens/second and each reasoning step produces 200 tokens, each generation step takes about 6.7 seconds. Across ten steps, that is over a minute of generation time. Context growth amplifies both of these. As the conversation history accumulates retrieved chunks and generated reasoning, the prompt that goes into each inference call gets longer. Longer prompts take more time to process, driving up TTFT. If you are using a model with a fixed context window and you are not truncating or summarizing, you will eventually hit that limit. ## Patterns that help Several patterns reduce the compound latency of agentic RAG. Most of them require either parallelizing work, reducing unnecessary overhead, or making smarter decisions about when to retrieve at all. ### Parallel retrieval If the agent needs to retrieve from multiple queries or multiple data sources at a given step, retrieve them in parallel rather than sequentially. This is the most straightforward optimization and often has a large impact. A step that would take 400ms + 400ms = 800ms in serial takes 400ms when both retrievals run simultaneously. ```python import asyncio async def parallel_retrieve(queries: list[str], retriever) -> list[list[dict]]: tasks = [retriever.aretrieve(q) for q in queries] return await asyncio.gather(*tasks) ``` The agent's reasoning step can identify multiple information needs at once, submit them as parallel retrieval queries, and integrate all results in the next step. This trades sequential latency for a small amount of over-retrieval, which is usually a good tradeoff. ### Prefetch the likely next query If the agent's reasoning pattern is predictable -- for example, it always follows a broad retrieval with a narrower clarification retrieval -- you can start the next retrieval before the current generation step has finished. While the LLM is generating its reasoning about the current retrieved chunks, you can begin embedding and searching for the likely follow-up. This requires some prediction of what the agent will want next, which is domain-specific. But even rough prefetching can eliminate full round-trips from the critical path. ### Stream generation and act on partial output If your agent framework allows it, start processing the generated output as it arrives rather than waiting for the complete response. This is particularly useful when the agent's output includes explicit signals that can be parsed incrementally -- a structured JSON output with query fields that appear early in the stream. With streaming, you can begin the next retrieval query as soon as the agent has emitted the query text, even if it has not finished generating the rest of its reasoning. This overlaps retrieval latency with generation latency rather than sequencing them. ```python async def streaming_step_with_prefetch(client, messages, retriever): buffer = "" next_query = None retrieval_task = None async for chunk in client.chat.completions.create( model="your-model", messages=messages, stream=True, ): delta = chunk.choices[0].delta.content or "" buffer += delta # As soon as we can parse a retrieval query from the stream, kick it off if next_query is None: parsed = try_parse_query(buffer) if parsed: next_query = parsed retrieval_task = asyncio.create_task(retriever.aretrieve(parsed)) results = await retrieval_task if retrieval_task else [] return buffer, results ``` ### Reduce reranking to where it matters Reranking is worth doing on the first retrieval, where recall is highest and precision is lowest. On later iterations where the agent is doing targeted lookups for specific information, the initial vector search is often precise enough without reranking. Skip the reranker on follow-up retrievals to save 50-300ms per step. ### Bound context growth explicitly The retrieved chunks that accumulate in the conversation history are often not all equally useful. Later steps do not need the full text of chunks that were retrieved in early steps -- they need the agent's reasoning about those chunks, which is more compact. Instead of appending raw retrieved chunks to the history at each step, have the agent summarize what it learned from each retrieval. This keeps the prompt size from growing linearly with iterations. Paired with a sliding window that keeps only the last few raw retrieval results, you can keep prompt size roughly constant across the loop, which holds TTFT and per-step cost stable. ```python def build_iterative_context(steps: list[dict], max_recent: int = 2) -> list[dict]: messages = [] # Older steps: include only the summary, not the raw chunks for step in steps[:-max_recent]: messages.append({"role": "user", "content": step["summary"]}) messages.append({"role": "assistant", "content": step["reasoning"]}) # Recent steps: include the full retrieved context for step in steps[-max_recent:]: chunks_text = "\n\n".join(c["text"] for c in step["chunks"]) messages.append({"role": "user", "content": f"Retrieved:\n{chunks_text}"}) messages.append({"role": "assistant", "content": step["reasoning"]}) return messages ``` ### Use a smaller model for reasoning, larger for synthesis Not every step in an agentic RAG loop requires your most capable model. Intermediate reasoning steps -- deciding what to retrieve next, summarizing what was retrieved, checking whether a source is relevant -- are often well within the capability of a smaller, faster model. Route to a larger model only for the final synthesis step where the agent is generating the response the user will actually see. This can cut inference cost and per-step latency substantially across the middle steps of a long loop. ```python STEP_MODEL_ROUTING = { "plan": "large-model", "reason": "small-fast-model", "summarize": "small-fast-model", "synthesize": "large-model", } ``` ## Measuring what matters To optimize an agentic RAG system, you need to measure the right things. A single end-to-end latency number for the full agent run hides where the time is actually going. Track these per step: - Retrieval latency (embedding time + search time + document load time) - TTFT for each generation call - Output token count per step (to estimate generation time) - Prompt token count per step (to track context growth) - Which step in the loop you are on (to identify whether later steps are slower) With this data, you can identify whether your bottleneck is retrieval or generation, and whether latency is growing across iterations (context growth) or staying flat (you have bounded it effectively). A simple structured log per step is enough to start: ```python import time import logging import json logger = logging.getLogger("rag_agent") def log_step(step_idx, retrieval_ms, ttft_ms, output_tokens, prompt_tokens): logger.info(json.dumps({ "step": step_idx, "retrieval_ms": retrieval_ms, "ttft_ms": ttft_ms, "output_tokens": output_tokens, "prompt_tokens": prompt_tokens, })) ``` Plot `prompt_tokens` over step index. If it is growing steeply, context is accumulating and you need summarization or truncation. If `retrieval_ms` dominates total step time, focus on parallel retrieval and reranker reduction. If `ttft_ms` is the main factor, faster inference is the lever. ## The asymmetry of optimization One useful observation about agentic RAG: retrieval and generation have different optimization levers, and they are not equally easy to pull. Retrieval latency is largely determined by infrastructure choices -- which vector database you use, how you size your embedding model, whether you run reranking. These are one-time architectural decisions that are hard to change once a system is in production, but they are also choices you can make deliberately at design time. Generation latency is determined by model size and inference throughput. The fastest way to reduce generation latency per step is to use a faster inference provider. A model that generates 200 tokens/second versus 30 tokens/second cuts generation time by 6x without any changes to your prompts, retrieval logic, or agent design. For agentic RAG with many generation steps, fast inference has a linear impact on total wall-clock time. This is why the two need to be considered together. Optimizing retrieval while leaving generation slow, or vice versa, still leaves you with a compound latency problem. The fastest step in a sequential chain does not matter if the other steps remain slow. --- General Compute provides inference at high tokens-per-second throughput across major open-source models, with low TTFT that keeps each step in an agentic loop fast. The OpenAI-compatible API means you can swap in a faster endpoint with minimal code changes: ```python from openai import OpenAI client = OpenAI( base_url="https://api.generalcompute.com/v1", api_key="your-api-key", ) ``` For agentic RAG workloads, see the [GeneralCompute docs](https://generalcompute.com/docs) for current model options and latency benchmarks. --- # Long-Running Agents: Managing Inference Costs Over Hours of Execution > When an agent runs for hours instead of seconds, inference costs and context management become the dominant engineering problems. Here's how to keep them under control. - Author: General Compute - Published: 2026-07-18 - Tags: agents, inference, cost, context-management, production - Canonical: https://www.generalcompute.com/blog/long-running-agents-managing-inference-costs-over-hours-of-execution --- Most agent use cases are short. A user asks a question, the agent makes a few tool calls, and the interaction finishes in under a minute. The engineering concerns at that scale -- TTFT, per-step latency, prompt construction -- are well understood. Long-running agents are different in character. Think of a research agent that autonomously gathers sources, synthesizes findings, and refines its conclusions over several hours. Or a background worker that monitors a codebase, runs tests, opens pull requests, and responds to review feedback over a full workday. Or a data pipeline agent that ingests, transforms, and validates records across a multi-hour batch window. These agents make hundreds or thousands of LLM calls, accumulate context that can push against model limits, and generate inference costs that add up in ways that are easy to miss until the bill arrives. This post covers the engineering patterns that matter specifically when your agent runs for hours: context management, cost accounting, model routing, and the checkpointing strategies that let you recover from failures without losing work or spending twice. ## How costs compound at long time scales In a short agent interaction, inference cost is mostly determined by the number of steps and the size of each LLM call. Both are bounded by the task. A 10-step coding agent with modest context costs a predictable amount. For long-running agents, the cost dynamics change in two important ways. First, context grows. Most agents include prior conversation history in each LLM request. Over hundreds of steps, that history can grow to tens of thousands of tokens. Since inference providers charge per token in and out, sending a 50,000-token context on every step is expensive -- and the cost per step grows over time as context accumulates, even if the step itself produces a small output. Second, the error budget for inefficiency shrinks. When an agent makes 15 LLM calls, a 2x cost inefficiency costs you twice the price of a short run. When an agent makes 2,000 LLM calls over 8 hours, the same inefficiency represents a large absolute dollar figure. The cost of ignoring prompt size, model selection, and token overhead at small scale becomes the cost of a failed project at large scale. A rough estimate for a research agent running for 6 hours with a capable frontier model, if you are not careful about context management: - 600 LLM calls averaging 8,000 input tokens each = 4.8M input tokens - At a typical frontier model price of $3-5 per million tokens, that is $14-24 in input tokens alone - Plus output tokens, plus retries, plus tool call parsing That is not always prohibitive, but it is easy to blow past budgets when agents are running continuously and no one is watching. ## Context management over long runs The central engineering problem in long-running agents is what to do with the accumulating conversation history. You have four main options, each with different cost and quality tradeoffs. **Sliding window**: Keep only the last N tokens of history in each prompt. Simple to implement, zero extra cost, no state management overhead. The downside is that the agent forgets everything outside the window. For tasks where early context matters -- a research plan established in the first hour, a constraint stated at the start -- a sliding window will cause the agent to contradict itself or repeat work. **Summarization**: At regular intervals (or when the context exceeds a threshold), have the agent summarize the conversation history into a compact representation, then replace the raw history with the summary. This preserves the gist of prior work at much lower token cost. The tradeoff is that summaries lose detail, and if the agent summarizes too aggressively, it may lose important nuance. A good heuristic: summarize every 30-50 steps, keeping the full raw history for the last 10-15 steps and a summary for everything before. **External memory**: Store conversation history and intermediate results outside the prompt -- in a vector database or a structured state store -- and retrieve relevant pieces at each step via semantic search. This is the most flexible approach for very long runs. It requires more infrastructure and adds retrieval latency, but it keeps prompt sizes bounded regardless of how long the agent has been running. It works especially well for tasks with structured intermediate state, like a research agent that accumulates sources and findings. **Hierarchical summarization**: A refinement of summarization, where you maintain summaries at multiple levels of detail -- a one-paragraph high-level summary updated every 50 steps, a more detailed recent-events summary updated every 10 steps, and raw history for the last 5 steps. The agent includes all three in its prompt, with the high-level summary providing continuity and the recent history providing specificity. This preserves more context than a single summary while keeping token counts manageable. Here is a basic sliding-window with summarization implementation in Python: ```python from dataclasses import dataclass, field from typing import List, Optional @dataclass class ContextManager: max_recent_tokens: int = 8000 summarize_every_n_steps: int = 30 recent_messages: List[dict] = field(default_factory=list) running_summary: Optional[str] = None steps_since_summary: int = 0 def add_message(self, role: str, content: str): self.recent_messages.append({"role": role, "content": content}) self.steps_since_summary += 1 def build_prompt_messages(self, system: str) -> List[dict]: messages = [{"role": "system", "content": system}] if self.running_summary: messages.append({ "role": "user", "content": f"Summary of prior work:\n{self.running_summary}" }) messages.append({ "role": "assistant", "content": "Understood. Continuing from where we left off." }) messages.extend(self.recent_messages[-20:]) return messages def should_summarize(self) -> bool: return self.steps_since_summary >= self.summarize_every_n_steps def apply_summary(self, new_summary: str): self.running_summary = new_summary self.recent_messages = self.recent_messages[-10:] self.steps_since_summary = 0 ``` The summarization call itself is worth doing with a smaller, cheaper model rather than your main reasoning model. Summarizing 2,000 tokens of history does not require the same capability as planning the next research step. ## Model routing to control cost Not every step in a long-running agent requires your most capable model. A research agent, for example, might use a large frontier model for initial planning and synthesis, but most of its work is routine: fetching a URL, extracting structured data from a document, checking whether a source is relevant. These steps can be done with a smaller, cheaper model at equivalent quality. Routing by step type is straightforward to implement if you structure your agent around named step types: ```python MODEL_ROUTING = { "plan": "large-frontier-model", "synthesize": "large-frontier-model", "summarize": "small-fast-model", "extract": "small-fast-model", "classify": "small-fast-model", "write": "medium-model", } def get_model_for_step(step_type: str) -> str: return MODEL_ROUTING.get(step_type, "medium-model") ``` The cost savings from routing can be substantial. If 70% of your agent's steps are routine extraction and classification tasks, and you route those to a model that is 10x cheaper per token, you reduce your overall inference cost by a large amount without meaningful quality loss. One pattern that works well is a two-stage approach for each step: run the cheap model first, and escalate to the expensive model only if the cheap model signals uncertainty or produces an output that fails validation. This keeps costs low for the majority of well-formed steps while ensuring quality on the edge cases. ## Cost accounting and budget enforcement For a 10-minute agent, you can mentally track costs. For an 8-hour agent, you need instrumentation. At minimum, track tokens in and out for every LLM call and accumulate totals in your agent's state. Check the running total before each step and halt if you have exceeded a configured budget: ```python @dataclass class CostTracker: input_token_cost_per_m: float output_token_cost_per_m: float budget_usd: float total_input_tokens: int = 0 total_output_tokens: int = 0 def record_call(self, input_tokens: int, output_tokens: int): self.total_input_tokens += input_tokens self.total_output_tokens += output_tokens @property def cost_usd(self) -> float: return ( self.total_input_tokens / 1_000_000 * self.input_token_cost_per_m + self.total_output_tokens / 1_000_000 * self.output_token_cost_per_m ) def check_budget(self): if self.cost_usd >= self.budget_usd: raise BudgetExceeded( f"Agent exceeded budget of ${self.budget_usd:.2f}. " f"Spent ${self.cost_usd:.2f} over {self.total_input_tokens + self.total_output_tokens} tokens." ) ``` A hard budget ceiling prevents runaway cost from loops, unexpected context growth, or model errors that cause the agent to spin. Budget enforcement should be checked before each LLM call, not just at the end, so the agent can save its state and exit gracefully rather than failing mid-step. Logging cost breakdowns by step type also helps you identify which steps are most expensive. Often 80% of cost comes from 20% of step types, and that is where optimization effort is best spent. ## Checkpointing for resumability A long-running agent that fails at hour 6 of an 8-hour run has wasted everything if it cannot resume. Checkpointing serializes agent state to durable storage at regular intervals, so a restart can pick up from the last checkpoint rather than from scratch. What to include in a checkpoint: - The running summary of prior work (or full history if manageable) - Intermediate results that are expensive to reproduce (gathered sources, generated artifacts) - The current step index and any step-specific state - The cost tracker state, so budget enforcement survives restarts - A task queue or plan, with completed steps marked Checkpoint frequency is a tradeoff between checkpoint overhead (each checkpoint is an I/O write plus potentially a summarization call) and the amount of work lost on failure. For most long-running agents, checkpointing every 10-20 steps is reasonable. ```python import json from pathlib import Path def save_checkpoint(state: dict, checkpoint_path: Path): tmp = checkpoint_path.with_suffix(".tmp") tmp.write_text(json.dumps(state, indent=2)) tmp.rename(checkpoint_path) # atomic rename avoids partial writes def load_checkpoint(checkpoint_path: Path) -> Optional[dict]: if checkpoint_path.exists(): return json.loads(checkpoint_path.read_text()) return None ``` The atomic rename pattern (write to a temp file, then rename) avoids corrupting the checkpoint if the process is killed mid-write. For agents running in distributed environments, use a durable store -- S3, a database, or a distributed cache -- rather than local files. The checkpoint key should include a run ID so that parallel runs do not overwrite each other. ## Rate limiting and throttling Inference providers have rate limits, typically expressed in requests per minute and tokens per minute. A long-running agent that runs at full speed can hit these limits, which causes requests to fail or queue. Unexpected rate limit errors mid-run can break agent logic if not handled gracefully. For long-running agents, it is often worth deliberately throttling below the rate limit ceiling, especially during periods when the agent's work is not time-sensitive. Spreading 2,000 LLM calls over 8 hours at a steady rate of 4 per minute is much less likely to cause problems than bursting to the rate limit and triggering backpressure. Exponential backoff with jitter is the standard pattern for handling rate limit responses: ```python import asyncio import random async def call_with_retry(client, messages, max_retries=5): for attempt in range(max_retries): try: return await client.complete(messages) except RateLimitError: if attempt == max_retries - 1: raise wait = (2 ** attempt) + random.uniform(0, 1) await asyncio.sleep(wait) ``` If your agent has natural phases -- a planning phase, a research phase, a synthesis phase -- you can also throttle between phases rather than between individual steps. This keeps each phase fast while keeping the overall rate manageable. ## Monitoring during long runs For short agent runs, you can check the output after the fact. For long runs, you want visibility while the agent is executing, both to catch runaway cost and to monitor progress. A minimal monitoring setup emits structured events from the agent loop to a log stream: ```python import logging import json logger = logging.getLogger("agent") def log_step(step_index, step_type, duration_ms, tokens_in, tokens_out, cumulative_cost): logger.info(json.dumps({ "step": step_index, "type": step_type, "duration_ms": duration_ms, "tokens_in": tokens_in, "tokens_out": tokens_out, "cumulative_cost_usd": cumulative_cost, })) ``` From there, you can stream these logs to any observability platform and set alerts on cumulative cost, error rate, or steps per hour. If the agent slows down significantly (steps per hour dropping), that usually indicates context growth driving up per-step latency, and it is a good trigger for a manual summarization pass. ## Putting it together The engineering for long-running agents is not exotic -- it is mostly careful accounting. Track context size and summarize before it gets out of hand. Route cheap steps to cheaper models. Enforce a budget ceiling and checkpoint often enough that failures are recoverable. Throttle to avoid rate limit failures. Log structured events so you can tell whether the agent is making progress. The cost and reliability properties that are acceptable to ignore at small scale become the main engineering constraints at long runtime scale. Getting these patterns in place before your agent runs for the first time at multi-hour scale is much easier than diagnosing a $200 failed run after the fact. --- General Compute's API supports OpenAI-compatible requests for all major open-source models, with high tokens-per-second throughput that keeps per-step latency low across long runs. For long-running agents, the practical benefit is that each individual step finishes faster, which means the agent makes more progress per hour for the same wall-clock budget. To get started: ```python from openai import OpenAI client = OpenAI( base_url="https://api.generalcompute.com/v1", api_key="your-api-key", ) ``` See the [GeneralCompute docs](https://generalcompute.com/docs) for available models and rate limit tiers. --- # Evaluating Agent Performance: Latency as a First-Class Metric > Most agent benchmarks measure accuracy and task completion. Here's why latency belongs in your evaluation framework alongside accuracy, and how to instrument your agent to collect the data you need. - Author: General Compute - Published: 2026-07-17 - Tags: agents, inference, latency, benchmarking, evaluation - Canonical: https://www.generalcompute.com/blog/evaluating-agent-performance-latency-as-a-first-class-metric --- When teams benchmark AI agents, the focus almost always lands on accuracy: did the agent complete the task correctly, call the right tools, produce valid outputs? These are important questions. But they leave out something that matters just as much in production deployments -- how long did it take? For single-turn LLM calls, latency is straightforward to measure. You send a request, you track time-to-first-token and total generation time, and you have what you need. For agents, the picture is more complicated. An agent might make a dozen LLM calls interleaved with tool calls, retrieval operations, and state management. Latency compounds across every step, and a slow model that scores well on accuracy benchmarks is often less useful in practice than a faster one that scores slightly lower. This post covers why latency belongs in your evaluation framework, how to measure it properly, and what a practical benchmarking setup looks like. ## Why accuracy alone is misleading Accuracy benchmarks for agents are typically run offline: you give the agent a fixed dataset of tasks, compare outputs to ground truth, and compute a score. This works for measuring capability but tells you nothing about what users actually experience. Consider a coding agent evaluated on SWE-bench. It might resolve 40% of GitHub issues correctly. That is a reasonable signal for capability. But the benchmark says nothing about whether each task takes 8 seconds or 80 seconds. For a developer waiting on a code suggestion, those 72 extra seconds are not an acceptable trade-off for the same accuracy score. The deeper problem is that agent latency and agent accuracy are orthogonal axes. A model that is fast but slightly less accurate and a model that is slow but slightly more accurate can score identically on a capability benchmark while having wildly different suitability for a given use case. Without latency data alongside accuracy data, you cannot make a principled choice between them. For real-time applications like voice agents, interactive coding assistants, and customer-facing chatbots, latency sets a hard ceiling on what is even possible. For batch processing like overnight report generation or background research, throughput and cost matter more than interactive latency. Your evaluation framework should reflect which mode you are actually building for, and accuracy alone does not encode that distinction. ## What to measure Agent latency is not a single number. There are several distinct metrics worth tracking, and each one tells you something different. **End-to-end task latency** is the wall-clock time from when the user submits a task to when the agent returns a final result. This is the number users experience directly. It should be your primary production metric. **Per-step LLM latency** measures how long each individual inference call takes. This includes time-to-first-token (TTFT), which determines how quickly the agent starts responding, and tokens-per-second (TPS), which determines how quickly it finishes. When an agent makes 10 LLM calls, per-step latency compounds directly into end-to-end latency. A model that is 300ms faster per call saves 3 full seconds across a 10-step pipeline. **Tool call latency** covers time spent executing tools: web searches, code execution, database queries, external API calls. Inference providers have no control over this, but it is often where agents spend surprising amounts of time. Separating tool latency from inference latency in your instrumentation lets you identify where the time is actually going before deciding what to optimize. **Step overhead** is the dead time between steps, spent on parsing LLM output, building the next prompt, and updating state. This is usually small but worth tracking to catch runaway prompt construction or expensive serialization. **P50 vs P99 latency** deserves particular attention. Agents are far more variable than single-turn calls because they take different numbers of steps on different inputs. A task that averages 5 steps might take 20 in the worst case. P99 latency tells you what users experience on the hardest inputs, and it is often three to five times the median. An agent with a fast median and a catastrophic P99 will still generate support tickets. ## Instrumentation Collecting these metrics requires explicit instrumentation. Most agent frameworks do not emit detailed timing data by default. A simple approach wraps your LLM client to capture per-call timing and accumulates traces across the run: ```python import time from dataclasses import dataclass, field from typing import List, Optional @dataclass class StepTrace: step_index: int step_type: str # "llm" | "tool" | "overhead" start_time: float end_time: float ttft: Optional[float] = None tokens_generated: Optional[int] = None @property def duration_ms(self) -> float: return (self.end_time - self.start_time) * 1000 @dataclass class AgentTrace: task_id: str start_time: float end_time: Optional[float] = None steps: List[StepTrace] = field(default_factory=list) @property def total_duration_ms(self) -> float: if self.end_time is None: return 0.0 return (self.end_time - self.start_time) * 1000 @property def llm_duration_ms(self) -> float: return sum(s.duration_ms for s in self.steps if s.step_type == "llm") @property def tool_duration_ms(self) -> float: return sum(s.duration_ms for s in self.steps if s.step_type == "tool") @property def llm_fraction(self) -> float: if self.total_duration_ms == 0: return 0.0 return self.llm_duration_ms / self.total_duration_ms ``` Wrapping your LLM client to record step traces looks like this: ```python class InstrumentedClient: def __init__(self, client, trace: AgentTrace): self._client = client self._trace = trace self._step_index = 0 def complete(self, messages, **kwargs): step = StepTrace( step_index=self._step_index, step_type="llm", start_time=time.perf_counter(), end_time=0.0, ) self._step_index += 1 first_token_time = None def on_first_chunk(): nonlocal first_token_time if first_token_time is None: first_token_time = time.perf_counter() result = self._client.complete(messages, on_chunk=on_first_chunk, **kwargs) step.end_time = time.perf_counter() if first_token_time is not None: step.ttft = (first_token_time - step.start_time) * 1000 step.tokens_generated = result.usage.completion_tokens self._trace.steps.append(step) return result ``` Once you are collecting traces, aggregating across runs gives you the distribution: ```python import statistics def summarize_traces(traces: List[AgentTrace]) -> dict: completed = [t for t in traces if t.end_time is not None] durations = sorted(t.total_duration_ms for t in completed) n = len(durations) return { "p50_ms": durations[n // 2], "p95_ms": durations[int(n * 0.95)], "p99_ms": durations[int(n * 0.99)], "mean_ms": statistics.mean(durations), "mean_steps": statistics.mean(len(t.steps) for t in completed), "mean_llm_fraction": statistics.mean(t.llm_fraction for t in completed), } ``` The `mean_llm_fraction` is particularly useful. If it is high (above 0.70), switching to a faster inference provider will have a large effect on end-to-end latency. If it is low, the bottleneck is tool calls or overhead, and you will need to look elsewhere. ## Designing a latency benchmark A good latency benchmark has a few structural requirements. **A representative task set** covers the actual range of tasks your agent handles. If your coding agent mostly does small bug fixes but your benchmark is all complex feature additions, you will overestimate latency for most real traffic. Stratify tasks by complexity so you can report latency separately for easy, medium, and hard inputs. **Controlled infrastructure** means running your benchmark with consistent hardware, consistent network conditions, and a consistent load on the inference provider. Agent latency is noisy. Run each task multiple times -- three to five runs is usually enough -- and use the median to reduce variance. **Accuracy and latency measured on the same tasks.** You need both axes to make principled model selection decisions. A scatter plot with task-level accuracy on one axis and task-level latency on the other shows the full picture more clearly than any single aggregate score. **Step count alongside latency.** If slow tasks are slow because they take more steps, that is a different problem from slow tasks being slow because each step is slow. Step count variation usually points to model reasoning quality: can the model plan efficiently enough to avoid unnecessary tool calls? Per-step latency variation points to the inference provider. A minimal benchmark runner: ```python import asyncio from typing import Callable, Awaitable async def run_benchmark( tasks: list[dict], agent_fn: Callable[[dict], Awaitable[dict]], runs_per_task: int = 3, ) -> list[dict]: results = [] for task in tasks: for run in range(runs_per_task): trace = AgentTrace(task_id=task["id"], start_time=time.perf_counter()) try: output = await agent_fn(task, trace=trace) success = True except Exception as e: output = {"error": str(e)} success = False trace.end_time = time.perf_counter() results.append({ "task_id": task["id"], "difficulty": task.get("difficulty", "unknown"), "run": run, "latency_ms": trace.total_duration_ms, "step_count": len(trace.steps), "llm_fraction": trace.llm_fraction, "success": success, "correct": task.get("evaluator", lambda o: None)(output), }) return results ``` ## The accuracy-latency tradeoff in practice Once you have latency data alongside accuracy data, you can reason about model selection more concretely. The tradeoff is not arbitrary -- there is typically a Pareto frontier where smaller models are faster but less accurate, and larger models are more accurate but slower. Where the optimal point sits depends on what you are building: For **interactive agents** where users are waiting in real time, you generally want end-to-end task latency under a few seconds for most tasks. A 5% accuracy drop that saves 8 seconds per task is usually worth taking. The user who gets a slightly less accurate answer in 2 seconds has a better experience than the user who waits 10 seconds for a slightly better one. For **background agents** that run asynchronously and deliver results later, optimize for cost and accuracy. Latency tolerance is much higher when users are not blocking on the result. For **streaming-capable agents** that can push intermediate results to the UI, TTFT matters more than total latency. If the first token arrives within 300ms, users perceive the agent as responsive even if the full output takes several seconds to finish. These distinctions are invisible if you only benchmark accuracy. ## How inference speed multiplies across steps In a single-turn LLM call, inference speed affects one interaction. In a multi-step agent, it affects every step. An agent that makes 10 LLM calls has 10 opportunities for inference latency to accumulate. If your current setup takes 800ms per LLM call, a 10-step agent run costs at least 8 seconds from inference alone -- not counting tool calls or overhead. Switching to an inference provider where calls take 150ms brings that to 1.5 seconds, a reduction that shows up directly in the end-to-end latency distribution your benchmark produces. The multiplier also works the other way. If your inference gets 2x slower -- for example, because you swap to a larger model or your provider has degraded performance -- your agent latency doubles. Monitoring per-step LLM latency in production, not just end-to-end latency, lets you catch this kind of regression before it shows up as user complaints. ## What your evaluation checklist should include When setting up agent evaluation, make sure you are tracking: - End-to-end task latency (p50, p95, p99) across your full task set - Per-step LLM latency and TTFT, broken out from tool call latency - Step count distribution by task and by task difficulty level - Accuracy measured on the same tasks as latency, so you can correlate them - Separate breakdowns for different task types: simple, complex, error recovery cases With these numbers, you can make informed decisions about model selection, inference provider, and whether optimizing for accuracy or for speed has more leverage for your specific workload. If you are not tracking latency at all right now, start with end-to-end task latency and step count. Those two numbers, measured over a representative task set, will tell you more about your agent's production behavior than any accuracy benchmark alone. --- General Compute's API runs popular open-source models -- including the ones commonly used in LangGraph, CrewAI, and AutoGen -- at high tokens-per-second on an OpenAI-compatible endpoint. If you want to run latency-aware agent benchmarks against fast infrastructure, the setup is a one-line base URL change: ```python from openai import OpenAI client = OpenAI( base_url="https://api.generalcompute.com/v1", api_key="your-api-key", ) ``` From there you can instrument the client with the trace patterns from this post and measure what your agent's latency distribution actually looks like. See the [GeneralCompute docs](https://generalcompute.com/docs) for model options and API details. --- # Browser Agents: Inference Speed as the Gating Factor > Browser agents take dozens of LLM calls to complete a single web task. Here's why inference latency determines whether they're usable in practice. - Author: General Compute - Published: 2026-07-16 - Tags: browser agents, inference speed, agentic AI, web automation, latency - Canonical: https://www.generalcompute.com/blog/browser-agents-inference-speed-as-the-gating-factor --- Browser agents are one of the more concrete demonstrations of agentic AI: a model that can navigate a webpage, fill out a form, extract information from a table, and complete a multi-step task without human intervention. They're also one of the clearest examples of how inference speed becomes a hard constraint rather than a nice-to-have. If you've built or experimented with a browser agent, you've probably hit the wall. The agent works -- it can browse, click, read, and act -- but it's painfully slow. A task that takes a human 30 seconds takes the agent several minutes. Users stop waiting. Demos fall flat in real time. The bottleneck is almost never the browser automation library. It's the inference calls. ## What a Browser Agent Actually Does To understand why latency compounds so badly, it helps to trace what happens during a single web task. Say the agent's job is to log into a site, find an invoice from last month, and extract the total amount. The agent loop looks something like this: 1. Take a screenshot (or parse the DOM) 2. Call the LLM: "Here is the current state of the page. What should I do next?" 3. Receive an action (click this element, type this text, scroll down) 4. Execute the action in the browser 5. Take another screenshot 6. Repeat until the task is complete or the agent gives up Each iteration requires at least one LLM call. A straightforward task might take 8-15 iterations. A more complex one -- navigating an unfamiliar UI, handling an error state, waiting for a redirect -- might take 20-30 or more. Now do the math. If each LLM call takes 2 seconds (a fairly typical latency for a capable model at moderate load), a 15-step task takes 30 seconds just in inference time, not counting browser overhead, DOM parsing, or network latency. If the calls take 4-5 seconds each, that's over a minute. For interactive use cases, that's a dead-end. ## The Screenshot-to-Action Loop Is Latency-Sensitive Most browser agents today use one of two approaches to perceive the page: pixel-based (a screenshot sent to a vision model) or text-based (the DOM or an accessibility tree extracted and sent as text). Either way, the model has to process a large context -- either an image or a serialized tree of elements -- and return a structured action. Vision-language models add another layer of cost. Processing an image alongside a long system prompt means longer time to first token and more total compute per call. If your agent uses a multimodal model for visual grounding, each step is slower than a text-only call at the same capability level. The text-DOM approach avoids image encoding, but the accessibility tree for a complex page can be thousands of tokens. You're still dealing with large inputs and large outputs (structured JSON actions with selectors and values), which means inference latency stays high. Neither approach magically becomes fast at 2+ seconds per call. You need the underlying model to be genuinely fast. ## Why You Can't Just Cache Your Way Out Prefix caching helps in some agentic contexts. If your system prompt is long and stable, caching the KV state for that prefix can shave meaningful time off each call. Most inference providers offer some form of this. But browser agents have a structural problem: the page state changes every step. The screenshot or DOM snapshot that becomes part of the prompt is different on every turn. That's new context each time, so you don't get the same cache hit rates you'd see in, say, a chatbot with a fixed system prompt and short user turns. The part of the prompt that's unique to each step -- the current page state -- is usually the largest part. A cache hit on your system prompt saves maybe 20-30% of the tokens. The other 70-80% are re-processed fresh each turn. So caching is useful but not sufficient. You need the base inference to be fast. ## Error Recovery Multiplies the Call Count A well-designed browser agent doesn't just march forward. When it encounters an unexpected state -- a CAPTCHA, a login that failed, a page that didn't load correctly -- it needs to recognize the problem and decide what to do. That's another LLM call (or several). Error recovery is often where browser agents fall apart in practice. In a demo, you control the environment. In production, the real web is messy: popups, rate limiting, session expiration, layout changes after A/B tests, modal dialogs that weren't there yesterday. Each unexpected state can easily add 3-5 extra steps to the task. At high base latency, those extra steps push the total time from "slow but acceptable" to "completely impractical." The cruel irony is that the harder tasks -- the ones where an agent would actually save a human meaningful time -- are precisely the ones with more error states, more edge cases, and more LLM calls. ## Parallelism Helps, But Has Limits One strategy for reducing total wall-clock time is to parallelize agent sub-tasks where possible. If the agent needs to visit three pages and extract data from each, you can spawn three concurrent sub-agents instead of visiting pages in sequence. This works well for loosely-coupled tasks and can give significant speedups. But many browser tasks are inherently sequential: you have to log in before you can navigate to the account page. You have to fill out step 1 of a form before step 2 appears. The dependency graph often forces serialization. Even with parallelism, the latency of each individual call matters. Parallelism reduces wall-clock time for independent subtasks, but it doesn't help at all for the sequential core of the task. If 15 of your 20 steps must happen in order, cutting inference latency in half cuts your total task time by 37%. That's meaningful, but you can only get there if base inference is fast. ## Practical Implications for Agent Architecture Given that inference speed is often the gating constraint, here's how it shapes design decisions in practice. **Model selection matters more than you might think.** A model that's 20% less accurate but 60% faster can actually produce better task completion rates in practice, because faster recovery from mistakes is cheaper than the cost of slowness on every step. The tradeoff isn't obvious from benchmarks that measure accuracy in isolation. **Streaming helps perceived speed.** If your agent displays its reasoning or intermediate steps, streaming the output lets users see progress while the next action is still being generated. This doesn't reduce total latency, but it makes the agent feel more interactive and builds trust that it's working. **Observation design is a lever.** Sending the full accessibility tree on every step is expensive. Some frameworks use a two-phase approach: a fast "what changed?" observation (a diff from the previous step) followed by a full observation only when the page changes substantially. This reduces input token counts and can meaningfully cut per-step latency. **Rate limiting and retries have a cost.** If your inference provider rate-limits your agent mid-task, you're waiting. If you're retrying on errors, you're adding calls. Pick an inference provider with high rate limits and reliable low latency at production scale, not just at demo scale. ```python # Example: a minimal browser agent step with timing instrumentation import time from playwright.sync_api import sync_playwright def agent_step(page, llm_client, task_description, dom_snapshot): step_start = time.time() # Build prompt from current DOM state prompt = build_prompt(task_description, dom_snapshot) # Time the inference call inference_start = time.time() response = llm_client.complete(prompt) inference_ms = (time.time() - inference_start) * 1000 # Parse and execute action action = parse_action(response) execute_action(page, action) total_ms = (time.time() - step_start) * 1000 print(f"step: {total_ms:.0f}ms total, {inference_ms:.0f}ms inference ({inference_ms/total_ms*100:.0f}% of step)") return action ``` When you add this kind of instrumentation, inference time almost always dominates. Browser actions (clicking, typing, navigating) are fast. DOM extraction and screenshot capture are fast. The LLM call is where the time goes. ## What Fast Inference Actually Unlocks At sub-300ms per inference call, browser agents start to feel qualitatively different. Task completion times drop into ranges where interactive use cases become viable. A user can kick off a task, watch it proceed, and course-correct if it goes wrong -- without the interaction feeling like waiting for a batch job. At 500ms-1000ms per call, you're in the zone where background tasks work but interactive ones don't. Users won't watch the agent work; they'll walk away and check back later. This is acceptable for some use cases and a dealbreaker for others. At 2+ seconds per call, most interactive use cases fail. Background automation is possible but slow. The agent is competing against a user who would just do the task themselves. The threshold depends on what users are trying to accomplish. A browser agent running overnight to scrape data from 500 pages can tolerate higher per-call latency. A browser agent helping a user fill out an application while they're watching is competing with the user's patience directly. ## The Infrastructure Question Running a fast browser agent in production isn't just a model selection question. You need inference infrastructure that can sustain low latency under concurrent load. A model that returns 200ms per call when one agent is running might return 1500ms when 50 are running simultaneously, if the underlying infrastructure isn't built for it. This is where managed inference APIs differ from self-hosted models. Self-hosting gives you control and potentially low latency on a warm, lightly-loaded system -- but capacity planning for variable agent workloads is genuinely difficult, and cold-start latency can be brutal if you're spinning up inference capacity on demand. Managed APIs with consistent, guaranteed latency SLAs make it easier to reason about agent performance in production. You can design around a known p95 latency budget instead of a best-case number that degrades unpredictably at scale. ## Building Toward Practical Browser Automation Browser agents are increasingly real. The tooling has matured (Playwright, Puppeteer, browser-use, Stagehand), the vision models have improved, and the prompting strategies are better understood. The remaining friction is mostly infrastructure: can you run enough inference, fast enough, cheaply enough, to make the economics work? The agents that will succeed in production are the ones built with latency as a first-class constraint from the start -- tight observation design, smart caching, good model selection, and inference infrastructure that doesn't become the bottleneck at scale. If you're building browser agents and running into latency walls, [General Compute's API](https://generalcompute.com) is worth benchmarking. Consistent sub-200ms TTFT at scale is what makes the step-by-step loop actually work. --- # Voice Agents and the 500ms Window: An Inference Architecture Guide > Real-time voice conversations require end-to-end latency under 500ms. This guide breaks down the inference architecture, latency budgets, and streaming patterns needed to hit that target reliably. - Author: General Compute - Published: 2026-07-15 - Tags: voice ai, inference, latency, agents, real-time, architecture - Canonical: https://www.generalcompute.com/blog/voice-agents-and-the-500ms-window --- Building a voice agent that feels natural in conversation is a harder engineering problem than it looks. The gap between a demo that works and a product that people actually want to use comes down to one number: 500 milliseconds. That number is not arbitrary. Human conversation turn-taking research (Levinson, 2016; Stivers et al., 2009) consistently finds that response delays above 500ms start to feel awkward, and anything above 1000ms breaks the social contract of conversation. People start to wonder if they were heard, if the call dropped, or if they should speak again. These patterns are consistent across languages and cultures. A voice agent that misses the 500ms window will feel wrong to users even if they can't articulate why. This post is a technical walkthrough of the entire inference architecture required to hit that target reliably. The short version: the LLM is almost always the bottleneck, streaming is mandatory, and model selection matters more than most teams realize. ## The Voice Pipeline A voice agent has three distinct processing stages between the user finishing a sentence and audio playing back: 1. **Automatic speech recognition (ASR)** -- converts the user's audio to text 2. **LLM inference** -- generates the response text 3. **Text-to-speech (TTS)** -- converts the response text to audio Each stage has its own latency profile, and they chain together. The total round-trip time is roughly: ``` total_latency = asr_latency + network_hops + llm_ttft + tts_first_chunk ``` Notice "llm_ttft" specifically, not total LLM generation time. This is important, and we will return to it. ### Realistic Latency Budgets Here is what each stage actually costs in a well-optimized production system: | Stage | Typical Range | Notes | |---|---|---| | ASR (cloud) | 80--200ms | Real-time streaming ASR can be faster | | ASR (local/edge) | 30--80ms | Faster-Whisper on GPU | | Network (ASR + LLM) | 20--60ms | Depends on geographic co-location | | LLM time-to-first-token | 80--400ms | The main variable | | TTS first audio chunk | 60--120ms | Streaming TTS systems | Add these up at the optimistic end: 30 + 20 + 80 + 60 = 190ms. That leaves comfortable headroom under 500ms. At the pessimistic end: 200 + 60 + 400 + 120 = 780ms. That is well over the threshold, and users will notice. The LLM TTFT range (80--400ms) is the largest source of variance. Everything else in the pipeline is relatively predictable once you pick your ASR and TTS providers. LLM inference latency is where architecture decisions have the most impact. ## Why TTFT Is the Metric That Matters Most discussions of LLM performance focus on tokens per second (throughput). For voice, that is the wrong metric. What matters is how quickly the model produces its first output token -- time-to-first-token -- because that is what determines when TTS can begin generating audio. Once you have the first token, you can start TTS and pipeline the rest. The user does not need to wait for the full LLM response before hearing anything. But until that first token arrives, everything is blocked. A model generating 100 tokens/second with a 400ms TTFT will sound worse in a voice application than a model generating 60 tokens/second with an 80ms TTFT. The throughput difference barely matters because most voice responses are short (50--150 tokens), and the latency difference means 320ms less dead air at the start. This is why inference provider selection matters so much for voice specifically. A provider optimized for throughput (maximizing aggregate tokens/second across all users) may batch requests in ways that add 100--200ms to individual TTFT. A provider optimized for TTFT will process your request immediately rather than waiting for a batch to fill. ## Streaming: The Architecture That Makes It Work Waiting for the full LLM response before starting TTS is a mistake that adds hundreds of milliseconds to every turn. The correct architecture pipelines LLM output directly into TTS in real time. Here is the basic pattern: ```python async def voice_response(user_text: str): llm_stream = await llm_client.chat.completions.create( model="qwen3-coder-7b", messages=[{"role": "user", "content": user_text}], stream=True, ) tts_client = TTSClient() text_buffer = "" async for chunk in llm_stream: token = chunk.choices[0].delta.content or "" text_buffer += token # Send to TTS when we have a natural sentence boundary if ends_with_sentence_boundary(text_buffer): await tts_client.synthesize_stream(text_buffer) text_buffer = "" # Flush remaining text if text_buffer: await tts_client.synthesize_stream(text_buffer) ``` The key is chunking at sentence boundaries. Sending individual tokens to TTS produces choppy, unnatural audio. Sending complete sentences produces natural prosody. The trade-off is that you wait slightly longer per chunk, but users hear audio within 2--3 sentences of the LLM starting to generate. In practice, many TTS systems expose their own streaming API that accepts a text stream and manages chunking internally. Systems like ElevenLabs, Deepgram Aura, and Cartesia handle this buffering for you. If you are building on Faster-Whisper or a self-hosted TTS system, you need to manage sentence boundaries yourself. ### The Sentence Boundary Problem Detecting sentence boundaries in streaming text is trickier than it sounds. Simple period detection fails on abbreviations ("Dr.", "U.S.", "3.5"), mid-sentence pauses, and bullet points. A practical approach: ```python import re SENTENCE_END = re.compile(r'(?<=[.!?])\s+(?=[A-Z])|(?<=[.!?])$') def ends_with_sentence_boundary(text: str) -> bool: # Minimum length to avoid flushing on short abbreviations if len(text) < 20: return False return bool(SENTENCE_END.search(text.strip())) ``` This is imperfect. For production, consider a lightweight ML-based sentence splitter, or tune heuristics to your specific domain (technical support conversations have different patterns than casual chat). ## Model Selection for Voice Most voice conversations do not require a 70B parameter model. Users ask questions, make requests, and have short exchanges. A well-prompted 7B--14B model handles the vast majority of voice interactions with acceptable quality, and it does so with significantly lower TTFT. The performance difference is real. On identical hardware, a 7B model might return a first token in 60--100ms. A 70B model on the same hardware might take 200--350ms. That is 200ms of difference applied to every single turn in every conversation. The right model selection process for voice: 1. Define your domain and the hardest 10% of queries you expect 2. Test a 7B and 14B model against those queries with your actual system prompt 3. Only go larger if quality genuinely requires it for that query distribution For many customer support, appointment scheduling, and information retrieval use cases, smaller models are sufficient. For complex reasoning or code generation over voice, you may need larger models and will need to accept higher latency or invest in faster infrastructure. ### Speculative Decoding If you need a larger model but want better TTFT, speculative decoding can help. A small draft model (1B--3B parameters) generates candidate tokens that a larger verifier model checks in parallel. Accepted tokens are kept; rejected tokens are regenerated. The result is 2--3x throughput improvement with the same output quality as the large model. The TTFT benefit for speculative decoding is more modest (it reduces time per generated token, not prefill time), but the reduced time-to-nth-token means audio starts playing sooner in practice. ## Infrastructure Patterns ### Co-location Network latency is often overlooked. If your ASR provider is on the East Coast, your LLM inference is in the Midwest, and your TTS is on the West Coast, you are adding 40--80ms of pure network latency per hop. For a 500ms budget, that is significant. Ideal setup: ASR, LLM inference, and TTS are in the same cloud region, or at least the same geographic area. For global products, consider regional deployments with traffic routing based on user location. ### Cold Starts Serverless GPU inference introduces cold start latency that can be 3--10 seconds for model loading. For voice, that means the first request to a cold worker will fail the 500ms target by a wide margin. This is not usually a problem for high-traffic applications (workers stay warm), but it is a problem for low-traffic applications or during traffic spikes. Options: - Keep minimum instances running at all times (adds cost, eliminates cold starts) - Use a dedicated inference provider with always-warm models - Build a fallback path that gracefully handles the occasional slow response ### Latency Percentiles p50 latency (median) looks great in benchmarks. Voice applications live and die by p95 and p99. If 5% of requests take 1200ms, users will notice those pauses. Monitor all percentiles, not just averages. A good inference provider should offer: - p50 TTFT: under 100ms - p95 TTFT: under 250ms - p99 TTFT: under 400ms If a provider cannot give you these numbers, test them yourself with a production-like load pattern. ## Practical Architecture Checklist Before deploying a voice agent, verify each of these: **ASR** - [ ] Using streaming ASR rather than batch mode where possible - [ ] Endpoint detection tuned to your use case (not too aggressive, not too loose) - [ ] ASR and LLM in the same region **LLM Inference** - [ ] Using TTFT-optimized inference (not throughput-maximized) - [ ] Model size right-sized for quality requirements (7B first, go larger only if needed) - [ ] Streaming enabled - [ ] System prompt is fixed and short (longer prompts increase prefill time) **TTS** - [ ] Streaming TTS, not batch synthesis - [ ] Sentence-boundary chunking implemented - [ ] TTS latency profiled under realistic load **Infrastructure** - [ ] p95/p99 TTFT benchmarked, not just p50 - [ ] Cold start strategy defined - [ ] Geographic co-location verified ## Measuring What Matters Set up instrumentation that tracks the full round-trip from end of user speech to start of agent audio. Many teams track individual component latencies but miss the end-to-end number. A minimal instrumentation setup: ```python import time class VoiceTurnMetrics: def __init__(self): self.user_speech_end = None self.asr_complete = None self.llm_first_token = None self.tts_first_audio = None def record(self, event: str): setattr(self, event, time.monotonic()) def to_dict(self): if not all([self.user_speech_end, self.tts_first_audio]): return {} return { "asr_latency": self.asr_complete - self.user_speech_end, "llm_ttft": self.llm_first_token - self.asr_complete, "tts_first_chunk": self.tts_first_audio - self.llm_first_token, "total_round_trip": self.tts_first_audio - self.user_speech_end, } ``` Log these per turn, aggregate by percentile, and set alerts when p95 total round-trip exceeds 450ms. That gives you 50ms of headroom before users start noticing. ## The TTFT Problem in Practice Here is what this looks like with real numbers from a production voice agent deployment. Assume: - ASR (streaming Deepgram): ~90ms - Network: ~25ms (same region) - TTS first chunk (Cartesia): ~70ms That accounts for 185ms. The remaining 315ms is the budget for LLM TTFT to hit a 500ms total. If the inference provider averages 200ms TTFT, you are fine at p50. But if p95 TTFT is 400ms, 5% of turns will take ~685ms -- 200ms over threshold. Switching to an inference provider with p95 TTFT of 200ms drops that 5% of turns to ~385ms, which is comfortably under threshold. The difference is entirely in the inference layer. ## Where to Go From Here If you are building voice agents, test your inference provider's TTFT under realistic load early in development, not at launch. The difference between providers is large enough to determine whether your product feels natural or frustrating. General Compute's inference API is designed for exactly this use case -- low TTFT at p95 and p99, not just p50. The same API endpoint works for both voice and text applications, so the migration path from an existing OpenAI-compatible integration is a single base URL change. Start with a 7B or 14B model and a streaming pipeline. Measure end-to-end round-trip latency at p50, p95, and p99. If you are hitting the 500ms target at p95, you are in good shape. If not, the bottleneck is almost always LLM TTFT -- and that is a solvable infrastructure problem. --- # Planning and Search with LLMs: Tree-of-Thought at Inference Scale > Tree-of-Thought turns a single LLM call into a branching search process. Here's how it works, why inference speed determines how deep you can search, and how to build ToT agents that run fast enough to be useful. - Author: General Compute - Published: 2026-07-14 - Tags: agents, inference, planning, tree-of-thought, reasoning - Canonical: https://www.generalcompute.com/blog/planning-and-search-with-llms-tree-of-thought-at-inference-scale --- Most LLM reasoning follows a straight line: you give the model a prompt, it produces a chain of thought, and it lands on an answer. This works well enough for many tasks, but planning problems are different. Planning requires the model to consider multiple possible futures, evaluate them, and commit to the best path. A single linear generation does not naturally support backtracking or comparison across branches. Tree-of-Thought (ToT) is an approach that structures LLM reasoning as a search process over a tree of partial solutions. Each node in the tree is a reasoning step or candidate plan. The model generates multiple continuations from each node, evaluates them, and expands the most promising ones. The result is more like a minimax search than a forward pass. The catch is that this approach multiplies your inference calls. A tree with branching factor `b` and depth `d` requires up to `b^d` generation steps before you apply any pruning, plus additional calls to score each node. For any agent where planning actually matters, inference speed is not a footnote -- it is the constraint that determines whether your search budget is useful or negligible. ## How Tree-of-Thought Works The original [Tree of Thoughts paper](https://arxiv.org/abs/2305.10601) from Princeton and Google DeepMind describes three components: 1. **Thought decomposition**: break the problem into a sequence of intermediate steps, where each step is small enough to evaluate independently. 2. **Thought generation**: at each node, generate multiple candidate next steps (either by sampling the same prompt k times, or by asking the model to propose k options in a single call). 3. **State evaluation**: score each candidate node -- either with a value heuristic ("rate this partial solution from 1-10") or by asking the model to vote across candidates. Search strategy sits on top of these components. Breadth-first search (BFS) expands all nodes at the current depth before going deeper. Depth-first search (DFS) commits to one branch and backtracks when it hits a dead end. Beam search keeps only the top-k nodes at each level. Here is the basic structure in Python: ```python from typing import Callable from dataclasses import dataclass, field @dataclass class ThoughtNode: state: str score: float = 0.0 children: list = field(default_factory=list) depth: int = 0 def tree_of_thought_bfs( problem: str, generate_fn: Callable[[str], list[str]], evaluate_fn: Callable[[str, str], float], max_depth: int = 3, beam_width: int = 3, ) -> ThoughtNode: root = ThoughtNode(state=problem) beam = [root] for depth in range(max_depth): candidates = [] for node in beam: thoughts = generate_fn(node.state) for thought in thoughts: child = ThoughtNode( state=thought, depth=depth + 1, ) child.score = evaluate_fn(problem, thought) node.children.append(child) candidates.append(child) # Keep the top beam_width nodes for the next level candidates.sort(key=lambda n: n.score, reverse=True) beam = candidates[:beam_width] return max(beam, key=lambda n: n.score) ``` The `generate_fn` and `evaluate_fn` are both LLM calls. With `beam_width=3` and `max_depth=3`, that is at minimum 9 generation calls and 9 scoring calls -- 18 calls total for a modest search tree. Deeper trees or higher branching factors scale this up quickly. ## The Inference Math To make ToT practical, you need to think carefully about how inference latency compounds. If each LLM call takes 2 seconds, a tree with depth 4 and beam width 3 requires roughly: - Generation: 3 × 4 = 12 calls × 2s = 24 seconds - Evaluation: 12 calls × 2s = 24 seconds - Total (sequential): ~48 seconds Most of these calls are independent and can run in parallel. The generation calls at each depth level can fire simultaneously once you have the parent nodes. The scoring calls for a given set of candidates can also run in parallel. With full parallelism, the wall-clock time collapses to roughly `max_depth × (generation_latency + evaluation_latency)` -- in this case, 4 × (2s + 2s) = 16 seconds. But at 500ms per call (achievable with fast inference), the same tree takes 4 × (0.5s + 0.5s) = 4 seconds. That difference matters a lot in interactive applications, and it determines how much search you can afford before a user gives up waiting. Here is a parallel version of the inner loop using `asyncio`: ```python import asyncio async def tree_of_thought_bfs_async( problem: str, generate_fn, # async callable evaluate_fn, # async callable max_depth: int = 3, beam_width: int = 3, ) -> ThoughtNode: root = ThoughtNode(state=problem) beam = [root] for depth in range(max_depth): # Generate candidates for all beam nodes in parallel generation_tasks = [generate_fn(node.state) for node in beam] all_thoughts_per_node = await asyncio.gather(*generation_tasks) # Flatten and score all candidates in parallel candidate_pairs = [ (node, thought) for node, thoughts in zip(beam, all_thoughts_per_node) for thought in thoughts ] score_tasks = [ evaluate_fn(problem, thought) for _, thought in candidate_pairs ] scores = await asyncio.gather(*score_tasks) # Build candidate nodes candidates = [] for (node, thought), score in zip(candidate_pairs, scores): child = ThoughtNode(state=thought, depth=depth + 1, score=score) node.children.append(child) candidates.append(child) candidates.sort(key=lambda n: n.score, reverse=True) beam = candidates[:beam_width] return max(beam, key=lambda n: n.score) ``` Even with full parallelism, each depth level still has a sequential dependency: you cannot generate depth-2 nodes until you have scored and selected the depth-1 beam. This is why per-call latency (time-to-first-token plus generation time) still matters, even when you parallelize aggressively within a level. ## Where ToT Beats Chain-of-Thought Tree-of-Thought is not universally better than chain-of-thought. It adds overhead and complexity, so you want to use it where the planning structure actually helps. **Combinatorial tasks with clear intermediate states.** The original paper uses the Game of 24 (combine four numbers with arithmetic to reach 24) and crossword puzzle solving. These problems have well-defined intermediate states and a clear evaluation function. **Code planning before writing.** Before generating code for a complex function, you can run a shallow ToT search over possible approaches: generate 3 high-level plans, score them on feasibility and simplicity, then generate the implementation from the best one. This is a 2-level tree with a branching factor of 3, totaling about 6 calls. Fast inference makes this feasible to do inline. **Multi-step reasoning with backtracking.** Tasks like "write a proof sketch for X" or "design a schema that satisfies constraints A, B, and C" benefit from being able to try partial solutions and abandon dead ends. Linear chain-of-thought cannot backtrack; it just keeps generating forward even when earlier reasoning was wrong. **Agent task decomposition.** When an agent receives a high-level task, it can use a ToT step to generate multiple decompositions, score each on estimated complexity and risk, and pick the one that looks most tractable before starting execution. ## Practical ToT with an OpenAI-Compatible API Here is a self-contained example using the OpenAI SDK against a fast inference endpoint. The task is planning a sequence of steps to refactor a codebase module. ```python import asyncio from openai import AsyncOpenAI client = AsyncOpenAI( base_url="https://api.generalcompute.com/v1", api_key="your-api-key", ) MODEL = "llama-4-maverick" async def generate_thoughts(state: str, n: int = 3) -> list[str]: """Generate n candidate next steps from the current plan state.""" response = await client.chat.completions.create( model=MODEL, messages=[ { "role": "system", "content": ( "You are a software architect. Given a partial refactoring plan, " "propose the next concrete step. Be specific and actionable." ), }, {"role": "user", "content": f"Current plan:\n{state}\n\nPropose the next step."}, ], n=n, temperature=0.8, max_tokens=200, ) return [choice.message.content for choice in response.choices] async def score_thought(problem: str, state: str) -> float: """Score a candidate plan state from 0.0 to 1.0.""" response = await client.chat.completions.create( model=MODEL, messages=[ { "role": "system", "content": ( "Rate the quality of this partial refactoring plan from 0 to 10. " "Consider correctness, safety, and completeness. " "Reply with a single integer and nothing else." ), }, { "role": "user", "content": f"Original task: {problem}\n\nPlan so far:\n{state}", }, ], max_tokens=5, temperature=0, ) try: score = int(response.choices[0].message.content.strip()) return max(0.0, min(10.0, score)) / 10.0 except ValueError: return 0.5 async def plan_with_tot(task: str, depth: int = 3, beam: int = 3) -> str: current_state = f"Task: {task}\n\nPlan steps:" beam_nodes = [current_state] for _ in range(depth): # Generate candidates in parallel gen_tasks = [generate_thoughts(s, n=beam) for s in beam_nodes] all_thoughts = await asyncio.gather(*gen_tasks) candidates = [ f"{parent}\n- {thought}" for parent, thoughts in zip(beam_nodes, all_thoughts) for thought in thoughts ] # Score all candidates in parallel score_tasks = [score_thought(task, c) for c in candidates] scores = await asyncio.gather(*score_tasks) ranked = sorted(zip(scores, candidates), reverse=True) beam_nodes = [plan for _, plan in ranked[:beam]] return beam_nodes[0] if __name__ == "__main__": task = ( "Refactor the authentication module to use JWT tokens instead of " "session cookies, without breaking existing user sessions." ) result = asyncio.run(plan_with_tot(task)) print(result) ``` A few notes on this implementation: - Passing `n=3` to the completions API generates three candidates in a single request, which is more efficient than three separate calls when the model and provider support it. - The scoring prompt asks for a single integer, which keeps output tokens low and evaluation fast. - The `beam` parameter controls the breadth-versus-depth trade-off. A beam of 1 is effectively greedy best-first search. A beam of 5 with depth 4 produces a wider, slower search. ## Controlling the Search Budget In production, you usually cannot give ToT an unlimited compute budget. There are a few knobs worth building into your implementation: **Time budget.** Set a wall-clock deadline. If the search is not complete by a threshold, return the best node found so far. This requires running the evaluation scores as they complete rather than waiting for the full gather. **Token budget.** Track total tokens across all calls in the tree. Once you hit a cap, stop expanding and return the current best leaf. **Early termination.** If a node's score exceeds a threshold (say, 0.9 out of 1.0), treat it as good enough and stop expanding siblings. This is most useful when your scoring function is reliable. **Depth-adaptive beam width.** Start with a wider beam at shallow depths (exploring broadly) and narrow it as you go deeper (committing to the best path). This is closer to how humans approach planning: consider many options early, then commit. ## The Inference Bottleneck The reason ToT is underused in production is not that it lacks theoretical merit -- it is that most inference APIs are too slow to make the approach feel responsive. If each call takes 2 to 3 seconds, even a modest 3-level tree with a beam of 3 takes 18 to 27 seconds total in the sequential case. Users will not wait for that. With sub-500ms inference, the same tree completes in 3 to 5 seconds end-to-end when parallelized properly. That is a usable latency budget for a coding agent or research assistant where the user expects to wait a moment for a plan before execution begins. The faster your inference, the deeper you can search within any given time budget, and the better your planning agent's decisions will be. A 10x speedup in per-call latency translates to roughly 10x more search depth (or equivalently, a much larger beam at the same depth) within the same wall-clock constraint. ## What This Means for Agent Design If you are building agents that need to make decisions -- not just respond to prompts -- the inference speed of your provider directly shapes how sophisticated your planning can be. A coding agent that plans with a 2-level ToT before writing code will produce better-structured implementations than one that jumps straight to generation. A research agent that explores multiple lines of inquiry in parallel will surface better answers than one following a single chain of thought. But these agents only become practical when each LLM call is fast enough to keep the total latency within what users will tolerate. Start simple: a 2-level tree with branching factor 3 and full parallelism requires only 6 LLM calls. With fast inference, that is a few seconds of wall clock time and already meaningfully better than greedy generation for complex planning tasks. Once you have that baseline working, you can tune depth, beam width, and evaluation prompts to fit your specific problem. --- If you want to try ToT on a fast inference endpoint, [General Compute's API](https://generalcompute.com) is OpenAI-compatible and designed for the kind of parallel, high-throughput workloads that planning agents require. Check out the [docs](https://generalcompute.com/docs) to get started with your API key. --- # The Cost of Agent Retries: When Slow Inference Kills Reliability > Retries are how agents recover from failures, but when inference is slow, each retry multiplies latency. Here's how to think about retry budgets and why faster inference makes your agent more resilient. - Author: General Compute - Published: 2026-07-13 - Tags: agents, inference, latency, reliability, retries - Canonical: https://www.generalcompute.com/blog/the-cost-of-agent-retries-when-slow-inference-kills-reliability --- Every agent fails sometimes. A model returns malformed JSON. A tool call targets a function that doesn't exist. A confidence check rejects the output. Retries are how agents handle these failures without crashing. But retries have a cost, and when inference is slow, that cost adds up fast enough to make your agent functionally unusable. This post is about how to think about retry budgets in agentic systems, why slow inference forces you into bad tradeoffs, and how inference speed changes what retry strategies are actually viable. ## Why agents retry An agent step can fail in several distinct ways, and each one calls for a different retry strategy. **Structural failures** are the most obvious. The model produces JSON that doesn't parse, or calls a tool with the wrong argument types. These are deterministic failures: the output is not usable as-is, and the agent needs to try again. Usually a retry with a richer error message ("you called `search_web` with an integer, but the query parameter must be a string") will fix the problem in one pass. **Tool execution failures** happen when the model's call is structurally valid but the tool itself fails. The web search times out. The database throws a connection error. The code sandbox returns a non-zero exit code. These are often transient. A retry with the same call will succeed after the transient clears, or fail again for a reason worth escalating. **Validation failures** are subtler. The model returns output that is valid JSON and passes schema validation, but a downstream check rejects it as wrong. A fact-checker finds a claim that contradicts the source documents. A test runner shows the generated code throws an exception. The agent needs to retry, but this time with the validation error folded back into the context so the model knows what it got wrong. **Confidence-based retries** are a pattern some orchestrators use: generate the same step multiple times (or generate a score alongside the output), and retry if confidence falls below a threshold. This is more expensive than deterministic retries but is useful when the error signal is not explicit -- when the output could be wrong without the agent noticing at the structural level. Each of these failure modes has different expected retry counts. Structural and transient failures usually resolve in one or two retries. Validation failures may need two or three. Confidence-based retries, by their nature, generate multiple candidates before selecting. In a multi-step agent with a pipeline of ten steps, the cumulative retry load is not trivial. ## The math of retries at slow inference speeds Suppose each of your agent's steps succeeds 90% of the time on the first attempt. That sounds reasonable. But in a five-step pipeline, the probability that all five steps succeed on the first try is 0.9^5 = 59%. Without retries, your agent fails four times out of ten. One retry per failed step changes the math. If each step has a 90% first-attempt success rate and you allow one retry, the per-step success rate becomes 1 - (0.1 * 0.1) = 99%. Five steps in series at 99% per step gives roughly 95% end-to-end success. That is viable. But now add timing. If each inference call takes 5 seconds, and a single step fails once before succeeding on retry: - First attempt: 5 seconds, fails - Retry: 5 seconds, succeeds - Total for this step: 10 seconds With five steps in a pipeline, and assuming roughly one failure across the pipeline (consistent with a 90% per-step success rate), the expected total latency becomes approximately 27.5 seconds: five steps at 5 seconds each for first attempts, plus one 5-second retry. With bad luck -- multiple steps failing -- you could easily be looking at 40 to 50 seconds. If inference takes 500 ms instead of 5 seconds, the same pipeline runs in under 3 seconds including the expected retry. That is not a marginal difference; it is the gap between a feature and a usable product. ## Retry budgets: capping latency instead of counting attempts Most retry logic is written as a maximum attempt count: "retry up to 3 times, then give up." This is simple to implement but is the wrong abstraction for agentic systems. The problem is that maximum attempt count does not compose well across steps. If each of ten steps retries up to 3 times, and your inference calls take 4 seconds each, your worst-case latency is 10 * 3 * 4 = 120 seconds. That failure mode will not show up in development; it takes load testing with realistic failure injection to observe. A better abstraction is a **retry budget**: a time limit for the entire agent run, not per step. Each attempt consumes from the budget, and when the budget is exhausted, the agent fails fast rather than continuing to retry. ```python import time class RetryBudget: def __init__(self, total_seconds: float): self.deadline = time.monotonic() + total_seconds def remaining(self) -> float: return max(0.0, self.deadline - time.monotonic()) def is_exhausted(self) -> bool: return time.monotonic() >= self.deadline def check(self): if self.is_exhausted(): raise TimeoutError("agent retry budget exhausted") ``` A step that respects the budget looks like: ```python MINIMUM_RETRY_TIME = 1.0 # don't attempt a retry if less than 1s remains def run_step_with_budget(step_fn, budget: RetryBudget, max_attempts: int = 3): last_error = None for attempt in range(max_attempts): budget.check() # fail fast if time is up try: return step_fn() except RecoverableError as e: last_error = e if budget.remaining() < MINIMUM_RETRY_TIME: break # not enough time to retry meaningfully raise last_error ``` The budget is shared across all steps in the pipeline. A step that consumes 15 seconds of retries leaves less time for subsequent steps to retry. This creates backpressure: a pipeline with one very flaky step naturally gets fewer retries later, rather than compounding latency without bound. This pattern also makes it easier to reason about SLAs. If your product promises a 30-second response time, set the budget to 25 seconds and you have a 5-second margin for overhead. ## How inference speed changes the retry math The budget framing makes the connection between inference speed and retry strategy explicit. With a 30-second budget and 5-second inference: - You can fit at most 6 inference calls total - In a 5-step pipeline, that means one retry across all steps - Any step that fails twice exhausts the retry allocation for the whole pipeline With a 30-second budget and 500 ms inference: - You can fit up to 60 inference calls - In a 5-step pipeline, each step could retry 11 times before hitting the budget - You can afford more aggressive confidence thresholds and more liberal retry policies The improvement is not just that each call is faster. The slack you gain from faster inference directly funds additional retries, which funds higher reliability. When inference is fast enough, retries become nearly free and you can design your agent to retry aggressively without worrying about the latency cost. When inference is slow, every retry is a real cost you are passing on to the user. ## Failure modes that slow inference amplifies Beyond raw retry counts, there are a few specific failure patterns that slow inference makes worse. **Cascading timeouts.** Many agent steps call external tools with their own timeouts. If your inference call for a step takes 8 seconds, and the downstream tool has a 10-second timeout, you have 2 seconds of headroom for the tool to execute. A tool that normally takes 3 seconds will time out, triggering a retry. With faster inference, that same tool has 9.5 seconds of headroom and completes without incident. **Context length growth during retries.** Each retry typically appends the failed attempt's output and an error message to the context before retrying. After two or three retries, the context has grown substantially. Longer contexts take longer to prefill on the next attempt, which slows inference further, which makes the next retry take longer still. This compounding effect is easy to miss in testing (where you rarely hit three consecutive failures on the same step) but shows up in production under load. **User abandonment.** This is the non-technical failure mode. An agent running under a budget-aware retry strategy will still sometimes run slowly, especially under load. Users who see no progress for 15 to 20 seconds tend to close the tab or restart the session. This creates a category of failures that your retry logic never gets to handle because the request is already gone. Faster inference reduces the base latency, which gives retries more room to happen before the user gives up. ## Designing retry logic for a multi-step agent Given all of this, here are the practical choices that matter most when building retry logic for a production agent. **Distinguish retryable from non-retryable errors.** Tool execution failures are usually retryable. Structural failures in the model's output are retryable with error feedback. Permission errors and missing tools are not retryable and should escalate immediately. Burning two retries on a non-retryable error is pure waste. **Pass error context on retry.** A retry that sends the same prompt as the original attempt will often fail the same way. The retry should include, at minimum, the error that caused the failure and a corrective instruction. For validation failures, include the specific assertion that failed. For tool argument errors, include the expected schema. ```python def run_with_error_context(agent_step, context: list, error_feedback_template: str): for attempt in range(3): result = call_model(context) error = validate(result) if not error: return result # fold the error back into context for the next attempt context = context + [ {"role": "assistant", "content": result.raw_output}, {"role": "user", "content": error_feedback_template.format(error=error)}, ] raise ValueError("step failed after 3 attempts") ``` **Set a deadline at the top of the pipeline, not at each step.** Per-step max attempt counts compose badly. A shared budget propagates the constraint correctly and makes the worst-case latency predictable and auditable. **Track retry counts in observability.** The average number of retries per step, and the distribution across percentiles, will tell you which steps are flaky and need attention. A step that retries 40% of the time is consuming nearly twice as much inference budget as a step with a 2% retry rate. Log it, alert on it, fix the underlying issue. **Invest in faster inference before investing in more complex retry logic.** Exponential backoff, jitter, circuit breakers -- these add real engineering cost and operational complexity. Faster inference solves much of the same problem with less code, because the retry budget runs out less often when each attempt is cheaper. ## Measuring the impact If you are not sure how much retries are costing your agent, the measurement is straightforward. Add a counter to each agent step that tracks the number of attempts before success (or final failure). Log it alongside per-step latency. Then look at the distribution. If your median step runs in one attempt but your 95th percentile runs in three, retries are adding significant latency to a small fraction of requests. Those are the requests your users notice most. Reducing per-attempt latency (by using faster inference) compresses the 95th percentile faster than anything you can do in retry logic alone. If retries are common -- say, 20% or more of steps need at least one retry -- you have a reliability issue that retry logic alone will not fix. You need to either improve the model's accuracy on the task, improve the quality of error feedback on retry, or reduce the complexity of the step so it is less likely to fail in the first place. Retries are the recovery mechanism, not the primary reliability mechanism. --- Agent reliability and inference speed are more closely coupled than they appear from the outside. The model accuracy numbers are the same regardless of how fast inference runs. But the retry budget your architecture can afford, the confidence thresholds you can set, and the failure modes you can absorb -- all of these scale with how much inference you can fit into the time you have. General Compute serves open models at very high tokens-per-second on an OpenAI-compatible API. If your agent is spending a significant fraction of its wall time on retries, the fastest way to test what changes is to run the same agent against a faster backend and measure the retry distribution directly. The API is at [generalcompute.com](https://generalcompute.com). --- # Whisper API for Real-Time Transcription: Alternatives to OpenAI Whisper > The OpenAI Whisper API works well for batch transcription but is not designed for real-time streaming. Here is how Faster-Whisper, Deepgram, and AssemblyAI compare on latency, accuracy, and pricing for live transcription use cases. - Author: General Compute - Published: 2026-07-12 - Tags: whisper, speech-to-text, real-time transcription, deepgram, assemblyai, voice-ai - Canonical: https://www.generalcompute.com/blog/whisper-api-for-real-time-transcription-alternatives-to-openai-whisper --- OpenAI's Whisper is the model most teams reach for when they first need speech-to-text. The weights are open, accuracy is strong across languages and accents, and the managed API requires no infrastructure. For transcribing a recording, it is a reasonable starting point. For real-time transcription of a live call or voice agent, it is the wrong tool, and the teams that discover this late spend a lot of time working around a constraint that a different choice would have avoided entirely. This post covers what the OpenAI Whisper API actually provides, where it stops working for live use cases, and how Faster-Whisper, Deepgram, and AssemblyAI compare as alternatives. The comparison covers accuracy, latency, pricing, and which use cases each one fits best. ## What the OpenAI Whisper API Provides The `whisper-1` model is available through OpenAI's audio transcriptions endpoint. You send a complete audio file, and you get back a transcript. The interface is simple: ```python from openai import OpenAI client = OpenAI(api_key="your_openai_key") with open("recording.mp3", "rb") as f: transcript = client.audio.transcriptions.create( model="whisper-1", file=f, response_format="text" ) print(transcript) ``` This works well for batch use cases: uploading a recorded meeting, transcribing a podcast episode, processing a support call after it ends. You pay per minute of audio (currently $0.006 per minute), and you get a complete transcript back after a few seconds of processing time. The constraints show up the moment you try to use it for anything interactive. **No streaming support.** The API requires a complete audio file as input. You cannot open a WebSocket connection and feed chunks of live audio as someone is speaking. You have to wait for the utterance to complete, upload it, wait for the API to respond, and then process the result. That round trip is tolerable for short clips but makes it impossible to show partial transcripts while someone is still talking. **Variable latency.** Transcription time through the managed API is not as consistent as a locally-hosted model. Under load, API response times can vary from a couple of seconds to notably longer, which is hard to plan around when you need predictable latency. **File size and format limits.** The API accepts files up to 25 MB. For long recordings this requires chunking, which introduces its own stitching logic. **Rate limits.** At higher volumes, the API rate limits require requesting increases. For a product that can spike unexpectedly, the ceiling is an operational risk. None of these are criticisms of the model itself, which is genuinely accurate. The managed API is designed for file-based transcription. If that is your use case, it is a fine choice. If you need real-time transcription, you need something else. ## Option 1: Faster-Whisper (Self-Hosted) Faster-Whisper is a reimplementation of the same Whisper model weights on top of CTranslate2, a C++ inference library. Running the same large-v3 model, it is typically four times faster than the reference PyTorch implementation, and it supports INT8 quantization to reduce memory usage further. The key difference from the managed API is that you host it yourself, which means you control the model, the hardware, and the concurrency. You can build a streaming interface on top of it by running inference on rolling audio buffers: ```python from faster_whisper import WhisperModel import numpy as np model = WhisperModel("large-v3", device="cuda", compute_type="int8_float16") buffer = np.array([], dtype=np.float32) for chunk in audio_stream: buffer = np.concatenate([buffer, chunk]) segments, _ = model.transcribe(buffer, beam_size=1, language="en") partial = " ".join(s.text for s in segments) emit_partial(partial) if utterance_ended(buffer): emit_final(partial) buffer = np.array([], dtype=np.float32) ``` Running `beam_size=1` uses greedy decoding instead of beam search, which cuts latency meaningfully in exchange for a small accuracy dip that is usually not noticeable in real-time use. **Where it makes sense:** teams that need to process high volumes of audio, want control over the model and infrastructure, or have data residency requirements that rule out sending audio to a third-party API. The operational overhead is real -- you're managing GPU instances, handling failures, and monitoring model performance -- but the per-minute cost at scale is much lower than any managed API. **Where it doesn't:** teams that want a fully managed service with no infrastructure to operate. Faster-Whisper is a library, not an API. You have to build the serving layer yourself. For accuracy, it matches the OpenAI API exactly because it uses the same weights. Word error rate on standard benchmarks (LibriSpeech clean, CommonVoice) is the same between the two implementations. ## Option 2: Deepgram Deepgram is a purpose-built speech-to-text API company with models trained specifically for transcription rather than adapted from a general-purpose audio model. Their current production model, Nova-2, is the default for most use cases. The key capability for real-time use is their streaming API, which uses WebSocket connections to accept audio chunks and return transcripts with very low latency. Partial transcripts arrive as the person is speaking, and final transcripts are emitted at speech boundaries: ```python import asyncio from deepgram import DeepgramClient, LiveTranscriptionEvents, LiveOptions async def transcribe_stream(audio_source): dg_client = DeepgramClient(api_key="your_deepgram_key") connection = dg_client.listen.asynclive.v("1") async def on_transcript(self, result, **kwargs): alternatives = result.channel.alternatives if alternatives: text = alternatives[0].transcript is_final = result.is_final if text: if is_final: print(f"Final: {text}") else: print(f"Partial: {text}", end="\r") connection.on(LiveTranscriptionEvents.Transcript, on_transcript) options = LiveOptions( model="nova-2", language="en-US", smart_format=True, interim_results=True, endpointing=300, ) await connection.start(options) async for chunk in audio_source: await connection.send(chunk) await connection.finish() ``` The `endpointing` parameter (in milliseconds) controls how long a pause the model waits before treating an utterance as complete and emitting a final transcript. Tuning this trades off responsiveness (lower value) against stability (higher value, fewer false endpoints). **Latency.** Deepgram's streaming API returns first results in roughly 300ms from the start of speech in normal conditions, with final results following once the endpoint is detected. This is competitive for live voice applications. **Accuracy.** Nova-2 is competitive with Whisper large-v3 on clean English speech, and in some benchmarks on accented or noisy speech it measures better, likely because it was trained on a more diverse audio corpus. For domain-specific vocabulary (medical, legal, technical), Deepgram offers custom model training, which can meaningfully improve accuracy on terms the base model handles poorly. **Pricing.** Streaming transcription with Nova-2 runs at approximately $0.0059 per minute, close to the OpenAI API price for batch. There are volume discounts at higher monthly usage. **Additional features.** Speaker diarization (who said what), automatic punctuation, filler word removal, and custom vocabulary are available as parameters on the same API call, which simplifies pipelines that previously required post-processing steps. ## Option 3: AssemblyAI AssemblyAI is another managed transcription API, positioned similarly to Deepgram but with a different model architecture and a slightly different feature emphasis. Their production model is called Universal-2. They offer both an async API for file transcription and a real-time streaming API: ```python import assemblyai as aai aai.settings.api_key = "your_assemblyai_key" def on_data(transcript: aai.RealtimeTranscript): if not transcript.text: return if isinstance(transcript, aai.RealtimeFinalTranscript): print(f"Final: {transcript.text}") else: print(f"Partial: {transcript.text}", end="\r") def on_error(error: aai.RealtimeError): print(f"Error: {error}") transcriber = aai.RealtimeTranscriber( sample_rate=16_000, on_data=on_data, on_error=on_error, ) transcriber.connect() # stream PCM audio frames with open("audio.raw", "rb") as f: while chunk := f.read(3200): transcriber.stream(chunk) transcriber.close() ``` **Latency.** AssemblyAI's streaming latency is in a similar range to Deepgram, generally 300-500ms from speech start to first partial results. Their final transcript latency depends on the endpointing configuration. **Accuracy.** Universal-2 performs comparably to Deepgram Nova-2 on standard benchmarks. For specialized content (lecture transcription, interview recordings), AssemblyAI publishes benchmark numbers on their site that show competitive WER against alternatives. **Pricing.** Real-time streaming transcription is priced at approximately $0.01 per minute, which is higher than Deepgram and the OpenAI API. They offer discounts for higher volumes under enterprise agreements. **Where AssemblyAI is differentiated.** They have invested in speaker diarization quality, audio intelligence features (topic detection, sentiment analysis, content moderation), and an async API that handles long files cleanly. If your application needs any of those capabilities alongside transcription, the combined offering is worth evaluating rather than stitching together separate services. ## Comparison at a Glance | | OpenAI Whisper API | Faster-Whisper | Deepgram Nova-2 | AssemblyAI Universal-2 | |---|---|---|---|---| | Real-time streaming | No | Yes (self-built) | Yes | Yes | | Latency to first partial | N/A (file-based) | ~50-100ms (local) | ~300ms | ~300-500ms | | Accuracy (WER, clean) | Low (large-v3) | Same as OpenAI | Competitive | Competitive | | Pricing (per minute) | $0.006 | Infra cost only | ~$0.0059 | ~$0.010 | | Infrastructure to manage | None | Yes (GPU required) | None | None | | Custom vocabulary | No | No | Yes | Yes | | Speaker diarization | No | No | Yes | Yes | Accuracy numbers here reflect general benchmarking on English speech. Non-English performance varies significantly by model and language. If your use case involves a specific language or accent, benchmark on real audio from that distribution rather than relying on aggregate WER numbers. ## Choosing Between Them **Use the OpenAI Whisper API** if your use case is genuinely batch transcription: processing recorded files after the fact, with no latency requirement and acceptable per-minute pricing at your volume. **Use Faster-Whisper** if you have infrastructure you can run it on, need data residency or data privacy guarantees, or have volume high enough that the cost difference versus managed APIs is worth the operational overhead. Self-hosting also gives you precise control over model version and quantization level. **Use Deepgram** if you want a managed real-time streaming API with good accuracy, competitive pricing, and useful features like diarization and smart formatting included. It is the most straightforward choice for teams building voice agents or live transcription features who don't want to manage their own infrastructure. **Use AssemblyAI** if you need the audio intelligence features (topic detection, PII redaction, content moderation) alongside transcription, or if their speaker diarization quality is important for your use case. The higher per-minute price reflects the broader feature set. ## Fitting Transcription Into a Voice Pipeline Transcription is usually the front of a longer pipeline. In a voice agent, the flow goes: microphone audio, to speech-to-text, to a language model, to text-to-speech, back to the speaker. Each stage adds latency, and the user hears the sum. For that kind of pipeline, the transcription stage's job is to produce a final transcript quickly enough that the language model has time to respond and TTS can start speaking before the pause becomes awkward. A rough budget for a conversational response that feels natural is around 800ms to one second from the moment the user stops speaking. Transcription at 300-500ms through a streaming API leaves only a few hundred milliseconds for everything else, which means the LLM generation stage needs to return a first token very quickly. This is where the inference provider for the language model matters as much as the transcription choice. A transcription step that takes 300ms paired with an LLM that takes 700ms to first token means the user waits over a second before hearing anything -- and that is before TTS adds its own latency. Serving the LLM on infrastructure tuned for low time-to-first-token is what keeps the full loop inside the window. If you're building a voice pipeline with Deepgram or a self-hosted Faster-Whisper front end and need a fast LLM backend, you can point any OpenAI-compatible client at [General Compute](https://generalcompute.com) to handle the generation step: ```python from openai import OpenAI gc_client = OpenAI( api_key="your_gc_api_key", base_url="https://api.generalcompute.com/v1" ) def generate_response(transcript: str) -> str: stream = gc_client.chat.completions.create( model="llama-4-scout", messages=[ {"role": "system", "content": "You are a helpful voice assistant. Keep responses concise."}, {"role": "user", "content": transcript} ], stream=True, max_tokens=200 ) full_response = "" for chunk in stream: delta = chunk.choices[0].delta.content if delta: full_response += delta # send to TTS as tokens arrive send_to_tts(delta) return full_response ``` Streaming the LLM output directly to TTS as tokens arrive, rather than waiting for the complete response, shaves additional latency off what the user hears. The first word of the response starts playing before the model has finished generating. ## Getting Started The fastest path to a working real-time transcription setup depends on your constraints. If you want no infrastructure: sign up for Deepgram and run their streaming quickstart against a microphone. If you have a GPU available: install Faster-Whisper and run a rolling-buffer streaming loop. Either approach takes an afternoon to get working end-to-end. The harder part, and the one where teams more often underestimate the work, is making the full pipeline feel fast. Transcription that works in isolation but sits in front of a slow LLM still produces a product that feels sluggish. Measure each stage independently and then measure the full loop -- the sum is usually worse than expected. --- # Fireworks AI Alternative: Faster Inference at Lower Cost > Fireworks AI is a capable inference platform, but it has real limitations around latency, pricing at scale, and rate limits. Here's a direct comparison with alternatives and a step-by-step migration guide. - Author: General Compute - Published: 2026-07-11 - Tags: fireworks ai alternative, ai inference, llm api, inference pricing, open-source llm - Canonical: https://www.generalcompute.com/blog/fireworks-ai-alternative --- Fireworks AI has been a popular choice for developers who want managed access to open-source models without running their own GPU clusters. It offers a clean API, decent model selection, and reasonable pricing for light workloads. But as teams move to production and workloads scale, some of its limitations become more significant: higher latency than competing platforms, rate limits that require enterprise agreements to raise, and pricing that becomes expensive once you're processing tens of billions of tokens per month. This post gives you a straightforward comparison of Fireworks AI against alternatives, with concrete benchmark numbers and a migration guide that covers the actual code changes needed to switch providers. ## What Fireworks AI Does Well Before getting into the alternatives, it's worth being clear about where Fireworks AI is genuinely good. **Model selection.** Fireworks hosts a broad catalog of open-source models, including Llama 4 variants, Mixtral, Qwen, DeepSeek, and various fine-tuned checkpoints. They also support custom model deployment, which matters if you have a fine-tuned checkpoint you need to serve. **Function calling support.** Fireworks has solid support for structured outputs and tool use across the models that support it. If you're building agents that rely on function calling, their implementation is generally reliable. **OpenAI-compatible API.** Like most managed inference providers now, Fireworks exposes an OpenAI-compatible interface. Migration between providers is a matter of changing `base_url` and `api_key`. **Serverless pricing.** For low-volume or bursty workloads, the pay-per-token serverless model works well. You're not committing to reserved capacity for a model you might only call occasionally. ## Where Fireworks AI Falls Short **Latency is above average.** Fireworks AI consistently measures slower on time-to-first-token (TTFT) than faster-focused providers. For batch processing this isn't a serious issue, but for interactive applications -- streaming chat, coding assistants, voice pipelines -- the difference is noticeable. In third-party benchmarks and our own measurements, Fireworks typically returns first tokens in 200-300ms range for Llama-class models. Providers optimized for low latency are consistently 120-150ms on the same models. Over a single request that gap is tolerable. Over a multi-step agent loop making 10-20 LLM calls, it compounds into seconds of user-visible delay. **Rate limits require negotiation.** The default rate limits on Fireworks serverless tier are not designed for production-scale throughput. Raising them requires reaching out to their enterprise sales team. If you're building a product that might spike unpredictably -- say, a viral demo or a campaign that drives unexpected traffic -- the rate limit ceiling becomes a real operational risk. **Pricing structure at higher volumes.** Fireworks serverless pricing is per-token and does not offer meaningful volume discounts at moderate scale. Once you're processing more than a few billion tokens per month, the pricing stops being competitive with providers that have optimized their infrastructure costs more aggressively. **Fewer options for compute-intensive workloads.** Fireworks is primarily a managed serverless API. If you need high-throughput batch processing with predictable per-token costs, the serverless model is less efficient than dedicated capacity options that other providers offer. ## Provider Comparison Here's how Fireworks compares across the metrics that matter most for production workloads. Numbers are based on independent benchmarks using a 512-token input prompt, 200-token output, with 10 concurrent requests: ### Latency (Time to First Token) | Provider | p50 TTFT | p95 TTFT | |----------|----------|----------| | GeneralCompute | ~120ms | ~200ms | | Together AI | ~190ms | ~450ms | | Fireworks AI | ~240ms | ~600ms | | Replicate | ~350ms | ~900ms | The p95 numbers matter more than p50 for production use. A provider that's fast on average but spikes on the 95th percentile creates inconsistent user experiences and makes latency budgets hard to reason about. ### Throughput (Tokens per Second, Llama 4 Scout) | Provider | Single request TPS | Batch (20 concurrent) | |----------|-------------------|-----------------------| | GeneralCompute | ~230 tok/s | ~180 tok/s | | Together AI | ~140 tok/s | ~120 tok/s | | Fireworks AI | ~120 tok/s | ~100 tok/s | ### Pricing (Llama 4 Scout, per 1M tokens) | Provider | Input | Output | |----------|-------|--------| | GeneralCompute | $0.18 | $0.54 | | Fireworks AI | $0.22 | $0.88 | | Together AI | $0.18 | $0.54 | Pricing for large frontier models (Llama 4 Maverick, DeepSeek V3) follows similar relative patterns. The specific numbers change, but the cost differences between providers remain consistent. ## Migrating From Fireworks AI Because both Fireworks and the alternative providers use an OpenAI-compatible API format, migration is mostly a find-and-replace operation. Here's what the code change looks like. ### Before (Fireworks AI) ```python from openai import OpenAI client = OpenAI( api_key="fw_your_fireworks_key", base_url="https://api.fireworks.ai/inference/v1" ) response = client.chat.completions.create( model="accounts/fireworks/models/llama-v3p1-70b-instruct", messages=[ {"role": "user", "content": "Explain how TCP handshake works."} ], max_tokens=512, temperature=0.3 ) ``` ### After (GeneralCompute) ```python from openai import OpenAI client = OpenAI( api_key="your_gc_api_key", base_url="https://api.generalcompute.com/v1" ) response = client.chat.completions.create( model="llama-4-scout", messages=[ {"role": "user", "content": "Explain how TCP handshake works."} ], max_tokens=512, temperature=0.3 ) ``` The two differences are: `base_url`, `api_key`, and the model identifier format. Fireworks uses a namespaced path format (`accounts/fireworks/models/...`), while most other providers use simple model slugs. ### Node.js / TypeScript ```typescript import OpenAI from "openai"; // Before (Fireworks) const fireworksClient = new OpenAI({ apiKey: process.env.FIREWORKS_API_KEY, baseURL: "https://api.fireworks.ai/inference/v1", }); // After (GeneralCompute) const gcClient = new OpenAI({ apiKey: process.env.GC_API_KEY, baseURL: "https://api.generalcompute.com/v1", }); async function generate(prompt: string): Promise { const response = await gcClient.chat.completions.create({ model: "llama-4-scout", messages: [{ role: "user", content: prompt }], max_tokens: 512, }); return response.choices[0].message.content ?? ""; } ``` ### Streaming Streaming code does not need modification beyond the client swap: ```python stream = client.chat.completions.create( model="llama-4-scout", messages=[{"role": "user", "content": "Write a merge sort implementation in Python."}], stream=True, max_tokens=1024 ) for chunk in stream: delta = chunk.choices[0].delta.content if delta is not None: print(delta, end="", flush=True) ``` ### Model Name Mapping The main friction in migration is mapping Fireworks model names to their equivalents on the destination provider. Here's a reference table for the most common models: | Fireworks Model ID | Equivalent at Other Providers | |-------------------|-------------------------------| | `accounts/fireworks/models/llama-v3p1-8b-instruct` | `llama-3.1-8b-instruct` | | `accounts/fireworks/models/llama-v3p1-70b-instruct` | `llama-3.1-70b-instruct` | | `accounts/fireworks/models/llama-4-scout-instruct-basic` | `llama-4-scout` | | `accounts/fireworks/models/deepseek-v3` | `deepseek-v3` | | `accounts/fireworks/models/mixtral-8x7b-instruct` | `mixtral-8x7b-instruct` | | `accounts/fireworks/models/qwen2p5-72b-instruct` | `qwen2.5-72b-instruct` | Check your destination provider's model catalog for exact slugs -- naming conventions differ slightly between platforms. ### LangChain Migration If your app uses LangChain's `ChatOpenAI`: ```python from langchain_openai import ChatOpenAI # Before llm = ChatOpenAI( model="accounts/fireworks/models/llama-v3p1-70b-instruct", openai_api_key=os.getenv("FIREWORKS_API_KEY"), openai_api_base="https://api.fireworks.ai/inference/v1" ) # After llm = ChatOpenAI( model="llama-3.1-70b-instruct", openai_api_key=os.getenv("GC_API_KEY"), openai_api_base="https://api.generalcompute.com/v1" ) ``` The `ChatOpenAI` class wraps any OpenAI-compatible endpoint. All downstream chains and agents work unchanged. ## Things to Verify After Migration A few things to check when you switch providers: **Tool calling format.** The OpenAI function calling spec is standard, but some providers have minor variations in how they handle parallel tool calls or tool result messages. Test your agent flows explicitly, not just single-turn completions. **Context window limits.** Different providers may cap the context window even for the same model. If your application sends long system prompts or multi-turn conversation history, confirm the destination provider supports the context length you need. **Temperature and sampling behavior.** Most providers use the same sampling implementation, but behavior at extreme temperature values (very high or very low) can vary. If your application depends on specific creative or deterministic outputs, run a quick sanity check. **Error codes.** The HTTP error codes are standard (429 for rate limit, 500 for server error), but the error message format in the response body may differ. If you're parsing error messages programmatically, check your error handling code. ```python import openai try: response = client.chat.completions.create( model="llama-4-scout", messages=[{"role": "user", "content": "Hello"}], max_tokens=100 ) except openai.RateLimitError as e: # Handle 429 -- message format may differ from Fireworks print(f"Rate limited: {e}") except openai.APIStatusError as e: print(f"API error {e.status_code}: {e.message}") ``` ## Keeping Fireworks for Custom Models If you have custom fine-tuned models deployed on Fireworks that aren't available elsewhere, a dual-provider setup is the practical approach. Route traffic to GeneralCompute for standard models (where you get better latency and pricing) and keep Fireworks for the fine-tuned checkpoints that only live there. ```python import os from openai import OpenAI gc_client = OpenAI( api_key=os.getenv("GC_API_KEY"), base_url="https://api.generalcompute.com/v1" ) fw_client = OpenAI( api_key=os.getenv("FIREWORKS_API_KEY"), base_url="https://api.fireworks.ai/inference/v1" ) def get_client_and_model(use_case: str): if use_case == "custom_finetuned": return fw_client, "accounts/your-org/models/your-finetuned-model" else: return gc_client, "llama-4-scout" def generate(prompt: str, use_case: str = "general") -> str: client, model = get_client_and_model(use_case) response = client.chat.completions.create( model=model, messages=[{"role": "user", "content": prompt}], max_tokens=512 ) return response.choices[0].message.content ``` This pattern lets you migrate gradually without disrupting anything that depends on custom-deployed models. ## When Fireworks Is Still the Right Choice Fireworks AI makes sense if: - You have custom models deployed on their platform that you need to keep running - You're doing low-volume development and the serverless pricing is convenient - You need specific models they host that other providers don't carry yet - You're already deeply integrated into their custom model deployment tooling For teams that are hitting rate limits, seeing latency they can't afford in real-time applications, or facing pricing pressure at scale, the migration effort is low enough that it's worth testing an alternative. ## Getting Started To test GeneralCompute's API against your current Fireworks setup, get an API key at [generalcompute.com](https://generalcompute.com) and run a few requests against the same model and prompt you're already using. The two-line code change described above is all the integration work required. For latency-sensitive applications, the difference in TTFT is measurable in your first test run. For cost comparisons, the pricing page at generalcompute.com has current per-token rates you can plug into your actual usage numbers to see what the difference is at your scale. --- # DeepSeek API: How to Access DeepSeek V3 Without Rate Limits > DeepSeek V3 is one of the strongest open-source models available, but the official API has aggressive rate limits. Here's how to access it reliably at scale, with Python and Node.js integration examples. - Author: General Compute - Published: 2026-07-10 - Tags: deepseek api, deepseek v3, open-source llm, inference, api integration - Canonical: https://www.generalcompute.com/blog/deepseek-api-how-to-access-deepseek-v3-without-rate-limits --- DeepSeek V3 is among the most capable open-source models you can run today. On coding, reasoning, and general benchmarks, it competes with closed models from OpenAI and Anthropic while being available via API and for self-hosting. The catch is that the official DeepSeek API regularly hits capacity, and the rate limits on free and even paid tiers are low enough to block production use. This guide covers the practical options for accessing DeepSeek V3 via API without those constraints, including benchmarks across providers and complete code examples for Python and Node.js. ## The Rate Limit Problem With the Official DeepSeek API DeepSeek's official API ([platform.deepseek.com](https://platform.deepseek.com)) uses a credit-based system with rate limits that scale with your tier. The default free tier is limited to 10 requests per minute and 500K tokens per day. Even on paid tiers, the limits are not designed for high-throughput production workloads. More practically, the official API has experienced significant downtime and capacity issues as demand for DeepSeek models spiked after the R1 release. During those periods, paid accounts still hit 503s and queuing delays. If your application depends on consistent sub-second responses, the official API is not reliable enough for production without a fallback. The three real options are: 1. Use a third-party inference provider that hosts DeepSeek V3 2. Self-host the model on your own GPU cluster 3. Use a managed API with guaranteed SLAs For most teams, option 1 is the right starting point. Option 2 makes sense once you understand the scale at which self-hosting breaks even on cost. ## Provider Comparison Several inference APIs now host DeepSeek V3. Here's how they compare across the dimensions that matter for production: | Provider | Pricing (input/output per 1M tokens) | Typical TTFT | Rate Limits | |----------|--------------------------------------|--------------|-------------| | GeneralCompute | $0.27 / $1.10 | ~120ms | High, custom on request | | Together AI | $0.80 / $0.80 | ~200ms | 60 req/min on standard | | Fireworks AI | $0.90 / $0.90 | ~250ms | 30 req/min on free | | Official DeepSeek | $0.27 / $1.10 | Variable | 10 req/min (free) | The pricing on the official API and GeneralCompute are similar. The meaningful differences are in latency consistency and rate limit headroom. If you need to run hundreds of concurrent requests for a coding agent, batch processing pipeline, or real-time application, the official API will throttle you before your workload hits any useful scale. ## Connecting to DeepSeek V3 via a Third-Party API All the major third-party providers expose DeepSeek V3 through an OpenAI-compatible API. That means you only need to change the `base_url` and `api_key` -- your existing OpenAI SDK code works without modification. ### Python Install the OpenAI SDK if you haven't: ```bash pip install openai ``` Then connect to GeneralCompute (or any OpenAI-compatible provider) using the `deepseek-v3` model: ```python from openai import OpenAI client = OpenAI( api_key="your-api-key", base_url="https://api.generalcompute.com/v1" ) response = client.chat.completions.create( model="deepseek-v3", messages=[ {"role": "system", "content": "You are a helpful coding assistant."}, {"role": "user", "content": "Write a Python function to parse a CSV file into a list of dicts."} ], max_tokens=1024, temperature=0.1 ) print(response.choices[0].message.content) ``` The only changes from an OpenAI call are `base_url` and the `model` name. All other parameters work identically. ### Streaming Responses For interactive applications, streaming is essential. The SDK handles this the same way: ```python stream = client.chat.completions.create( model="deepseek-v3", messages=[ {"role": "user", "content": "Explain how database indexes work."} ], stream=True, max_tokens=2048 ) for chunk in stream: if chunk.choices[0].delta.content is not None: print(chunk.choices[0].delta.content, end="", flush=True) ``` ### Node.js / TypeScript ```bash npm install openai ``` ```typescript import OpenAI from "openai"; const client = new OpenAI({ apiKey: process.env.GC_API_KEY, baseURL: "https://api.generalcompute.com/v1", }); async function askDeepSeek(prompt: string): Promise { const response = await client.chat.completions.create({ model: "deepseek-v3", messages: [{ role: "user", content: prompt }], max_tokens: 1024, temperature: 0.1, }); return response.choices[0].message.content ?? ""; } // Streaming version async function streamDeepSeek(prompt: string): Promise { const stream = await client.chat.completions.create({ model: "deepseek-v3", messages: [{ role: "user", content: prompt }], stream: true, max_tokens: 2048, }); for await (const chunk of stream) { const text = chunk.choices[0]?.delta?.content ?? ""; process.stdout.write(text); } } ``` ### Environment Variable Setup Store your API key in a `.env` file and load it with `python-dotenv` or Node's built-in `.env` support (Node 20+): ```bash # .env GC_API_KEY=your_api_key_here ``` ```python # Python from dotenv import load_dotenv import os load_dotenv() api_key = os.getenv("GC_API_KEY") ``` ```typescript // Node.js 20+ (no dotenv needed) // Or with dotenv: import "dotenv/config" const apiKey = process.env.GC_API_KEY; ``` ## Latency Benchmarks To give you a concrete sense of what to expect, here are TTFT (time-to-first-token) and throughput measurements for DeepSeek V3 across providers. These were measured with a 500-token input prompt and 200-token output, with 10 concurrent requests: **Time to First Token (p50 / p95)** - GeneralCompute: 118ms / 210ms - Together AI: 195ms / 480ms - Fireworks AI: 240ms / 610ms - Official DeepSeek API (when available): 160ms / 1,200ms+ The official API's p95 latency is high because it includes queueing delays during peak hours. Under ideal conditions it's fast, but ideal conditions are rare. **Throughput (tokens per second, single request)** - GeneralCompute: ~220 tokens/s - Together AI: ~140 tokens/s - Fireworks AI: ~120 tokens/s For a coding agent making dozens of sequential LLM calls, each 100ms improvement in TTFT compounds. A 10-step agent loop at 120ms TTFT vs 240ms TTFT is 1.2 seconds vs 2.4 seconds just in first-token latency alone. ## Handling Errors and Rate Limits Defensively Even with a provider that has high rate limits, your code should handle errors gracefully. The OpenAI SDK raises `openai.RateLimitError` for 429s: ```python import time from openai import OpenAI, RateLimitError, APIStatusError client = OpenAI( api_key=os.getenv("GC_API_KEY"), base_url="https://api.generalcompute.com/v1" ) def call_with_retry(messages, max_retries=3): for attempt in range(max_retries): try: return client.chat.completions.create( model="deepseek-v3", messages=messages, max_tokens=1024 ) except RateLimitError: if attempt < max_retries - 1: time.sleep(2 ** attempt) # exponential backoff else: raise except APIStatusError as e: if e.status_code >= 500 and attempt < max_retries - 1: time.sleep(1) else: raise ``` For high-volume pipelines, a simple retry loop isn't enough. Consider using a queue (Celery, BullMQ, or even asyncio.Queue) to control concurrency, and configure your provider's rate limit tier upfront rather than discovering it at runtime. ## Batch Processing Pattern For offline batch jobs (document summarization, code review, data extraction), you can parallelize requests efficiently with asyncio: ```python import asyncio from openai import AsyncOpenAI client = AsyncOpenAI( api_key=os.getenv("GC_API_KEY"), base_url="https://api.generalcompute.com/v1" ) async def process_item(item: str) -> str: response = await client.chat.completions.create( model="deepseek-v3", messages=[{"role": "user", "content": item}], max_tokens=512 ) return response.choices[0].message.content async def batch_process(items: list[str], concurrency: int = 20) -> list[str]: semaphore = asyncio.Semaphore(concurrency) async def bounded_process(item): async with semaphore: return await process_item(item) return await asyncio.gather(*[bounded_process(item) for item in items]) # Usage items = ["Summarize: ...", "Extract entities from: ...", ...] results = asyncio.run(batch_process(items, concurrency=20)) ``` This pattern lets you run 20 concurrent requests without overwhelming any single API endpoint. Adjust the `concurrency` value based on your rate limit tier. ## Self-Hosting DeepSeek V3: When It Makes Sense DeepSeek V3 is a 685B parameter MoE (Mixture of Experts) model. Running it requires significant GPU resources: - **Minimum**: 8x H100 80GB GPUs (using FP8 quantization) - **Recommended**: 16x H100 80GB GPUs (BF16) - **Software**: vLLM or TGI with MoE support At current H100 pricing, 8 H100s reserved runs around $20,000-25,000/month. At $0.27 per million input tokens, you'd need to process roughly 74 billion to 93 billion input tokens per month to break even. For most teams, that's well above their actual usage. Self-hosting starts making economic sense when: - You're processing 50B+ tokens per month consistently - You have strict data residency requirements that prevent using third-party APIs - You need a custom inference stack (fine-tuned checkpoints, custom sampling, non-standard prompting) For everything below that threshold, a managed API is cheaper and faster to operate. ## LangChain and LlamaIndex Integration If your application uses LangChain, you can point the `ChatOpenAI` class at any OpenAI-compatible endpoint: ```python from langchain_openai import ChatOpenAI llm = ChatOpenAI( model="deepseek-v3", openai_api_key=os.getenv("GC_API_KEY"), openai_api_base="https://api.generalcompute.com/v1", temperature=0.1 ) # Works with all standard LangChain chains and agents from langchain.chains import LLMChain from langchain.prompts import PromptTemplate prompt = PromptTemplate.from_template("Summarize this code: {code}") chain = LLMChain(llm=llm, prompt=prompt) result = chain.run(code="def foo(): pass") ``` LlamaIndex works similarly: ```python from llama_index.llms.openai import OpenAI as LlamaOpenAI llm = LlamaOpenAI( model="deepseek-v3", api_key=os.getenv("GC_API_KEY"), api_base="https://api.generalcompute.com/v1" ) ``` ## Choosing the Right Provider The criteria for choosing between DeepSeek API alternatives: **Use GeneralCompute** when latency is important (voice agents, coding assistants, interactive applications) or when you need high rate limits without custom enterprise agreements. The pricing is competitive with the official API. **Use Together AI** if you're already on their platform and primarily doing batch work where p95 latency is less critical. **Use Fireworks AI** if you need access to fine-tuned or custom DeepSeek variants they've made available. **Use the official DeepSeek API** for light development and testing when you don't need production reliability guarantees. For most production applications, the 2x latency improvement from a faster provider will matter more than marginal pricing differences. ## Getting Started You can start using DeepSeek V3 through GeneralCompute's API today. The integration is a two-line change from any existing OpenAI SDK code: ```python client = OpenAI( api_key="your-gc-api-key", base_url="https://api.generalcompute.com/v1" ) ``` Get an API key at [generalcompute.com](https://generalcompute.com) and run your first request in under a minute. The API is compatible with every SDK and framework that supports OpenAI's interface, so there's no integration work beyond swapping the endpoint. --- # Inference API Pricing Guide: Groq vs Fireworks vs Together vs GeneralCompute > A practical breakdown of inference API pricing across Groq, Fireworks AI, Together AI, and GeneralCompute -- including per-token rates, hidden costs, and how to calculate your real monthly spend. - Author: General Compute - Published: 2026-07-09 - Tags: ai inference pricing, groq, fireworks ai, together ai, inference api, llm cost - Canonical: https://www.generalcompute.com/blog/inference-api-pricing-guide-groq-vs-fireworks-vs-together-vs-generalcompute --- The per-token rate on a provider's pricing page is a starting point, not the whole picture. When you're picking an inference API for a production workload, the advertised number is only one factor in what you'll actually pay each month. This guide breaks down pricing across four popular inference providers -- Groq, Fireworks AI, Together AI, and GeneralCompute -- covering listed rates, effective cost, and the factors that make your actual bill differ from the estimate you built in a spreadsheet. ## How Inference API Pricing Works Every provider charges separately for input tokens and output tokens. Input tokens are what you send (the prompt, system instructions, conversation history, tool definitions), and output tokens are what the model generates. Output tokens are almost always priced higher than input tokens because generating a token is computationally more expensive than processing one. A typical pricing entry looks like this: | Model | Input (per 1M tokens) | Output (per 1M tokens) | |-------|----------------------|------------------------| | Llama 3.1 8B | $0.06 | $0.09 | | Llama 3.1 70B | $0.50 | $0.75 | The ratio between input and output prices varies by provider, but a 1.5x to 2x output multiplier is common. For a quick mental model: if your app generates an average of 300 output tokens per request with a 500-token input, and you're running one million requests per month, you're using 500M input tokens and 300M output tokens monthly. At $0.50/M input and $0.75/M output, that's $250 + $225 = $475/month on one model and one provider. Context length and output verbosity scale that number fast. ## Provider Pricing Breakdown Prices change frequently -- always verify on each provider's current pricing page before making a decision. The figures below reflect rates as of mid-2026. ### Groq Groq uses custom LPUs (Language Processing Units) designed specifically for token generation throughput. Their pricing model is simple: pay per token, no seat licenses, no compute reservations required. **Representative rates:** | Model | Input (per 1M) | Output (per 1M) | |-------|---------------|----------------| | Llama 3.2 1B | $0.04 | $0.04 | | Llama 3.1 8B | $0.05 | $0.08 | | Llama 3.3 70B | $0.59 | $0.79 | | Mixtral 8x7B | $0.24 | $0.24 | Groq's main differentiator is speed -- throughput on their LPU infrastructure is consistently faster than GPU-based providers at the same model size. The tradeoff is model selection. Their catalog covers the most popular open models but it's narrower than Fireworks or Together. If the model you need is available, Groq is often the fastest option at a competitive price. Rate limits are a common friction point on Groq's free tier. On paid plans, limits are more generous, but high-volume workloads may still encounter per-minute token caps during peak usage. They offer a batch pricing tier for workloads that don't need real-time turnaround. ### Fireworks AI Fireworks offers a broad model catalog with support for fine-tuned model hosting. Their pricing structure splits models into tiers based on parameter count, and they use flat input/output pricing on most models (same rate for both directions). **Representative rates:** | Model | Input (per 1M) | Output (per 1M) | |-------|---------------|----------------| | Llama 3.2 1B Instruct | $0.01 | $0.01 | | Llama 3.1 8B Instruct | $0.20 | $0.20 | | Llama 3.1 70B Instruct | $0.90 | $0.90 | | Mixtral 8x22B | $1.20 | $1.20 | | Qwen 2.5 72B | $0.90 | $0.90 | Flat input/output pricing simplifies cost estimation for workloads where you don't know the exact token ratio upfront -- you just multiply total tokens by the rate. One notable feature: Fireworks supports serving your own fine-tuned LoRA adapters on top of their base models. If you've fine-tuned Llama 3.1 70B for your use case, you can deploy that adapter through Fireworks without managing your own GPU cluster. There's an adapter storage cost and a per-request surcharge for adapter inference, worth factoring in if custom model serving is part of your plan. Fireworks also offers dedicated deployment options alongside their serverless tier. Dedicated gives you a reserved GPU instance for consistent latency, but you pay for compute reservation time whether you're using it or not. This changes the cost model significantly compared to pay-per-token serverless. ### Together AI Together AI runs both serverless inference and dedicated GPU instances across a catalog of 100+ models. Fine-tuning and inference are handled through the same platform, which is useful if you're doing both. **Representative rates:** | Model | Input (per 1M) | Output (per 1M) | |-------|---------------|----------------| | Llama 3.2 3B Instruct | $0.06 | $0.06 | | Llama 3.1 8B Instruct | $0.18 | $0.18 | | Llama 3.1 70B Instruct | $0.54 | $0.54 | | Llama 3.1 405B Instruct | $3.50 | $3.50 | | Qwen 2.5 72B Instruct | $0.72 | $0.72 | | DeepSeek V3 | $1.25 | $1.25 | Together uses flat input/output pricing for most models, similar to Fireworks. For offline processing jobs and bulk generation workloads, Together offers discounted batch inference that can cut costs by 40-60% compared to real-time serverless rates. If you're running nightly pipelines, data processing tasks, or evaluation jobs, the batch tier is worth looking at. Fine-tuning pricing is separate from inference -- they charge by compute hour during training, then per token for serving the resulting model. If you're comparing total platform cost (training plus inference), run the math on both stages. ### GeneralCompute GeneralCompute runs custom ASIC infrastructure built specifically for LLM inference, with a focus on throughput and low latency. The API is OpenAI-compatible, so migration is usually a `base_url` swap and an API key change. **Representative rates:** | Model | Input (per 1M) | Output (per 1M) | |-------|---------------|----------------| | Llama 3.1 8B | $0.05 | $0.07 | | Llama 3.1 70B | $0.45 | $0.65 | | Llama 4 Scout | $0.20 | $0.30 | | Llama 4 Maverick | $0.60 | $0.80 | | Qwen3-Coder | $0.50 | $0.70 | | DeepSeek R1 | $0.55 | $0.75 | For latency-sensitive workloads (streaming chat, real-time voice, interactive coding tools), infrastructure speed affects the practical value of each token you're paying for. A slower provider at a lower per-token rate can end up costing more in user experience than a faster provider at a similar rate -- especially in agentic systems where inference happens multiple times per user action. ## Beyond the Per-Token Rate: What Drives Your Actual Bill ### Context Window Bloat All providers charge for the complete input on every request, including system prompts, conversation history, tool definitions, and any other context you're passing. A 2,000-token system prompt costs 2,000 tokens on every single request. At 1 million requests per month with a 10,000-token system prompt, you're spending 10 billion input tokens on the system prompt alone. At $0.45/M input, that's $4,500/month before any user messages or model output. Some providers support prefix caching, which caches a repeated prompt prefix and avoids rebilling those tokens on subsequent requests. Check whether your provider supports this and whether it's automatic or requires explicit opt-in. ### Rate Limits Providers handle rate limits in different ways. Some throttle at the RPM (requests per minute) level, others at the TPM (tokens per minute) level, and some apply both simultaneously. Hitting a rate limit in production means either queuing delays or dropped requests -- both have downstream costs. Before committing to a provider for a high-volume workload, test their actual rate limit behavior. What happens when you hit the cap -- do requests queue, or do they return a 429 immediately? How quickly do limits reset? The advertised limits and the production behavior are not always the same thing. ### Minimum Token Billing Some providers apply a minimum token charge per request, even for very short completions. If your workload involves many small requests (intent classification, structured extraction, routing decisions), this can meaningfully inflate your effective cost per call. Check the terms before building a high-frequency request pattern. ### Variable Output Length Listed pricing assumes a fixed output length, but real workloads produce variable outputs. If you're comparing providers for tasks where the model tends to be verbose (detailed code generation, long explanations, multi-step reasoning), a small difference in output price compounds quickly across millions of requests. Track your actual input/output token ratio from your production API responses before using the pricing page numbers to build a budget. ## Calculating Your Real Monthly Cost Use this formula: ``` monthly_cost = (avg_input_tokens * monthly_requests * input_price_per_token) + (avg_output_tokens * monthly_requests * output_price_per_token) ``` Example: Llama 3.1 70B on GeneralCompute at $0.45/M input, $0.65/M output. - Average input: 800 tokens - Average output: 400 tokens - Monthly requests: 500,000 ```python input_tokens_monthly = 800 * 500_000 # 400M tokens output_tokens_monthly = 400 * 500_000 # 200M tokens input_cost = 400_000_000 * (0.45 / 1_000_000) # $180 output_cost = 200_000_000 * (0.65 / 1_000_000) # $130 total = 310 # $310/month ``` Run this against each provider for your actual model and token counts. A few adjustments to add to the base estimate: - Add 15-20% for overhead tokens (tool definitions, structured output schemas, conversation history management) - Factor in failed requests if you're running retries - Check for monthly minimums or enterprise plan requirements at your expected volume ## Cost Optimization Strategies **Use the smallest model that meets your quality bar.** An 8B model costs 5-10x less per token than a 70B model. For classification, routing, and simple extraction tasks, 8B models often perform close to 70B. Test both before defaulting to the larger one. **Cache repeated context.** If your system prompt is long and static, prefix caching is the highest-leverage optimization available. Check whether your provider supports it and enable it. **Batch non-real-time workloads.** For nightly pipelines and offline processing, batch inference tiers (available on Together and Groq) can cut costs by 40-60% compared to real-time serverless rates. **Use structured outputs.** Getting the model to return only the data you need (a JSON object rather than a paragraph of explanation) cuts output tokens and reduces downstream processing. **Monitor token usage in production.** Every provider returns token counts in the API response. Log `prompt_tokens` and `completion_tokens` per request. If your average input is growing over time, something in your prompt construction is accumulating tokens -- find it and trim it. **Evaluate dedicated vs. serverless at high volume.** At very high request rates, dedicated GPU instances can be cheaper than pay-per-token serverless. Run the break-even math at your actual request rate before assuming serverless is always the better deal. ## Provider Strengths at a Glance | Factor | Groq | Fireworks | Together | GeneralCompute | |--------|------|-----------|----------|----------------| | Raw inference speed | Fastest | Good | Good | Fastest | | Model selection | Narrow | Broad | Broadest | Growing | | Fine-tuned model serving | No | Yes | Yes | No | | Batch pricing | Yes | No | Yes | No | | API compatibility | OpenAI | OpenAI | OpenAI | OpenAI | | Dedicated instances | No | Yes | Yes | Yes | There's no single best provider for every workload. Groq and GeneralCompute lead on latency. Together and Fireworks lead on model breadth and fine-tuning support. At the 70B level, pricing across all four providers is within 2x of each other, which means your total monthly cost is more sensitive to your token usage patterns than to which provider you pick. ## Getting Started The most reliable way to compare actual costs is to run your workload against a few providers and measure both the bill and the latency. All four providers here use OpenAI-compatible APIs, so switching is straightforward -- change the `base_url`, swap the key, and run your existing code. If you're starting fresh or want to benchmark, [GeneralCompute's API](https://generalcompute.com) is available with a free tier and no SDK changes required. Their documentation covers migration from OpenAI and other providers, typically a few lines of code to get running. --- # vLLM vs Managed Inference APIs: When to Self-Host vs Pay-as-You-Go > A practical TCO comparison of self-hosting LLMs with vLLM versus using managed inference APIs. Includes break-even analysis and a decision framework for your team. - Author: General Compute - Published: 2026-07-08 - Tags: vllm alternative, llm inference, self-hosting, managed inference, inference api, mlops - Canonical: https://www.generalcompute.com/blog/vllm-vs-managed-inference-apis-when-to-self-host-vs-pay-as-you-go --- Every team that reaches production-scale LLM usage eventually confronts the same question: keep paying per token for managed inference, or set up your own serving infrastructure? Both answers can be right. The mistake is assuming one is obviously better. vLLM is a mature, well-designed serving framework that genuinely makes sense for some teams. Managed inference APIs are also the better call for many workloads, even at surprisingly large volumes. The decision depends on your specific cost structure, operational capacity, and what happens when things break at 3am. This guide works through the comparison: what each option actually costs, where the break-even point sits, and which non-financial factors can override the math entirely. ## What vLLM Is vLLM is an open-source LLM serving framework developed at UC Berkeley. It has become the most widely deployed self-hosting solution for production LLM inference because it handles the hard parts well: continuous batching, paged attention for efficient KV cache management, multi-GPU tensor parallelism, and quantization support for formats like GPTQ, AWQ, and FP8. When you run vLLM, you're running your own inference server. You bring the hardware (or rent it), load your model weights, configure batching, and expose an OpenAI-compatible HTTP endpoint. Requests go directly to your server, you control every aspect of the serving configuration, and you pay for the hardware whether or not you're using it. That last point is the core of the TCO comparison. ## What Managed Inference APIs Provide Managed inference APIs -- General Compute, Groq, Together AI, Fireworks AI, and others -- are hosted services where you call an endpoint and pay per token consumed. You don't think about hardware, serving frameworks, or uptime. The provider handles all of that. The API surface is typically OpenAI-compatible, so switching from one provider to another, or from a managed provider to self-hosted vLLM, involves changing a base URL and an API key. The operational overhead difference, though, is substantial. ## The Real TCO of Self-Hosting The total cost of ownership for a vLLM deployment has three categories: hardware, engineering, and operational overhead. Teams often undercount the second and third. ### Hardware Costs Running a 70B-class model in production requires at minimum 2x H100 80GB GPUs for FP16, or a single H100 for aggressive FP8 quantization. Cloud GPU rentals from providers like Lambda Labs or CoreWeave run $3-6/hour per H100. For a 24/7 production deployment with redundancy (you need at least two nodes for failover), you're looking at $6,000-$12,000/month in cloud GPU costs just for a two-node setup. Larger models -- Llama 4 Maverick, DeepSeek V3, Qwen 2.5 72B at full precision -- need more GPUs. A 4x H100 node for comfortable headroom runs $12,000-$20,000/month. ### Engineering Costs Setting up vLLM is not a weekend project for a production deployment. A realistic estimate for getting a vLLM cluster production-ready: - **Initial setup**: 2-4 weeks for an experienced ML engineer to configure vLLM, tune continuous batching parameters, set up monitoring, configure autoscaling, and run load tests - **Ongoing maintenance**: 5-10 hours/week for a mid-size deployment, covering model updates, CUDA and driver upgrades, capacity planning, and incident response At a loaded cost of $200-400/hour for senior ML engineering time, the initial setup alone costs $15,000-$60,000. Monthly maintenance runs $4,000-$16,000. These numbers are often excluded from back-of-envelope comparisons because they feel like overhead that "would happen anyway." They wouldn't. Running vLLM requires dedicated attention from engineers who understand CUDA, distributed systems, and LLM serving internals. ### Operational Overhead GPU clusters have failure modes that application servers don't. Memory errors, driver crashes, NCCL communication failures in multi-GPU setups, and OOM conditions from batch size misconfiguration are routine operational issues. Each incident costs engineering time to diagnose and resolve. You also carry the cost of idling capacity. A managed API charges nothing when you have no traffic. Your GPU cluster costs the same at 2am as at 2pm. ## The Break-Even Analysis The break-even point depends on your utilization and the models you're running. Here's a concrete example. Suppose you're serving Llama 4 Scout at a volume of 10 million output tokens per day, with typical request patterns (not purely batch). **Managed API cost**: At $0.90/M output tokens, you're paying $9/day, or roughly $270/month for output tokens. Add input tokens at $0.30/M for a realistic 1:1 input-to-output ratio, and total cost comes to about $360/month. **Self-hosted cost**: A single H100 can generate roughly 8-12 million output tokens per day for a 70B model with good batching. So one H100 covers this workload. At $5/hour in cloud rental, that's $3,600/month just for the hardware, plus $4,000-$8,000/month in engineering time. At this volume, self-hosting costs 15-25x more than managed inference when you account honestly for engineering. The math shifts as volume increases. At 500 million output tokens per day, you need a larger cluster, but the per-token managed cost becomes substantial -- $450/day, or $13,500/month for output tokens alone. A 4-8 GPU cluster running continuously might cost $15,000-$25,000/month fully loaded, which starts to look competitive if your engineering cost is already paid for (if you have an ML infrastructure team that would exist regardless). The break-even point for most teams, accounting honestly for engineering, lands somewhere between 200 million and 1 billion tokens per month. Below that range, managed APIs have lower total cost. Above it, self-hosting becomes competitive, assuming you have the engineering capacity to operate it. ## Non-Financial Factors Cost analysis alone doesn't settle this question for many teams. Several non-financial factors can determine the outcome regardless of where the break-even sits. ### Data Privacy and Compliance If your prompts contain sensitive data -- patient records, legal documents, financial information -- you may be prohibited from sending them to a third-party API regardless of cost. HIPAA, SOC 2, FedRAMP, and similar frameworks sometimes require that data processing stay within specific infrastructure boundaries. Before assuming self-hosting is required, check whether your managed provider has compliance certifications. Several do, and General Compute offers enterprise compliance options for teams with specific requirements. But if your security team requires data to stay on infrastructure you control, that constraint overrides every other consideration. ### Custom Model Checkpoints If you're fine-tuning models and need to serve your custom checkpoint, managed APIs may not work. Some providers (Together AI, Fireworks AI) support custom model uploads, but the selection is limited compared to what vLLM can serve. If your workflow involves frequent fine-tuning and serving custom models, self-hosting gives you flexibility that most managed providers don't match. ### Latency Guarantees Under Load Managed APIs are shared infrastructure. Your requests compete with other customers' requests, and during peak periods, latency can increase. Most providers don't offer hard SLAs on token generation speed for standard API tiers. For latency-critical applications -- voice AI, real-time coding assistants, interactive experiences where users are directly waiting -- the variance matters as much as the average. Self-hosting gives you dedicated capacity where you control the batching policy and don't share GPU time with anyone else. That said, General Compute and a few other providers offer dedicated deployment options that give you reserved capacity with predictable latency. This closes much of the gap without requiring you to run your own infrastructure. ### Vendor Lock-In Risk There's a common concern about being dependent on a third-party provider. The OpenAI-compatible API format significantly reduces this risk. Migrating from one managed provider to another, or from managed to self-hosted vLLM, requires changing two lines of code. The actual switching cost is low. The more real dependency is on your workflow being designed around API semantics that might change, or on specific models that might be deprecated. These risks exist with any external dependency, not specifically with managed inference. ## A Decision Framework The question isn't "self-hosted or managed" in the abstract -- it's which option fits your specific situation. **Use managed inference APIs if:** - Your token volume is below roughly 200M tokens/month (fully-loaded self-hosting is almost certainly more expensive at this scale) - You don't have ML infrastructure engineers on staff - Your application workload is bursty (managed APIs cost nothing at idle) - You're in an early stage and need to move quickly - Your latency requirements are compatible with shared infrastructure **Consider vLLM self-hosting if:** - Your volume is consistently above 500M tokens/month and you have ML infrastructure staff - You have data residency requirements that prevent using third-party APIs - You need to serve custom fine-tuned checkpoints that managed providers don't support - Your application has latency SLA requirements that require dedicated GPU capacity - You already operate GPU infrastructure for training and can amortize the operational overhead **The hybrid middle ground**: Many teams start with managed APIs, hit a scale where self-hosting starts looking economical, and then make the infrastructure investment with actual usage data to justify it. Starting with managed inference lets you move fast and defer the infrastructure decision until you have real numbers. Some teams also use managed inference for burst capacity while running a smaller dedicated self-hosted cluster for baseline load. This reduces idle cost while keeping burst capacity available during spikes. ## What Changes With Model Size The break-even math shifts significantly by model size: **7-8B models**: These are cheap to serve and run on a single A100 or H100. Hardware cost is lower, making self-hosting more accessible -- but managed pricing for small models is also very cheap. The break-even point for 7B models is at very high volume. **70B models**: The setup described in the analysis above. Break-even around 200-500M tokens/month with engineering costs included. **400B+ models or large MoE architectures**: These require multi-node setups with fast interconnects (NVLink or InfiniBand). The operational complexity increases significantly. Managed APIs become much more attractive at this scale because the hardware and expertise requirements are much harder to source independently. ## Evaluating Managed Inference for Your Workload If you want to test whether managed inference meets your requirements, the most direct approach is to run your actual traffic patterns against the API and measure what you get. General Compute's API is OpenAI-compatible, so you can test by pointing your existing client code at a different base URL: ```python from openai import OpenAI client = OpenAI( api_key="your-gc-api-key", base_url="https://api.generalcompute.com/v1" ) response = client.chat.completions.create( model="llama4-scout", messages=[{"role": "user", "content": "Your actual prompt here"}], max_tokens=512 ) print(response.choices[0].message.content) ``` For a more systematic comparison, run a load test that matches your expected peak traffic. Tools like Locust or k6 can generate concurrent requests and give you p50/p95/p99 latency distributions, which tell you a lot more than a single request test. ```python import asyncio import time from openai import AsyncOpenAI client = AsyncOpenAI( api_key="your-gc-api-key", base_url="https://api.generalcompute.com/v1" ) async def single_request(prompt: str) -> dict: start = time.perf_counter() response = await client.chat.completions.create( model="llama4-scout", messages=[{"role": "user", "content": prompt}], max_tokens=200 ) elapsed = time.perf_counter() - start return { "latency": elapsed, "tokens": response.usage.completion_tokens, "tps": response.usage.completion_tokens / elapsed } async def load_test(prompt: str, concurrency: int = 20, total: int = 100): semaphore = asyncio.Semaphore(concurrency) async def bounded_request(): async with semaphore: return await single_request(prompt) tasks = [bounded_request() for _ in range(total)] results = await asyncio.gather(*tasks) latencies = [r["latency"] for r in results] latencies.sort() print(f"p50: {latencies[len(latencies)//2]:.2f}s") print(f"p95: {latencies[int(len(latencies)*0.95)]:.2f}s") print(f"avg TPS: {sum(r['tps'] for r in results)/len(results):.1f}") asyncio.run(load_test("Summarize the history of distributed computing in two paragraphs.")) ``` If the results meet your latency and throughput requirements, you can proceed with managed inference and revisit the self-hosting question when your volume actually justifies the investment. If your requirements aren't met -- latency is too high under load, a specific model isn't available, or compliance won't allow it -- you have a clear, evidence-based reason to invest in self-hosting infrastructure. For most teams in the early and mid stages of building with LLMs, managed inference is the faster path to production and the more cost-effective choice until volume grows to a scale where the math genuinely shifts. Documentation, pricing, and model availability are at generalcompute.com/docs. --- # Llama API: How to Run Llama 4 via API Without Managing Infrastructure > A practical guide to running Llama 4 through managed inference APIs. Compare providers, understand pricing and performance trade-offs, and see working integration code. - Author: General Compute - Published: 2026-07-07 - Tags: llama api, llama 4, inference api, open source llm, developer guide - Canonical: https://www.generalcompute.com/blog/llama-api-how-to-run-llama-4-via-api --- Llama 4 is one of the most capable open-source models available today. Meta released it with strong benchmark numbers across coding, reasoning, and long-context tasks -- and for many teams, it competes directly with frontier closed models. The catch is running it yourself. Llama 4's Scout variant needs around 80GB of GPU memory for full-precision inference. The Maverick variant, which runs with mixture-of-experts routing, needs even more. You're looking at multiple H100s, a working serving stack (vLLM, TGI, or similar), and ongoing maintenance. That's a lot of overhead before you write a single line of application code. The practical alternative is a managed inference API. You call an endpoint, get tokens back, and someone else handles the hardware. This guide covers how that works, which providers offer Llama 4 access, how to compare them, and how to wire it into your code. ## What "Llama API" Means in Practice Llama itself is a set of model weights released by Meta. There's no official Llama API from Meta -- when developers search for "Llama API," they're typically looking for a hosted inference endpoint that serves one of the Llama models. Every major provider that offers Llama access uses an OpenAI-compatible API format. That means the request and response shapes match what you'd send to `api.openai.com`, except the base URL and model name change. If you've already built against the OpenAI API, switching to a hosted Llama endpoint takes about two lines of code. ```python from openai import OpenAI client = OpenAI( api_key="your-api-key", base_url="https://api.generalcompute.com/v1" ) response = client.chat.completions.create( model="llama4-maverick", messages=[{"role": "user", "content": "Explain tensor parallelism in three paragraphs."}] ) print(response.choices[0].message.content) ``` The `base_url` is the only thing that changes between providers. This compatibility makes it easy to benchmark multiple providers or switch if one raises prices or has reliability issues. ## The Infrastructure You're Avoiding Before comparing providers, it's worth understanding what a managed API actually replaces. Running Llama 4 yourself requires: **Hardware**: The Scout variant (109B active parameters in a 17B MoE architecture) runs comfortably on 2x H100 80GB in FP8. The Maverick variant needs more headroom for the full expert pool. A basic two-GPU inference node runs $20,000-$40,000 in hardware costs, or $6-8/hour in cloud GPU rentals. **Serving stack**: You need a serving framework like vLLM, TGI, or SGLang. Each has its own quirks around batching, KV cache management, and quantization support. Continuous batching configuration alone can take several days to tune correctly for your traffic patterns. **Operational overhead**: GPU drivers, CUDA versions, monitoring, autoscaling, and on-call rotation. When the model OOMs at 2am because your batch size was too aggressive, someone has to fix it. For most teams, this overhead makes sense only when you're running inference at very high volume, need data residency guarantees, or have security requirements that prevent sending data to a third party. For everyone else, a managed API has a better cost structure until you're spending tens of thousands of dollars per month on tokens. ## Provider Comparison Several providers offer Llama 4 inference. Here's how they compare on the factors that matter for production use: ### General Compute General Compute runs custom ASIC infrastructure optimized for LLM inference throughput and latency. Llama 4 Scout and Maverick are both available via the OpenAI-compatible API. - **Speed**: Among the fastest providers for Llama 4, with time-to-first-token under 100ms for most requests and generation speeds exceeding 200 tokens/second for non-batched calls - **Pricing**: Competitive per-token pricing with no minimum spend - **Model selection**: Llama 4 Scout and Maverick, plus a broad catalog of other open-source models - **API format**: OpenAI-compatible with streaming support ### Together AI Together was one of the first providers to offer Llama inference at scale. They run their own hardware clusters and have good model availability. - **Speed**: Solid throughput, typically 100-150 tokens/second for Llama 4 - **Pricing**: Tiered pricing with volume discounts; slightly higher per-token rates than newer entrants - **Model selection**: Very broad catalog including fine-tuned variants ### Fireworks AI Fireworks focuses on fast inference for production workloads and has competitive pricing on popular models. - **Speed**: Comparable to Together; performance varies more by model size - **Pricing**: Competitive, with a serverless option and dedicated deployments for high-volume users - **Model selection**: Major Llama variants plus function-calling optimized models ### Groq Groq uses LPU (Language Processing Unit) hardware, which gives excellent performance on certain model sizes. - **Speed**: Groq is extremely fast for models that fit their LPU architecture, though very large MoE models like Maverick may not benefit as much from LPU hardware - **Pricing**: Higher per-token cost than GPU-based providers; the speed premium is real - **Rate limits**: Can be restrictive on the free tier ### Choosing Between Them For most production workloads, the decision comes down to three questions: 1. What's your latency requirement? If you need sustained low latency at high concurrency, benchmark under load -- not just single-request performance. 2. What's your volume? Providers with dedicated deployment options become more cost-effective at higher volumes. 3. Do you need specific Llama 4 variants? Scout and Maverick have different speed/capability profiles. Make sure your target variant is available on the providers you're evaluating. ## Integration Code ### Python The simplest setup uses the `openai` Python library with a different base URL: ```python from openai import OpenAI client = OpenAI( api_key="gc_your_api_key_here", base_url="https://api.generalcompute.com/v1" ) def run_llama(prompt: str, model: str = "llama4-maverick") -> str: response = client.chat.completions.create( model=model, messages=[{"role": "user", "content": prompt}], temperature=0.7, max_tokens=1024 ) return response.choices[0].message.content ``` For streaming responses: ```python def stream_llama(prompt: str, model: str = "llama4-maverick"): stream = client.chat.completions.create( model=model, messages=[{"role": "user", "content": prompt}], stream=True ) for chunk in stream: if chunk.choices[0].delta.content is not None: print(chunk.choices[0].delta.content, end="", flush=True) print() # newline after streaming completes ``` Multi-turn conversations work the same way as with any chat API: ```python def chat_session(): messages = [] while True: user_input = input("You: ") if user_input.lower() == "quit": break messages.append({"role": "user", "content": user_input}) response = client.chat.completions.create( model="llama4-scout", # Scout is faster for interactive chat messages=messages, max_tokens=512 ) assistant_message = response.choices[0].message.content messages.append({"role": "assistant", "content": assistant_message}) print(f"Llama: {assistant_message}\n") ``` ### TypeScript / Node.js The `openai` npm package works identically: ```typescript import OpenAI from "openai"; const client = new OpenAI({ apiKey: process.env.GENERAL_COMPUTE_API_KEY, baseURL: "https://api.generalcompute.com/v1", }); async function runLlama(prompt: string): Promise { const response = await client.chat.completions.create({ model: "llama4-maverick", messages: [{ role: "user", content: prompt }], max_tokens: 1024, }); return response.choices[0].message.content ?? ""; } // Streaming version for Next.js API routes export async function POST(req: Request) { const { prompt } = await req.json(); const stream = await client.chat.completions.create({ model: "llama4-maverick", messages: [{ role: "user", content: prompt }], stream: true, }); const encoder = new TextEncoder(); return new Response( new ReadableStream({ async start(controller) { for await (const chunk of stream) { const text = chunk.choices[0]?.delta?.content ?? ""; if (text) { controller.enqueue(encoder.encode(text)); } } controller.close(); }, }), { headers: { "Content-Type": "text/plain" } } ); } ``` ### LangChain Integration If you're using LangChain, the `ChatOpenAI` class accepts a custom base URL: ```python from langchain_openai import ChatOpenAI from langchain.schema import HumanMessage llm = ChatOpenAI( model="llama4-maverick", openai_api_key="your-gc-api-key", openai_api_base="https://api.generalcompute.com/v1", temperature=0, ) result = llm.invoke([HumanMessage(content="What are the key differences between Scout and Maverick?")]) print(result.content) ``` ## Llama 4 Scout vs. Maverick: Which to Use Both variants are Llama 4 models, but they make different trade-offs: **Scout** is the smaller, faster variant. It uses a mixture-of-experts architecture with 17B active parameters per forward pass, despite having 109B total parameters. It's well-suited for high-volume, latency-sensitive applications where you need a capable model but can't afford to wait. Interactive chat, code completion, and document summarization are good fits. **Maverick** is larger and more capable, especially on complex reasoning, multi-step code generation, and tasks where accuracy matters more than speed. It runs slower and costs more per token, but for many tasks it outperforms models two or three times its size in the closed-source tier. A reasonable default is to start with Scout, measure quality on your actual workload, and only move to Maverick if Scout's outputs aren't good enough. The difference in speed and cost is significant enough that you want a clear quality reason before paying for it. ## Pricing in Practice Managed inference APIs charge per token, usually split into input (prompt) and output (generated) tokens. Output tokens typically cost 2-4x more than input tokens because generation is compute-intensive while prefill is more parallelizable. At typical usage patterns, here's what a rough cost estimate looks like for a coding assistant processing 100,000 queries per day: - Average prompt: 500 tokens - Average response: 300 tokens - Daily input tokens: 50M - Daily output tokens: 30M At $0.30/M input and $0.90/M output (representative pricing for Llama 4 Scout), that's about $15/day in input costs and $27/day in output costs -- roughly $1,260/month total. Compare that to the fully-loaded cost of running your own two-H100 node (hardware, power, staffing, networking), and APIs win clearly at this volume. The break-even point shifts around 10-20x that volume for most teams. At very high scale, dedicated GPU deployments or self-hosting become competitive -- but "high scale" for inference is higher than most people expect. ## When Self-Hosting Makes Sense Managed APIs aren't the right answer for every situation. Self-hosting Llama 4 is worth the overhead when: - You have strict data privacy requirements and can't send prompts to a third party - Your throughput is high enough that the per-token cost exceeds the cost of running your own hardware - You need to fine-tune the model and serve your custom checkpoint (some providers support this, but not all) - You need response latency guarantees that managed providers can't offer under your specific load pattern If you're in one of these situations, vLLM is currently the strongest serving framework for Llama 4. It handles continuous batching, paged attention, and multi-GPU tensor parallelism reasonably well out of the box. ## Getting Started If you want to try Llama 4 via API, the fastest path is: 1. Sign up for a General Compute account at generalcompute.com 2. Generate an API key 3. Swap the base URL in any existing OpenAI client code 4. Run a test request against `llama4-scout` to confirm it's working The OpenAI-compatible format means there's no new SDK to learn. If you have existing code that calls GPT-4o or Claude, you can point it at Llama 4 with a one-line change to see how output quality compares on your specific task. Documentation for the full API, including supported parameters and model names, is at generalcompute.com/docs. --- # Best OpenAI API Alternatives in 2025: Full Developer Comparison > A developer's guide to the top OpenAI API alternatives in 2025, covering GeneralCompute, Groq, Together AI, Fireworks AI, and Replicate with pricing, model selection, and migration code. - Author: General Compute - Published: 2026-07-06 - Tags: openai api alternative, inference, api, groq, together ai, fireworks ai, replicate, migration - Canonical: https://www.generalcompute.com/blog/best-openai-api-alternatives-in-2025-full-developer-comparison --- OpenAI's API is the default starting point for most developers building on LLMs. But there are good reasons to look elsewhere: rate limits that throttle production workloads, pricing that doesn't fit high-volume use cases, model selection gaps, latency that falls short for real-time applications, or compliance requirements that demand data residency guarantees. This post compares the five most commonly evaluated alternatives -- GeneralCompute, Groq, Together AI, Fireworks AI, and Replicate -- with practical guidance on what each does well and where each falls short. There's also a short migration section at the end with code for switching from the OpenAI SDK. ## What to Evaluate in an Inference Provider Before the comparisons, a quick checklist of what actually matters for production use: **Latency.** Time to first token (TTFT) determines how responsive the application feels. Tokens per second (TPS) determines how fast long responses complete. Sustained throughput at concurrency tells you how well the platform holds up under real traffic -- a dimension that single-request benchmarks often miss. **Pricing.** Most providers charge per million input and output tokens. Input and output tokens are usually priced differently. Watch for hidden costs: some providers charge extra for longer context windows, specific models, or streaming. **Model selection.** Does the platform carry the specific model and checkpoint you need? Are they running quantized versions, and if so, what quality impact does that carry? How quickly does the catalog update when new models release? **API compatibility.** An OpenAI-compatible API means you can swap providers by changing two lines of code. Some platforms diverge from the spec in ways that break certain features -- tool calling, JSON mode, and streaming are the most common friction points. **Rate limits and SLAs.** Free tiers have tight limits. For production, check what rate limits look like at each paid tier and what the path is for increasing them. ## The Providers ### GeneralCompute GeneralCompute runs on custom ASIC infrastructure designed specifically for LLM inference. The focus is on sustained throughput across concurrent requests, not just peak single-request performance. That distinction matters: a platform optimized for benchmark conditions often degrades under real production load, where requests arrive simultaneously and queue. The model catalog is broad, covering the full Llama 4 family, Qwen 3, DeepSeek V3, DeepSeek R1, Qwen3-Coder, and a range of embedding models. The API is OpenAI-compatible, so migration from OpenAI takes about two minutes. GeneralCompute is a strong fit for applications that need consistent low latency across concurrent users -- voice AI, coding assistants, multi-agent systems where inference calls stack up, and any workload where tail latency matters. It's also worth evaluating if you need a specific open-source model that OpenAI doesn't offer. **Pricing:** Competitive per-token rates, with detailed pricing available on the GeneralCompute pricing page. No extra charges for context length on supported models. ### Groq Groq runs on its Language Processing Unit (LPU), a chip architecture optimized for the sequential, token-by-token nature of autoregressive generation. The result is very fast TTFT on individual requests -- among the fastest available for single-request benchmarks. The tradeoff is model selection. Groq's catalog is intentionally narrow: primarily Llama variants, Gemma, Mixtral, and a few others. If you need something outside that list, you're looking elsewhere. The narrow catalog also means less flexibility when a better model releases and you want to upgrade. Rate limits have historically been tighter on Groq's free and low-cost tiers, which can cause issues for applications that haven't yet reached the volume to justify a paid plan. At scale on paid tiers, throughput is solid. Groq is a good fit when you need the lowest possible TTFT for a single-threaded flow and one of their supported models meets your requirements. ### Together AI Together AI is a GPU-based inference cloud with one of the widest open-source model catalogs of any managed provider. If a model exists and has significant community use, Together probably hosts it. They also support fine-tuning, which makes them useful if you want to train on Together's infrastructure and then serve the result there. Performance is generally good but not class-leading on latency. Their differentiation is breadth: model variety, fine-tuning support, and a developer experience that makes it easy to try different models in the same application. They also support some specialized endpoints, including image generation. Together is worth evaluating when you need a specific model that other providers don't carry, or when you want fine-tuning and inference on the same platform without managing separate pipelines. **API compatibility:** OpenAI-compatible. Model names differ, but the endpoint structure and SDK behavior align closely. ### Fireworks AI Fireworks AI is a GPU inference platform that has invested heavily in compound AI use cases -- applications that combine multiple model calls, tools, and retrieval steps in a single workflow. Their FireFunction models are specifically trained for function calling and tool use, and they've benchmarked well for structured output and tool call reliability. They support a solid range of open-source models and have been competitive on latency. Vision models are also supported via FireLLaVA and similar. Where Fireworks has carved out a distinct position is reliability for tool-calling-heavy applications. If your application makes a lot of function calls and you've had inconsistent results with other providers, Fireworks is worth testing. **API compatibility:** OpenAI-compatible. Tool calling and JSON mode both work as expected against the spec. ### Replicate Replicate is structured differently from the others. It's primarily a model hosting marketplace where anyone can publish a model and others can run it via API. This gives it an enormous selection -- thousands of models, including image generation, video, audio, and a long tail of fine-tuned text models. For text models specifically, the major open-source releases are there, but so are many community variants. The API is not OpenAI-compatible. Replicate has its own request/response format and a different SDK. Migrating to Replicate requires more code changes than switching between the other providers on this list. Pricing on Replicate is per-second of compute time for most models, which makes cost estimation harder than per-token pricing. For image and video generation, per-second pricing is standard and easy to reason about. For text inference, it's less intuitive. Replicate is a good fit when you need access to models that aren't hosted elsewhere, especially non-text modalities or highly specific fine-tuned checkpoints. ## Comparison Table | Provider | API Compatibility | Model Catalog | Latency Focus | Fine-Tuning | Best For | |---|---|---|---|---|---| | GeneralCompute | OpenAI-compatible | Broad (Llama 4, Qwen 3, DeepSeek, embeddings) | Sustained throughput at concurrency | No | Production apps, voice AI, agents | | Groq | OpenAI-compatible | Narrow (Llama, Gemma, Mixtral) | Single-request TTFT | No | Lowest latency for supported models | | Together AI | OpenAI-compatible | Very broad + community models | Good, not class-leading | Yes | Fine-tuning + serving, wide model selection | | Fireworks AI | OpenAI-compatible | Good, tool-calling-optimized models | Competitive | No | Tool-heavy applications, structured output | | Replicate | Own API/SDK | Enormous (including image/video/audio) | Varies by model | Yes (community) | Non-text modalities, niche fine-tuned models | ## Migrating from OpenAI For any provider with an OpenAI-compatible API (GeneralCompute, Groq, Together, Fireworks), the migration is two lines. **Python:** ```python from openai import OpenAI # Before (OpenAI) client = OpenAI(api_key="sk-...") # After (GeneralCompute) client = OpenAI( api_key="your-gc-api-key", base_url="https://api.generalcompute.com/v1" ) # After (Groq) client = OpenAI( api_key="your-groq-api-key", base_url="https://api.groq.com/openai/v1" ) # After (Together AI) client = OpenAI( api_key="your-together-api-key", base_url="https://api.together.xyz/v1" ) # After (Fireworks AI) client = OpenAI( api_key="your-fireworks-api-key", base_url="https://api.fireworks.ai/inference/v1" ) response = client.chat.completions.create( model="llama-4-maverick", # model names differ by provider messages=[{"role": "user", "content": "Hello"}] ) ``` **Node.js / TypeScript:** ```typescript import OpenAI from "openai"; const client = new OpenAI({ apiKey: process.env.GC_API_KEY, baseURL: "https://api.generalcompute.com/v1", }); const response = await client.chat.completions.create({ model: "llama-4-maverick", messages: [{ role: "user", content: "Hello" }], }); ``` The main thing to update beyond the base URL and API key is the model name. Each provider uses its own naming convention. A quick mapping for common OpenAI models: | OpenAI Model | Open-Source Equivalent | Notes | |---|---|---| | gpt-4o | llama-4-maverick | Comparable capability, faster on GeneralCompute | | gpt-4o-mini | llama-4-scout | Lower cost, fast generation | | gpt-3.5-turbo | llama-3.1-8b-instruct | Budget-friendly, fast | | text-embedding-3-small | nomic-embed-text | Check MTEB scores for your retrieval task | For Replicate, the migration is more involved since the API structure is different. Their Python SDK has a `replicate.run()` interface rather than the OpenAI-compatible completions format. If you're moving from OpenAI to Replicate specifically for a model they host, plan for a refactor of the API layer. ## How to Choose If your primary constraint is **latency at scale** -- you're building a voice assistant, a coding agent, or anything that stacks multiple inference calls -- evaluate GeneralCompute first. The sustained throughput under concurrency is the relevant benchmark, not single-request TTFT. If you need **the absolute lowest single-request latency** and one of Groq's supported models fits your requirements, Groq is worth testing. The narrow model catalog is the limiting factor. If you need **fine-tuning on the same platform** as serving, Together AI covers both. Their catalog is wide enough that you're unlikely to hit a model gap, and the fine-tuning pipeline is well-documented. If your application is **tool-calling-heavy** and you've had reliability issues with JSON mode or function call parsing elsewhere, Fireworks has invested in that use case specifically. If you need **a model that isn't hosted anywhere else** -- a specific community fine-tune, an image or video generation model, or something from the long tail -- Replicate is the right call. Accept the non-compatible API as a tradeoff. ## Validating Before You Commit Before migrating a production workload, run your own benchmarks. Use representative prompt lengths and concurrency levels that match your actual traffic. Measure TTFT, p50/p90/p99 TPS, and error rates under sustained load. Providers that look similar on a simple benchmark often diverge significantly when the traffic pattern gets realistic. Most providers on this list offer a free tier or trial credits. Running a short load test -- even a few hundred requests over a few minutes -- will surface queue behavior and rate limit interactions that you won't see in single-request tests. --- If you're evaluating GeneralCompute, the [API documentation](https://generalcompute.com/docs) covers setup, model names, and the full endpoint reference. You can also test latency directly from the playground before writing any code. --- # GeneralCompute vs Groq: Speed, Pricing, and Model Selection Compared > A developer's comparison of GeneralCompute and Groq covering inference speed, model availability, API pricing, and when to choose each platform. - Author: General Compute - Published: 2026-07-05 - Tags: inference, groq alternative, benchmarks, pricing, api - Canonical: https://www.generalcompute.com/blog/generalcompute-vs-groq --- If you're looking for a Groq alternative, or trying to decide between the two, this post covers what actually matters for production use: inference speed, model availability, pricing structure, API compatibility, and developer experience. Both GeneralCompute and Groq are purpose-built inference platforms. Neither runs models on standard GPU clouds -- both have invested in custom silicon designed specifically for fast LLM token generation. That shared approach makes the comparison genuinely interesting, which is why it comes up so often among developers evaluating inference providers. ## What Each Platform Is Built On Groq runs on its Language Processing Unit (LPU), a chip architecture designed around sequential computation rather than the massively parallel approach of GPUs. LPUs are optimized for the autoregressive nature of token generation, where each new token depends on all the previous ones. The result is very low latency per token on individual requests. GeneralCompute runs on custom ASIC infrastructure purpose-built for LLM inference. The design prioritizes both per-token latency and sustained throughput across concurrent requests. That distinction matters in production: a platform can look fast in a single-request benchmark and slow down significantly when handling many requests simultaneously. The architectural difference shapes the performance characteristics of each platform in ways that become apparent under realistic workloads. ## Inference Speed: What to Actually Measure Speed comparisons for inference APIs depend heavily on what you're measuring. There are two core metrics: - **Time to First Token (TTFT)**: How long from sending the request until the first token arrives. Matters for chat interfaces, voice AI, and any application where the user is waiting. - **Tokens per Second (TPS)**: How fast the model generates tokens after that first one. Matters for long-form generation and high-throughput workloads. There's a third metric that often goes unmeasured in published comparisons: **sustained throughput under concurrency**. A platform might benchmark well on a single isolated request but queue and slow down when multiple users hit it simultaneously. Groq performs well on TTFT for individual requests. Their LPU architecture is well-suited to this. Where things get more variable is at scale -- when you're sending many concurrent requests, queuing and rate limits introduce latency that single-request benchmarks don't surface. GeneralCompute's ASIC infrastructure is designed for sustained throughput across concurrent load. A coding assistant serving 100 developers at once has different requirements than a benchmark script running one request at a time. Platforms that optimize for the latter don't always hold up under the former. The most useful benchmark you can run is one that matches your actual traffic pattern: realistic prompt lengths, realistic concurrency, and measured at the 50th, 90th, and 99th percentile (not just average). Averages hide tail latency, and tail latency is what your users experience on bad requests. ## Model Selection This is where the two platforms diverge most noticeably. Groq offers a focused selection of models -- primarily Llama variants, Mixtral, Gemma, and a handful of others. They've prioritized making fewer models work exceptionally fast rather than supporting a broad catalog. If the model you need is on their list, that works well. If not, you're looking elsewhere. GeneralCompute supports a broader range of open-source models, including the latest Llama 4 variants (Scout and Maverick), Qwen 3, DeepSeek V3, and specialized models for coding, reasoning, and embeddings. The catalog depth matters when your application has specific model requirements -- Qwen3-Coder for a code generation feature, DeepSeek R1 for a reasoning task, or a specialized embedding model for retrieval. A few questions worth asking when evaluating model selection on any platform: 1. Does the platform support the specific model and checkpoint you need right now? 2. Are they running the full-precision model or a quantized version, and if quantized, which format and what's the quality impact? 3. How quickly does the platform add newly released models? The third point matters more than it might seem. When a significantly better model releases, you want to be able to swap it in without changing providers. Platforms that lag on new model support create friction every time the landscape shifts. ## API and SDK Compatibility Both platforms offer OpenAI-compatible APIs. If you're already using the OpenAI Python or Node.js SDK, switching the base URL and API key is all it takes to get started. **Python:** ```python from openai import OpenAI # GeneralCompute client = OpenAI( api_key="your-gc-api-key", base_url="https://api.generalcompute.com/v1" ) # Groq client = OpenAI( api_key="your-groq-api-key", base_url="https://api.groq.com/openai/v1" ) response = client.chat.completions.create( model="llama-4-maverick", messages=[{"role": "user", "content": "Explain KV cache"}] ) ``` **Node.js:** ```typescript import OpenAI from "openai"; const client = new OpenAI({ apiKey: process.env.GENERALCOMPUTE_API_KEY, baseURL: "https://api.generalcompute.com/v1", }); ``` Both work with LangChain, LlamaIndex, Vercel AI SDK, and other frameworks that support OpenAI-compatible endpoints -- no custom client code required. Where they differ is in which features beyond core completions are supported. Streaming, function calling, JSON mode, and embeddings support vary in implementation depth between providers. Check the specific endpoints your application uses before committing. "OpenAI-compatible" covers a range of completeness, and edge cases (like parallel tool calls or multi-modal inputs) sometimes aren't fully covered. ## Pricing Inference pricing changes frequently enough that specific numbers here would be stale within months. Instead, here's the framework for comparing providers. **Input vs output token pricing**: Both platforms charge separately for input and output tokens. Output tokens cost more because they require sequential computation that can't be parallelized. Models used for long-form generation or multi-step reasoning cost meaningfully more per request than models used for short classification tasks, even at the same per-token rate. **Rate limits and capacity**: Groq's free tier is generous for prototyping, but production workloads often hit rate limits before cost becomes the primary concern. Verify the concurrency and requests-per-minute limits at each pricing tier before you're relying on the API in production. **Reserved vs shared capacity**: Some providers offer dedicated capacity options where your requests don't compete with other users for compute. For latency-sensitive applications, this can matter more than the per-token price. **Volume discounts**: At scale, headline per-token rates matter less than whether the provider offers volume pricing. Check what the effective cost looks like at your projected monthly token volume. For current pricing on both platforms, check their respective documentation. The numbers shift often enough that a comparison table would mislead more than it would help. ## Developer Experience and Documentation Documentation quality shows up in production debugging. Both platforms have improved here, but they're at different points. Groq's documentation covers the quickstart path well. The API reference handles the main endpoints. It gets thinner in edge cases: behavior at rate limits, specific error codes, and best practices for high-throughput applications aren't as thoroughly covered. GeneralCompute's documentation covers the OpenAI-compatible API, model-specific guides, and integration examples for major frameworks. The emphasis is on production deployment patterns -- not just how to make the first API call, but how to handle concurrency, manage context, and optimize for specific use cases like voice AI or coding assistants. Both platforms have Discord communities for developer support. Enterprise tiers with SLAs are available on both. If uptime commitments and response time guarantees matter for your application, verify what's included at each support tier before signing up. ## Rate Limits and Availability This practical concern often doesn't come up in comparisons but matters in production. Groq's free tier works well for development. Production workloads hit ceilings quickly, and the upgrade path to higher limits involves moving to paid tiers with specific concurrency caps. Verify the exact limits for your expected request volume at each pricing level. GeneralCompute offers dedicated capacity options for high-throughput workloads. This sidesteps the "noisy neighbor" problem where your latency increases when the shared platform is under heavy load from other users. For applications where consistent latency is a hard requirement -- voice AI, real-time coding assistants, anything with a user waiting -- predictable capacity matters more than peak theoretical speed. ## When to Choose Groq Groq makes sense when: - The model you need is in their supported catalog - Your workload is low-concurrency or you're benchmarking individual requests - You're prototyping and the free tier covers your volume - You're already integrated and the switching cost isn't justified by the expected gain ## When to Choose GeneralCompute GeneralCompute is the better fit when: - You need a broader model selection, including newer releases and specialized models - Your workload involves high concurrency across many simultaneous users - You're building voice AI or real-time applications where sustained throughput matters - You want dedicated capacity with predictable latency under load - The model you need isn't on Groq's supported list ## Migrating Between the Two Moving from Groq to GeneralCompute (or vice versa) is straightforward given the shared OpenAI-compatible API surface: 1. Generate an API key from the new provider 2. Update your `base_url` and `api_key` environment variables 3. Map model names (identifiers differ between providers) 4. Run your test suite against the new endpoint The main issue is model naming. Groq uses identifiers like `llama3-70b-8192` while GeneralCompute uses names like `llama-4-maverick`. Keep your model name in an environment variable rather than hardcoded so you can swap it without code changes. ```python import os from openai import OpenAI MODEL = os.getenv("LLM_MODEL", "llama-4-maverick") BASE_URL = os.getenv("LLM_BASE_URL", "https://api.generalcompute.com/v1") API_KEY = os.getenv("LLM_API_KEY") client = OpenAI(api_key=API_KEY, base_url=BASE_URL) ``` This pattern makes it easy to A/B test providers or swap based on availability. ## The Decision Framework For most teams, the right approach is to test both with your actual workload before committing. Run a few hundred requests with realistic prompt lengths and concurrency levels. Measure TTFT and TPS at the 50th and 99th percentiles. Compare against your cost budget at expected monthly volume. If you're still prototyping and don't yet have production traffic patterns to benchmark against, either platform will get you started quickly. Once you have real data on what your application needs, the choice becomes concrete. ## Getting Started with GeneralCompute If you want to evaluate GeneralCompute as a Groq alternative, setup takes about five minutes. Sign up at generalcompute.com, grab an API key, and update your base URL. No client library changes, no schema migration -- same OpenAI SDK, different endpoint. The [GeneralCompute docs](https://generalcompute.com/docs) cover model selection, streaming, function calling, and optimization guides for specific use cases including coding assistants, voice AI pipelines, and high-throughput batch workloads. --- # Scaling Laws in LLMs: What They Mean for Inference Cost in Production > A practical breakdown of Chinchilla and Kaplan scaling laws, what they predict about model quality and size, and how to use that knowledge to make smarter inference cost decisions. - Author: General Compute - Published: 2026-07-04 - Tags: scaling laws, inference, LLMs, cost optimization, production AI, model selection - Canonical: https://www.generalcompute.com/blog/scaling-laws-llms-inference-cost-production --- Scaling laws are the closest thing machine learning has to first principles. They describe a predictable relationship between model size, training data, compute budget, and the quality of the resulting model. If you're choosing which model to run in production or trying to estimate what a smaller model will cost you in accuracy, understanding scaling laws will save you a lot of expensive trial and error. This post covers what the major scaling law papers actually say, how to interpret them for inference decisions (not just training), and what the shift toward smaller, more compute-optimal models means for your cost-per-query at scale. ## What Scaling Laws Say (and What They Don't) The foundational work here is the 2020 paper from Kaplan et al. at OpenAI, often called the "GPT-3 scaling laws paper." The core finding: language model loss improves as a power law with respect to three variables -- model size (number of parameters), training compute (total FLOPs), and dataset size (number of tokens). Double any of these while holding the others fixed and you get a predictable, measurable improvement in perplexity. Crucially, the Kaplan paper found that model size mattered more than dataset size when you had a fixed compute budget. Their recommendation: train very large models on relatively fewer tokens. This reasoning drove the design of GPT-3 (175B parameters, trained on roughly 300B tokens). Then in 2022, DeepMind published "Training Compute-Optimal Large Language Models" -- commonly called the Chinchilla paper. Hoffmann et al. trained over 400 models across a wide range of sizes and token counts and reached a different conclusion: **Kaplan's models were significantly undertrained**. The Chinchilla result: for a fixed compute budget, you should train a model with roughly 20 tokens per parameter. A 10B parameter model should see 200B tokens. A 70B model should see about 1.4 trillion tokens. GPT-3 at 175B was trained on 300B tokens. By Chinchilla's math, a 70B model trained on the same compute budget would outperform it, because you're spending those FLOPs more efficiently. Chinchilla (the actual model) proved this out, matching or beating models two to four times its size on benchmarks. Llama 2 and Llama 3, as well as Mistral's models, followed the same principle -- train smaller models longer on more tokens. ## The Training vs. Inference Trade-off Here's where scaling laws connect directly to your production infrastructure costs. Kaplan's original framing was about training efficiency: given a compute budget, how do you minimize loss? But inference has a very different cost structure. When you deploy a model, you pay per forward pass. The cost of a single inference call scales roughly linearly with the number of parameters -- a 70B model costs roughly twice as much per token to serve as a 35B model at similar batch sizes. So the Chinchilla result has a practical implication that goes beyond benchmarks: smaller compute-optimal models are cheaper to run and, for the same training compute spent, are equally capable. If Chinchilla-style training means a 7B model can do what a 30B undertrained model could do, your inference costs drop by 4x. This is why the open-source ecosystem largely moved toward Chinchilla-optimal training after 2022. Llama 2 7B was trained on 2 trillion tokens. Llama 3 8B was trained on 15 trillion. These models punch above their weight in benchmarks precisely because they follow better compute allocation. ## How to Use Scaling Laws for Model Selection The practical question for most teams is: given the task, what's the smallest model I can get away with? Scaling laws give you a useful mental model here, but they're not a substitute for benchmarking. What they tell you: **Capability scales with effective compute, not raw parameters.** A 7B model trained on 15T tokens likely outperforms a 13B model trained on 1T tokens. When comparing models from different training runs, parameter count alone is a poor proxy for quality. **Task difficulty matters.** Scaling laws predict average loss on a language modeling objective. Your specific task -- code generation, structured output, multi-step reasoning -- may not scale the same way. Coding and math tasks tend to require more parameters to solve reliably, even holding training compute constant. **Beyond a certain scale, gains flatten.** The power law curves are not linear -- they have diminishing returns. Going from 7B to 70B often gives you meaningful gains on hard reasoning tasks. Going from 70B to 700B gives you much less, for ten times the inference cost. Unless you specifically need frontier-model capabilities, you're often paying for marginal improvements. A reasonable decision framework: - For classification, extraction, summarization, or simple Q&A: a well-trained 7B or 8B model is usually sufficient and costs a fraction of a 70B model to serve. - For complex reasoning, multi-step planning, or tasks that require broad knowledge: 30B to 70B range is the practical sweet spot. - For frontier tasks that require SOTA performance: you're looking at 70B+ dense models or large MoE models like DeepSeek V3, which achieve high capability through sparse routing rather than dense scale. ## Small Language Models and the Production Case One consequence of Chinchilla-optimal training is that "small" doesn't mean "bad" anymore. Models like Phi-3 Mini (3.8B), Qwen 2.5 3B, and Llama 3.2 3B are trained on enormous token counts relative to their size. On many benchmarks they match models that were considered large just two years ago. For production inference, this matters a lot. Small models have: **Lower latency.** Fewer parameters mean fewer matrix multiplications per token. At the same hardware, a 3B model generates tokens roughly five to ten times faster than a 70B model. If you're building voice AI or anything with real-time requirements, this isn't a nice-to-have -- it's required. **Higher throughput.** You can fit more concurrent requests on the same GPU. A 3B model on a single A100 can handle 50+ concurrent sequences; a 70B model needs multiple GPUs just to load the weights. **Lower cost per query.** Combining lower hardware requirements with higher throughput, the effective cost per 1M tokens for a 3B model is roughly 10 to 20 times cheaper than a 70B model on equivalent hardware. The practical catch is that small models fail more on hard tasks. If your application involves complex multi-step reasoning, legal analysis, nuanced code generation, or long-horizon planning, you'll likely see meaningful quality degradation at 3-8B. Benchmark your actual task, not just generic leaderboard scores. ## Inference Cost Estimation Using Scaling Law Intuitions You can use scaling law thinking to estimate what you'll pay before you run a single query. The cost of a forward pass through a transformer scales roughly as: ``` cost ≈ 2 * num_parameters * num_tokens ``` in FLOPs (the factor of 2 accounts for multiply-accumulate operations). At a fixed hardware throughput (tokens/second/GPU), you can convert this to latency and cost. For rough budget planning: | Model Size | Relative FLOP Cost per Token | Typical TPS (single A100) | |------------|------------------------------|---------------------------| | 3-8B | 1x | 80-150 tok/s | | 13-14B | 2-3x | 40-70 tok/s | | 30-34B | 5-6x | 15-30 tok/s | | 70B | 10x | 8-15 tok/s | | 405B+ | 50x+ | 1-3 tok/s (multi-GPU) | These are rough figures that vary significantly with quantization, batch size, and hardware. But the relative ratios hold up well enough for planning. If you're generating 100M tokens per day: - At 7B (with good inference infra): you might pay $0.10-0.20 per 1M tokens - At 70B: that same output costs 5-10x more in compute At scale, the model size choice is one of the highest-leverage cost decisions you'll make. ## Quantization and the Effective Scaling Budget Scaling laws assume you're running models in full precision (typically BF16 or FP16). Quantization complicates this but in a mostly favorable direction. INT4 or INT8 quantization reduces memory bandwidth requirements, which is often the actual bottleneck in inference (not raw compute). This means you can run a larger model at a given throughput target, effectively shifting the cost curve. A 70B model quantized to INT4 fits in roughly 40GB of VRAM -- a single high-end GPU. Running it at INT4 costs about the same per token as running a 30B model at FP16. So quantization partially offsets the cost disadvantage of larger models, as long as your quality tolerance includes the small perplexity hit from reduced precision. The practical takeaway: when comparing models for production deployment, compare: 1. The benchmark results at your target quantization level (not just FP16) 2. The throughput numbers at that quantization level on your target hardware 3. The quality on your actual task, not just generic benchmarks Scaling laws predict quality at training precision. Quantization introduces a small, predictable degradation that's usually worth the infrastructure savings. ## What "Compute-Optimal" Means at Inference Time There's an emerging extension of Chinchilla reasoning to inference: the idea of "inference-optimal" models, sometimes called test-time compute scaling. The basic insight: if you know how many inference calls you'll make, you can treat inference FLOPs as part of your total compute budget and optimize accordingly. A smaller model that you call five times might be cheaper than a large model called once, if the aggregated outputs reach the same quality. This is one motivation behind chain-of-thought prompting and reasoning models. Llama 3 8B with a careful reasoning prompt can sometimes match a 70B model on specific tasks, at a fraction of the inference cost -- though this doesn't generalize to all tasks and adds latency from longer output sequences. The more principled version of this is speculative decoding, where a small draft model generates candidate tokens and a larger verifier model checks them in parallel. The draft model does most of the work; the verifier runs less often. You get output that matches the larger model's quality at throughput closer to the small model's speed. This is a direct application of inference-budget thinking derived from scaling law intuitions. ## Practical Recommendations If you're making model selection decisions for a production system, here's how to apply this: **Don't treat parameter count as a quality signal in isolation.** A Llama 3 8B outperforms many older 30B models. Check training token counts and benchmark results for your specific task category. **Start small and move up only when you see quality failures.** Deploy a 7-8B model first. Measure task performance on a representative sample. Escalate to 30-70B only for the failure cases, using a router if needed. **Factor in quantization from the start.** INT8 is nearly lossless for most tasks. INT4 is acceptable for many workloads. Plan your memory budget around quantized model sizes so you're not over-provisioning hardware for FP16 weights you don't need. **Benchmark your actual task.** Scaling laws predict average capability on language modeling. Your task -- structured JSON extraction, code completion with a specific style guide, domain-specific summarization -- may behave very differently from MMLU or HumanEval. Run evals on real examples before committing to a model. **Consider total cost of ownership.** A smaller model at higher request rates will sometimes require more instances than a larger model with caching and batching optimizations. Model the full system, not just the per-token cost. --- If you want to run these experiments without managing your own GPU cluster, [General Compute's API](https://generalcompute.com) gives you access to a range of open-weight models -- from sub-10B to 70B+ -- on fast inference infrastructure with OpenAI-compatible endpoints. You can benchmark models side by side, measure latency at your target throughput, and find the right size for your workload before committing to a deployment decision. --- # Mixture of Experts (MoE) Models: Why They're Dominating 2025 > MoE models activate only a fraction of their parameters per token, giving you large-model quality at a fraction of the compute cost. Here's how the architecture works, why it won, and what the load balancing challenges actually look like. - Author: General Compute - Published: 2026-07-03 - Tags: moe models, mixture of experts, architecture, deep learning, inference, mixtral, deepseek - Canonical: https://www.generalcompute.com/blog/mixture-of-experts-moe-models-why-theyre-dominating-2025 --- If you look at the models that have dominated benchmark leaderboards over the past two years, most of them are Mixture of Experts (MoE) architectures. Mixtral, DeepSeek V3, Grok-1, Llama 4 Maverick -- all MoE. The architecture has become the default for anyone building at the frontier, and there's a clear reason why: MoE lets you scale model capacity without scaling compute proportionally. This post explains how MoE works, what the gating and routing mechanics look like in practice, and where the hard problems actually are. If you've seen the term and want to understand what's really happening under the hood, this covers it. ## The core idea: conditional computation A standard transformer processes every token with the same set of parameters. Every token passes through the same attention heads and the same feed-forward network (FFN). This means scaling quality requires scaling compute, roughly linearly. MoE breaks that constraint by making parts of the network conditional. Instead of one shared FFN, an MoE layer has N separate FFN networks called "experts," plus a small router network that decides which experts each token uses. For any given token, only k of the N experts are active. The others don't run. You get a model with far more total parameters, but each forward pass only touches a fraction of them. This is sometimes called "sparse" computation, in contrast to the "dense" computation of a standard transformer. The sparsity is what creates the efficiency: you can make the model much larger without making it proportionally slower. ## How the router and gating work The router is usually a simple single linear layer. It takes the token's hidden state as input and outputs a logit score for each expert. These scores determine which experts handle the token. The selection process looks like this: ```python gate_logits = router(hidden_state) # [num_experts] top_k_indices = torch.topk(gate_logits, k=2).indices top_k_weights = F.softmax(gate_logits[top_k_indices], dim=-1) # output is a weighted sum of the chosen experts output = sum( top_k_weights[i] * experts[top_k_indices[i]](hidden_state) for i in range(k) ) ``` The router runs on every token, at every MoE layer, in every forward pass. The expert outputs are combined using the gating weights (the softmax scores), so the contribution of each expert is proportional to how strongly the router scores it for that token. Most practical MoE models use k=2, meaning two experts are active per token per layer. Some recent fine-grained designs use higher k with smaller experts, but the principle is the same: a small number of experts activate, the rest are skipped. The attention layers in an MoE model are almost always kept dense -- every token still goes through the same attention computation. The sparsity applies only to the FFN portion. In modern transformers, the FFN accounts for two-thirds or more of the total parameter count, so this is where most of the savings come from. ## Why this produces better models The interesting result, confirmed across many experiments, is that MoE models consistently outperform dense models of equivalent active-parameter count. A model with 8B active parameters arranged as MoE will often beat a 8B dense model on standard benchmarks. The intuition is that different experts can specialize. Over training, the router learns to direct certain types of tokens or patterns to experts that have become good at handling them. One expert might specialize in code, another in mathematical reasoning, another in natural language. The model develops more diverse internal representations than a dense model of the same active size. This specialization is emergent, not explicit. You don't assign experts to domains. The routing and the expert weights are learned jointly, and specialization arises from gradient flow. There's evidence of it in post-hoc analyses (you can sometimes identify expert specialization by measuring what input distributions each expert activates on), but it's not guaranteed to be clean or interpretable. The practical implication is that you can match the quality of a dense 70B model with an MoE model that has 70B active parameters at most -- but you can spread those "total" parameters across many more experts, increasing total capacity significantly. DeepSeek V3 pushes this hard: 671B total parameters, 37B active per token. It consistently matches or beats models with much larger active-parameter counts. ## The models that made MoE mainstream **Mixtral 8x7B** (Mistral, 2023) was the first widely adopted open MoE model. It has 8 experts per layer and activates 2, giving it 46.7B total parameters with 12.9B active per token. It outperformed Llama 2 70B on most benchmarks while being much cheaper to serve at comparable quality. This demonstrated that MoE was practical to deploy and not just a research curiosity. **Mixtral 8x22B** followed in 2024, scaling to 141B total parameters with 39B active. It pushed the quality ceiling further while keeping active-parameter compute manageable. **Grok-1** (xAI, 2024) was released as an open-weights model with 314B total parameters and 86B active per token. It uses 8 active experts out of 64, making it one of the largest open-weights MoE models at the time of release. **DeepSeek V3** is currently the most influential MoE model in production. 671B total parameters, 37B active, with 256 routed experts plus 2 shared experts per layer. The fine-grained routing (many small experts rather than few large ones) combined with hardware-aware training let it match or beat GPT-4-class models on most benchmarks while being significantly cheaper to train. DeepSeek's architecture paper was heavily studied for the engineering choices it documents. **Llama 4** adopted MoE across its Maverick and Scout variants. Maverick has 400B total parameters with 17B active per token. This marked MoE's full arrival into Meta's core model family, which had been dense through Llama 3. The pattern across all these models is similar: MoE was chosen specifically to increase total capacity beyond what was practical with dense architectures at the available compute budget. ## Training challenges MoE models are harder to train than dense models, and the main difficulty is load balancing. If the router learns to prefer a small subset of experts for most tokens, those experts receive more gradients and improve faster. The other experts get fewer gradients, improve slower, and the router has even less reason to route to them. This is a collapse mode called "expert collapse" or "routing collapse," and it's a real failure case in MoE training. The standard solution is an auxiliary load-balancing loss added to the training objective: ```python # fraction of tokens routed to each expert router_probs = softmax(gate_logits, dim=-1) # [batch, num_experts] fraction_per_expert = router_probs.mean(dim=0) # target: uniform distribution (1/num_experts per expert) load_balance_loss = num_experts * (fraction_per_expert * fraction_per_expert).sum() total_loss = task_loss + alpha * load_balance_loss ``` The coefficient alpha controls how strongly you enforce balance. Too low and experts collapse; too high and the routing loses expressivity because the router is forced to distribute uniformly regardless of input, defeating the purpose of specialization. Getting this balance right is one of the main hyperparameter challenges in MoE training. DeepSeek V3 used an expert-level bias term that adjusts routing probabilities based on recent expert utilization history, which allowed them to maintain balance without aggressively penalizing the task loss. Training stability is also harder. The sparse routing creates high variance in gradient estimates, particularly for the router. Careful initialization, gradient clipping, and monitoring of routing statistics (which experts are active, how uniform the distribution is across the batch) are all more important than in dense training. ## Inference and load balancing challenges At inference time, load balancing takes a different form. During training you can compute the auxiliary loss and push the router toward uniformity. At inference time, you just have the trained router making decisions for whatever input comes in. If your workload happens to trigger many tokens that all route to the same expert, some GPUs sit idle while one is overwhelmed. For large MoE models served with expert parallelism (each GPU holds a subset of experts), this shows up as throughput instability. The all-to-all communication that ships tokens to their assigned expert GPUs becomes uneven, and the latency of a batch is determined by the slowest expert, not the average. Production serving addresses this several ways: - **Capacity capping**: each expert accepts only up to a capacity factor times the average load. Tokens that exceed the cap get dropped to their second-choice expert or skipped entirely. This caps worst-case latency at the cost of a small quality hit. - **Auxiliary routing tables**: some serving stacks maintain a soft router that can redirect traffic away from overloaded experts in real time. This adds complexity but can smooth out load spikes. - **Batching strategy**: with large enough batches and diverse enough inputs, the law of large numbers tends to even out routing. Single-request serving at low batch sizes sees more variance; high-throughput batch serving sees more natural balance. In practice, the load balancing issue is real but manageable. The serving frameworks (vLLM, SGLang, TensorRT-LLM) have all invested in efficient dispatch kernels that minimize the overhead from irregular routing patterns. The main signal to watch is per-expert token counts within a batch -- if they're highly non-uniform, you'll see throughput well below theoretical peak. ## Memory vs. compute trade-off One thing that catches people off guard: MoE models are not smaller in memory than their dense equivalents. All the experts have to be loaded into GPU memory, even if only 2 of 256 are active for any given token. DeepSeek V3 at 671B parameters requires roughly the same GPU memory as a 671B dense model would. The economics work because the compute savings are large. A 671B MoE model with 37B active parameters does about 37B/671B = 5.5% of the FLOPs per token of a theoretical 671B dense model. The memory footprint stays large; the compute footprint drops dramatically. For throughput-bound workloads -- where you're processing many requests and the bottleneck is GPU compute rather than memory bandwidth -- this is a good trade. For latency-bound workloads at very low batch sizes, the trade is less attractive. You're paying full memory bandwidth cost on every decode step regardless of which experts activate, because the attention layers are still dense. The per-token compute savings help, but they don't fully offset the memory bandwidth cost of the large model. This is why you'll see MoE models favored for high-throughput serving scenarios and why dense models remain competitive at small scale. A 7B dense model outpaces a 671B MoE model on a single GPU for interactive, single-user latency because it's simply much smaller and faster to move through memory. ## Where MoE fits in the architecture landscape The practical picture in 2025 is that MoE has become the default for frontier-scale models. If you're training a model with more than a few hundred billion parameters and you want to be compute-efficient, MoE is the expected choice. Dense models at that scale are too expensive to train and too slow to serve. For smaller models, dense is still dominant. The routing overhead, training complexity, and memory requirements of MoE don't pay off below roughly 30B total parameters. The sweet spot for MoE is models where you want 70B+ quality but can't afford to activate 70B parameters per token. The interesting design space is in the middle: models in the 30B to 150B total parameter range, where MoE and dense are both viable and the tradeoffs are genuine choices. Here the routing granularity (few large experts vs. many small experts), the activation ratio, and the serving infrastructure all matter for how the model performs in production. ## Starting with MoE models If you want to use MoE models without setting up the infrastructure yourself, General Compute runs DeepSeek V3 and Llama 4 Maverick on production inference hardware with expert parallelism and the dispatch kernels already in place. The API is OpenAI-compatible; switching from a dense model is a one-line change. ```python from openai import OpenAI client = OpenAI( api_key="your-api-key", base_url="https://api.generalcompute.com/v1" ) response = client.chat.completions.create( model="deepseek-v3", # 671B MoE, 37B active messages=[{"role": "user", "content": "Explain the gating mechanism in MoE models."}] ) print(response.choices[0].message.content) ``` For deeper coverage of how MoE behaves specifically at inference time -- including the all-to-all communication, per-layer dispatch, and expert parallelism topology -- see [Mixture of Experts at Inference Time](/blog/mixture-of-experts-at-inference-time). You can get started at [generalcompute.com](https://generalcompute.com). --- # KV Cache in LLM Inference: How It Works and Why It Matters > The KV cache is one of the most important mechanisms in transformer inference. This post explains what it is, how prefill and decode phases use it, how paged attention manages it, and what MQA/GQA and quantization do to shrink it. - Author: General Compute - Published: 2026-07-02 - Tags: kv-cache, inference, attention, transformers, performance - Canonical: https://www.generalcompute.com/blog/kv-cache-in-llm-inference-how-it-works-and-why-it-matters --- Every time a transformer generates a token, it needs to attend to every token that came before it. Without any optimization, this means recomputing the key and value vectors for the entire history on every single step. At sequence length 1,000, generating one token requires recomputing 1,000 sets of keys and values across every layer. At 10,000 tokens, the compute cost becomes prohibitive. The KV cache solves this problem by storing those key and value vectors after they're first computed, so subsequent steps can reuse them. The name comes from the K (key) and V (value) matrices in the attention mechanism. The Q (query) matrix does not need to be cached because it's only ever computed for the current token. Understanding the KV cache is useful because it sits at the center of most of the important tradeoffs in LLM serving: memory vs. compute, throughput vs. latency, batch size vs. context length. ## Attention and Why K and V Are Special In a standard multi-head attention layer, each token's hidden state produces three vectors: a query Q, a key K, and a value V. The attention score for the current token is computed by taking the dot product of its Q with the K of every other token in the sequence, passing through softmax, then using those scores to weight-sum the V vectors. The result is a new hidden state for the current token. During autoregressive generation, you're computing one token at a time. Each new token needs to produce its Q and attend to all previous Ks, then use all previous Vs to form its output. Crucially, the Ks and Vs from previous tokens don't change between steps -- they depend only on the token and its position in the sequence. There's no reason to recompute them. The KV cache stores each token's K and V vectors after they're computed, indexed by layer and position. When generating the next token, you compute Q, K, V for just the new token, append the new K and V to the cache, and run attention over the full cached K and V sequences. Compute per decode step stays constant regardless of sequence length. ## Prefill and Decode: Two Different Problems LLM inference has two distinct phases, and the KV cache behaves differently in each. **Prefill** is the processing of the input prompt. The full prompt is available at once, so the model processes all tokens in parallel using regular matrix operations. At the end of prefill, the KV cache is populated with one entry per input token, per layer. Prefill is compute-bound: you're doing a lot of matrix multiplications in parallel and GPU utilization is high. **Decode** is the generation phase, where tokens are produced one at a time. Each decode step extends the KV cache by one entry and runs attention over the entire cached sequence. Decode is memory-bandwidth-bound, not compute-bound. The arithmetic intensity is very low: read a large cache from HBM, do a small amount of computation, write one new entry. GPU utilization during decode is far lower than during prefill, especially for single-request workloads. This asymmetry matters for serving. A user submitting a 32K-token document and requesting a 1,000-token summary will see a prefill phase that takes a fraction of the total time, followed by a decode phase that stretches out. Time-to-first-token (TTFT) is dominated by prefill. Token generation speed (tokens per second after the first token) is a decode concern. ## How Much Memory Does the KV Cache Use? The cache size for a single request depends on the model architecture and context length: ``` cache_bytes = 2 * num_layers * num_kv_heads * head_dim * seq_len * dtype_bytes ``` The factor of 2 accounts for K and V. For Llama 3 70B with 80 layers, 8 KV heads (it uses GQA, covered below), head dimension 128, FP16, and a 32K context: ``` 2 * 80 * 8 * 128 * 32,000 * 2 bytes = ~10.5 GB per request ``` A single H100 with 80GB of HBM can hold the model weights plus a small number of concurrent requests at that context length. Scale to 128K context and the per-request cache more than quadruples. This is the binding constraint in long-context serving, often more limiting than the weights themselves. For a smaller model like Llama 3 8B (32 layers, 8 KV heads, 128 head dim), the cache at 32K context in FP16 is: ``` 2 * 32 * 8 * 128 * 32,000 * 2 bytes = ~4.2 GB per request ``` Still substantial relative to the 8B model's roughly 16GB weight footprint at FP16. And this scales linearly with context: double the context length, double the cache. ## Paged Attention: Managing the Cache Efficiently A naive KV cache implementation pre-allocates memory per request based on the maximum possible context length. This wastes memory when requests end up shorter than expected, creates fragmentation when requests finish at different times, and prevents sharing memory across requests with common prefixes (like system prompts). Paged attention, introduced in the vLLM paper, borrows the virtual memory paging concept from operating systems. Instead of a contiguous block per request, the cache is divided into fixed-size pages. Each page holds a fixed number of tokens' K and V vectors for a given layer. A page table maps logical positions (request ID, layer, token position) to physical pages in a shared pool. The practical benefits are meaningful: - No per-request pre-allocation. Pages are assigned as needed, one at a time. - No wasted memory from early termination. Pages from a finished request return to the pool immediately. - Prefix sharing. Multiple requests with the same system prompt can share the same physical pages for that prefix, with copy-on-write semantics if they diverge. Modern serving frameworks -- vLLM, SGLang, TensorRT-LLM -- all implement paged attention or a close variant. It's effectively the standard for production serving. The attention kernel has to handle non-contiguous memory, which requires careful implementation, but the serving efficiency gains justify it. ## MQA and GQA: Fewer KV Heads The most direct way to shrink the KV cache is to have fewer keys and values to store. Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) do this by reducing the number of independent KV heads while keeping query heads at full count. In standard Multi-Head Attention (MHA), each attention head has its own Q, K, and V. With 64 heads, you store 64 K vectors and 64 V vectors per token per layer. MQA (Shazeer, 2019) uses a single shared K and V pair across all query heads. You keep all 64 Q heads for expressivity, but cache only one K and one V. That's a 64x reduction in the KV cache, with a measurable quality regression on harder benchmarks. GQA (Ainslie et al., 2023) is the practical middle ground: group query heads into g groups, with one shared K and V per group. Llama 3 uses 64 query heads and 8 KV heads (g = 8), giving an 8x cache reduction versus MHA. This preserves most of MHA's quality while making long-context serving economically viable. Essentially every modern open model uses GQA at this point. The quality tradeoff comes from expressivity. In full MHA, each head independently specializes its keys and values. Sharing K and V across heads reduces that specialization. GQA lands in a range where the quality impact is small enough to be acceptable for almost all production workloads. For a deeper look at how DeepSeek's Multi-Head Latent Attention (MLA) extends this idea using low-rank K/V projection for even larger cache reductions, see [KV Cache Compression: MLA and Beyond](/blog/kv-cache-compression-mla-and-beyond). ## KV Cache Quantization Rather than changing the architecture, KV cache quantization keeps the same number of heads but stores them at lower precision. FP16 is the standard baseline, but the cache is often a good candidate for quantization because the quality sensitivity to precision is lower than for the model weights themselves. **FP8** is the current production default where hardware supports it. Hopper-generation GPUs (H100, H200) have native FP8 support in their tensor cores, so you get 2x compression versus FP16 with essentially no dequantization overhead. The attention kernel can operate directly on FP8 values. Most modern serving frameworks support FP8 KV caches out of the box, and the quality impact is negligible for typical workloads. **INT8** gives the same 2x compression and works on older hardware, but the attention kernel has to dequantize on the fly. The overhead is small, and INT8 KV caches are well-supported across frameworks. **INT4 and lower** are possible with careful per-group quantization schemes. KIVI (2024) demonstrated 2-bit quantization with per-group scales and acceptable quality loss. The memory savings are substantial (8x versus FP16), making ultra-low-bit caches worth watching for deployments with very long contexts or very high concurrency. Mainstream frameworks don't yet ship this by default. KV cache quantization composes with GQA and paged attention. You can use GQA to have fewer heads, quantize those heads to FP8, and manage memory allocation with paged attention. The compression ratios stack multiplicatively. ## What This Means in Practice When you're designing a serving setup, the KV cache is often the binding constraint rather than the model weights. Here's why: model weights are static. You load them once and they stay in memory. The KV cache is dynamic -- it grows with context length, varies by request, and competes with other concurrent requests for the same HBM pool. The number of concurrent requests you can serve depends on: - The maximum context length you support - The number of KV heads in the model (model architecture) - The dtype you're using for the cache (quantization) - How efficiently you're managing memory (paged vs. contiguous allocation) If your model supports 128K context but your average request uses 4K, contiguous pre-allocation wastes most of your memory budget. Paged attention fixes this. If you're on an H100 and using FP16 for the KV cache, switching to FP8 doubles your effective cache capacity at no quality cost. If you're selecting a base model and haven't checked whether it uses GQA, the cache size could be 8x larger than necessary. Long-context deployments amplify all of this. A model serving 1M-token contexts needs very careful cache management -- the per-request footprint can exceed the weight footprint by a large margin. This is the engineering problem that motivated both MLA in DeepSeek-V3 and the focus on FP8 KV caches across the industry. Here's a rough comparison to make the stakes concrete. Suppose you're serving a 70B model on a node with 640GB of HBM (8xH100) and want to support 32K context: | Configuration | Cache per request | Concurrent requests at 50% HBM budget | |---|---|---| | MHA, FP16 | ~84 GB | ~3 | | GQA (8 KV heads), FP16 | ~10.5 GB | ~30 | | GQA, FP8 | ~5.2 GB | ~61 | The model weights take roughly 140GB at BF16 (or less with quantization), leaving around 320GB for cache if you budget 50% for it. The table above shows how architecture and quantization choices translate directly into serving capacity. ## Putting It Together The KV cache is what makes autoregressive generation tractable at practical sequence lengths. Without it, each decode step would require recomputing all prior context from scratch. With it, each step reads the existing cache from memory, appends one entry, and moves on. Compute per step is constant; memory grows linearly. The techniques built on top -- paged attention, GQA, quantization -- are all about using that memory more efficiently. They allow more concurrent requests, longer contexts, or both, within the same hardware budget. They don't change the fundamental mechanics, just the economics. If you're debugging why inference costs more than expected at longer context lengths, or why throughput drops off when concurrency increases, measuring KV cache utilization is usually the right place to start. The numbers often explain the bottleneck more directly than any other metric. General Compute's infrastructure is designed for high-throughput, low-latency inference across long contexts. If you're working with memory-intensive serving workloads and want to evaluate the tradeoffs, you can get started with the API at [generalcompute.com](https://generalcompute.com). --- # Agentic AI in Production: Inference Requirements for Multi-Step Workflows > Building a single agentic AI demo is straightforward. Scaling it to production -- with observability, error handling, rate limits, and hundreds of concurrent agents -- requires thinking carefully about your inference layer from the start. - Author: General Compute - Published: 2026-07-01 - Tags: agentic system, ai agents, inference, production, observability, rate limiting, scaling - Canonical: https://www.generalcompute.com/blog/agentic-ai-in-production-inference-requirements --- The difference between a working agentic AI demo and a reliable production system is mostly invisible on the happy path. Both complete tasks, both produce reasonable outputs, both look great in a demo. The problems show up under load, at the edges, and in the places you didn't instrument. This post covers what actually changes when you move an agentic system from prototype to production: the inference requirements, the observability gaps, the error handling patterns, and what happens when you scale from one agent to a thousand. ## What Makes an Agentic System Different A traditional LLM API call is a single request-response. You send a prompt, you get a completion, you move on. An agentic system is a loop: the model generates an action, the system executes it, the result comes back, the model generates the next action. Repeat until done. This loop changes everything about how you think about inference: - **Latency compounds.** A 10-step agent with 500ms per LLM call has 5 seconds of pure inference time, before any tool execution. Shave 200ms off each call and you save 2 full seconds per task. - **Errors propagate.** A failed tool call mid-chain doesn't just lose one response. It can corrupt the agent's context, trigger unnecessary retries, or leave external systems in a partial state. - **Context grows.** Each iteration adds tokens. A naive implementation that appends every tool call result to the context window can hit token limits several steps in. - **Concurrency multiplies costs.** One agent doing 10 LLM calls costs the same as 10 agents each doing 1. When you scale to 100 agents, you're suddenly making 1,000 concurrent inference requests. ## Observability: You Can't Debug What You Can't Trace The first time a production agent fails in a non-obvious way, you'll wish you had traced every step. A good observability setup for an agentic system captures: - The full prompt and response at each step - Tool calls and their results - Token counts per step (input and output separately) - Wall-clock latency per LLM call - The agent's current goal or plan state if you're using an explicit planning step - Any errors and the full context at the time they occurred OpenTelemetry is a reasonable choice here. Most inference providers expose request IDs you can attach to spans. The pattern looks like this: ```python import opentelemetry.trace as trace tracer = trace.get_tracer("agent") def run_agent_step(messages, tools): with tracer.start_as_current_span("agent_step") as span: span.set_attribute("input_tokens", count_tokens(messages)) response = client.chat.completions.create( model="qwen-coder-32b", messages=messages, tools=tools ) span.set_attribute("output_tokens", response.usage.completion_tokens) span.set_attribute("finish_reason", response.choices[0].finish_reason) return response ``` For longer-running agents, you'll also want to log state checkpoints. If an agent is running for 10 minutes and fails at step 47, you don't want to replay from the beginning. Periodic snapshots of the agent's context and memory let you resume or debug from a known point. One common mistake: logging only the final result. When things go wrong, the final result tells you almost nothing. You need the intermediate steps, especially the tool call arguments and results that led to the failure. ## Error Handling: Retries Are Not Enough Agentic systems fail in ways that chat apps don't. Here are the failure modes worth designing around explicitly. **Tool call failures.** The agent asks a tool to do something, the tool fails, and now the agent has to decide what to do. If you don't handle this explicitly, the model will often try the same call again, fail again, and spin. You need explicit error handling at the tool call level that communicates useful information back to the model: what failed, why, and what it should try instead. ```python def execute_tool(tool_name, args): try: result = tools[tool_name](**args) return {"success": True, "result": result} except ToolTimeoutError: return { "success": False, "error": "Tool timed out after 10s. Try a simpler query or break it into smaller parts." } except PermissionError as e: return { "success": False, "error": f"Permission denied: {e}. Check that the required credentials are configured." } ``` The quality of your error messages to the model matters a lot here. Vague errors like "tool failed" give the model nothing to work with. Specific messages with actionable context let the model adapt its plan. **Context overflow.** Long-running agents will eventually hit token limits. Handle this before it becomes an error. Monitor token usage at each step and implement a summarization step when you approach the limit. A common pattern: when the context hits 70-80% of the model's limit, summarize the history into a compact "progress so far" block and replace the full history with it. You lose some detail, but the agent can continue. **Hallucinated tool calls.** Models sometimes call tools with invalid arguments, or call tools that don't exist. Validate tool call arguments against your schemas before executing, and return a clear error message when the model generates an invalid call. Most models self-correct when given explicit feedback about what went wrong. **Loops and stuck states.** An agent can get into a loop where it repeatedly tries the same approach without making progress. Track the last N tool calls and detect when the pattern is repeating. If the agent is stuck, inject a system message prompting it to try a different approach, or surface it to a human for review. ## Rate Limiting: Scaling From 1 Agent to 1,000 A single agent making 10 LLM calls per task is manageable. A hundred agents making 1,000 calls per minute is a different situation. Rate limits are usually expressed as requests per minute (RPM) and tokens per minute (TPM). At scale, you'll hit both. Here's how to handle them. **Token estimation before calls.** Count your input tokens before making each call. This lets you predict whether you're approaching TPM limits and schedule calls accordingly. Tiktoken works for most models: ```python import tiktoken def estimate_tokens(messages, model="gpt-4"): enc = tiktoken.encoding_for_model(model) total = 0 for msg in messages: total += len(enc.encode(msg["content"])) + 4 # per-message overhead return total ``` **Request queuing.** Don't fire all agent requests simultaneously. Use a queue with a rate limiter. The `asyncio.Semaphore` pattern works well in Python: ```python import asyncio class RateLimitedClient: def __init__(self, max_concurrent: int, rpm_limit: int): self.semaphore = asyncio.Semaphore(max_concurrent) self.rpm_limit = rpm_limit self.request_times = [] async def create_completion(self, **kwargs): async with self.semaphore: now = asyncio.get_event_loop().time() self.request_times = [t for t in self.request_times if now - t < 60] if len(self.request_times) >= self.rpm_limit: wait = 60 - (now - self.request_times[0]) await asyncio.sleep(wait) self.request_times.append(now) return await self.client.chat.completions.create(**kwargs) ``` **Exponential backoff on 429s.** Even with rate limiting, you'll occasionally get 429 responses. Retry with exponential backoff and jitter: ```python import random import asyncio async def call_with_backoff(fn, max_retries=5): for attempt in range(max_retries): try: return await fn() except RateLimitError: if attempt == max_retries - 1: raise wait = (2 ** attempt) + random.uniform(0, 1) await asyncio.sleep(wait) ``` **Prioritization.** When you're running a mix of agent types, not all requests are equally urgent. A user-facing agent responding in real time needs low latency. A background batch job can wait. Build priority levels into your queue so high-priority requests jump ahead of lower-priority ones. ## Scaling: The Architectural Decisions That Matter Scaling an agentic system to thousands of concurrent agents requires a few architectural choices. **Async by default.** Synchronous agent execution blocks threads. If you're running agents in Python, use `asyncio` throughout. Each agent step involves network I/O for both the LLM call and typically one or more tool calls -- both excellent candidates for async execution. **Stateless agent workers.** Don't store agent state in memory on the worker process. Put it in a fast key-value store like Redis. This lets you run any number of stateless worker processes, restart them without losing in-progress agents, and distribute work across machines. ``` Worker 1: pick up agent_id=42 from queue, load state from Redis, run step, save state, enqueue next step Worker 2: pick up agent_id=43 from queue, load state from Redis, run step, save state, enqueue next step ``` **Parallel tool execution.** When an agent can make multiple tool calls that don't depend on each other, run them in parallel. If the agent wants to call a search API and a database lookup simultaneously, there's no reason to do them sequentially. This requires parsing the tool calls to identify independent ones, but the latency savings are significant for agents with wide tool use. **Agent timeouts at every level.** Set timeouts on individual LLM calls, on tool executions, and on the total agent run. Without these, a hung LLM call can hold up an entire worker indefinitely. The total task timeout should be aggressive enough to surface problems quickly. ## Observability Infrastructure at Scale When you're running 1,000 concurrent agents, you can't read traces individually. You need aggregated metrics that tell you the health of the system at a glance: - **P50/P95/P99 latency per agent step** -- watch P99 for outliers that affect user-facing tasks - **Token usage per agent run** -- spot agents that are consuming excessive context - **Error rate by tool** -- identify tools that are failing more than expected - **Stuck agent count** -- how many agents have been running longer than the expected maximum - **Retry rate** -- high retry rates indicate either rate limiting or flaky tools Feed these into a dashboard with alerting. A sudden spike in P99 latency or error rate often indicates an infrastructure issue with your inference provider or a newly deployed change that's causing more LLM calls than expected. ## Inference Provider Requirements for Agentic Systems The inference provider you choose affects almost everything above. For production agentic systems, the requirements are more demanding than for simple chat apps. **Low and consistent latency.** High variance in response times is worse than uniformly higher latency for agentic systems. When step 3 takes 800ms but step 7 takes 5 seconds, it's hard to build reliable timeout logic. Look for providers with low p99 latency, not just low median. **High rate limits.** At 1,000 concurrent agents making 10 calls each, you need a provider that can handle sustained high throughput without degrading. Most providers offer enterprise tiers with higher limits -- verify before you're blocked in production. **Reliable function calling.** Agentic systems depend on the model reliably generating valid tool calls. Test your specific tool schemas with your target model before committing to it. Some models handle complex nested schemas better than others. **Good error messages.** When the API returns an error, the message should tell you what happened and what to do. Cryptic error codes require looking up docs at the worst possible moment; good error messages let you handle failures programmatically. ## Putting It Together A production-ready agentic system needs: - Distributed tracing across all agent steps - Explicit error handling at the tool call level, with informative messages back to the model - Context management to avoid token limit failures mid-run - Rate limiting and request queuing to handle concurrent agents - Stateless workers backed by persistent state storage - Timeouts at every level of the stack Most of these aren't difficult to implement individually. The challenge is getting all of them in place before you need them, because the failure modes are subtle and often only appear under load. If you're scaling an agentic system and running into inference bottlenecks, General Compute's API is worth evaluating. The combination of low latency and high throughput limits reduces the rate limiting complexity and makes the concurrency math more manageable. You can get started at [generalcompute.com](https://generalcompute.com). --- # How to Build a Real-Time Coding Assistant With Open-Source Models > A practical guide to building a coding assistant LLM using Qwen3-Coder or DeepSeek Coder, covering model selection, codebase indexing, and VS Code integration. - Author: General Compute - Published: 2026-06-30 - Tags: coding assistant, open-source, llm, vs code, qwen3-coder, deepseek coder, tutorial - Canonical: https://www.generalcompute.com/blog/how-to-build-a-real-time-coding-assistant-with-open-source-models --- Building a coding assistant LLM from open-source components is practical today. The models are good enough, the tooling has matured, and the OpenAI-compatible API format means most of the plumbing is already written. This guide walks through the full stack: picking the right model, indexing your codebase for context retrieval, and wiring everything into VS Code. The reason teams build their own instead of using commercial tools like GitHub Copilot typically comes down to three things: code stays on your infrastructure, you can fine-tune on proprietary patterns, and at scale the cost difference is meaningful. ## Choosing a Model The two strongest open-source coding models right now are Qwen3-Coder and DeepSeek Coder V2. They have different strengths, and for a real-world coding assistant you'll likely want both. ### Qwen3-Coder Qwen3-Coder is Alibaba's latest coding model and performs well at: - **Fill-in-Middle (FIM) completion** -- the format that powers single-line autocomplete, where the model sees code before and after the cursor and fills the gap - **Multi-language support** -- TypeScript, Python, Rust, Go, and Java all work reliably - **Following explicit formatting constraints**, which matters when generating code that needs to match surrounding style The 7B and 14B variants are practical for most teams. The 7B model fits in 8GB of VRAM and is fast enough for real-time autocomplete. ### DeepSeek Coder V2 DeepSeek Coder V2 uses a mixture-of-experts architecture. The Lite variant (16B active parameters out of 236B total) scores well on HumanEval and SWE-bench. It does better at: - Multi-step reasoning about code architecture - Generating longer, contextually coherent functions - Code chat: explaining, refactoring, and debugging The tradeoff is that it requires more inference infrastructure than Qwen3-Coder 7B. Managed inference APIs make this transparent, but if you're self-hosting, the Lite variant needs more VRAM than a 7B model. ### Which to Use for Each Task For **autocomplete/FIM**, use Qwen3-Coder 7B or 14B. The FIM format is well-supported and the model is fast. For **code chat, refactoring, and explanation**, DeepSeek Coder V2 Lite or Qwen3-Coder 32B produce noticeably better output. A practical approach is to use both: a smaller fast model for inline autocomplete and a larger one for the chat panel. With an API that supports multiple models per account, you can switch model per request without maintaining two inference servers. ## Codebase Indexing A coding assistant that only sees the current file is limited. The real value comes when the model has context about the broader codebase: what functions exist, how types are defined, what patterns are used elsewhere. The problem is that a 500K-line codebase won't fit in any context window. The solution is retrieval: index the codebase upfront, find the relevant chunks at query time, and inject them into the prompt. ### Step 1: Parse with Tree-sitter Instead of splitting code at arbitrary line counts, use [Tree-sitter](https://tree-sitter.github.io/tree-sitter/) to parse it into an AST and split at meaningful boundaries (functions, classes, methods). ```python import tree_sitter_python as tspython from tree_sitter import Language, Parser PY_LANGUAGE = Language(tspython.language()) parser = Parser(PY_LANGUAGE) def extract_functions(source_code: str) -> list[dict]: tree = parser.parse(bytes(source_code, "utf8")) root = tree.root_node functions = [] for node in root.children: if node.type == "function_definition": name_node = node.child_by_field_name("name") functions.append({ "name": name_node.text.decode() if name_node else "", "body": source_code[node.start_byte:node.end_byte], "start_line": node.start_point[0], }) return functions ``` Tree-sitter has grammar packages for TypeScript, Rust, Go, Java, and most other languages. The same parsing logic works across languages; you swap the grammar package. ### Step 2: Embed the Chunks Embed each function or class using a code-aware embedding model. [Nomic Embed Code](https://huggingface.co/nomic-ai/nomic-embed-code) is a solid open-source option: 768-dimensional embeddings, fast, and it understands code semantics better than general-purpose text embedders. ```python from sentence_transformers import SentenceTransformer embed_model = SentenceTransformer("nomic-ai/nomic-embed-code") def embed_chunks(chunks: list[str]) -> list[list[float]]: return embed_model.encode(chunks, batch_size=32).tolist() ``` Store these in a vector database. For a local single-developer setup, SQLite with the `sqlite-vec` extension works well and avoids running a separate process. ```python import sqlite_vec import sqlite3 db = sqlite3.connect("codebase.db") db.enable_load_extension(True) sqlite_vec.load(db) db.enable_load_extension(False) db.execute(""" CREATE VIRTUAL TABLE IF NOT EXISTS code_chunks USING vec0( chunk_id INTEGER PRIMARY KEY, file_path TEXT, chunk_text TEXT, embedding float[768] ) """) ``` For teams where multiple developers share an index, Qdrant or Chroma work as standalone services. ### Step 3: Retrieve at Query Time When a user triggers autocomplete or asks a question, retrieve the top-k most relevant chunks and inject them into the prompt. ```python def get_relevant_context(query: str, top_k: int = 5) -> list[str]: query_embedding = embed_model.encode([query])[0].tolist() results = db.execute(""" SELECT chunk_text FROM code_chunks ORDER BY vec_distance_cosine(embedding, ?) LIMIT ? """, (query_embedding, top_k)).fetchall() return [row[0] for row in results] ``` Keep the retrieval fast. A query to a local SQLite index should run in under 10ms. If you're hitting a remote vector DB, cache embedding lookups for repeated queries. ## The Inference Layer Both Qwen3-Coder and DeepSeek Coder expose an OpenAI-compatible API, so you can use the standard `openai` SDK with any compatible endpoint. ```python from openai import OpenAI client = OpenAI( base_url="https://api.generalcompute.com/v1", api_key="your-api-key" ) ``` ### FIM for Autocomplete Fill-in-Middle is the completion format where you provide the code before and after the cursor, and the model fills what goes in between. Qwen3-Coder uses this special token format: ``` <|fim_prefix|>{code before cursor}<|fim_suffix|>{code after cursor}<|fim_middle|> ``` FIM requests go through the completions endpoint (not chat completions): ```python def get_fim_completion(prefix: str, suffix: str, model: str = "qwen3-coder-7b") -> str: prompt = f"<|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>" response = client.completions.create( model=model, prompt=prompt, max_tokens=128, temperature=0.1, stop=["<|fim_pad|>", "<|endoftext|>"] ) return response.choices[0].text ``` Keep `temperature` low for autocomplete (0.0 to 0.2) and cap `max_tokens` tightly. For single-line completion, 64 tokens is enough. For function-level completion, 256 is a reasonable ceiling. You want a fast response, not a complete file. ### Chat Mode with Retrieval For the chat panel, use the chat completions endpoint with retrieved context injected into the system prompt: ```python from typing import Generator def ask_about_code(question: str, context_chunks: list[str]) -> Generator[str, None, None]: context = "\n\n".join([f"```\n{chunk}\n```" for chunk in context_chunks]) stream = client.chat.completions.create( model="deepseek-coder-v2-lite", messages=[ { "role": "system", "content": ( "You are a coding assistant. Use the following code context " f"to answer questions accurately:\n\n{context}" ) }, {"role": "user", "content": question} ], stream=True ) for chunk in stream: delta = chunk.choices[0].delta.content if delta: yield delta ``` Streaming matters here. If the chat panel waits for the full response before rendering anything, users perceive it as slow even when the actual generation time is reasonable. Yielding tokens as they arrive makes the interface feel much more responsive. ## VS Code Integration There are two ways to surface this in VS Code: using an existing framework that supports custom backends, or building a minimal extension directly. ### Option 1: Continue (Fastest Path) [Continue](https://continue.dev) is an open-source VS Code extension that handles all the UI: inline completion, diff viewer, chat panel, slash commands, and keyboard shortcuts. You configure it to point at any OpenAI-compatible API. Add this to `~/.continue/config.json`: ```json { "models": [ { "title": "DeepSeek Coder (Chat)", "provider": "openai", "model": "deepseek-coder-v2-lite", "apiBase": "https://api.generalcompute.com/v1", "apiKey": "your-api-key" } ], "tabAutocompleteModel": { "title": "Qwen3-Coder (Autocomplete)", "provider": "openai", "model": "qwen3-coder-7b", "apiBase": "https://api.generalcompute.com/v1", "apiKey": "your-api-key" }, "contextProviders": [ { "name": "codebase", "params": {} }, { "name": "diff", "params": {} }, { "name": "open", "params": {} } ] } ``` This gets you a working coding assistant in a few minutes. Continue handles codebase indexing internally using its own embedding pipeline, so you can skip the custom indexing step if you don't need fine-grained control over chunking. ### Option 2: Custom VS Code Extension If you need more control -- custom indexing logic, internal auth, specific keybindings, or your own chat UI -- a lightweight VS Code extension is straightforward to build. Here's a minimal inline completion provider in TypeScript: ```typescript import * as vscode from 'vscode'; import OpenAI from 'openai'; const client = new OpenAI({ baseURL: 'https://api.generalcompute.com/v1', apiKey: process.env.GC_API_KEY ?? '', }); let debounceTimer: ReturnType | undefined; class FIMCompletionProvider implements vscode.InlineCompletionItemProvider { provideInlineCompletionItems( document: vscode.TextDocument, position: vscode.Position, ): Promise { return new Promise((resolve) => { clearTimeout(debounceTimer); debounceTimer = setTimeout(async () => { const prefix = document.getText( new vscode.Range(new vscode.Position(0, 0), position) ); const suffix = document.getText( new vscode.Range(position, new vscode.Position(document.lineCount, 0)) ); try { const prompt = `<|fim_prefix|>${prefix}<|fim_suffix|>${suffix}<|fim_middle|>`; const response = await client.completions.create({ model: 'qwen3-coder-7b', prompt, max_tokens: 128, temperature: 0.1, stop: ['<|fim_pad|>', '<|endoftext|>'], }); const text = response.choices[0]?.text ?? ''; resolve(text ? [new vscode.InlineCompletionItem(text)] : []); } catch { resolve([]); } }, 150); }); } } export function activate(context: vscode.ExtensionContext) { context.subscriptions.push( vscode.languages.registerInlineCompletionItemProvider( { pattern: '**' }, new FIMCompletionProvider() ) ); } ``` The 150ms debounce fires the request only after the user has paused typing. Without it you'd send a request on every keystroke, which wastes API calls and often returns completions for a half-finished token. ## Latency Targets and What Affects Them Users notice autocomplete latency above roughly 200ms. If a suggestion appears quickly it feels like the editor is helping; at 500ms or more, users start ignoring it and the feature loses its value. Things that affect latency: - **Model size**: Qwen3-Coder 7B is meaningfully faster than 14B or 32B. Use the smallest model that produces acceptable quality for your use case. - **Max tokens**: Cap it low. For single-line autocomplete, 64 tokens is enough. For function-level completion, 256 works. Higher ceilings don't improve quality for autocomplete -- they just slow the response. - **Streaming**: For autocomplete, streaming doesn't help because you need the full completion before showing it. For chat, stream from the first token. - **Retrieval overhead**: Your vector DB query needs to be fast. A local SQLite index runs in under 10ms. If you're querying a remote service, cache frequent lookups. On a fast inference API, Qwen3-Coder 7B can produce a 64-token autocomplete in 50-150ms end to end. That's within the range where the latency is invisible to most users. ## The Full Request Flow For autocomplete: 1. User pauses typing for 150ms. 2. Extract the prefix (everything before the cursor) and suffix (everything after) from the current file. 3. Optionally retrieve relevant chunks from the codebase index to add as context. 4. Format the FIM prompt and send to the completions endpoint. 5. Return the completion text to VS Code's inline completion API. For chat: 1. User sends a message in the chat panel. 2. Retrieve the top-k most relevant chunks from the codebase index. 3. Build a messages array with the retrieved context in the system prompt. 4. Stream the response back to the chat panel token by token. ## Getting Started The fastest path to a working coding assistant: 1. Install the [Continue VS Code extension](https://marketplace.visualstudio.com/items?itemName=Continue.continue). 2. Get an API key at [generalcompute.com](https://generalcompute.com). 3. Add the config above pointing at `qwen3-coder-7b` for autocomplete and `deepseek-coder-v2-lite` for chat. 4. Open any project, position your cursor, and press `Tab` or `Cmd+I`. For teams that need custom indexing, the Tree-sitter parsing, sqlite-vec embeddings, and OpenAI client code above give you a solid foundation to build on. The components are all independent: you can swap in a different embedding model, a different vector store, or a different inference backend without rewriting the rest. --- # Multi-Turn Conversations in LLM APIs: Best Practices for Agents > How to manage conversation history in LLM APIs without blowing up your context window or your bill. Covers sliding window, summarization, memory patterns, and cost optimization for production agents. - Author: General Compute - Published: 2026-06-29 - Tags: agents, inference, multi turn conversation, llm api, memory - Canonical: https://www.generalcompute.com/blog/multi-turn-conversations-llm-apis-best-practices-agents --- Most LLM tutorials show a single-turn request: send a prompt, get a response, done. Production agents don't work that way. A coding assistant, a customer support bot, or an autonomous research agent all maintain conversation state across many turns. That state management is where most production bugs live. This post covers how multi-turn conversation works at the API level, what goes wrong as conversations grow, and the practical patterns for keeping agents reliable and cost-efficient over long sessions. ## How Multi-Turn Works at the API Level OpenAI-compatible APIs (including General Compute's) represent conversation history as a list of message objects, each with a `role` and `content`. You pass this entire list with every request: ```python messages = [ {"role": "system", "content": "You are a helpful coding assistant."}, {"role": "user", "content": "How do I reverse a string in Python?"}, {"role": "assistant", "content": "You can use slicing: `s[::-1]`."}, {"role": "user", "content": "What about in JavaScript?"}, ] response = client.chat.completions.create( model="qwen3-coder", messages=messages ) ``` The model has no memory between API calls. You're responsible for assembling and sending the history each time. This is simple when conversations are short, but it means every token in that history list counts against your context limit and shows up in your bill. The token math is straightforward: if you have 40 turns of 200 tokens each, your history alone is 8,000 tokens before you've written a single new prompt. At 100 turns, you're looking at 20,000 tokens of overhead per request, and if that's in a tight loop, costs compound fast. ## The Context Window Ceiling Every model has a maximum context length. Exceed it and the API returns an error. Common limits: - 8K--32K tokens: older or smaller models - 128K tokens: most current production models - 1M+ tokens: extended-context models (Llama 4, Gemini 1.5) Even with a 128K window, a long-running agent that makes tool calls, collects outputs, and reasons over results can burn through context in a few dozen steps. And large contexts cost more per token on most providers. The practical ceiling for cost-effective operation is often lower than the technical limit. A 128K-token context window is available, but filling it completely on every request gets expensive fast. ## Strategy 1: Sliding Window The simplest approach is to keep only the N most recent turns: ```python def trim_messages(messages: list, max_turns: int = 20, system_msg: dict = None) -> list: # Always preserve the system prompt non_system = [m for m in messages if m["role"] != "system"] # Keep only the most recent turns # A "turn" is a user+assistant pair, so we keep max_turns * 2 messages trimmed = non_system[-(max_turns * 2):] if system_msg: return [system_msg] + trimmed return trimmed ``` This caps your context cost at a known maximum. The downside is that old context gets dropped entirely. For many tasks -- coding help, Q&A, step-by-step workflows -- this is fine. Users rarely need the model to recall something from 30 turns ago. Where sliding window breaks down is in long research or planning tasks where earlier decisions constrain later ones. Dropping the context for "I decided to use PostgreSQL because the team already has it running" means the model might suggest SQLite three turns later. A few practical notes on sliding windows: - Always keep the system prompt. It should never roll off. - Prefer dropping from the oldest end, not the middle. - Drop user+assistant pairs together. Dropping just the user message while keeping the assistant response creates a confused context where the model appears to have answered nothing. - Set your window size based on your model's context limit minus your expected response length and tool outputs. Leave headroom. ## Strategy 2: Summarization Rather than dropping old context, you compress it. When the conversation exceeds a threshold, you call the model to summarize the history so far, then replace those messages with a single summary message: ```python async def summarize_older_history(client, messages: list, keep_recent: int = 10) -> list: if len(messages) <= keep_recent + 1: # +1 for system return messages system_msg = next((m for m in messages if m["role"] == "system"), None) non_system = [m for m in messages if m["role"] != "system"] to_summarize = non_system[:-keep_recent] to_keep = non_system[-keep_recent:] summary_prompt = [ {"role": "user", "content": ( "Summarize the following conversation history concisely. " "Preserve key decisions, facts established, and any important context " "that might be needed later.\n\n" + "\n".join(f"{m['role'].upper()}: {m['content']}" for m in to_summarize) )} ] summary_response = await client.chat.completions.create( model="qwen3-coder", messages=summary_prompt, max_tokens=500 ) summary_text = summary_response.choices[0].message.content summary_message = { "role": "user", "content": f"[Earlier conversation summary: {summary_text}]" } result = [] if system_msg: result.append(system_msg) result.append(summary_message) result.extend(to_keep) return result ``` Summarization preserves semantic content at the cost of an extra API call and some latency. It works well when: - The early conversation contains important decisions or constraints - You're running an agent over a long session (hours, not minutes) - Exact phrasing from earlier turns doesn't matter, only the meaning The cost of the summarization call itself needs to factor into your budget. On fast inference providers, this adds latency on the order of a few hundred milliseconds, which is usually acceptable as a background operation. ## Strategy 3: External Memory For long-running or stateful agents, storing memory outside the context window entirely is often the right call. Instead of feeding all history to the model, you retrieve only what's relevant to the current step. The basic pattern: ```python from typing import TypedDict class MemoryStore: def __init__(self, embedding_client, vector_db): self.embedder = embedding_client self.db = vector_db async def store(self, turn: dict): text = f"{turn['role']}: {turn['content']}" embedding = await self.embedder.embed(text) self.db.upsert({"text": text, "embedding": embedding, "turn_id": turn["id"]}) async def retrieve(self, query: str, top_k: int = 5) -> list[str]: query_embedding = await self.embedder.embed(query) results = self.db.query(query_embedding, top_k=top_k) return [r["text"] for r in results] async def build_context_with_memory(store, current_query: str, recent_messages: list) -> list: relevant_memories = await store.retrieve(current_query) memory_block = "\n".join(relevant_memories) memory_message = { "role": "system", "content": f"Relevant context from earlier in the conversation:\n{memory_block}" } return [memory_message] + recent_messages ``` This approach lets an agent maintain context across sessions that span hours or days without ever hitting a context limit. The tradeoff is complexity: you need an embedding model, a vector store, and retrieval logic, and retrieval quality determines what the agent "remembers." For most agents, a hybrid approach works well: keep a short sliding window of recent messages for immediate context, and use vector retrieval for older information. ## Managing Tool Call History Agents that make tool calls accumulate large message sequences. Each tool call produces a `tool_calls` message from the assistant and a `tool` role message with the result. These can be long -- a web search result, a file read, a database query. A few approaches for keeping tool call history manageable: **Summarize large tool results before storing them.** If a tool returns 10,000 tokens of raw data, have the model extract the relevant parts before adding it to the messages list: ```python async def store_tool_result(client, tool_name: str, raw_result: str, max_tokens: int = 300) -> str: if count_tokens(raw_result) <= max_tokens: return raw_result response = await client.chat.completions.create( model="qwen3-coder", messages=[{ "role": "user", "content": f"Extract only the key facts from this {tool_name} result:\n\n{raw_result}" }], max_tokens=max_tokens ) return response.choices[0].message.content ``` **Drop tool messages from the sliding window, not conversation turns.** Tool results are often single-use: they mattered when the model used them, but future turns rarely need to see the raw output. You can drop old tool result messages more aggressively than you drop user/assistant turns. **Keep tool call structure even when dropping content.** The model needs to know what tools were called to avoid repeating work. If you drop a tool result, replace it with a placeholder: `"[result from search_web call #3 -- dropped to save context]"`. This tells the model what happened without paying for the full token cost. ## Cost Optimization at Scale If you're running many concurrent agents or high-turn sessions, the token costs compound. A few specific levers: **Count tokens before sending.** Don't wait for an API error to discover you've exceeded the context limit. Most tokenizers have a fast local token count. Budget 20--30% headroom for the response: ```python import tiktoken def estimate_tokens(messages: list, model: str = "gpt-4") -> int: enc = tiktoken.encoding_for_model(model) total = 0 for message in messages: total += 4 # overhead per message for value in message.values(): if isinstance(value, str): total += len(enc.encode(value)) return total + 2 # reply priming ``` **Use prefix caching if your provider supports it.** Prefix caching reuses KV cache across requests for shared prompt prefixes -- like a system prompt or a long document you reference on every turn. General Compute supports prefix caching, which means you only pay for the new tokens added to a cached prefix, not the full context on each call. **Route by conversation length.** Short conversations with a few turns are fine on larger, more capable models. Very long conversations where you've already summarized most of the context into a compact form can be handled by a smaller, faster, cheaper model. Building a routing layer that chooses model based on current context length pays off at scale. **Set token budgets per agent.** In production, an agent that enters a bad loop can run up a large bill before a human notices. Set a hard limit on the total tokens consumed per session and terminate or escalate when it's hit. ## Choosing the Right Pattern The choice between sliding window, summarization, and external memory comes down to your latency and fidelity requirements: | Pattern | Latency added | Memory fidelity | Complexity | |---|---|---|---| | Sliding window | None | Low (drops context) | Very low | | Summarization | 200--500ms per summary | Medium (semantic) | Low | | External memory | 50--150ms per retrieval | High (retrieved) | Medium | | Hybrid | 50--500ms | High | Medium | For most stateless or short-session agents, a sliding window with a reasonable turn limit (10--20 turns) is all you need. For long-running research or planning agents where earlier context matters, add summarization or external memory. The pattern you choose also affects which models you can use. Sliding window works with any model. Summarization benefits from a fast model to keep the extra call from adding perceptible latency. External memory requires an embedding model alongside your generation model. ## Putting It Together A production-grade conversation manager might look like this: ```python class ConversationManager: def __init__(self, client, max_recent_turns: int = 15, summary_threshold: int = 30): self.client = client self.max_recent = max_recent_turns self.summary_threshold = summary_threshold self.messages = [] self.system_prompt = None def set_system_prompt(self, content: str): self.system_prompt = {"role": "system", "content": content} def add_turn(self, role: str, content: str): self.messages.append({"role": role, "content": content}) async def get_context(self) -> list: msgs = self.messages # Summarize if we have too many turns if len(msgs) > self.summary_threshold: msgs = await summarize_older_history( self.client, msgs, keep_recent=self.max_recent ) self.messages = [m for m in msgs if m["role"] != "system"] # Trim to sliding window non_system = [m for m in msgs if m["role"] != "system"] trimmed = non_system[-(self.max_recent * 2):] result = [] if self.system_prompt: result.append(self.system_prompt) result.extend(trimmed) return result async def chat(self, user_message: str) -> str: self.add_turn("user", user_message) context = await self.get_context() response = await self.client.chat.completions.create( model="qwen3-coder", messages=context ) assistant_message = response.choices[0].message.content self.add_turn("assistant", assistant_message) return assistant_message ``` This isn't a complete production system -- you'd want persistence, error handling, token counting, and logging on top of this. But the structure shows how the pieces fit: the manager owns the message list, applies the appropriate trimming strategy, and keeps the context window under control. ## What Actually Goes Wrong in Production A few failure modes that come up repeatedly: **Forgetting to carry the system prompt.** When you build a trimmed context, it's easy to accidentally omit the system prompt. The agent loses its persona, constraints, and instructions. Always explicitly prepend the system prompt to your trimmed message list. **Dropping half a turn.** If you drop the user message but keep the assistant response, the model sees it answered a question that wasn't asked. Drop complete turns (user + assistant pair) together. **Summarizing too infrequently.** If you wait until you're near the context limit to summarize, you're summarizing a huge chunk of history in one go. Summarize earlier and more frequently to keep the summary task manageable. **Not testing at the edges.** Most conversation bugs only appear at turn 40 or turn 80, not in your unit tests. Include integration tests that run realistic long sessions to catch context management bugs before they hit production. --- General Compute's API is OpenAI-compatible, so these patterns work directly with our endpoint by swapping the base URL. Our inference speeds make strategies like in-context summarization faster -- when the extra summarization call takes 200ms instead of 2 seconds, you can afford to run it more frequently and keep history tighter. Check out the [General Compute docs](https://generalcompute.ai/docs) to get started. --- # Your AI Agent Is Only as Good as Its Inference Speed > Agent latency compounds across every sequential step. This post covers the multiplier effect, how model routing can cut costs without sacrificing quality, and why parallelizing calls is one of the highest-leverage improvements you can make to an agentic system. - Author: General Compute - Published: 2026-06-28 - Tags: agents, inference, latency, coding-agents, model-routing - Canonical: https://www.generalcompute.com/blog/your-ai-agent-is-only-as-good-as-its-inference-speed --- When you build an LLM chat interface, you are writing a wrapper around one API call. When you build an agent, you are writing a wrapper around twenty. The difference matters because every shortcoming in your inference layer shows up once in a chat app and twenty times in an agent. The way people discover this is usually not in benchmarks. It is in production, when the thing that felt fast in evals suddenly feels unusable when a real user is waiting for it to do a real task. The model did not get slower. The workload did. ## The multiplier in concrete terms A ReAct agent doing a moderately complex task -- finding a bug, writing a unit test, refactoring a function -- makes somewhere between 10 and 40 LLM calls, depending on how many tool results it has to read and how many retries it encounters. Each call has a time to first token (TTFT) and a decode tail. If TTFT is 600ms and decode runs at 60 tokens per second, a 100-token tool call takes about 2.3 seconds. Do that 20 times and you have 46 seconds of pure inference, before tool execution, before tests run, before the user sees a single useful result. If you bring TTFT down to 150ms and decode up to 200 tokens per second, the same call takes 0.65 seconds. 20 calls is 13 seconds. Same agent, same prompts, same model quality. The user experience goes from "this feels broken" to "this is fast enough to use." The ratio is not 1:1 because tool execution and other fixed overhead set a floor. But the inference contribution is large enough that a 3-4x improvement in per-call speed produces a 2-3x improvement in end-to-end task time. In most agentic workloads, inference accounts for 60-75% of wall clock. ## Why model size is not the only variable The instinct when an agent feels slow is to reach for a smaller model. If you are using a 70B parameter model and it is slow, try a 13B. This sometimes works. It often does not, because the latency problem is not always about model size. The variables that actually matter: - **Inference backend and hardware.** A 70B model on hardware tuned for low-latency short generations will often outperform a 13B model on a batch-throughput-optimized stack, in the regimes that agents actually hit. The model's parameter count sets a floor, but the serving infrastructure determines how close to that floor you actually get. - **TTFT vs decode balance.** Agents make many short generations. A call that produces 80 tokens of JSON is dominated by TTFT, not decode throughput. If you optimize only decode throughput (as most batch benchmarks do), you miss the dominant cost for agent calls. - **Structured output overhead.** Many inference stacks are slower at constrained generation (tool calls with specific JSON schemas) than at freeform text. The degradation ranges from subtle to a factor of two. If your agent makes 15 tool calls per task, this overhead compounds significantly. ## Model routing: right-sizing inference per step Not every step in an agent loop has the same quality requirements. A planning step that sets the overall structure of the agent's approach needs the best reasoning you can get. A file lookup step that produces a one-line tool call does not. Model routing assigns different steps to different model sizes based on what each step actually demands. A routing layer for a code agent might look like: ```python def select_model(step_type: str) -> str: routes = { "plan": "llama4-maverick", # complex reasoning, needs the big model "tool_call": "llama4-scout", # structured output, smaller is fine "code_generation": "qwen3-coder", # specialized, fast, strong at code "verify": "llama4-scout", # quick correctness check } return routes.get(step_type, "llama4-scout") ``` The savings add up quickly. If 60% of your agent's calls are short tool selections and verifications, and you can handle those with a model that runs at 3x the speed and a fraction of the cost, you have materially changed the economics and the wall clock without touching the quality of the steps that actually need the bigger model. The risk with routing is getting the categorization wrong. A verification step that should be quick but ends up needing to understand subtle code semantics can fail silently if you have routed it to a small model without the depth to catch the issue. In practice, you learn the boundaries of each step type in your specific workload through testing, not by assuming the routing in someone else's architecture will work for yours. ## Parallel calls: the obvious optimization most agents skip Sequential agent loops assume that step N+1 always depends on the result of step N. Often this is true. Often it is not. A code agent might need to: - Read the file that contains the bug - Read the test file for that module - Check if there is an existing utility function that handles similar logic These three reads are independent. They do not depend on each other's results. A naive sequential loop runs them one at a time, paying 3x TTFT and 3x decode for what could be 1x of each. Dispatching parallel calls changes the profile: ```python async def gather_context(file_paths: list[str]) -> list[str]: tasks = [read_file_with_agent(path) for path in file_paths] results = await asyncio.gather(*tasks) return results ``` The wall clock time goes from sum to max. If each read takes 0.8 seconds and you have three of them, sequential is 2.4 seconds and parallel is 0.8 seconds. The tricky part is that most agent frameworks are built around linear loops. They dispatch one call, wait, process the result, dispatch the next. Adding parallelism means either using a framework that supports it natively (LangGraph has some support for parallel branches, and CrewAI can run workers concurrently) or building a custom dispatch layer. Neither is trivial, but the payoff is real. Some practical rules for parallel calls: 1. File reads and search queries are almost always parallelizable. 2. Any step that gathers information from independent sources can run in parallel. 3. Candidate generation (produce two or three plans and pick the best) is parallelizable, but costs proportionally more inference. 4. Steps that depend on each other's outputs cannot run in parallel. Forcing parallelism there breaks correctness. ## Coding agent case study To make this concrete, consider a specific task: an agent that takes a failing test as input, locates the bug, and fixes the code so the test passes. The task involves: 1. Parse the failing test output and extract the error. 2. Locate the relevant source file. 3. Read the source file and the test file. 4. Understand what the test expects and what the code does instead. 5. Generate a fix. 6. Apply the fix. 7. Run the test. 8. Verify the output. On a fast inference backend (150ms TTFT, 200 tokens/sec decode): | Step | Output tokens | TTFT | Decode | Total | |------|--------------|------|--------|-------| | Parse error | 60 | 150ms | 300ms | 450ms | | Locate file | 40 | 150ms | 200ms | 350ms | | Read source | 30 | 150ms | 150ms | 300ms | | Read test | 30 | 150ms | 150ms | 300ms | | Understand + plan | 200 | 200ms | 1000ms | 1200ms | | Generate fix | 400 | 200ms | 2000ms | 2200ms | | Apply patch | 60 | 150ms | 300ms | 450ms | | Run tests | (subprocess) | -- | -- | 3000ms | | Verify | 100 | 150ms | 500ms | 650ms | Total for a clean one-shot success: roughly 9 seconds. Three optimizations apply directly here: - Steps 3 and 4 (reading source and test) can run in parallel. Wall clock for that pair drops from 600ms to 300ms. - The file location step could use a smaller routed model, since it is producing a file path from search results, not reasoning about code. - Prefix caching from step 4 onward reduces TTFT on subsequent steps, because the source and test content is already in the KV cache. With those changes, the fast backend gets to about 7 seconds for a one-shot success. Now consider a two-iteration case, which is more common than one-shot success in real workloads. The fix has a subtle error and the test fails the first time. Without the parallelism and routing, two iterations cost roughly 20 seconds. With the optimizations, about 16 seconds: 7 for the first pass, 6 for the second (context is warmer), plus two test runs. That is acceptable for a developer at a keyboard. The equivalent on a slow inference backend (2.5x the latency per call) pushes past 40 seconds, which is past the attention threshold for most interactive use. ## What to evaluate in an inference provider If you are putting agents into production and latency is the binding constraint, the things to evaluate are not the same as for a batch processing workload. **TTFT at short context.** Most providers publish throughput numbers. Ask for TTFT at 2k, 4k, and 8k input tokens. This is what your tool calls will actually look like. **Latency at low batch sizes.** A provider that is fast at batch size 32 may have poor single-request latency. Agents often run at low concurrency per user session. **Prefix cache effectiveness.** Ask whether the KV cache is preserved across requests in the same conversation. If not, every step pays full prefill cost, including for the system prompt and accumulated context. **Structured output latency.** Generate 50 tool calls with realistic JSON schemas and measure the p50 and p95. The p95 tells you what your retry cases will feel like, and retries are not rare in production. ## The compounding effect The argument for fast inference in agents is ultimately about compounding. When each step is slow, every inefficiency multiplies across the loop. When each step is fast, that multiplication works in your favor: more steps fit in the same wall clock, more validation is affordable, and more retry tolerance is available before the user gives up. A coding agent that can run 10 steps in 20 seconds is qualitatively different from one that can run 30 steps in the same time. The second one handles meaningfully harder tasks. The difference between them is not model quality. It is the infrastructure serving the model. If you are building agents and hitting latency walls, [General Compute's API](https://generalcompute.com) is optimized for exactly the access pattern agents need: short generations, aggressive prefix caching, and low TTFT at the context lengths agents actually hit. It is OpenAI-compatible, so the migration is a one-line config change, and the latency difference shows up immediately in the metrics that matter for your loop. --- # What Are Agentic AI Systems? How to Build Them With Fast Inference > Agentic AI systems chain LLM calls into autonomous loops that plan, act, and observe. This guide covers the core components, the main reasoning patterns (ReAct, Plan-and-Execute, Tree of Thoughts), and how inference speed shapes what you can actually build. - Author: General Compute - Published: 2026-06-27 - Tags: agents, agentic ai systems, inference, langchain, langgraph - Canonical: https://www.generalcompute.com/blog/what-are-agentic-ai-systems --- A chatbot answers a question. An agentic AI system accomplishes a goal. The difference is that the system does not wait for the next user message between each step. It plans, uses tools, checks the result, and keeps going until the task is done or it determines it cannot continue. This sounds deceptively simple. In practice, building an agent that behaves reliably in production involves understanding the components that make up the loop, the reasoning patterns that govern how the loop runs, and the infrastructure constraints that determine which designs are actually feasible. Fast inference sits at the center of most of those constraints. ## What an agentic system actually is An agentic AI system is a program that runs an LLM inside a loop. Each iteration of the loop consists of three phases: observe, think, and act. **Observe** means taking in the current state. That might be a user message, the output of the last tool call, an error from a failed action, or a retrieved document. **Think** means passing that state to the model and generating a decision. The model might decide to call a tool, generate a piece of code, ask a clarifying question, or declare the task complete. **Act** means executing the decision and capturing the output to feed back into the next observation. The loop continues until a stopping condition is met: the goal is achieved, a maximum step count is reached, the model outputs a stop signal, or an error is unrecoverable. What distinguishes an agent from a simple chain of prompts is that the next step is not predetermined. The model decides at each iteration what to do based on what it just learned. This is what makes agents powerful: they can handle tasks that require variable-length reasoning where you cannot know in advance how many steps the task will take. ## Core components Every agentic system, regardless of framework, has the same basic parts. **The model.** The LLM that drives decisions. In most systems this is a single model used for all steps, though some architectures use a larger model for planning and a smaller one for execution. **The context window.** Everything the model knows at any given step. This includes the system prompt, the conversation history, recent tool outputs, and any retrieved documents. Managing what goes into the context window is one of the harder engineering problems in agents, because context windows are finite and most agent tasks generate more content than fits. **The tool registry.** The set of actions available to the model. Tools are functions the agent can call: searching the web, reading a file, querying a database, writing code, sending an email. The model generates a structured call (usually JSON), the runtime executes it, and the result comes back as an observation. **The memory system.** How state persists across steps and across sessions. Short-term memory is the context window. Long-term memory is typically a vector database, a key-value store, or a structured log that the agent can query. The design of the memory system determines what the agent can remember and how fast retrieval is. **The executor.** The runtime that actually runs the loop, dispatches tool calls, manages retries, and handles errors. This is where frameworks like LangGraph, CrewAI, and AutoGen live. ## Reasoning patterns The pattern you choose for structuring the agent's thinking determines how the model moves from observation to action. There are three main patterns in current use. ### ReAct ReAct (Reasoning and Acting) is the most widely deployed pattern. The model is prompted to emit a reasoning trace before each action: ``` Thought: I need to find the current price of AAPL stock. Action: search_web("AAPL stock price today") Observation: Apple Inc. (AAPL) is trading at $212.40 as of market close. Thought: I have the price. I can now answer the question. Action: finish("AAPL is trading at $212.40.") ``` The interleaved reasoning makes the model's decisions more interpretable and often more reliable. When the model has to explain what it is doing before doing it, it tends to catch mistakes in its own plan before they turn into failed tool calls. ReAct works well for tasks with clear structure: fetch data, transform it, report it. It is less effective for tasks that require genuine multi-step planning, because the model only looks one step ahead. ### Plan-and-Execute Plan-and-Execute separates planning from acting. In the first phase, the model receives the goal and produces a complete plan as a list of steps. In the second phase, a separate agent (or the same model in a different mode) executes each step, optionally revising the plan when something unexpected happens. ```python # Phase 1: planning plan = planner_llm.generate( system="You are a planning agent. Break this task into steps.", user=goal ) # returns: ["Step 1: ...", "Step 2: ...", ...] # Phase 2: execution for step in plan: result = executor_llm.generate( system="You are an execution agent. Complete this step.", user=step, context=accumulated_results ) accumulated_results.append(result) ``` This pattern is useful when the task has enough structure that you can plan it upfront, and when the execution of each step is relatively independent. It handles long-horizon tasks better than ReAct because the full plan is explicit and can be inspected or edited. The downside is rigidity. If step 3 produces a result that invalidates step 4, a pure Plan-and-Execute system either fails or requires a replanning pass, which adds another LLM call. ### Tree of Thoughts Tree of Thoughts extends the idea of chain-of-thought by exploring multiple reasoning paths simultaneously and using search (BFS, DFS, or best-first) to find the best solution. Each node in the tree is a partial solution. The model generates multiple candidate next steps from each node, evaluates them, and expands the most promising ones. This is computationally expensive because each node requires at least one LLM call, and the tree can grow large. ``` Goal ├── Plan A │ ├── A.1 (evaluated: good) │ │ ├── A.1.1 (evaluated: good) │ │ └── A.1.2 (evaluated: poor, pruned) │ └── A.2 (evaluated: poor, pruned) └── Plan B └── B.1 (evaluated: good) └── B.1.1 (evaluated: best, selected) ``` Tree of Thoughts is rarely used in production today because the cost in LLM calls is too high at current inference speeds. It is more of a research result than a practical pattern. That said, as inference gets faster, the cost of tree search drops proportionally, and some teams are starting to use lightweight versions (two or three candidates per step rather than full search) in production for high-value tasks. ## Frameworks: LangGraph, CrewAI, AutoGen You do not need a framework to build an agent, but most teams use one because they handle the boilerplate: state management, tool dispatch, retry logic, streaming, and observability. ### LangGraph LangGraph models agents as directed graphs where nodes are functions (often LLM calls or tool calls) and edges define control flow. The graph state is a typed dictionary that each node reads from and writes to. ```python from langgraph.graph import StateGraph, END from typing import TypedDict class AgentState(TypedDict): messages: list tool_calls: list final_answer: str | None def call_model(state: AgentState): response = llm.invoke(state["messages"]) return {"messages": [response]} def call_tool(state: AgentState): results = execute_tools(state["tool_calls"]) return {"messages": results} def should_continue(state: AgentState): last = state["messages"][-1] if last.tool_calls: return "call_tool" return END graph = StateGraph(AgentState) graph.add_node("call_model", call_model) graph.add_node("call_tool", call_tool) graph.add_conditional_edges("call_model", should_continue) graph.add_edge("call_tool", "call_model") ``` LangGraph is a good fit when the agent's control flow is nontrivial: human-in-the-loop steps, branching on tool results, parallel sub-tasks. The graph abstraction makes the flow inspectable and testable. ### CrewAI CrewAI organizes agents into crews: groups of specialized agents that collaborate to complete a task. Each agent has a role, a goal, and a backstory that shapes how it behaves. A crew has a process (sequential or hierarchical) that determines how agents hand off work. ```python from crewai import Agent, Task, Crew, Process researcher = Agent( role="Research Analyst", goal="Find accurate information about the given topic", tools=[search_tool, browser_tool] ) writer = Agent( role="Content Writer", goal="Write a clear summary based on research findings", tools=[] ) research_task = Task( description="Research the current state of LLM inference hardware.", agent=researcher ) write_task = Task( description="Write a 500-word summary of the research findings.", agent=writer ) crew = Crew( agents=[researcher, writer], tasks=[research_task, write_task], process=Process.sequential ) result = crew.kickoff() ``` CrewAI is useful for tasks that map naturally onto multiple specialized roles. The role-based framing helps when you want different agents to behave differently within the same pipeline. ### AutoGen AutoGen from Microsoft takes a conversation-centric view. Agents communicate with each other through a shared message protocol. The runtime coordinates who speaks next and handles things like code execution in sandboxed environments. ```python from autogen import AssistantAgent, UserProxyAgent assistant = AssistantAgent( name="assistant", llm_config={"model": "your-model", "api_base": "https://api.generalcompute.com/v1"} ) user_proxy = UserProxyAgent( name="user_proxy", human_input_mode="NEVER", code_execution_config={"work_dir": "coding", "use_docker": False} ) user_proxy.initiate_chat( assistant, message="Write and test a Python function that sorts a list of dicts by a given key." ) ``` AutoGen handles code generation and execution well. The UserProxyAgent can run code locally or in a container and feed the result back to the assistant, which is the core loop for code agents. ## Why inference speed determines what you can build The patterns above all share a property: each step requires at least one LLM call, and steps are sequential because each step depends on the output of the previous one. This is where inference speed becomes a hard constraint, not a preference. Consider a ReAct agent doing a task that takes 10 steps. If each step pays 600ms in time-to-first-token and generates about 80 tokens at 80 tokens per second (1 second of decode), each call costs roughly 1.6 seconds. The 10-step task takes at minimum 16 seconds, before tool execution. Swap in a model with 150ms TTFT and 200 tokens per second decode, and the same step costs 0.55 seconds. The 10-step task now takes about 5.5 seconds. Same framework, same tools, same prompts -- roughly 3x faster end to end. The less obvious effect is on architecture. When steps are cheap, you can afford patterns that would be too slow otherwise: - Running two candidate plans in parallel and selecting the better one - Adding a validation step after every tool call - Using a smaller, faster model for simple steps and a larger model only for the hard ones - Attempting Tree of Thoughts search at small branching factors When steps are expensive, you are forced to minimize the number of LLM calls. You write prompts that try to do more in one shot. You skip validation. You accept that failed steps are costly and design around them instead of catching them early. The agents that handle 20-step tasks today are often constrained by what fits in the latency budget, not by what would produce the best result. As inference gets faster, those constraints loosen. ## Practical considerations for production **Context management.** Agent tasks accumulate a lot of text. Tool results can be long. Conversation history grows with each step. Left unmanaged, context overflow will truncate earlier observations and break the agent's reasoning. The standard approaches are summarization (periodically compress earlier history into a shorter summary) and retrieval (store old observations externally and pull them back when relevant). **Structured output reliability.** Agents depend on the model generating valid tool call JSON. Parsing failures cause retries, which compound latency. Use a serving stack that supports grammar-constrained decoding or native function calling to minimize parse failures. **Observability.** Debugging a 15-step agent from logs is painful. You want traces that show each step's input, output, token count, latency, and whether it was a retry. Most frameworks have integrations with tracing tools (LangSmith, LangFuse, Arize). Set these up before you go to production. **Failure modes.** Agents fail in ways that chat doesn't. The model can get stuck in a loop, calling the same tool repeatedly without making progress. It can hallucinate tool arguments. It can lose track of the original goal after many steps. You need timeouts, step count limits, and sanity checks. Budget: how many LLM calls am I willing to spend before forcing a stop? **Testing.** Unit testing individual tools is straightforward. End-to-end testing agents is harder because the path through the agent is non-deterministic. Build a small suite of tasks with known correct outcomes, run the agent against them, and measure success rate and step count. Regression testing becomes important when you change the model or the prompts. ## Getting started If you are new to building agents, start with ReAct and LangGraph. ReAct is well-studied and its failure modes are understood. LangGraph gives you enough structure to build something maintainable without too much abstraction overhead. Wire it up to a fast inference endpoint from the start. The biggest mistake teams make is building against a slow model and then trying to optimize later. Latency shapes architectural decisions early in development. If your first prototype runs in 3 seconds per step, you will design around that budget. If it runs in 300ms per step, you will build differently. General Compute's API is OpenAI-compatible, so pointing an existing LangGraph or AutoGen setup at our endpoint is a one-line change: ```python from openai import OpenAI client = OpenAI( api_key="your-key", base_url="https://api.generalcompute.com/v1" ) ``` From there, the rest of your agent code stays the same. The latency difference shows up in the wall clock time on your first test run. Agentic systems are still maturing. The patterns are stabilizing, the frameworks are still changing, and best practices around memory and evaluation are not fully settled. But the fundamentals -- loop, observe, think, act -- are not going anywhere. Building fluency with those fundamentals and the constraints that shape them is the most durable investment you can make as the space evolves. --- # SOC 2 Compliant AI Inference: What Enterprise Teams Need to Know > A practical guide to SOC 2 compliance for AI inference: Type I vs. Type II, data residency, audit logging, and a checklist for evaluating inference providers. - Author: General Compute - Published: 2026-06-26 - Tags: enterprise, security, compliance, soc2, inference - Canonical: https://www.generalcompute.com/blog/soc-2-compliant-ai-inference-what-enterprise-teams-need-to-know --- When an enterprise team decides to integrate AI inference into a product, the conversation usually starts with latency and cost. But if your company operates in healthcare, finance, legal, or any sector with meaningful data governance requirements, you hit a different question quickly: is this inference provider SOC 2 compliant? This post walks through what SOC 2 compliance means in the context of AI inference specifically, where Type I and Type II differ, what data residency requirements look like in practice, and what your security or legal team will likely ask before signing off on any vendor. --- ## What SOC 2 Actually Is SOC 2 (Service Organization Control 2) is an auditing framework developed by the American Institute of CPAs (AICPA). It assesses whether a service provider has adequate controls in place around five Trust Service Criteria: Security, Availability, Processing Integrity, Confidentiality, and Privacy. For most AI inference providers, Security and Confidentiality are the relevant criteria. Availability matters too if your SLA requires it. Processing Integrity and Privacy apply in narrower circumstances. SOC 2 is not a certification in the way ISO 27001 is. It is an audit report produced by a third-party CPA firm that attests to your controls at a point in time (Type I) or over a period of time (Type II). The report itself is not public by default -- vendors share it under NDA as part of a security review. --- ## Type I vs. Type II: The Practical Difference **Type I** reports assess whether controls are designed correctly as of a specific date. The auditor is essentially saying: "On this date, we examined the controls this company says they have, and they appear to be appropriately designed." **Type II** reports cover a period of time, typically six to twelve months. The auditor is saying: "We examined whether these controls were actually operating effectively over this period." That requires evidence -- logs, access reviews, incident reports, change records -- not just documentation of what controls exist. From an enterprise buyer's perspective, Type II is the meaningful one. Type I is a reasonable starting point for newer vendors, but it tells you very little about whether controls held up under real operations. A company can have excellent policies on paper and still have gaps in how they actually function day-to-day. Type II catches that gap. When evaluating an inference provider, ask specifically: do you have a SOC 2 Type II report, and how recent is it? Reports more than eighteen months old should prompt additional questions, since the controls landscape changes and audits expire in practice if not renewed. --- ## The Five Trust Service Criteria Applied to AI Inference Here is how each criterion maps to the specific concerns an enterprise has with an inference provider: **Security (CC)**: This covers logical and physical access controls, network security, change management, and incident response. For an inference provider, this includes how they manage access to the infrastructure running your model requests, how they segment customer workloads, and how they detect and respond to breaches. **Availability**: Uptime commitments and the controls that support them. If your use case is latency-sensitive (voice agents, real-time applications), you need to understand whether the provider's availability controls align with your SLA requirements. A SOC 2 report alone does not guarantee uptime -- it attests to whether the controls around availability are in place. **Processing Integrity**: Ensures that system processing is complete, accurate, timely, and authorized. For inference, this matters if you need assurance that requests are processed without corruption or unauthorized modification. **Confidentiality**: This is critical for AI inference. It covers whether data shared with the provider is protected from unauthorized disclosure. The key question is whether your prompt data and completion data are logged, retained, used for training, or visible to provider employees. **Privacy**: Applies when personal data is involved. If users are sending prompts containing PII -- names, email addresses, health information, financial data -- this criterion becomes relevant. It maps to GDPR, CCPA, and similar frameworks. --- ## Data Residency in AI Inference Data residency is separate from SOC 2 compliance but comes up in the same security review. It refers to where your data is physically stored and processed. For AI inference, "data" has a few distinct components: **Prompt data** is what you send in the request. This is often the most sensitive -- it may contain user-submitted content, proprietary documents, or PII depending on your application. **Completion data** is what the model returns. In most inference architectures, completions are generated and streamed back without persistent storage on the provider's side. **Logs and telemetry** are what providers retain for debugging, billing, and abuse detection. This is where residency gets complicated. Even if a provider does not store your prompts long-term, their logging infrastructure may capture metadata about requests. When reviewing a provider for data residency compliance: - Ask explicitly whether prompt data is logged, and if so, where and for how long. - Confirm whether any data is transmitted outside your required region (EU, US, etc.). - Understand whether the provider uses third-party sub-processors and whether those sub-processors have their own data residency commitments. - Review the DPA (Data Processing Agreement) alongside the SOC 2 report -- the DPA is the contractual commitment; the SOC 2 is the attestation that controls back it up. For EU-based companies working with GDPR requirements, make sure the DPA includes standard contractual clauses (SCCs) if data processing occurs in the US. GDPR compliance and SOC 2 compliance are not the same thing, and providers sometimes conflate them in marketing materials. --- ## Audit Logging: What You Need and What Providers Offer Your security team will want to know that you have an audit trail for AI inference activity. This serves several purposes: debugging production issues, responding to security incidents, and demonstrating compliance in an audit. The specific logging requirements depend on your industry. Here is what enterprise teams typically ask for: **Request-level logs**: Timestamps, endpoint called, model used, token counts, response codes. This is the minimum. Most providers expose this through a usage API or dashboard. **Identity and attribution**: Which API key or user identity initiated each request. If you have multiple teams or applications sharing an inference account, you need to know which requests belong to which context. Use separate API keys per application or use organizational access controls if the provider supports them. **Data access logs**: Evidence that only authorized parties can access your account data, and that the provider's own employees access logs only under defined conditions (support requests, security investigations). SOC 2 Type II reports cover this for the provider side. For your side, your own SIEM or logging infrastructure should capture API activity. **Retention policies**: How long do you need logs? Healthcare applications under HIPAA typically require six years. Financial services may have different requirements. Confirm whether the provider's log retention aligns with your retention policy, or whether you need to export and store logs independently. If your compliance requirements are strict, consider building a logging proxy in front of the inference API. This lets you capture and retain request/response data under your own policies without relying on what the provider exposes through their dashboard. --- ## What to Ask an Inference Provider Before Signing a Contract Here is the practical checklist your security or legal team should work through: **Compliance documentation** - Do you have a current SOC 2 Type II report? From what period? - Are you willing to share it under NDA for our security review? - Do you have an ISO 27001 certification? (Often required in addition to SOC 2 for multinational enterprise buyers.) - Do you have a penetration test report from the past twelve months? **Data handling** - Is prompt data retained? If so, for how long and in what form? - Is prompt data used to train models? If yes, can we opt out? - What happens to our data if we terminate the contract? - Do you have a data breach notification process? What is the timeline for notifying customers? **Infrastructure and access controls** - Where are inference servers physically located? - Can you commit to processing our data only in a specific region (US, EU, etc.)? - Do your employees have access to customer prompt data? Under what conditions? - How is customer workload isolation handled at the infrastructure level? **Contracts and legal** - Do you have a standard DPA? Does it include SCCs for EU data? - Do you have a BAA (Business Associate Agreement) available for HIPAA use cases? - What SLA do you offer, and what remedies exist for SLA breaches? - What does your incident response process look like, and how are customers notified? **Operational controls** - How are access credentials to the production environment managed? - What is your change management process for infrastructure changes? - Do you have a formal vulnerability disclosure or bug bounty program? --- ## HIPAA and Other Regulatory Overlays SOC 2 is often the floor, not the ceiling. Depending on your use case: **Healthcare (HIPAA)**: If prompt data includes PHI (protected health information), you need a BAA with the inference provider. SOC 2 does not cover HIPAA -- a provider can be SOC 2 Type II certified and still not eligible to sign a BAA if their infrastructure or controls are not set up to handle PHI appropriately. **Financial services (SOX, PCI-DSS)**: If AI inference is used in a system that touches financial reporting or cardholder data, additional controls are required. SOC 2 may satisfy some requirements, but review your specific regulatory framework. **Government (FedRAMP)**: For US federal government use cases, FedRAMP authorization is typically required. This is a substantially different and more demanding framework than SOC 2. Most commercial inference providers do not have FedRAMP authorization. --- ## Building an Enterprise-Ready AI Inference Setup The infrastructure choices you make on your side matter as much as the provider's compliance posture. A few patterns that enterprise teams use: **Dedicated API keys per environment**: Keep separate keys for production, staging, and development. Rotate keys regularly. Store them in a secrets manager (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager), not in environment variables or source code. **Request proxying for data control**: Route inference requests through your own service before they reach the provider. This lets you redact or mask sensitive fields in prompts, enforce rate limits per user, capture logs in your own system, and add your own authentication layer. **Egress monitoring**: If you route through a proxy or API gateway, monitor for unusual patterns -- unexpected spikes in token usage, requests from unexpected IP ranges, or model access that does not match application behavior. These are early signals of a compromised key or unauthorized use. **Contractual review cadence**: SOC 2 reports have effective periods. Set a calendar reminder to request an updated report from your inference provider each year, and re-review the DPA if there are changes to data handling practices. --- ## Evaluating Providers for Enterprise Readiness When you put a provider through a security review, you are assessing both their documentation and the quality of their responses. A provider that is genuinely prepared for enterprise sales will have these materials ready: the SOC 2 report, a DPA template, answers to a standard security questionnaire (many enterprises use HECVAT or the SIG Lite), and a clear data handling FAQ. Providers that are not enterprise-ready will either lack these materials or provide vague answers. The willingness to be specific in a security review is itself a signal about how seriously they take operational controls. General Compute maintains a SOC 2 Type II report and is happy to work through security reviews with enterprise teams. If you are evaluating inference providers and want to see our security documentation or discuss specific compliance requirements for your use case, reach out directly at [generalcompute.com](https://generalcompute.com) or through our enterprise sales contact. Building AI infrastructure that your security and legal teams can actually approve is not a side project -- it is what makes the difference between a proof of concept and a production deployment. --- # Embedding Models in Production: Choosing the Right One for Your App > A practical guide to picking an embedding model for production: MTEB benchmarks, head-to-head comparisons of BGE, Nomic, E5, Cohere, and OpenAI models, multilingual considerations, and code to get started. - Author: General Compute - Published: 2026-06-25 - Tags: embedding model, rag, vector search, sentence transformers, mteb, production ai, python - Canonical: https://www.generalcompute.com/blog/embedding-models-in-production --- Choosing an embedding model feels like a small decision, but it shapes retrieval quality, latency, cost, and how well your application handles edge cases. Swap from a weak model to a strong one and your RAG pipeline's answer quality can improve noticeably -- without changing the chunking strategy, vector database, or LLM. The choice matters. This guide covers the MTEB leaderboard and how to read it, the main open-source and managed options (BGE, Nomic, E5, Cohere, OpenAI), what multilingual support actually looks like in practice, and a decision framework to help you pick the right model for what you're building. ## What Embedding Models Actually Do An embedding model converts text into a dense vector -- a fixed-length list of floats. Texts with similar meanings end up with vectors that are close together in that high-dimensional space. When you search a vector database, you're asking: "which stored vectors are closest to the query vector?" The quality of embeddings determines whether semantically related passages actually cluster together. A weak embedding model might place "myocardial infarction" far from "heart attack" because it hasn't learned the semantic overlap. A strong one puts them close. Three properties matter most in production: - **Retrieval quality**: how well the model ranks relevant documents above irrelevant ones - **Latency**: how fast it can embed a batch of texts - **Dimension count**: higher dimensions usually mean better quality but more storage and slower similarity search Most models offer a fixed set of these trade-offs. Understanding them helps you stop defaulting to whatever was in the tutorial you read last month. ## The MTEB Leaderboard [MTEB](https://huggingface.co/spaces/mteb/leaderboard) (Massive Text Embedding Benchmark) is the standard benchmark for embedding models. It covers 58 datasets across 8 task categories: classification, clustering, pair classification, reranking, retrieval, semantic textual similarity (STS), summarization, and bitext mining. For most RAG use cases, retrieval and reranking scores are what you care about. The leaderboard shows an average across all tasks, which can obscure a model that is excellent at retrieval but weak at clustering (or vice versa). Practical things to keep in mind when reading MTEB numbers: **Higher average does not always mean better for your task.** A model tuned for sentence similarity might score well overall but be mediocre at document retrieval. Filter the leaderboard by the task type that matches your use case. **English vs. multilingual.** MTEB has separate leaderboards for multilingual models (MMTEB). The top English models often underperform on non-English text. If you serve multiple languages, check the multilingual scores specifically. **Model size and inference time are not shown.** A model scoring 65 average with 128M parameters is very different from one scoring 66 with 7B parameters. You have to check those separately. **Dataset contamination is a real concern.** Some models are trained on data that overlaps with MTEB test sets. Take leaderboard positions with some skepticism; try to find models with demonstrated performance on data similar to yours. ## The Main Contenders ### BGE (BAAI/bge-*) BGE models from BAAI (Beijing Academy of Artificial Intelligence) have been consistently strong on MTEB since their release. The family includes several variants worth knowing: - **bge-large-en-v1.5** (1024 dims, 335M params): top-tier English retrieval, solid choice for English-only applications - **bge-m3** (1024 dims, 568M params): multilingual, supports 100+ languages, also strong on English -- arguably the best general-purpose open-source choice in 2026 - **bge-small-en-v1.5** (384 dims, 33M params): compact, fast, reasonable quality for applications where latency is the primary constraint BGE models support three retrieval modes via different pooling strategies: dense retrieval, sparse retrieval (like BM25), and multi-vector (ColBERT-style). `bge-m3` can do all three in one pass, which is handy for hybrid search pipelines. Licensing: Apache 2.0 -- safe for commercial use. ```python from sentence_transformers import SentenceTransformer model = SentenceTransformer("BAAI/bge-m3") # BGE models benefit from an instruction prefix for asymmetric retrieval tasks passages = ["How does attention work?", "What is a transformer?"] queries = ["explain attention mechanism"] # Encode passages and queries separately when doing retrieval passage_embeddings = model.encode(passages, normalize_embeddings=True) query_embeddings = model.encode(queries, normalize_embeddings=True) ``` ### Nomic Embed (nomic-ai/nomic-embed-text-v1.5) Nomic's embed model gets less attention than BGE but deserves serious consideration for production use. Key attributes: - 768 dimensions, 137M parameters - Supports matryoshka representation learning (MRL) -- you can truncate to 256, 128, or 64 dimensions with a relatively small quality penalty, which cuts storage and search costs - Strong MTEB retrieval scores that are competitive with models 2-3x larger - Apache 2.0 license, fully open weights The MRL feature is genuinely useful. If you're storing hundreds of millions of vectors, being able to use 256-dimensional embeddings instead of 768 with a 5-8% quality reduction can meaningfully reduce your vector database costs. ```python from sentence_transformers import SentenceTransformer # The model uses task-specific prefixes model = SentenceTransformer("nomic-ai/nomic-embed-text-v1.5", trust_remote_code=True) def embed_documents(texts): return model.encode( [f"search_document: {t}" for t in texts], normalize_embeddings=True ) def embed_query(query): return model.encode( f"search_query: {query}", normalize_embeddings=True ) ``` Note the task prefixes: `search_document:` for things you're indexing and `search_query:` for queries. Nomic trained the model with these prefixes and skipping them degrades retrieval quality. ### E5 (intfloat/e5-*) The E5 family from Microsoft covers a range of sizes. The variants most people use: - **e5-large-v2** (1024 dims): strong English performance, good for applications that don't need multilingual support - **multilingual-e5-large** (1024 dims): excellent multilingual retrieval, competitive with bge-m3 on many languages - **e5-mistral-7b-instruct** (4096 dims): a 7B model fine-tuned from Mistral, currently one of the top-performing models on MTEB -- but it costs significantly more to run The 7B instruct variant is interesting but mostly useful when you're indexing at batch time and have GPU resources available. Running 7B-scale inference for query embedding in a latency-sensitive path is painful. Most teams end up using it for offline indexing and a smaller model for real-time query encoding. Like Nomic, E5 models expect task prefixes: "query: " and "passage: " for asymmetric retrieval tasks. ```python from sentence_transformers import SentenceTransformer model = SentenceTransformer("intfloat/multilingual-e5-large") # Prefix required for retrieval tasks docs = [f"passage: {text}" for text in document_texts] queries = [f"query: {q}" for q in query_texts] doc_embeddings = model.encode(docs, normalize_embeddings=True, batch_size=32) query_embeddings = model.encode(queries, normalize_embeddings=True) ``` ### Cohere Embed v3 Cohere's managed embedding API is the strongest commercial option for multilingual use cases. Embed v3 supports 100+ languages with consistent quality across all of them -- something that remains hard for open-source models to match at this scale. Dimensions: 1024 (with support for 256 and 64 via MRL truncation in the `embed-multilingual-v3.0` variant). The API requires you to specify an `input_type` parameter: `"search_document"`, `"search_query"`, `"classification"`, or `"clustering"`. This matters -- the model was trained with these modes and you'll see lower quality if you skip it. ```python import cohere co = cohere.Client("your-api-key") def embed_documents(texts): response = co.embed( texts=texts, model="embed-multilingual-v3.0", input_type="search_document", ) return response.embeddings def embed_query(query): response = co.embed( texts=[query], model="embed-multilingual-v3.0", input_type="search_query", ) return response.embeddings[0] ``` The downside is cost and vendor lock-in. At high volume, managed API pricing adds up relative to self-hosting bge-m3 on your own GPU. ### OpenAI text-embedding-3-small and text-embedding-3-large OpenAI's third-generation embedding models are widely used because they work well and require no infrastructure -- you're already using the OpenAI SDK for your LLM calls. They also support dimension reduction via MRL truncation using the `dimensions` parameter. - **text-embedding-3-small** (1536 dims, or truncated): good quality at low cost, roughly $0.02 per million tokens - **text-embedding-3-large** (3072 dims, or truncated): near-state-of-the-art on MTEB, $0.13 per million tokens For English-only applications at moderate scale, `text-embedding-3-small` is hard to beat on the cost-quality curve. At very high scale, self-hosted open-source models become cheaper. ```python from openai import OpenAI client = OpenAI(api_key="your-api-key") def embed_texts(texts, model="text-embedding-3-small", dimensions=1536): response = client.embeddings.create( input=texts, model=model, dimensions=dimensions, ) return [item.embedding for item in response.data] ``` ## Multilingual Options: What Actually Works If your application needs to handle text in multiple languages, your embedding model needs multilingual training data. Simply using an English-focused model on French or Chinese text produces poor retrieval. The options in order of quality-to-cost: 1. **bge-m3** -- best open-source multilingual option. Strong across European languages, Chinese, Japanese, Korean, Arabic. Apache 2.0. 2. **multilingual-e5-large** -- competitive with bge-m3, especially for retrieval. MIT license. 3. **Cohere embed-multilingual-v3.0** -- consistently strong across 100+ languages, including lower-resource ones that bge-m3 and e5 may not cover well. Managed API. 4. **OpenAI text-embedding-3-large** -- solid multilingual performance, especially for languages with significant presence in web text. One nuance: cross-lingual retrieval (query in English, documents in French) is harder than same-language retrieval. Cohere tends to handle this better than open-source models, but bge-m3 is competitive for the major language pairs. If you're building for a single non-English language with a large user base (Chinese, German, Japanese), check whether there's a language-specific model that outperforms the multilingual options. Models like `Alibaba-NLP/gte-Qwen2-7B-instruct` often outperform general multilingual models for Chinese text. ## Comparison Table | Model | Dims | Params | License | MTEB Avg | Best For | |---|---|---|---|---|---| | BAAI/bge-m3 | 1024 | 568M | Apache 2.0 | 66.1 | Multilingual, hybrid search | | nomic-embed-text-v1.5 | 768 | 137M | Apache 2.0 | 62.4 | Cost-effective production use | | intfloat/multilingual-e5-large | 1024 | 560M | MIT | 64.6 | Multilingual retrieval | | intfloat/e5-mistral-7b-instruct | 4096 | 7B | MIT | 66.6 | Highest quality, offline indexing | | Cohere embed-multilingual-v3.0 | 1024 | N/A | Commercial | ~65 | Enterprise multilingual | | OpenAI text-embedding-3-small | 1536 | N/A | Commercial | 62.3 | Low-cost managed option | | OpenAI text-embedding-3-large | 3072 | N/A | Commercial | 64.6 | High-quality managed option | MTEB averages vary depending on which tasks are included in the calculation. Treat these as rough guidance rather than precise rankings. ## How to Choose A few questions that narrow the field quickly: **Do you need multilingual support?** Use bge-m3, multilingual-e5-large, or Cohere. Rule out English-only models regardless of their English MTEB scores. **Do you need to self-host?** Open-source models (BGE, Nomic, E5) can run on your own infrastructure. If you need air-gapped deployment or have data residency requirements, this is non-negotiable. **What's your throughput?** For batch indexing jobs, a larger model (e5-mistral-7b, bge-m3) is fine. For real-time query encoding at high QPS, you want something smaller -- bge-small, nomic-embed -- or a managed API that handles scaling for you. **How many vectors are you storing?** Higher-dimensional models use more storage. 100M vectors at 1024 dimensions (float32) is 400GB before any database overhead. At that scale, MRL truncation (Nomic to 256 dims, OpenAI to 512) can cut storage costs meaningfully with modest quality loss. **What's your latency budget for indexing?** If you're re-embedding your corpus frequently, throughput matters. bge-small processes roughly 3x-5x more texts per second than bge-large on the same hardware. ## A Practical Starting Point For most RAG applications in English: start with `nomic-embed-text-v1.5` or `BAAI/bge-large-en-v1.5`. Both are strong, Apache-licensed, and well-supported by `sentence-transformers`. For multilingual applications: `BAAI/bge-m3`. It's the most versatile open-source option and handles hybrid search without needing a separate sparse retrieval model. If you want a managed API and aren't concerned about lock-in: `text-embedding-3-small` is the easiest path. Upgrade to `text-embedding-3-large` if you're seeing retrieval quality issues. Here's a minimal setup you can adapt: ```python from sentence_transformers import SentenceTransformer from openai import OpenAI # Option 1: Self-hosted with sentence-transformers class LocalEmbedder: def __init__(self, model_name="BAAI/bge-m3"): self.model = SentenceTransformer(model_name) def embed(self, texts, is_query=False): # bge-m3 uses instruction prefixes for asymmetric retrieval if is_query: texts = [f"Represent this sentence for searching relevant passages: {t}" for t in texts] return self.model.encode(texts, normalize_embeddings=True, batch_size=64).tolist() # Option 2: OpenAI managed API class OpenAIEmbedder: def __init__(self, model="text-embedding-3-small"): self.client = OpenAI() self.model = model def embed(self, texts, is_query=False): response = self.client.embeddings.create(input=texts, model=self.model) return [item.embedding for item in response.data] ``` Both classes expose the same interface, so you can swap them without changing downstream code. ## Evaluating Your Choice Don't rely solely on MTEB scores. Run your own evaluation on a sample of your actual data: 1. Take 50-100 representative queries from your application. 2. For each query, manually identify the 3-5 most relevant documents in your corpus. 3. Embed everything with the candidate model and compute retrieval metrics: Recall@5 (fraction of relevant docs in the top 5 results) and MRR (mean reciprocal rank). 4. Compare models head-to-head on these numbers. A model that scores 2 points higher on MTEB but performs worse on your specific domain is the worse choice. Your evaluation data is more predictive than a benchmark built on generic web text. ## Getting the Generation Side Right Choosing a strong embedding model solves the retrieval half of the problem. The generation half -- turning retrieved chunks into accurate, fast answers -- depends on your LLM inference layer. Slow inference hurts RAG specifically: every user query goes embedding + vector search + LLM generation in sequence. If the LLM step takes 3 seconds, your total latency is already too high for most interactive applications regardless of how fast retrieval is. GeneralCompute's API provides low-latency generation on Llama 4, Qwen3, and other leading open models, and it's OpenAI-compatible -- change `base_url` and `api_key` and your existing code works: ```python from openai import OpenAI client = OpenAI( api_key="your-generalcompute-api-key", base_url="https://api.generalcompute.com/v1", ) ``` Solid embeddings plus fast generation gives you the retrieval quality and response speed that make a RAG application actually usable. Start with [generalcompute.com](https://generalcompute.com) to get API access. --- # How to Migrate From OpenAI to GeneralCompute in 10 Lines of Code > GeneralCompute's API is fully OpenAI-compatible. Here's exactly what to change in Python, Node.js, and LangChain, plus a validation checklist to make sure nothing breaks. - Author: General Compute - Published: 2026-06-24 - Tags: migrate openai sdk, python, nodejs, langchain, tutorial, api - Canonical: https://www.generalcompute.com/blog/how-to-migrate-from-openai-to-generalcompute-in-10-lines-of-code --- GeneralCompute's API is OpenAI-compatible. That means if your app already calls OpenAI, the migration is mostly a matter of changing a base URL and swapping an API key. You don't need to rewrite your prompt logic, message formatting, streaming setup, or error handling. This guide covers the exact diffs for Python, Node.js, and LangChain, along with a validation checklist for confirming the migration worked. ## Why the Migration Is Short The OpenAI SDK -- both the Python and JavaScript versions -- accepts a `base_url` (or `baseURL`) parameter that overrides where requests go. Most libraries built on top of the OpenAI API accept the same parameter. Because GeneralCompute implements the same REST interface (same endpoints, same request/response shapes, same streaming protocol), pointing your existing client at `https://api.generalcompute.com/v1` is enough to switch providers. The only things you need to update are: 1. The base URL 2. Your API key 3. The model name (if you're switching to a model with a different identifier) ## Python Migration ### Before ```python from openai import OpenAI client = OpenAI( api_key="sk-..." ) response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Explain KV caching in one paragraph."}] ) print(response.choices[0].message.content) ``` ### After ```python from openai import OpenAI client = OpenAI( api_key="your-generalcompute-api-key", base_url="https://api.generalcompute.com/v1" ) response = client.chat.completions.create( model="llama-4-maverick", messages=[{"role": "user", "content": "Explain KV caching in one paragraph."}] ) print(response.choices[0].message.content) ``` Two lines changed: `api_key` and `base_url`. Everything else -- message format, response parsing, error handling -- stays identical. ### Streaming in Python Streaming works the same way: ```python from openai import OpenAI client = OpenAI( api_key="your-generalcompute-api-key", base_url="https://api.generalcompute.com/v1" ) with client.chat.completions.stream( model="llama-4-maverick", messages=[{"role": "user", "content": "Write a haiku about latency."}] ) as stream: for text in stream.text_stream: print(text, end="", flush=True) ``` The `.stream()` context manager, the `text_stream` iterator, and the flush pattern all carry over unchanged. ### Using Environment Variables The cleaner approach for production code is to keep provider config in environment variables: ```python import os from openai import OpenAI client = OpenAI( api_key=os.environ["GENERALCOMPUTE_API_KEY"], base_url=os.environ.get("LLM_BASE_URL", "https://api.generalcompute.com/v1") ) ``` This pattern also makes it easy to switch back to OpenAI for testing -- just set different env vars rather than touching code. ## Node.js / TypeScript Migration ### Before ```typescript import OpenAI from "openai"; const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY, }); const response = await client.chat.completions.create({ model: "gpt-4o", messages: [{ role: "user", content: "What is speculative decoding?" }], }); console.log(response.choices[0].message.content); ``` ### After ```typescript import OpenAI from "openai"; const client = new OpenAI({ apiKey: process.env.GENERALCOMPUTE_API_KEY, baseURL: "https://api.generalcompute.com/v1", }); const response = await client.chat.completions.create({ model: "llama-4-maverick", messages: [{ role: "user", content: "What is speculative decoding?" }], }); console.log(response.choices[0].message.content); ``` Two lines changed. Note that the Node.js SDK uses `baseURL` (camelCase) while the Python SDK uses `base_url` (snake_case). ### Streaming in Node.js ```typescript import OpenAI from "openai"; const client = new OpenAI({ apiKey: process.env.GENERALCOMPUTE_API_KEY, baseURL: "https://api.generalcompute.com/v1", }); const stream = client.chat.completions.stream({ model: "llama-4-maverick", messages: [{ role: "user", content: "Describe transformer attention." }], }); for await (const chunk of stream) { const text = chunk.choices[0]?.delta?.content ?? ""; process.stdout.write(text); } ``` The async iterator pattern works identically. If you're using `.on("content", ...)` event handlers or `stream.finalMessage()`, those work too. ## LangChain Migration LangChain's `ChatOpenAI` class accepts the same `base_url` and `api_key` overrides. ### Before ```python from langchain_openai import ChatOpenAI llm = ChatOpenAI( model="gpt-4o", openai_api_key="sk-..." ) result = llm.invoke("What are the trade-offs of quantization?") print(result.content) ``` ### After ```python from langchain_openai import ChatOpenAI llm = ChatOpenAI( model="llama-4-maverick", openai_api_key="your-generalcompute-api-key", openai_api_base="https://api.generalcompute.com/v1" ) result = llm.invoke("What are the trade-offs of quantization?") print(result.content) ``` The parameter is `openai_api_base` here (not `base_url`) because LangChain wraps the OpenAI client and exposes its own parameter names. Everything downstream -- chains, memory, callbacks, streaming -- works without modification. ### LangChain Streaming ```python from langchain_openai import ChatOpenAI from langchain_core.messages import HumanMessage llm = ChatOpenAI( model="llama-4-maverick", openai_api_key="your-generalcompute-api-key", openai_api_base="https://api.generalcompute.com/v1", streaming=True ) for chunk in llm.stream([HumanMessage(content="Explain MoE routing.")]): print(chunk.content, end="", flush=True) ``` ### LangChain with LCEL If you're using LangChain Expression Language, nothing changes in your chain definitions: ```python from langchain_openai import ChatOpenAI from langchain_core.prompts import ChatPromptTemplate from langchain_core.output_parsers import StrOutputParser llm = ChatOpenAI( model="llama-4-maverick", openai_api_key="your-generalcompute-api-key", openai_api_base="https://api.generalcompute.com/v1" ) prompt = ChatPromptTemplate.from_messages([ ("system", "You are a helpful engineering assistant."), ("human", "{question}") ]) chain = prompt | llm | StrOutputParser() print(chain.invoke({"question": "How does continuous batching work?"})) ``` The `|` chaining syntax, prompt templates, and output parsers all remain the same. ## Other Libraries Most libraries that wrap the OpenAI API follow the same pattern: **LlamaIndex:** ```python from llama_index.llms.openai import OpenAI llm = OpenAI( model="llama-4-maverick", api_key="your-generalcompute-api-key", api_base="https://api.generalcompute.com/v1" ) ``` **Instructor (structured outputs):** ```python import instructor from openai import OpenAI client = instructor.from_openai( OpenAI( api_key="your-generalcompute-api-key", base_url="https://api.generalcompute.com/v1" ) ) ``` **Vercel AI SDK:** ```typescript import { createOpenAI } from "@ai-sdk/openai"; const generalcompute = createOpenAI({ apiKey: process.env.GENERALCOMPUTE_API_KEY, baseURL: "https://api.generalcompute.com/v1", }); ``` The pattern is consistent: find the parameter that sets the API base URL, point it at GeneralCompute, and swap in your key. ## Validation Checklist After making the change, run through this checklist before deploying: **Basic connectivity** - [ ] A simple non-streaming completion returns a response - [ ] Response has the expected shape (`choices[0].message.content`) - [ ] No authentication errors (confirm your API key is correct) **Streaming** - [ ] Streaming completions produce chunks incrementally - [ ] Stream terminates cleanly (no hanging connections) - [ ] `finish_reason` is present on the final chunk **Your specific use case** - [ ] System prompts work as expected - [ ] Multi-turn conversations (passing a `messages` array with history) produce coherent responses - [ ] Tool/function calling works if your app uses it - [ ] JSON mode works if your app uses `response_format: { type: "json_object" }` **Error handling** - [ ] Rate limit errors surface as expected exceptions - [ ] Invalid model names return a clear error (not a silent failure) **Performance** - [ ] TTFT (time to first token) meets your expectations - [ ] Token generation speed is acceptable for your use case If anything in the checklist fails, the most common causes are: a mismatched model name (check the [GeneralCompute docs](https://generalcompute.com) for supported model identifiers), a base URL with a trailing slash that conflicts with path construction, or a library that hard-codes the OpenAI domain somewhere you haven't overridden yet. ## Common Gotchas **Trailing slash in base URL.** Some SDKs are sensitive to whether `base_url` ends with `/v1` or `/v1/`. If you get 404s on endpoints, try removing or adding the trailing slash. **Library-specific parameter names.** As shown above, different libraries use `base_url`, `baseURL`, `api_base`, or `openai_api_base`. Check the library's docs if the standard override isn't working. **Model name format.** OpenAI uses names like `gpt-4o` and `gpt-4o-mini`. GeneralCompute uses names like `llama-4-maverick` and `llama-4-scout`. Make sure you've updated the `model` parameter -- this is the one change that doesn't fail silently, but it's easy to miss. **Embeddings.** If your app calls `client.embeddings.create(...)`, note that embedding models have different identifiers. Check the docs for the embedding model name you need. **Hardcoded URLs.** Search your codebase for `api.openai.com` to catch any direct HTTP calls that bypass your client configuration. ## Getting Started Get a GeneralCompute API key at [generalcompute.com](https://generalcompute.com). The migration is small enough that it's worth running in a branch and A/B testing the response quality before fully committing. Most teams find the behavior is equivalent for standard chat and completion tasks, with meaningfully faster token generation. If you run into anything that behaves differently from what you expected, the [GeneralCompute docs](https://generalcompute.com) cover the full API surface including supported parameters and model options. --- # Using Vercel AI SDK With GeneralCompute: Full Integration Guide > A complete guide to wiring the Vercel AI SDK's useChat and useCompletion hooks to GeneralCompute's OpenAI-compatible API, with streaming in Next.js App Router and edge runtime deployment. - Author: General Compute - Published: 2026-06-23 - Tags: vercel ai sdk, nextjs, streaming, tutorial, react, edge runtime - Canonical: https://www.generalcompute.com/blog/using-vercel-ai-sdk-with-generalcompute --- The Vercel AI SDK is the most popular way to add streaming AI responses to Next.js apps. It ships `useChat` and `useCompletion` hooks that handle everything from token streaming to loading states to conversation history, and it connects to any OpenAI-compatible API. GeneralCompute exposes an OpenAI-compatible API, which means the SDK connects to it with a two-line config change. You get GeneralCompute's inference speeds with the same hooks and components you already know. This guide walks through: - Setting up the Vercel AI SDK to use GeneralCompute as its provider - Building a chat interface with `useChat` - Building a text generation UI with `useCompletion` - Deploying your API routes to the Vercel edge runtime for minimum latency ## How the SDK Works The Vercel AI SDK separates the provider (where you call the model) from the hooks (how you handle the response in your UI). On the server you define a route handler that calls the model and streams back the response. On the client, `useChat` or `useCompletion` hits that route and handles the stream, exposing React state you bind to your UI. This architecture keeps API keys on the server and makes the client code trivially simple. ## Installation ```bash npm install ai @ai-sdk/openai ``` The `ai` package is the core SDK. `@ai-sdk/openai` is the official OpenAI provider, which works with any OpenAI-compatible endpoint -- including GeneralCompute's. ## Configuring the Provider Create a shared provider config you can import into any route handler: ```typescript // lib/generalcompute.ts import { createOpenAI } from "@ai-sdk/openai"; export const generalcompute = createOpenAI({ apiKey: process.env.GENERALCOMPUTE_API_KEY, baseURL: "https://api.generalcompute.com/v1", }); ``` The `createOpenAI` factory accepts `baseURL` to override where requests go. From here, use `generalcompute("model-name")` anywhere you'd normally write `openai("gpt-4o")`. Add your API key to `.env.local`: ``` GENERALCOMPUTE_API_KEY=your_api_key_here ``` ## Building a Chat Interface With `useChat` `useChat` is the SDK's primary hook for multi-turn conversations. It manages message history, handles the streaming response, and exposes loading and error states. ### The Route Handler First, define the server-side route that the hook will call. In the Next.js App Router, this goes in `app/api/chat/route.ts`: ```typescript // app/api/chat/route.ts import { streamText } from "ai"; import { generalcompute } from "@/lib/generalcompute"; export const runtime = "edge"; export async function POST(req: Request) { const { messages } = await req.json(); const result = streamText({ model: generalcompute("llama-4-maverick"), messages, system: "You are a helpful assistant. Be concise.", }); return result.toDataStreamResponse(); } ``` `streamText` calls the model and returns a streaming result. `toDataStreamResponse()` formats it as the Vercel AI SDK's data stream protocol, which `useChat` on the client knows how to consume. The `export const runtime = "edge"` line tells Vercel to deploy this function to its edge network. More on that below. ### The Chat Component ```typescript // app/chat/page.tsx "use client"; import { useChat } from "ai/react"; export default function ChatPage() { const { messages, input, handleInputChange, handleSubmit, isLoading } = useChat({ api: "/api/chat", }); return (
{messages.map((message) => (

{message.role === "user" ? "You" : "Assistant"}

{message.content}

))} {isLoading && (

Thinking...

)}
); } ``` `useChat` gives you `messages` (the full conversation array), `input` (controlled input value), `handleInputChange` and `handleSubmit` (event handlers you attach directly to the form), and `isLoading` (true while a response is streaming). That's the complete chat UI. The hook handles everything else: appending your message to the history, sending the request, accumulating streamed tokens into the message object, and updating React state on each token. ## Building a Text Generation UI With `useCompletion` `useCompletion` is better suited for single-shot generation tasks: summarization, text transformation, content generation from a prompt. Unlike `useChat`, it doesn't maintain a conversation history. ### The Route Handler ```typescript // app/api/completion/route.ts import { streamText } from "ai"; import { generalcompute } from "@/lib/generalcompute"; export const runtime = "edge"; export async function POST(req: Request) { const { prompt } = await req.json(); const result = streamText({ model: generalcompute("llama-4-scout"), prompt, maxTokens: 1024, }); return result.toDataStreamResponse(); } ``` ### The Completion Component ```typescript // app/complete/page.tsx "use client"; import { useCompletion } from "ai/react"; export default function CompletionPage() { const { completion, input, handleInputChange, handleSubmit, isLoading } = useCompletion({ api: "/api/completion", }); return (