Inference is fragmenting

Every mature compute market eventually fragments. Databases started as one product, Oracle, and split into OLAP, OLTP, time-series, vector, graph, and key-value systems as workloads diversified. Networking silicon split between switching, routing, deep-packet inspection, and SmartNICs. Mobile chips fragmented from a single application processor into dedicated NPUs, ISPs, modems, and security enclaves. In each case, the fragmentation was a response to the same pressure: as the market grew, workload diversity outpaced what any single architecture could serve well, and specialized silicon won the slices where its architectural choices made it the obvious answer.

Inference is at the beginning of this transition. Training will stay on general-purpose GPUs for the foreseeable future, because training is a workload that rewards exactly what GPUs do well. But inference is splitting into workload classes with very different demands. Voice has different requirements than batch. On-device is a different problem than hyperscale serving. Agents are a different problem than chatbots. No single chip wins all of these slices, and the chips that win individual slices already exist. They are taped out, funded, benchmarked, and shipping.

What does not exist is the channel that gets them into production. Chips do not win markets on architectural merit. They win on deployed capacity, and deployed capacity is a financing problem before it is an engineering problem. NVIDIA understands this better than anyone, which is why the fastest-growing infrastructure companies in the world, the asset-heavy neoclouds, are in practice extensions of NVIDIA's balance sheet: funded with NVIDIA equity, collateralized by NVIDIA GPUs, and backstopped by NVIDIA purchase guarantees. Every specialized chip that would win its slice of inference faces the same wall. The customers who want its performance will not buy racks of first-generation silicon, and the neoclouds that could deploy it at scale are structurally committed to a single vendor.

General Compute is the asset-heavy neocloud built to be the balance sheet for specialized silicon. We buy, deploy, and operate the chips our customers want to run on but cannot or will not own themselves, and we sell the result as inference. Our fleet is set by customer demand, not vendor allegiance. Today that demand is SambaNova's SN50 and Etched's Sohu, two architectures that dominate different regions of the inference Pareto curve. Behind them is a growing bench of credible silicon, and we are open to any chip that earns customer pull.

This is not a speculative model. We run production agent workloads on SambaNova SN40L today, powered by low-cost energy in existing US colocation. On GPT-OSS-120B, that stack delivers time-to-first-token 2.6x faster and end-to-end latency 4.6x faster than Together AI, one of the strongest GPU-based inference providers in the market. That is what one piece of well-chosen silicon already buys against a well-engineered general-purpose stack. The rest of this paper explains why the market is fragmenting, which chips win the fragments, why those chips cannot reach the market alone, and how General Compute is built to carry them there.

The clearest evidence that inference is fragmenting is the workload we serve today. The inference infrastructure the industry built from 2022 to 2025 was built for chatbots. A user types a question, a model generates a response, the user reads it. Throughput matters because you want to serve many users at once. Tail latency matters less because a human is reading the output as it streams, and at reading speed, any decode rate above about fifty tokens per second feels instant.

Agents invert every one of these assumptions.

The input is long. An agent's prompt is not a user question, it is a system prompt, a tool catalog, a memory buffer, retrieved context, and the accumulated history of prior steps. Tens of thousands of tokens is normal. Hundreds of thousands is not unusual. Every step re-sends most of this context, which makes prefill a recurring cost rather than a one-time setup.

The output is short and structured. Agents do not write essays, they emit tool calls, JSON objects, reasoning traces, and intermediate plans. Generations of a few hundred tokens are typical. The model spends most of its time starting and stopping rather than producing sustained output.

The workload is sequential. A chatbot conversation has natural pauses while the user reads and types. An agent trajectory has none. The moment one step finishes, the next begins. There is no human in the loop absorbing latency, so every millisecond of decode delay is a millisecond added to the end-to-end task time, and those milliseconds compound across every step.

The batch is small. Chatbot serving economics rely on batching dozens or hundreds of concurrent user requests together to amortize the cost of reading model weights from memory. Agent workloads are often batch size one: a single long trajectory, running as fast as possible, with no sibling requests to pack alongside. The throughput optimizations that define modern GPU serving stacks simply do not apply.

Agents and chatbots are two workload classes from the same model families, and they already reward different silicon. That divergence is not an anomaly. It is the first visible crack of a fragmentation that runs through the entire inference market, and it repeats along every axis: batch versus interactive, dense versus sparse, long-context versus short-form. The chips that win these slices will not be the same chip.

The dominant framing of LLM inference treats it as a single problem. Tokens go in, tokens come out, faster is better. This framing has produced a generation of infrastructure optimized for a weighted average of fundamentally different computations, and it is starting to break down.

Start with the split inside a single request. Prefill is compute-bound: processing a long prompt is a dense parallel matrix multiplication, and the bottleneck is FLOPs. This is exactly what modern GPUs are built for. Decode is memory-bound: generating each token requires reading the entire model's weights and KV cache from memory to do a trivial amount of math, and the bottleneck is the latency of data movement. Arithmetic intensity collapses, and a GPU doing decode at batch size one is a Ferrari in a parking lot, expensive silicon doing almost nothing. HBM bandwidth has scaled more slowly than compute for several GPU generations, so this gets worse with every new part, not better.

Now widen the lens, because prefill versus decode is only one axis. The full inference market is a Pareto curve. On one axis is interactivity: how many tokens per second a single user or a single agent trajectory experiences. On the other is cost: how many dollars it takes to produce a token at that speed. Every serving stack is a point on this plane, and no chip dominates the whole frontier. A chip that wins maximum interactivity does it by keeping weights close to compute, which is expensive per token. A chip that wins minimum cost per token does it by amortizing memory reads across huge batches, which caps per-user speed. General-purpose GPUs sit in the broad middle of the curve: genuinely good over a wide band, dominant at almost no single point on it.

The industry's answer has been to serve the entire curve from that middle. Batch aggressively, use continuous batching and PagedAttention to pack sequences, and accept that the extremes are underserved. This works when the market is one undifferentiated pool of chatbot traffic. It fails as the market fragments, because the value migrates to the extremes: agents that need the highest interactivity money can buy, and high-volume products that need the lowest cost per token physics allows. You cannot batch your way to either end of the curve from the middle.

A fragmented market is won region by region, by silicon built for each region. The interesting question is no longer whether such silicon exists. It does. The question is who puts it into production.

Two architectures anchor our fleet today, and they dominate opposite ends of the curve.

SambaNova SN50 owns the top of the interactivity frontier. SN50 is the fifth generation of SambaNova's dataflow architecture. Instead of a small compute die pulling data from a bandwidth-constrained external memory on every token, a dataflow chip distributes memory and compute across a reconfigurable grid, with a three-tier hierarchy of on-chip SRAM, HBM, and high-capacity DDR that keeps weights and KV cache resident where they are needed. Decode is precisely the workload this wins: the memory hierarchy is not a cliff to be climbed on every token, and per-user token rates reach levels that memory-bound GPUs cannot touch regardless of batch discipline.

The numbers are independently verified. SemiAnalysis benchmarked a single 16-chip SambaRack SN50 on MiniMax M2.7 at roughly 800 tokens per second of per-user interactivity in its maximum-speed configuration, and roughly 400 tokens per second in a balanced configuration that simultaneously approaches the aggregate throughput of B200 systems on the same model. Artificial Analysis has validated SN50 results as well. On GPT-OSS-120B, SN50 reaches interactivity rates of 1,900 tokens per second, and on 600B+ parameter models it moves total rack throughput from roughly 800 tokens per second on SN40L to roughly 15,000, nearly a twenty-times generational step. A SambaRack draws about 20 kilowatts, air-cooled, sized for models up to 10 trillion parameters and context up to 10 million tokens, with model hot-swapping in milliseconds. SambaNova's most recent round raised $1 billion at an $11 billion valuation, and the SN50 ships to customers in the second half of 2026. Our LOI covers dedicated SN40L capacity today and SN50 as it comes online.

Etched Sohu attacks the curve from the other end. Sohu is an inference ASIC built on TSMC's 4nm process with 144GB of HBM3E per chip, and its bet is concentrated in two innovations. Low-voltage compute runs the chip's math blocks at well below half of conventional operating voltage, which collapses power per operation and lets Etched put radically more silicon to work inside the same power and thermal envelope. Cluster-scale memory pools memory across chips into a single shared low-latency space over a proprietary interconnect, so an eight-chip server holds 400 to 600 billion parameter models without sharding penalties. Etched reports server-level throughput of hundreds of thousands of tokens per second on 70B-class models and sustained utilization near 80 percent of peak FLOPs, figures measured on their own hardware and not yet independently benchmarked, though more than $1 billion in signed customer contracts suggests the people who have seen the silicon up close find the numbers persuasive. The company raised at a $10.3 billion valuation in July 2026, and first racks ship this summer. We have an executed LOI with Etched.

On the Pareto curve, Sohu's economics let it undercut NVIDIA across the band of moderate interactivity where most production traffic lives today: same speeds, meaningfully lower cost per token. And its ceiling is higher: it can push per-user interactivity past what GPU serving stacks deliver, though not as far as SN50's dataflow architecture reaches. The two chips are not competitors in our fleet. Sohu dominates the cost-per-token region and the middle band that GPUs hold today; SN50 dominates the extreme-interactivity region that agents are making valuable. Between them, the general-purpose middle of the curve gets squeezed from both sides. That is what fragmentation looks like when it arrives in silicon.

Behind these two is a bench of credible specialized silicon, including Positron and TensorDyne, each staking out its own region of the curve. We evaluate every serious architecture, and the fleet follows customer demand. The point is not any single chip. The point is that the fragmentation thesis is no longer a forecast. The chips are here, the benchmarks are hardening, and the capital behind them is real.

If these chips are so good, the natural question is why they are not already everywhere. The answer is that in compute infrastructure, architectural merit does not deploy itself. Somebody has to buy the hardware, and the structure of the market determines who can.

Consider how NVIDIA actually maintains its position. The asset-heavy neoclouds, CoreWeave, Nebius, and their peers, are the fastest-growing buyers of accelerators in the world, and they are structurally single-vendor. CoreWeave is guiding roughly $33 billion of capital expenditure in 2026 against a debt load that passed $24 billion in the first quarter, with quarterly interest expense alone exceeding a quarter of revenue. That debt is raised through delayed-draw term loans collateralized by the GPUs themselves and by investment-grade customer contracts. NVIDIA holds equity stakes of roughly $2 billion in each of CoreWeave and Nebius, and has committed a multi-billion-dollar backstop obligating it to purchase CoreWeave's unsold capacity into the next decade. The loop is elegant: NVIDIA invests in the neocloud, the neocloud borrows against NVIDIA hardware to buy more NVIDIA hardware, and NVIDIA guarantees the downside. It is a remarkable machine, and every part of it is vendor-specific. The lenders underwrite GPUs because GPUs are liquid collateral with a deep resale market. The backstop exists because NVIDIA wrote it. None of this transfers to anyone else's silicon.

Now look at the same market from a specialized chipmaker's side. Their natural customers, the AI products that would benefit most from their performance, do not want to own hardware. Buying racks of first-generation silicon means capital expenditure, datacenter operations, power contracts, and single-vendor risk on an unproven resale market, all to serve what is often one workload class. Enterprises will not do it. The hyperscalers build their own chips. And the existing neoclouds cannot do it, because their financing, their vendor relationships, and their allocation-based competitive advantage are all denominated in NVIDIA hardware. A chip can win its slice of the Pareto curve decisively and still have no path to deployed capacity.

This is the actual moat, and it is worth being precise about it. NVIDIA's dominance in inference is no longer primarily about CUDA, and it was never only about the chips. It is about being the only silicon vendor whose hardware comes with a capital markets apparatus attached. The way to compete with that is not a better chip. Better chips exist. The way to compete is to build the missing piece of the machine: a balance sheet that works for silicon NVIDIA did not make.

General Compute is that balance sheet. We are asset-heavy on purpose, in a market where asset-heavy has come to mean NVIDIA-exclusive. We buy the specialized silicon our customers demand, deploy it into existing US colocation on low-cost power, operate it as a production inference cloud, and sell the output as tokens through a single API. The customer gets the performance of silicon they would never rack themselves. The chipmaker gets what NVIDIA built for itself and no one else has: a committed, operating, revenue-generating deployment channel. We sit in the middle and own the assets, because owning the assets is the product.

The fleet is demand-driven. Our customers' pull so far has been for SambaNova and Etched, and both are under executed LOI: dedicated SN40L capacity serving production traffic today with a path to SN50 as it ships, and Sohu racks as Etched's first systems come online. Positron, TensorDyne, and others are on the horizon as customer demand and silicon maturity warrant. We are open to any chip that earns a place on the curve and has customers asking for it. What we are not is a general-purpose cloud. Every accelerator we deploy is there because it dominates a region of the Pareto curve for a workload our customers run.

The deployment model is built for speed. Everything we operate is air-cooled: SN40L and SN50 racks at roughly 20 kilowatts, Sohu servers, and the GPU prefill nodes in our disaggregated architecture all fit existing US colocation at standard rack power densities. The infrastructure most GPU clouds are scaling into does not: GB200-class density requires purpose-built liquid-cooled datacenters with direct-to-chip cooling loops, and the wait for new liquid-cooled capacity in North America is measured in years. We already hold options on 15 megawatts of air-cooled power at facilities that exist today, enough for our Q4 2026 architecture and the growth beyond it. Our path from signed capacity to serving traffic is measured in installation timelines, not construction permits.

The serving architecture is where the fleet thesis pays off technically. In Q4 2026 we move to a disaggregated stack: prompts hit GPU nodes for the compute-bound prefill phase, hand off the KV cache, and decode runs on the silicon whose region of the curve matches the request. Latency-critical agent trajectories decode on SN50. Cost-sensitive high-volume traffic decodes on Sohu. The user sees one API; the hardware sees each workload landing on the chip built for it, and we provision each pool independently as the traffic mix shifts. A single-vendor cloud cannot route across the curve, because it only owns one point on it.

Underneath all of it is power. Our capacity runs on low-cost energy, because power at scale is the dominant operating cost of an inference cloud and we buy it wherever it is cheapest. Efficient silicon, inexpensive power, and air-cooled deployment into existing colocation are what make our pricing defensible over the long run rather than a short-term discount. Distribution went live through OpenRouter at our May launch, and the first cohort of production customers is onboarding on SN40L now, the same customers the Q4 2026 architecture serves with dramatically more headroom.

If the economics of an agent are bounded by latency, raising the ceiling on latency changes what gets built. The agent products in market today are shaped by the latency budget available to them. Coding agents make a handful of tool calls before returning control to the user because each call costs seconds. Research agents retrieve a small number of documents because longer trajectories become unusable. Voice agents constrain themselves to shallow reasoning because the turn-taking budget is brutal. These look like product choices, but most of them are latency constraints wearing product clothing. When decode runs at SN50 speeds, a coding agent takes thirty tool calls instead of five, a research agent reads fifty documents instead of five, and long-horizon agents cross the threshold where multi-hour trajectories are economically viable.

If the economics of an AI product are bounded by cost per token, lowering the floor changes what gets built just as surely. Products that are uneconomical at GPU serving costs, always-on assistants, high-volume extraction, ambient intelligence in consumer software, become viable when tokens cost what Sohu-class silicon can produce them for. The two ends of the Pareto curve unlock two different product generations, and neither is reachable from the middle.

There is a third unlock, and it is the one we think investors should weigh most heavily. Every specialized architecture that wins its slice on merit but dies for lack of a deployment channel is capability the world loses, and margin that defaults back to the incumbent. The inference market is fragmenting on a trajectory every mature compute market has followed, the silicon for the fragments exists, and the only missing piece is the balance sheet willing to carry it. That piece is small compared to the value it gates, which is exactly what makes it a business.

General Compute is being built for the chips that deserve a market and the customers who deserve those chips.

If that is the future you see too, we would like to talk.