Why TileRT Cannot Catch SambaNova SN50: The Limits of Software-Emulated Dataflow on the Wrong Hardware

TileRT is a software layer that remodulates execution on existing accelerators to behave more like a dataflow processor. The idea is reasonable: if you can tile compute graphs, fuse operations, and schedule producers and consumers efficiently in software, you might approximate what purpose-built dataflow silicon does in hardware.

The problem is that TileRT runs on chips that were designed around different assumptions. The memory hierarchy, interconnect topology, and scheduler all reflect the host chip's original purpose. Software can reshape how work is dispatched, but it cannot change where memory lives, how wide the on-chip buses are, or how the hardware arbitrates between concurrent workloads. Those constraints are baked into silicon, and they determine the performance ceiling regardless of how smart the software layer is.

The SambaNova SN50 is designed from scratch for dataflow execution. This post explains what that means in practice, where TileRT hits structural limits that software cannot overcome, and what the performance delta looks like in concrete terms.

The CUDA Moat Was Never About Inference

TileRT is, among other things, strong validation that kernel-by-kernel execution models cannot deliver fast inference. CUDA is fundamentally a kernel-by-kernel paradigm. Every operation dispatches a kernel, completes, and returns control to the runtime before the next operation begins. TileRT exists precisely to work around this: it intercepts the dispatch sequence, fuses kernels, and tries to impose producer-consumer ordering that CUDA's runtime was not designed to provide.

If the winning inference architecture is dataflow, then for fast inference there is no CUDA moat. You have to work around CUDA rather than leverage it. TileRT is a workaround, not a use, of the CUDA programming model. The moat that matters for inference is not a programming model moat. It is a hardware architecture moat, and it belongs to silicon designed from the start for streaming dataflow execution.

What TileRT Actually Does

The Software Remodulation Idea

TileRT works by intercepting a model's compute graph and retiling it into execution blocks that can be mapped across processing elements with better producer-consumer locality. Instead of dispatching independent kernels to a GPU runtime and relying on the hardware scheduler, TileRT tries to control the execution order explicitly so that the output of one operation flows directly into the input of the next without going through global memory.

This is a real optimization. Kernel-at-a-time execution wastes bandwidth because each kernel reads inputs from DRAM, computes, and writes outputs back to DRAM. If you can fuse multiple kernels and keep intermediate tensors in L2 cache or on-chip SRAM, you cut a significant fraction of memory traffic.

What TileRT Runs On Today and Why That Matters

TileRT is deployed on GPU-class hardware and some general-purpose accelerators. These chips have a fixed memory hierarchy: registers, then L1 cache per SM, then L2 cache shared across SMs, then HBM. The interconnect between SMs uses a mesh or crossbar that was sized and designed for GPU workloads, not for the streaming tensor traffic that dataflow processors are built around.

This matters because the TileRT runtime cannot widen the buses, resize the on-chip caches, or reorder the levels of the memory hierarchy. It can control when data moves, but it cannot change where data must live or how long it takes to move there.

Why Software Alone Cannot Change Memory Hierarchy or Interconnect Topology

When TileRT fuses two operations, it keeps intermediate values on-chip if they fit in the available cache. But "available cache" is determined by what the host chip provides. An H100 SM has 256 KB of shared memory per block, and the L2 is 50 MB shared across all SMs. These are fixed constraints.

If a fused tensor chain exceeds what fits on-chip, data spills to HBM. At 3.35 TB/s bandwidth, the H100 is fast, but that bandwidth is shared across every concurrent workload on the chip, and the round-trip to HBM is still orders of magnitude slower than on-chip SRAM access on a purpose-built dataflow unit.

What a SambaNova Dataflow Unit Does in Silicon

The RDU/PCU/PMU Fabric

The SambaNova SN50 is built around a Reconfigurable Dataflow Unit (RDU). The RDU fabric consists of Pattern Compute Units (PCUs) and Pattern Memory Units (PMUs) arranged so that compute and memory are physically adjacent. Each PCU has its own PMU, and the two are connected by short, high-bandwidth local wires.

This is structurally different from a GPU, where compute (SM) and memory (L2/HBM) are separated by a shared bus that all SMs compete for. On the RDU, each PCU pulls data from its adjacent PMU without contending with neighboring PCUs. The effective memory bandwidth per compute unit is much higher because the paths are shorter and not shared.

Streaming Tensors Between Stages Without Global Memory Round-Trips

On the SN50, the compiler schedules a full model graph and assigns each operation to specific PCUs and PMUs at compile time. The output tensor of one layer streams directly to the PCU handling the next layer through the on-chip interconnect. There are no global memory round-trips between layers unless the data explicitly needs to be stored.

For transformer inference, this means the attention output can stream into the feed-forward network without touching HBM. The KV cache lives in PMUs close to the PCUs that consume it. Token generation becomes a continuous pipeline through on-chip fabric rather than a sequence of kernel launches separated by HBM accesses.

Compiler-Scheduled Dataflow vs Runtime Kernel Dispatch

The SambaNova compiler resolves the full execution schedule at compile time. At runtime, the hardware executes a dataflow graph without a runtime scheduler making decisions. There is no kernel launch overhead, no GPU driver interaction per operation, and no scheduler arbitrating between concurrent warps.

TileRT operates at a different level. Even when it fuses operations, it ultimately dispatches them through the host chip's runtime. On a GPU, that means CUDA kernel launches with their associated overhead, driver calls, and dependency tracking by the CUDA runtime. Each launch takes on the order of 5-20 microseconds. Across thousands of decoder steps in a long generation, that overhead accumulates.

A Pulled Quote from the SambaNova Perspective

"A persistent engine kernel means the model is statically expanded ahead of time into one resident program. This is indeed extremely hard. We have been working on this problem for 9 years in our compiler."

-- SambaNova team (source)

This is the crux. TileRT's approach relies on collapsing the whole decode graph into a single persistent kernel. That is a compiler problem SambaNova has been solving for nearly a decade with the right hardware target, and it is not something you catch up on in a release cycle. The difficulty is not just algorithmic. It is that the compiler must reason about a hardware fabric designed for the target. Writing a persistent kernel scheduler for a chip whose memory hierarchy was not designed for it is the hard part, and that is what TileRT is attempting.

Where TileRT Hits the Wall

Kernel Launch and Scheduler Overhead the Host Chip Still Imposes

Even a highly optimized TileRT graph eventually dispatches work to the host chip's execution units. On a GPU, each dispatch is a kernel launch. The CUDA driver tracks dependencies, allocates registers, and schedules blocks across SMs. This overhead is small per kernel but it is non-zero and it compounds.

For short sequences or small batch sizes where inference is already memory-bound, kernel launch overhead can represent 10-30% of total execution time. TileRT cannot eliminate this because it does not control the host chip's dispatch mechanism.

Memory Bandwidth and Cache Behavior TileRT Cannot Rewrite

TileRT improves cache utilization within the constraints of the host chip's cache hierarchy. But it cannot make the L2 larger, cannot place PMU-equivalent memory next to each compute unit, and cannot change the bandwidth between cache levels.

For a 70B parameter model at BF16, loading weights alone requires moving roughly 140 GB of data. On an H100, HBM bandwidth is 3.35 TB/s, which puts the theoretical minimum at about 42ms per decode step if the chip is purely memory-bound. The SN50's PMU fabric reduces the effective distance data travels to reach compute, cutting this floor for the operations that can stay on-chip.

Fixed Interconnect Topology Limits Producer-Consumer Fusion

TileRT's fusion strategy works best when the intermediate tensor between two fused operations fits in the cache available on the SM (or equivalent compute unit) executing both operations. When tensors are too large to fit, the fusion either spills to L2 or HBM or gets split in ways that limit the potential speedup.

On the SN50, the compiler assigns PMU capacity to each operation at compile time and routes data through the PCU-PMU fabric based on what the hardware physically supports. The compiler knows exactly how much on-chip capacity each stage has and can plan accordingly. TileRT operates within whatever the runtime presents, which varies based on what else is running on the chip and how the OS and driver decide to allocate resources.

Debugging a Software-Emulated Dataflow Graph

When something goes wrong in a TileRT execution, the error surfaces through multiple layers: the user's model code, the TileRT graph transformation layer, the host chip's runtime, and sometimes the hardware itself. Stack traces can reference symbols across all of these layers. Performance counters on the host chip reflect the hardware's view of what happened, not TileRT's view of what it intended. Non-determinism from the host scheduler means that a timing-dependent bug may not reproduce reliably.

On the SN50, the compiler produces a static execution schedule. Profiling tools see the same graph structure that the compiler reasoned about. When a performance issue appears, it maps back to a specific PCU-PMU pair in the compiled graph, and the compiler's dataflow model matches what the hardware executed.

The Tensor Parallel Ceiling at 8

No GPU inference deployment on InferenceX, or in public benchmarks, is scaling tensor parallel beyond 8 GPUs for high-speed decoding, despite 72-GPU scale-up domains like NVL72 being physically available. Data parallel does not count, because it does not reduce per-request latency. TileRT's demonstrated mappings still leave significant HBM bandwidth unutilized within TP8.

Until GPU-based dataflow emulation can saturate HBM bandwidth across a large model-parallel domain (16, 32, 64+ GPUs), it will not keep up with dataflow chips that can spread a single model across the full scale-up domain. The interconnect topology that limits TP scaling on GPU clusters is not a software problem. It is a fabric design problem.

Batching at Interactive Latency and 140K Context

The Premium Inference bar is high batch per chip at interactive latency at long context. Median context on InferenceX is now around 140K tokens. Dataflow emulation on GPUs, even on Rubin-class hardware, will struggle to hit this combination because KV cache pressure at 140K context blows past on-chip memory and forces HBM traffic that a static persistent kernel cannot hide.

On the SN50, the same speed target can be met with much higher batch sizes per chip at 140K context. The PMU fabric keeps KV cache close to the compute that consumes it. The difference is not bandwidth per second. It is where the bandwidth has to travel.

Generalization Across Models

Statically expanding a full model into one resident persistent kernel is a hard compiler problem, as the SambaNova team has noted. It has to be redone for every model architecture, every context length regime, every batch shape. TileRT is starting on this problem. SambaNova has been building compiler infrastructure for reconfigurable dataflow for nine years, on hardware that was designed for the target.

The generalization problem is easier when the hardware matches the abstraction. A compiler targeting a chip with fixed memory hierarchy and runtime dispatch must work around the hardware. A compiler targeting a chip whose PCU-PMU fabric was designed around dataflow scheduling reasons directly about the target it controls.

SN50 vs TileRT: The Expected Delta

The following table summarizes the expected performance difference between TileRT on a representative GPU-class accelerator and the SambaNova SN50 for transformer inference workloads.

The latency advantage comes from eliminating global memory round-trips between layers. The throughput advantage compounds the latency gain with the absence of kernel dispatch overhead and the higher effective bandwidth per compute unit. The power figure reflects that the SN50 moves less data over shorter distances, which is the primary driver of power consumption in memory-bound inference workloads.

At 10% of the power draw, the cost advantage from power alone is significant before accounting for capital costs or utilization differences.

The Software Ecosystem Argument

SN50 Tooling Designed Around Dataflow From Day One

SambaNova's compiler, profiler, and debugger all operate on the same dataflow graph model. The profiler shows time spent in each PCU-PMU pair. The debugger inspects the compiled graph directly. When you optimize a model for the SN50, the tools reflect the same abstractions the hardware executes.

This coherence matters for production engineering. When a model runs slower than expected, the profiler points to a specific stage in the graph with specific capacity constraints. The feedback loop between profiling and optimization is tight.

TileRT Debuggability Problem

TileRT introduces a layer of abstraction between the user's model and the hardware. This layer is valuable, but it adds complexity to the debugging process. A performance regression in TileRT may appear in the host chip's performance counters as increased HBM bandwidth, but the cause may be a fusion boundary in TileRT's graph transformation that was not visible in the profiler.

When errors occur, they can surface at any layer: the model code, TileRT's graph rewriting logic, the host runtime, or the hardware. Reproducing these errors consistently is harder because the host chip's scheduler introduces non-determinism that TileRT does not fully control.

Why an Integrated Stack Beats a Retrofit

The SN50 was not originally designed for CPUs and later adapted for dataflow. The hardware architecture, compiler, and tooling were co-designed around the dataflow execution model. TileRT is a retrofit: it applies dataflow ideas to hardware built for a different model. The retrofit can capture some of the benefits, but it operates within the constraints of hardware decisions made before dataflow was the goal.

When TileRT Still Makes Sense

TileRT's real value is fungibility. If you are an inference provider and you buy some systems for prefill and high-throughput decoding, and other systems for fast decoding, and demand shifts toward more fast decoding, you can reallocate GPU capacity with TileRT to serve that demand, albeit at a much higher cost per million tokens. This is a real operational advantage during workload transitions.

The counter-argument: dataflow chips are also fungible at the edges. If workloads shift from fast decode toward higher throughput and medium interactivity, dataflow chips can be reallocated too, at competitive tokenomics. The right design point is fungibility at the edges, not at the extremes: cover enough of the Pareto curve to adapt without over-specializing, but keep enough specialization to differentiate. If both platforms can flex at the edges but dataflow silicon wins on the primary metric by 2-3x latency and 10x throughput and roughly 10x power, the fungibility argument gives buyers more reason to buy dataflow, not less.

Beyond fungibility, there are other situations where TileRT remains a reasonable choice.

Research and prototyping. TileRT runs on widely available hardware. For teams that are exploring model architectures or serving strategies before committing to a specific hardware platform, TileRT on a GPU cluster is more accessible than procuring SN50 capacity.

Workloads where the gap is smaller. For very large batch sizes where the workload is heavily compute-bound rather than memory-bound, the dataflow fusion advantage narrows. Compute-bound workloads spend their time in matrix multiplications where the GPU's tensor cores are already well-utilized, and the benefit from eliminating HBM round-trips is smaller relative to the total compute time.

TileRT is a well-executed engineering effort and it advances the state of GPU inference. The takeaway for the industry is not that TileRT is bad. It is that the fact TileRT has to exist at all is validation of the dataflow thesis. Kernel-by-kernel execution is not the endgame for inference. The question is which stack has the right hardware target and enough compiler maturity to close the generalization problem, and today that stack is dataflow silicon.

What This Means for Inference Buyers in 2026

Latency-Bound Workloads

For real-time applications where latency is the primary constraint (voice AI, interactive coding assistants, chatbots targeting sub-200ms TTFT), the 2-3x latency advantage of purpose-built dataflow silicon is decisive. TileRT can improve GPU latency, but it cannot close that gap because the gap comes from structural differences in the memory hierarchy and interconnect, not from software inefficiency.

For these workloads, the hardware choice matters more than the software layer on top of it.

Throughput-Bound Workloads at Scale

For batch inference workloads where throughput per dollar is the primary metric, the power advantage compounds the throughput advantage. At roughly 10x lower power consumption per token, the SN50 can run at higher sustained throughput without hitting thermal limits. At scale, the power cost becomes a significant fraction of total cost of ownership, and a 10x difference in tokens-per-watt is not recoverable through software optimization.

The Debuggability Tax Compounds Over Time

In the first months of deployment, the debuggability difference between an integrated dataflow stack and a software-emulated one is manageable. Engineering teams learn the tooling and develop workflows for TileRT's abstraction layers.

Over time, as the system scales and edge cases accumulate, the cost of debugging a multi-layer abstraction grows. Production incidents that would take an hour to diagnose on an integrated stack can take a day on a retrofitted one. For teams running inference at scale, this operational overhead is real and measurable in engineering hours.

FAQ

Is TileRT the same as SambaNova's SambaFlow?

No. SambaFlow is SambaNova's own compiler and runtime stack designed specifically for the RDU hardware. TileRT is a separate software layer developed to bring dataflow-like execution patterns to GPU-class hardware. They share some conceptual goals (producer-consumer fusion, compiler-scheduled execution) but differ fundamentally in that SambaFlow has the hardware it needs to execute its schedule, while TileRT must work within the constraints of hardware that was not designed for it.

Can TileRT close the gap with future GPU generations?

Future GPU generations will continue to increase HBM bandwidth and add more on-chip SRAM. These improvements will narrow the gap in some workloads. But the structural difference remains: as long as TileRT runs on hardware with a shared global memory hierarchy and a runtime scheduler that introduces per-kernel overhead, purpose-built dataflow silicon will maintain an advantage on memory-bound inference workloads. The gap may shrink from 10x throughput to 5x, but closing it entirely would require the GPU to adopt a fundamentally different memory architecture.

Why is debugging a TileRT-based system harder than a native dataflow stack?

The core problem is abstraction boundary mismatch. TileRT's graph model does not have a 1:1 correspondence with the host chip's execution model. A performance issue visible in TileRT's graph (a fusion boundary that forces a write to HBM) shows up in the GPU's performance counters as generic HBM bandwidth, without a clear pointer back to TileRT's graph structure. A developer debugging performance must reason about two models simultaneously: TileRT's dataflow view and the GPU's kernel-dispatch view. On the SN50, these views are the same, because the hardware executes the compiler's dataflow graph directly.

---

If your workload is latency-sensitive or throughput-constrained, the inference provider's hardware architecture matters as much as the software stack on top of it. GeneralCompute runs on custom ASIC infrastructure designed specifically for inference, not repurposed training hardware. Try the GeneralCompute API to see what purpose-built inference hardware delivers for your workload.