This piece cuts through the hype of specialized AI hardware with a counterintuitive claim: you don't need a new chip to achieve ultra-low latency, you just need to rewrite the software that runs on the chips we already have. Dylan Patel argues that the industry's rush to buy purpose-built inference accelerators may be premature, as a software breakthrough called TileRT is already shattering performance records on standard Nvidia GPUs. For leaders managing compute budgets, the implication is stark—massive capital expenditure might be avoidable if the bottleneck isn't silicon, but rather the operating system's inefficiency.
The Latency Ceiling
Patel begins by dismantling the assumption that GPUs are inherently too slow for real-time, interactive AI. He notes that while traditional models prioritize batched throughput—processing thousands of requests at once—interactive applications like voice assistants require a different metric: tokens per second per user. "GPUs perform exceptionally well at high throughput and low-to-medium interactivity, but their architecture is less suited for ultra-low-latency inference," he writes. This distinction is critical because the current market is bifurcating between systems designed for cost-efficiency and those designed for human-like responsiveness.
The author highlights a frustrating reality in hardware design: while memory bandwidth has skyrocketed, memory latency has remained stagnant. "While GPU memory bandwidth increases by roughly 2–3× each generation, memory latency has not improved at all," Patel observes. This creates a paradox where newer, more expensive hardware often fails to deliver faster interactive responses because the software overhead of launching and synchronizing thousands of tiny tasks eats up the gains. This framing is effective because it shifts the blame from the hardware manufacturer to the software stack, a nuance often lost in hardware-centric reporting.
"The traditional GPU programming model launches and synchronizes many individual kernels, whose setup and teardown overhead becomes significant at ultra-high levels of interactivity."
Critics might argue that software optimization has diminishing returns compared to the raw architectural shifts of dedicated dataflow chips. However, Patel's data suggests that the gap is not yet closed, leaving room for software to bridge the divide.
The Persistent Kernel Solution
The core of Patel's analysis focuses on TileRT, a project that fundamentally changes how code executes on a GPU. Instead of treating the GPU as a collection of independent workers that wake up, do a task, and go to sleep, TileRT compiles the entire model into a single, persistent engine. "TileRT statically compiles the entire decode graph into a single persistent kernel on NVIDIA GPUs, maximizing overlap across computation, memory loads and stores, and communication," he explains. This approach draws on historical lessons from kernel optimization, where the overhead of context switching can destroy performance, similar to how early operating systems struggled with process scheduling before modern pre-emptive multitasking became standard.
By keeping the engine resident on the GPU and overlapping data movement with computation, TileRT eliminates the "stop-and-go" nature of traditional inference. Patel illustrates this with a striking analogy: "A bus amortizes its cost across many passengers but makes each passenger wait for shared stops. A race car carries only one or two people and reaches the destination faster, but at much higher cost per passenger." He positions TileRT not as a bus, but as a "private rocket ship with room for just one passenger," designed specifically for the latency-sensitive decode phase of AI generation.
The results presented are difficult to ignore. On an eight-GPU B200 server, TileRT achieved 340 tokens per second per user, nearly double the previous best on comparable hardware. "TileRT at FP8 outperforms the best previously recorded GLM-5.1 result by 4.5× at 1k/1k and 3.0× at 8k/1k," Patel writes. This evidence suggests that the "specialized chip" narrative may be a solution in search of a problem that software can solve.
"A CUDA graph optimizes the launching of kernels, while TileRT abolishes the kernel as the unit of execution."
This is the piece's most technically significant claim. It suggests a paradigm shift where the unit of execution moves from the kernel to the entire graph, effectively turning the GPU into a dedicated inference engine without changing the silicon.
The Trade-Off and the Ecosystem
Patel is careful not to present TileRT as a silver bullet. He explicitly acknowledges the trade-off: extreme interactivity comes at the cost of aggregate throughput. While TileRT delivers blistering speed for a single user, it cannot match the total volume of tokens a conventional engine can process across thousands of users simultaneously. "TileRT provides much higher per-user speed, but the conventional GB300 point completes more aggregate work per GPU," he admits. This honesty strengthens his credibility, as it avoids the trap of claiming one technology solves every problem.
The proposed solution is a disaggregated architecture, where the system splits the workload. High-throughput engines like vLLM handle the initial "prefill" phase (reading the prompt), while TileRT takes over the "decode" phase (generating the response). "With PD disaggregation inference technique, the hyperspecialized TileRT engine handles latency-sensitive decode while throughput-optimized engines such as vLLM and SGLang continuing to serving prefill," Patel writes. This hybrid approach allows data centers to use their existing Nvidia fleets more efficiently, potentially disrupting the total addressable market for companies like Cerebras and Groq that rely on selling proprietary hardware for low-latency tasks.
Bottom Line
Patel's argument is compelling because it leverages the massive installed base of Nvidia GPUs to challenge the necessity of a hardware arms race. The strongest part of the analysis is the demonstration that software architecture, not just silicon, dictates the limits of interactivity. However, the biggest vulnerability lies in the scalability of the "single-passenger" model; if the cost of running a dedicated decode node for every user becomes prohibitive, the economic case for specialized hardware may still win out. The industry should watch closely to see if TileRT can scale its single-user speed to a multi-user environment without losing its latency advantage.
"TileRT abolishes the kernel as the unit of execution."