YC Paper Club: inference costs 12x more than training, say AI infra builders

Why inference now outweighs training

Opening the session, the host (from YC) argued that inference deserves far more attention than training because of the sheer volume of compute it consumes over a model's lifetime. Using DeepSeek's published numbers as the only public data point, he noted DeepSeek V3 took about 2.8 million H800 hours to train and runs roughly 43,000 H800 hours per day, implying inference uses about 6x more flops than training per year, or "12 times more important" over a two-year useful life. He pointed to recent numbers implying DeepSeek's inference margins reach 80%, crediting heavy inference-side innovation, and noted inference has produced over $220 billion in market cap in the past decade.

He broke inference innovation into three buckets: model-level changes (MoE, SSMs, multi-headed latent attention, quantization, distillation), kernel and runtime innovations (radix attention, flash attention, page attention), and harness-level tricks (routing, speculative decoding, recursive language models, compaction). He cited DeepSeek figures from the prior day: 60,000 tokens/sec per GPU on prefill excluding cache hits, 1.5 million tokens/sec with a 96% cache hit rate, time-to-first-token around 400ms (P99 about 1 second), and 14,000 tokens/sec per GPU on decode — illustrating why decode is the bottleneck driving chip innovation.

Dan Fu (Together AI): the lifecycle of a request

Dan Fu, who runs research at Together AI, walked through how a request moves from API call to GPU and back, covering routing (balancing load while trying to keep KV caches resident on the same GPU), tokenization (handling tool calls), scheduling, and the split between prefill (compute-bound) and decode (memory-bound) stages. He said Together serves open-model traffic at scale, noting that at one point the company handled "1/8 of total traffic that Google is doing" and that open models now account for more than half of tokens routed through OpenRouter.

He detailed three optimization levers: kernels (including an internal project, tentatively still called "Together Compiler," using agents to auto-optimize kernels against real production traffic), parallelism (citing larger GPU scale-up domains like 72-GPU NVLink racks enabling expert parallelism, tied to DeepSeek's cited "$38 per hour" figure), and quantization. On quantization, he described working with MiniMax to fix a failed attempt to quantize their M3 model to NVFP4, where Together's own recipe matched original FP8 performance while MiniMax's own recipe did not.

Jeff Morgan (Ollama): not all open-model tokens are equal

Jeff Morgan, founder and CEO of Ollama, argued that the same open model can perform very differently depending on which provider serves it, because of differences in quantization, tool-calling implementation, sampling, and kernel choices. He said research has found the quality gap between the best and worst provider serving the same model can be larger than the gap between open and closed models, citing a 2024 paper that tested 31 different Llama endpoints and found some performed closer to GPT-3.5 than to other Llama deployments of the same model.

He warned this problem compounds in agentic workloads: a modest 15-step coding task with a 10% per-step deviation could compound toward an 80% error rate. He outlined several ways builders can monitor provider quality: expensive end-to-end evals (as run by Artificial Analysis), distribution tests comparing provider outputs to a reference model, log-probability drift checks, and simply monitoring tool-call failure rates and output-length patterns over time. He also noted some labs, including DeepSeek with its new "DeepInfer" stack, are now publishing their own serving stacks and quantized weights so providers can replicate official performance more reliably.

"The gap between the worst provider for a model and the best provider is actually bigger than the gap between closed and open models." — Jeff Morgan

Joe Fioti (Luminal): compiling by search, not heuristics

Joe Fioti, co-founder and CEO of Luminal, described his company's approach of using "equality saturation" to compile models rather than relying on hand-written heuristics or large kernel-engineering teams. Luminal reduces PyTorch models to roughly 14–15 primitive operations, builds an e-graph that captures many equivalent program forms, and then measures real hardware runtime to pick the fastest version rather than relying on a cost model, which he said is "many, many PhDs worth of work" to build accurately for modern GPUs.

He said the compiler only searches "1% of 1%" of the possible space yet reaches near-optimal results, since the goal is a local optimum close to the global one rather than the global optimum itself. He said Luminal currently runs roughly 80–90 trillion tokens a month across GPUs, TPUs, Trainium, and other ASICs with a team of eight people, and that the company has solved compiler-company monetization, which he noted has historically killed many compiler startups.

"The best cost model is no cost model and we should actually just go and run these things on the real hardware." — Joe Fioti

Liangsheng Yin (SGLang): memory architecture for agents

Liangsheng Yin, a core developer of SGLang, traced the project's origin to the "Radix Attention" paper, built to let inference engines automatically reuse shared prefixes across parallel agent workloads via a radix-tree memory structure. He said SGLang later added overlapped (zero-overhead) scheduling to hide CPU-GPU synchronization bubbles, which he said can roughly double throughput on Blackwell-class hardware versus vanilla scheduling, and hierarchical caching across GPU, host, and disk tiers (L1/L2/L3), which he said lowered time-to-first-token by 56% in their experiments.

He described a newer "unified radix tree" and "unified memory pool" designed to handle hybrid architectures mixing full attention, sliding-window attention, and recurrent states, since these require coordinated, flexible cache-sharing rules rather than fixed memory ratios per component. A "compaction" operation defragments physical GPU memory by repacking fragments and remapping virtual page IDs with low-latency copies, avoiding wasted memory from mismatched pool boundaries.

Open questions and areas of disagreement

Speakers did not converge on several points. Dan Fu said he is not a hardware specialist and could not explain the technical jump from 8-GPU to 72-GPU NVLink interconnects in detail. On future model architectures, he noted that ideas once dismissed as dead ends, such as SSM-style layers, sparse attention, diffusion language models, and looped models, are already being used in leading models, suggesting rapid, unpredictable uptake of new architectures.

Jeff Morgan acknowledged he does not know of a startup yet dedicated purely to monitoring LLM serving quality, calling it "a really interesting idea" rather than an existing solution, and flagged an open risk that quantized models could be tuned to "hill climb" public evals without preserving true quality, a problem he said is partly addressed only through private, bespoke customer-specific test suites.

Joe Fioti noted that Luminal's compiler explicitly does not attempt to replace speculative decoding or parallelism-scheme innovation, leaving those to inference engines like vLLM and SGLang, which the other speakers' teams build.