Founders say general-purpose LLMs, not robot-specific models, may crack robotics

From coding agents to robot control

The hosts frame the episode around MIT professor Philip Isola's essay suggesting the rise of "robot use agents," general-purpose models that can control different robots with little robot-specific training. Hamming (Waddle Labs) and Jay (RoboCurve) say their recent viral demos, including an LLM unscrewing a bottle cap and coordinating between multiple robots, helped inspire that framing. Waddle Labs builds a harness for LLMs to control robots and collects data to train better models; RoboCurve is described as an evals company for physical AI, measuring robots, VLA models, and world-action models across embodiments like hands, grippers, arms and humanoids.

Jay traces the lineage back to Google DeepMind's RT2 paper, which fine-tuned a language model pre-trained on web text and images to output "effector poses" instead of English. He argues RT2's strength came from benefiting from pre-training on broad web data, not from architecture. Hamming compares this to the "chain of thought" moment in language models: early vision-language-action models had to output an action immediately, with no room for reasoning, whereas newer systems can think in code before acting.

The bitter lesson and where VLAs may be limited by data

Vincent argues the "bitter lesson" for robotics isn't about architecture but about which data is useful, since VLA-specific data has remained a longstanding bottleneck with "very very slow progress." He suggests the real unlock is transferring data from modalities models already understand well, like coding and computer use, into robotics rather than relying on scarce robot-specific data.

Hamming adds that models have simply gotten much better at tool use and writing complex policies as code since RT2, largely through in-context learning rather than architectural change. He says the group wants to be "bitter-lesson pilled," pouring computer-use and coding data into robot models rather than training narrow robotics foundation models.

Code as policies and the limits of in-context learning

The conversation revisits "code as policies" research, including Google DeepMind work from around 2022 and the Voyager Minecraft agent, where coding agents were given libraries of functions (pick up object, move to pose) and could one-shot complex tasks without additional robot data, because they already understood sequencing from coding training.

Francois describes experiments testing how far in-context learning (ICL) alone can improve performance, finding gains are "non-monotonic" and "cap out very quickly," typically after 20 to 40 examples, with performance degrading once the model's context window is exceeded. He argues ICL is powerful in low-data regimes but is ultimately constrained by a model's post-training and context length, distinguishing it from heavier approaches like LoRA or full fine-tuning/RL.

"It caps out very quickly. And so after like 20, 30, maybe 40 examples, it is basically saturated and more examples back into the context, not improve." — Francois

Demos, harnesses, and turning actions into reusable skills

The group walks through a demo of the model Astra directly controlling robot arms to move a block into a bowl using camera feeds and tool calls. Hamming notes that direct control by Astra isn't always optimal; repetitive tasks are better served by compiled "skills" or code rather than keeping a large model in the loop each step, partly because of latency. Jay says fable-class LLM latency has been improving roughly 2x per month, and that real-time control could be possible "by end of the year" if trends continue.

They discuss building "harnesses" that consolidate in-context learning into reusable skills or code, describing this as a form of distillation for future agents, with variation points such as VLM-based failure detection embedded in otherwise deterministic code graphs.

The platonic representation hypothesis and predictions for general-purpose robots

Referencing Philip Isola's "platonic representation hypothesis," the group discusses the idea that strong language models and strong robotics models may converge on similar representations of the world as training data scales, implying a single strong general-purpose model could outperform narrower robotics-specific ones. They speculate that Astra's strength may partly come from heavy pre-training on computer-use and CAD data, which teaches spatial concepts like orbiting objects or directional relationships useful for robot control, alongside egocentric video data used elsewhere in the field.

"There's some consensus within the frontier labs and also in the robotics foundation models companies that we will have general purpose robots within the next two years or even earlier." — Jay

Jay defines this as a system that can follow any natural-language instruction to do "what a competent teenager could do with their bare hands," generalizing to unseen tasks and environments, and suggests society is largely unprepared for this. Vincent cautions that visible challenges remain, particularly latency, and floats an analogy to sleep-based memory consolidation as a model for how growing libraries of learned skills might eventually be compressed back into model weights rather than kept purely in context.