🧠 From Scaffolding to Self-Improvement: How Agent Harnesses Became the Unlock for AGI-Level Performance

📊 The Research That Was "Too Basic" for Top-Tier Conferences

Just a month ago, a Reddit thread dismissed prompt engineering as "not a research problem." Another commenter called it "scaffolding" unworthy of serious academic attention. Yet agent harnesses—the frameworks that orchestrate how models interact with tools, memory, and execution environments—have delivered 18% performance improvements between iterations and have become the difference between solving ARC-AGI and not.

The evidence is stark: Claude Opus achieved 30% on ARC-AGI's private holdout set with basic inference. With an advanced harness, that figure jumped to 95%. Nvidia's AVO configuration hit 100%. These aren't incremental gains—they're step-function improvements unlocked purely through better orchestration, not larger models.

🧩 The Static Harness Era: Building Blocks of Intelligence

Early harnesses were rudimentary: a simple while-not-end-of-sequence loop with top-P sampling. Over time, researchers added crucial primitives:

• In-context learning (ICL): Few-shot examples to guide behavior
• Chain-of-thought (CoT): Smearing compute across multiple tokens to improve reasoning
• Tool-calling (Tool Former): JSON-based function invocation—letting models call Python, APIs, or external systems
• Memory management (MemGPT): CRUD operations on context itself, separating active context from persistent storage
• Skills (Voyager): Reusable, composable procedures distilled into markdown files
• Multi-agent reflection (ReAct, Self-Refine): Internal evaluators that critique and improve outputs before submission
• Recursive agents (RLM): Orchestrator agents spawning sub-agents with their own contexts, creating hierarchical task decomposition

These primitives remain static—defined by the harness architect, not improved by the agent itself. But they laid the foundation for what came next.

🔄 The Self-Improving Era: When Harnesses Learn to Optimize Themselves

The last six months have seen a shift: harnesses that rewrite their own scaffolding. Instead of fixed system prompts and tool lists, these frameworks use genetic programming and meta-learning to evolve:

"We're not just giving the agent tools—we're letting it redesign the toolbox."

DSPY (Demonstrate, Search, Predict) introduced the concept of optimizing system prompts via gradient-free search over training examples. Darwin Machines went further: agents now modify the harness code itself, choosing which tools to expose, how to structure context, and when to spawn sub-agents—all through an evolutionary archive of configurations tested against fitness functions.

Continual Harness added test-time training—agents can now update their own weights via small-batch learning (Dagger-style online learning) on newly encountered tasks. This closes the loop: agents don't just adapt their prompts or tools—they fine-tune their underlying intelligence in real time.

🏗️ Prime Agent: A Self-Improving RLM Harness Built for Long-Horizon Tasks

Prime Agent, developed by researchers at Princeton and Prime Intellect, exemplifies the new generation of harnesses. It treats the LLM as a von Neumann computer rather than a Turing machine—capable of reading and writing to external memory, not just sequentially processing a tape.

Key innovations:

• Persistent sub-agents: Parent sessions can spawn child agents that remain in RAM, preserving context across long tasks without re-prompting
• IPython ripples: Live Python environments where agents manipulate variables in memory, avoiding token-heavy context bloat
• Agentic garbage collection: Automatic cleanup of stale variables and inactive sub-agents to prevent RAM overflow
• Inter-agent messaging: Siblings and parents can coordinate directly, sharing context without reloading
• CRUD on harness state: Agents can update their own system prompts, skill libraries, and tool lists based on task history

Performance results:

• ARC-AGI: GPT-4o achieved 78% (vs. 25.7% with the Responses API alone); Claude Opus hit 95.5%
• Cost efficiency: Programmatic context management reduced per-task costs significantly vs. alternatives like Air Agent
• Long-horizon persistence: A 7-day Factorio run used 633 agents, 23 million output tokens, continuously progressing through the tech tree without getting stuck

The harness avoided the "babysitting" problem common in early agents: it sustained multi-day workloads by refining its own memory and skills, not by burning through token budgets.

💻 Open Jarvis: Bringing Agent Intelligence On-Device

Open Jarvis, from Stanford's Hazy Research Lab, flips the script: what if the entire agent stack—inference, execution, memory, learning—ran locally?

The thesis: Cloud-bound personal AI is costly, privacy-invasive, and energy-intensive. Local models like Qwen 2.5 32B are now only 6–12 months behind frontier models in capability, while offering 800x cost reduction and dramatic latency improvements.

Open Jarvis primitives:

• User interfaces: Desktop and Slack integration, matching cloud UX
• Agentic logic: Composable reasoning with tool orchestration
• Intelligence engines: Qwen, Gemma, GGUF—swappable on-device models
• Inference backends: Ollama, Llama.cpp, vLLM, optimized for Apple Silicon and Nvidia GPUs
• Learning systems: Both prompt-based (DSPY) and weight-based (LoRA, SFT) adaptation

The optimization loop: Cloud models like Claude Opus or GPT-4.5 Turbo configure the local stack—choosing the right model, harness structure, and tool set—then deploy it for inference. This hybrid approach captures frontier intelligence in the setup phase while keeping runtime costs near zero.

"We're letting cloud LLMs design the blueprint, then running it entirely on your laptop."

🤝 QM: YC's Open-Source Agent for Work

YC's internal agent, QM, demonstrates how harnesses scale to organizational use. It powers Slack-based assistance and web UIs for 50+ employees, handling email triage, document editing, database queries, and event planning.

Evolution of YC's agent stack:

• January 2025: "General Agent"—one-size-fits-all system prompt with tools in a loop
• June 2025: Hooked Cloud Code to Slack, enabling VM-based code changes and CI pipeline execution
• April 2026: Deployed 50+ Hermes agents in VMs—powerful but difficult to manage
• May 2026: Built QM to centralize intelligence and treat sandboxes as resources, not homes

QM's architectural choices:

• Brain in Postgres: All session state centralized, accessible across sandboxes
• Dynamic sandbox allocation: Agents choose machine resources based on task complexity
• Model switching: Agents select providers to avoid refusals (e.g., switching from Opus to open models for security research)
• Grind budgets: Agents are not allowed to give up before hitting token or time thresholds, forcing deeper exploration
• Human-reviewed bulk upserts: Database writes require approval—though users increasingly "rubber stamp" them

Key lessons: Agents still struggle with social context—leaking privileged information into public channels remains a risk. Robust permissioning systems are essential for multiplayer agent environments.

🔬 The Harness as Operating System

The meta-narrative emerging from these projects: harnesses are becoming agentic operating systems. They manage memory hierarchies (L1 = model weights, L2 = active context, L3 = file storage), allocate compute (spawning sub-agents, choosing GPU clusters), and evolve their own architectures.

The distinction between "tool" and "skill" is blurring. Is outputting a function a tool call or a skill? Does it matter if the agent can rewrite both?

The practical plateau: Instead of measuring performance at arbitrary token budgets, researchers now ask: "At what point do we only get incremental gains?" This shift toward long-horizon evaluation reveals where harnesses truly shine—not in one-shot queries, but in sustained, multi-day research, debugging, and planning.

🚀 What This Means for Builders

If you're building agents today:

• Think beyond static loops: CRUD on system prompts, tool lists, and memory is table stakes
• Expose programmability: IPython shells, MCP-compatible tools, and sub-agent APIs unlock capabilities models can't yet express natively
• Design for persistence: Sessions that survive laptop closures, context that survives compaction—agents need to live in their environments
• Optimize the harness, not just the model: The same weights perform 18% better with better scaffolding
• Run standardized evals: Prime Intellect's verifiers package and public benchmarks (ARC-AGI, Oolong, Emulator Bench) let you compare apples to apples

🎯 The Takeaway

Harnesses were dismissed as "not research." Now they're the difference between solving ARC-AGI and not. Between agents that give up after five minutes and agents that run productively for a week. Between cloud-only systems and privacy-preserving, cost-efficient local stacks.

The models are getting smarter. But the harness is what decides whether that intelligence gets unleashed—or bottlenecked by brittle scaffolding.

As one researcher put it: "We keep measuring perplexity and IQ, but we're not leveraging test-time experience. Harnesses are how we close that gap."

📚 Further exploration: Prime Agent, Open Jarvis, and QM are all open source. Try them. Break them. Improve them. The era of self-improving agents is here—and the harness is the unlock.