Research project

SoL-Pi:Scaling Auto-Research Loops for Efficient Agent Harnesses

Spend less without getting less done.

Layered code papers traced by recursive research paths toward a verified result

Introduction

The arc of coding agents has been short and steep. They moved from completing lines to resolving issues across repositories, acting on feedback and even modifying the tools around them. With modern harnesses, thousands of agents can now work together continuously for a week without human intervention. This raises a basic question: over an unsupervised run of hours or days, does every token carry the work forward, or does redundancy grow with trajectory length?

The question deepens with recursive self-improvement (RSI). Once AI can work on complex software systems, it can also work on the systems that produce AI. RSI may therefore accelerate progress toward more general intelligence. Yet RSI is itself token-intensive: every attempt to produce a better system spends tokens, whether or not it succeeds. Before scaling RSI, should we first ask AI to make AI more efficient?

To address this problem, we built a scalable RSI pipeline. Agents construct executable environments for harness rollouts from public data and publicly available software environments. Other agents observe how models explore within the resulting trajectories, then turn those observations into auto-research loops that improve the harness under an efficiency objective. The mechanisms that survive these searches are combined into SoL-Pi, an agent harness focused on token efficiency. For a professional researcher working on a single problem, SoL-Pi saves $8.75–$13.50 per hour vs. native Codex and Claude Code harnesses, and $4.36–$5.71 per hour vs. Pi.*

Last but not least, SoL-Pi points to a broader view of RSI: its lasting value may lie not in any single artifact, but in a search process that can scale across public environments to discover reusable improvements.

Pi is the lightweight, extensible coding-agent harness that serves as our research substrate. EdgeBench is a 51-task suite of long-horizon executable agent work; we reserve its tasks, verifiers, and feedback for final held-out evaluation. * The savings range reflects the model backend and is calculated using official API-equivalent pricing.

Method

SoL-Pi sends 152 proposed directions through auto-research loops; four mechanisms survive into the final harness. Use backup video · Download video

We treat harness improvement as an open-ended RSI problem. We search across different environments for changes that can transfer to unseen settings. The auto-research process generates mechanism ideas in batches, sends each selected idea into an independent loop, and carries forward candidates that survive validation.

The target is constrained efficiency: we search for reductions in cost or token use subject to a predeclared capability-preservation criterion. The research procedure remains fixed. Surviving mechanisms define the next research frontier.

Auto-research proposal pool

152proposed ideas4surviving mechanisms

Pause motion

C1Use observed context pressure instead of global capsC4Promote facts by evidence and keep the prompt prefix stableC7Preserve compact contract anchors across compactionC10Pop verified subgoal frames into result cardsC13Disable dormant mechanisms at configuration timeC16Mask old observations selectivelyC19Keep the authoritative memory journal append-onlyC22Treat curated working state as a first-class productP1Make progress control evidence-basedP4Recover artifacts without destructive cleanupP7Add a focused repair controller after failed checksP10Demand a check that can distinguish the broken stateP13Escalate after a narrow fail-to-pass flip only with broader evidenceP16Escalate repeated blocked completion with an exact commandP19Prefer selective soft boundaries to hard exploration blocksP22Debounce automatic checks and keep evaluators subordinate to evidenceP25Delete action-free-turn guards when the event pool is emptyT2Repair malformed tool payloads within a bounded schemaT5Execute deterministic command sequences with `run-steps`T8Preflight only the capabilities the task will needT11Manage long commands as explicit lifecyclesT14Inject environment facts from deterministic sourcesT17Deduplicate exact provider tool-call IDsT20Interpret command outcomes by contract, not exit code aloneT23Make `run` a language-neutral task managerT26Register agents and capabilities as dataD3Route delegation by evidence need, not task length aloneD6Avoid generative edit delegates without a verifier contractD9Optimize topology under a hard non-regression constraintD12Split model roles by decision densityD15Make each Actor own a role and context historyR3Minimize solution scope explicitlyR6Test hiding visible reasoning summaries separatelyR9Route reasoning effort with verified progress signalsR12Do not use few-shot, RAG, or keyword gates as the primary leverR15Charge escalation for cache discontinuityM3Require isolated, paired, and sealed gatesM6Use issue history and blind failure mining as proposal evidenceM9Replay alternatives from critical checkpointsM12Judge patches with fail-before/pass-after testsM15Bind reviewer evidence safely and prove mechanism activationM18Give analyzers a layered evidence ladderM21Use a read-only orchestrator and a filesystem queueM24Gate V2 candidates on consumption, dormant behavior, and distribution shiftM27Put a bounded authoritative gate on the final diffM30Fail closed on negative or ambiguous terminal statesM33Compile trajectories into structural triage viewsM36Resolve live service identity before operating on itM39Establish role architecture before optimizing promptsM42Choose tasks with mechanism headroom before scalingM45Repair data trust before single-model capability tuning

C2Reset from a verified replacement checkpointC5Keep an immutable archive with targeted rehydrationC8Reassemble tiers only on explicit triggersC11Reduce tool output before its first prompt insertionC14Measure provider reasoning as its own context axisC17Query an external context historianC20Detect and repair prompt-cache prefix churnC23Build observation packs before paying for full bodiesP2Treat the tail as an explicit state transitionP5Break stalls only when the next action is legal and informativeP8Couple successful mutations to their verifierP11Do not generalize long-context output pruning from train hitsP14Check blast radius after the first consequential editP17Separate verified, declared, and advisory completion modesP20Repay mutation validation debt in ReAct loopsP23Break repeated diagnostic loops without stopping real progressP26Use the final gate to protect evidence, not to claim capabilityT3Anchor edits and expose artifact-correction actionsT6Do not add a persistent REPL without demonstrated state reuseT9Coalesce waiting and repeated callsT12Load specialist tools and schemas only when neededT15Apply unified diffs strictly before one unique fallbackT18Emit streaming updates only at semantic boundariesT21Promote high-frequency discovery into bounded primitivesT24Deliver asynchronous completion at turn boundariesD1Delegate oversized result compression with deterministic fallbackD4Bound child work and measure parent reuseD7Make child execution isolated, bounded, and default-offD10Hand off fresh context between research, implementation, and reviewD13Buy extra attempts only behind a hard verifierR1Adapt reasoning and verbosity to phase and evidenceR4Make provider capabilities part of runtime policyR7Do not treat blunt brevity or early compaction as a mechanismR10Merge narration into one event-driven status lineR13Preserve exact model identity and real effort ceilingsM1Mine paired trajectories with counterexamples and controlsM4Keep a failure memory for future proposal roundsM7Preserve negative evidence and unavailable historical candidatesM10Select among rollout summaries with a recursive tournamentM13Spend a second full rollout only on high-risk tasksM16Use longer-step rescue only as a diagnosticM19Keep service-tier savings in a separate cost laneM22Keep the research funnel separable and stoppableM25Separate scheduling, execution, and monitoring authorityM28Seal the manifest before computing coverage or metricsM31Prevent proposal examples and private tasks from entering evaluationM34Optimize quality and transcript economics as separate axesM37Use one worktree boundary for isolation, rollback, and comparisonM40Admit expensive mechanisms through paired, low-cost gatesM43Turn expert steering into a falsifiable diagnosis protocolM46Isolate protocol reforms before making them mandatory

C3Maintain memory tiers on different clocksC6Use layered context epochs and event-driven compactionC9Evict stale observations deterministically before adding a summarizerC12Invalidate reads precisely after mutationC15Recover ReAct context overflow without splitting tool atomsC18Carry recent, still-valid reads across compaction verbatimC21Use progress cards when a long subtask has no clean boundaryC24Gate ObservationPack by expected lifetime valueP3Track verification and acceptance debt separatelyP6Require validation evidence before claiming completionP9Give counterexamples precedence and break repeated failure loopsP12A static-reading streak is too weak a validation triggerP15Budget repair per error fingerprintP18Checkpoint workspace and process state, not only messagesP21Make final output an explicit state transitionP24Discover the smallest verifier after mutationT1Fix tool-call transport before tuning behaviorT4Batch independent probes without hiding causalityT7Do not rely on a passive observation-pressure latchT10Slim the tool surface around common action sequencesT13Retry a thinking-only length stop onceT16Use provider schema constraints only when they are realT19Keep generic action batches thinner than a workflow engineT22Design the core surface around `search`, `view`, `edit`, and `run`T25Add a semantic layer before polishing byte-level readsD2Delegate bounded subproblems early enough to matterD5Route models by subproblemD8Decide routing during the first two discovery actionsD11Try a cheap scout before a conditional reviewerD14Consult without handing over the execution chainR2Distill tool-family policy, not only tool syntaxR5Preserve provider reasoning items across tool callsR8Defer semantic prompt and schema deduplicationR11Derive thresholds from the current trajectory, not the benchmarkR14Probe a model before adding compatibility shimsM2Separate hypothesis authorship from evidence reviewM5Attribute cost to actions, phases, and providersM8Evaluate a mechanism at its exact first firing pointM11Feed new attempts multiple prior failure summariesM14Search harness levers with frozen layered workflowsM17Run paired optimization loops for alternative PI transportsM20Express orchestration with a few native language primitivesM23Prove a capability gap before designing a leverM26Train on mixed distributions and estimate baseline noise firstM29Make the evidence ledger transactional and replayableM32Bind review and repair packets end to endM35Stop when the evidence substrate is invalidM38Put single-turn mechanisms in the kernel and cross-turn policy outsideM41Decompose packaged failures before rejecting their componentsM44Classify replay by the first changed decision

CContext24PProgress26TTools26DDelegation15RPrompt & policy15MImprovement & evaluation46

Parallel auto-research loops

Inside each lineageAuto-research loop

One idea moves through Trajectory Rollouts, Map–Reduce Analysis, and Proposal. One agent implements the proposal, then another agent reviews it. Reviewer returns to Implementation when revisions are needed. The idea then moves through In-Trajectory Validation and Held-Out Validation. In-Trajectory Validation can return to Implementation or Trajectory Rollouts. Held-Out Validation evaluates the frozen candidate on a fully isolated test set.

Trajectory RolloutsGenerate exploration trajectories

Map–Reduce AnalysisMap separately, then reduce evidence

ProposalPropose one mechanism

ImplementationOne agent implements

ReviewerAnother agent reviews

In-Trajectory ValidationTraining set screen

Held-Out ValidationFrozen candidate · isolated test set

Parallel across ideas; iterative within each lineage.

At the outer level, SoL-Pi fans out a broad idea pool into independent mechanism lineages. The first pass starts with 152 proposed directions. Oracle Analysis, a pre-rollout stage that estimates opportunity from existing trajectories, screens them before the system spends rollout budget. Each selected direction makes a concrete claim about avoidable work in the harness, so the system can reject a weak lineage without discarding progress elsewhere.

The six proposal families record where a hypothesis begins, not where a surviving mechanism is eventually implemented. ObservationPack, for example, starts in the Context family as C23 and C24, then becomes a mechanism at the observation boundary.

Within each lineage, we follow Karpathy's conventional autoresearch experiment loop: propose a change, implement it, run the experiment, read the result, then keep or discard the change and repeat. We make three adaptations for harness research.

First, implementation runs as a Ralph Loop, an iterative implementation loop that continues until an explicit exit condition passes. A separate reviewer checks the implementation and its behavioral contract. Failed reviews return the proposal for revision.

Second, map-reduce analysis handles multiple exploration trajectories. Independent analyzers inspect each trajectory, and a reducer merges their evidence before the next mechanism proposal.

Third, validation uses two isolated splits. We develop and screen candidates on the training set. Once we freeze the mechanism and acceptance rule, the system evaluates them on a held-out test set. Held-out trajectories never enter subsequent analysis, and no agent inside the auto-research loop sees the held-out results. A failed held-out evaluation rejects the frozen candidate without becoming feedback for repair or another search episode.

From compiled workflows to disposable skill loops

Workflow evolutionThree designs

Design	Advantage	Limitation
01Compiled workflow	Clear handoff	Brittle beyond the fixed graph
02Code orchestration	Runtime flexibility	Code and tests accumulate
03Disposable skill loop	Easy parallel scaling	Requires a reliable template

Orchestrating auto-research across hundreds of ideas and hundreds of research iterations is not straightforward. Our workflow went through three designs, each differing in where the orchestration lives, how long its code survives, and what breaks first at scale.

Compiled workflow. Each agent flow was described in YAML and compiled into an executable workflow, so the orchestration lived in a configuration file and was fixed once compiled. The explicit graph made collaboration and handoff straightforward. At scale the fixed graph was the limit: a precompiled workflow could not cover every edge case, runs repeatedly stopped for human repair, and operators could not reconstruct the context of hundreds of concurrent attempts.

Code orchestration. A lead agent wrote the coordination code, opened sessions, passed messages among them, and assembled each workflow at runtime, so the orchestration lived in code that persisted across experiments. This removed the fixed graph. At scale the coordinator was the limit: keeping every loop compatible meant adding branches, tests, and verification scripts to a long-lived and unbounded codebase, and launching a new experiment could require more than ten hours of changes to it.

Disposable skill loop. We maintain one minimal loop template together with the instructions for using it. Each experiment instantiates a fresh copy, sets its parameters, runs it, and discards its modified orchestration code when the loop ends, so orchestration code survives only for the length of one run. Scaling becomes repeated template instantiation rather than continued expansion of one coordinator. Nothing here is free: the template itself has to be kept correct, since every experiment inherits it. That single shared dependency has been easier to maintain than either a fixed graph or a growing coordinator.

Two environment families separate search from evaluation

We build 535 executable training environments in two families. The first contains 495 tasks with ground-truth trajectories mined from GitHub issue-PR pairs. The second contains 40 synthetic tasks defined by executable verifiers.

For the first family, the issue specifies the task and the linked PR provides the accepted patch and change history, which serve as the ground-truth trajectory. We restore the repository to the commit immediately before the fix, install its dependencies in an offline image, and withhold the PR and the maintainer's regression test from the agent. We retain the environment only if the test fails before the PR and passes after it, keeping the task, trajectory, and verifier consistent.

The second family is verifier-driven. We first generate an executable verifier that defines success, then construct an environment around it for open-ended exploration without a reference trajectory. These 40 environments mainly use Terminal-Bench-style verifiers. To keep EdgeBench a valid held-out benchmark, we do not use EdgeBench data or verifiers as sources or templates for environment synthesis. EdgeBench tasks and feedback remain outside harness search; we use them only for held-out validation. This separation makes its score a stronger measure of generalization to unseen task and verifier designs.

Capability floors constrain efficiency gains

What the search learns is a reusable harness mechanism. The capability floor decides which candidates survive. A cheaper candidate fails if it saves by stopping early, skipping necessary verification, or removing evidence required to finish the task.

Each loop applies two acceptance gates. First, every capability metric must stay within its predeclared tolerance. Second, at least one efficiency metric must improve. Among candidates that pass the capability floor, the loop retains nondominated results. The gate applies to one mechanism at a time, so the small losses it permits can accumulate once mechanisms combine: the assembled harness retains roughly 94% of Pi's average score. What the gate rules out is savings that come from getting less done.

What auto-research loops discovered

Auto-research outcomes

The diagram shows the base harness nodes Agent, Context Memory, Action, and Observation. Hover, focus, or tap a node to reveal the auto-research outcome for that component. The animation traces Task through Agent, Action, Environment, and Observation, with Context Memory exchanging information with Agent before the cycle repeats.

Environment

Task

Hover, focus, or tap a component to reveal the auto-research outcome that modifies it.

Auto-research produced four narrow mechanisms. They map onto tools, context management, observation compression, and multi-agent delegation.

ToolsAction Fusionone intent, one turn

Action Fusion combines an edit and its follow-up command in a two-lane comparison — Action Fusion keeps an edit and its follow-up command inside one local sequence, removing the middle model decision.

Base Pi rollouts exposed a recurring sequence: after editing a file, the coding agent often issued a command to test, build, or run the change. Action Fusion turns that sequence into one tool call. The harness applies the edit and runs the command locally, then returns one combined observation without another model round trip.

ContextOnline Context Compactcompact at subtask boundaries

Online Context Compact compresses completed subtasks before later requests replay the full history — Online Context Compact treats a completed subtask as a possible compaction point, then waits until later requests can repay the rewrite.

KV-cache reuse usually pushes compaction late into a run. Online Context Compact uses a different clock: it decomposes a task into subtasks and reconsiders compaction whenever one finishes. Online Context Compact turns semantic completion into a compaction trigger, but acts only when expected future savings can repay the rewrite.

ObservationsObservationPackpreserve access, remove repetition

ObservationPack replaces repeated tool output with a compact handle while preserving recall — ObservationPack replaces repeated full output with a stable handle while keeping the original available for paged recall.

In base Pi, a large file or tool result reappeared in every later request and occupied both context and cache. ObservationPack changes that lifecycle by archiving the payload locally. It leaves a handle and short excerpt in context, then recalls exact pages only when needed. The harness keeps the payload accessible without replaying it.

DelegationEvidence-Preserving Reducerdelegate reading, verify evidence

Evidence-Preserving Reducer turns a long diagnostic log into a verified short receipt — Evidence-Preserving Reducer accepts a compact diagnostic receipt only when its evidence can be checked against the archived log.

In build and test trajectories, often only a few lines of a long log changed the next decision. Evidence-Preserving Reducer uses that boundary to delegate the first reading to a cheaper agent. It binds the resulting receipt to the archived log and verifies every quoted line before the frontier agent sees it. Delegation no longer requires trust in a fluent summary.

Results

Grouped bars comparing average EdgeBench score for native harness, Pi, and SoL-Pi on GPT-5.6 Sol and Claude Opus 5 — Average score

Grouped bars showing lower official API-equivalent cost for SoL-Pi across both model backends — Official API-equivalent cost

Grouped bars showing lower total token use for SoL-Pi across both model backends — Total tokens

In this work, we used frontier AI models to scale auto-research loops and build SoL-Pi, a new agent harness. All comparisons ran each model backend at xhigh, the highest reasoning-effort setting in this evaluation. Across EdgeBench tasks, SoL-Pi retains roughly 94% of Pi's average score on both model backends, and with GPT-5.6 Sol it exceeds the model's native Codex harness. The efficiency gains are larger: compared with Pi, it uses 45-49% fewer tokens and costs about one-third less; compared with the model-native harnesses, it uses 35-64% fewer tokens at list-price API cost 50-54% lower.

Discussion

Token efficiency favors reusable improvements

Optimizing a harness on a closed task set creates an easy path to benchmark overfitting. A search loop can move task-related information into the harness or add special rules for patterns that recur in the training tasks. The score may improve even when the change stops working outside that set.

Token efficiency directs the search toward work that recurs across tasks. Removing repeated context, combining actions that require no intermediate decision, and avoiding repeated transmission of large tool outputs all target waste in the interaction loop. They do not depend on knowing the answer to a particular task, so they have a better chance of transferring to new tasks and models.

An agent can also reduce token use simply by doing less. SoL-Pi therefore applies a predeclared capability floor: token savings count as an efficiency improvement only when task quality remains within the allowed range.

EdgeBench makes long-horizon efficiency measurable

We chose EdgeBench because it exercises agents over trajectories that run for several hours. Among the public benchmarks we could find at the time, it was the only one that consistently supported roughly two to twelve hours of continuous reasoning and tool use. Common alternatives such as Terminal-Bench 2.1 and SWE-bench usually finish within an hour.

That longer time horizon is essential for measuring token efficiency. On shorter tasks, context replay, large tool outputs, cache writes, and extra model turns have little time to accumulate. Small per-turn savings can be hidden by startup costs and variation between tasks. Over several hours, the same sources of waste recur many times, making harness-level efficiency differences easier to measure.

Shorter benchmarks remain useful for many questions, but they do not match our research question. SoL-Pi asks whether a harness can reduce repeated overhead throughout a long trajectory while preserving task quality. EdgeBench provides the time horizon needed for that difference to become measurable.

Breadth escapes local search basins

We scale auto-research along two axes. A depth-first strategy (DFS) keeps one lineage alive and repeatedly refines its latest candidate. A breadth-first strategy (BFS) expands many independent ideas, then gives each promising branch an auto-research loop to implement, test, and harden it.

In our runs, we saw a qualitative pattern: after roughly five to ten iterations, even GPT-5.6 Sol at xhigh could get stuck in a local basin, making smaller adjustments to the same design instead of changing direction. Breadth-first search behaved differently. Most ideas failed validation, but increasing the diversity of starting ideas occasionally exposed an unusual candidate that a later loop could implement and harden. Those jumps produced some of the most useful harness changes in this project. The conversion rate remained low: only about one out of every forty starting ideas survived validation. Breadth made those rare candidates discoverable.

Breadth and isolation support generalization

In our experience, two conditions appear especially important. The first is a diverse pool of ideas created through broad search. The second is robust validation: evaluation environments should be fully isolated from search or training and should reflect the variation and failure modes of real-world deployment. Without both, auto-research can improve performance in one environment without improving the system beyond it.

Humans set priors and inspect survivors

As AI systems begin to improve the machinery around themselves, the human role in recursive self-improvement remains unsettled. SoL-Pi is best understood as a hybrid auto-research loop. Humans supplied early priors about harness principles and mechanisms. Before scaling the idea pool, they filtered out directions with limited exploratory value for this project, such as searches over base-harness hyperparameters. Once an idea entered an auto-research loop, the loop ran without human intervention through research and validation. When a candidate survived, humans returned to understand the mechanism the agent had discovered and refactor its code into a clean, maintainable implementation.

Whether fully autonomous RSI is ultimately the right end state remains an open question. In an open-ended loop, drift in goals, evidence, or implementation can compound before humans notice it, and the consequences may be difficult to reverse. The relationship between human judgment and machine-scale research therefore needs sustained study. Where should people provide priors, gate the search budget, review accepted changes, or clean up implementations? Where should they step back?

Task diversity may scale harness quality

SoL-Pi also ran a preliminary closed loop: agents gathered or synthesized tasks, constructed executable environments, collected trajectories, validated candidates, and updated their own harness.

We call this direction pretraining the harness: before deployment, harness search sees an expanding stream of self-generated tasks and environments and retains mechanisms that survive across that distribution. This line of research is still at an early stage. We expect to see a scaling law for harnesses analogous to those for models: as compute and the diversity of rollout environments grow, harnesses should become more capable, robust, and efficient.

Efficiency for efficiency

Efficiency may itself become recursive: a more efficient harness could lower the cost of the auto-research used to build its successor. We plan to use SoL-Pi as the starting point for the next research cycle, where lower per-run costs could let a fixed budget cover more executable environments, trajectories, and research ideas. In this view, efficiency is both an outcome of harness research and a resource for expanding the search that follows, so a more efficient harness may help discover an even more efficient one. We call this possibility efficiency for efficiency. It is a long-term research vision rather than a compounding effect demonstrated by the present study.

Authors

Haozhe Liu,Tian Ye,Sensen Gao,Qihang Cao,Yitong Li,Mingchen Zhuge,Duomin Wang,Ruihua Zhang,Jiawang Bian,Lei Zhu,Ligeng Zhu,Enze Xie,Song Han

Equal contribution. Core contributors.

SoL-Pi:Scaling Auto-Research Loops for Efficient Agent Harnesses

Introduction