🔍 Read the full analysis: Terminal-RSI And The Challenge Of Measuring AI Self-Improvement on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A preprint from Tencent’s Hy Frontier lab and four universities reports that a 27-billion-parameter terminal agent improved on several benchmarks through five human-designed training stages. The results do not show an AI independently rewriting and improving itself: researchers selected the tasks, rewards, training settings and checkpoints. Long-horizon performance remained limited, with the final model completing an average of 5 of 46 tasks.
A 26-page preprint from Tencent’s Hy Frontier lab and four universities reports that a 27-billion-parameter AI agent improved its scores on several command-line task benchmarks after five training stages. Despite online claims that the system achieved recursive self-improvement without human involvement, the paper describes a researcher-designed pipeline: people set the stages and rewards, ran the training, and chose which checkpoints to retain.
The final system, called Terminal-RSI-27B, is built on the open Qwen3.8-27B model and was trained using a pool of 3,000 tasks. Its sequence was Base, Recursive Self-Rewrite (RSR), Recursive Policy-Refine (RPR), then a second RSR and RPR stage. RSR uses attempts that pass a verifier as examples for supervised fine-tuning; RPR uses reinforcement learning to make higher-scoring behavior more likely. The researchers controlled the training process throughout.
On Terminal-Bench 4.0, the reported score rose from 0.00% to 19.84%. The paper also reports an equal-weight average across four Terminal-Bench suites increasing from 20.68% to 50.58%. Those figures are averages of three evaluation runs. The paper’s control provides evidence that stage order mattered: reinforcement learning applied directly to the base model remained at 0.00% on Terminal-Bench 4.0 in the recorded checkpoints, while reinforcement learning following the first self-rewrite stage reached 16.67%.
Results were weaker on longer tasks. The final model completed an average of 5 of 46 long-horizon tasks, compared with none for the base model. On SWR100, a scientific-workflow test, the score rose from 3% to 6% and did not increase after the third stage. Terminal-Bench 2.1 also showed no gain in the final stage. The source report says all benchmark scores are Mean@3, or the arithmetic mean of three independent runs, rather than the best result.
Infographic 01 · The results, stage by stage
Real gains on the benchmarks, and a small long-horizon result
A 27 billion parameter open model, trained in five stages that alternate learning from its own verified successes with reinforcement learning.
From solving nothing to about one task in five, over five training stages. Each score is the mean of three evaluation runs (Mean@3).
Base → final, per benchmark
Bar = final score; base in the label
Source: Yang et al., Terminal-RSI, arXiv 2610.22192 v3 (30 Oct 2026), Table 2 and Figure 5. All values as printed in the paper.
Powered by Thorsten Meyer AI
Why Verified Completion Matters
The paper addresses a practical problem for AI systems that operate in command-line environments: an agent can make partial progress, then stop or claim it has finished without completing the requested work. Its reported training results suggest that how success is rewarded can affect whether an agent learns to complete tasks rather than accumulate partial credit.
The researchers compared reward designs using task checks. Under the earlier approach, the source report says failed attempts received 50.09% of all reward, allowing unsuccessful runs to earn substantial credit. The revised reward kept every failure below every success. That distinction matters to developers evaluating agents: a system’s own statement that work is done is not reliable evidence that the requested result exists. The paper defines success by whether the complete protected verifier passes.
However, the reported gains do not establish that the system can improve itself autonomously or reliably handle long, open-ended work. The final model’s 5-of-46 average on the long-horizon benchmark points to a substantial gap between improvements on test suites and dependable performance across extended tasks.
Products Worth Considering
Jackery Explorer 1000 v2 Portable Power Station, 1070Wh, 1500W
Sonos Era 100 SL Compact Microphone-Free Speaker, Black
Logitech MX Creative Console with 9 Customizable Keys, Graphite
As an affiliate, we earn on qualifying purchases.
How the Training Stages Worked
A terminal agent operates through a command line: it can inspect files, run commands, observe outputs and try again. The benchmark tasks described in the source include work such as repairing a build or fixing a data pipeline. To judge completion, the paper uses hidden tests that inspect the final state rather than accepting the agent’s own report of success.
The RSR stages turn verified successful attempts into training examples. According to the source report, the model rewrites a successful attempt as a step-by-step plan; a review process screens the plan for leaked answers, and the plan is run again from scratch. Only attempts that pass again are retained. The RPR stages use reinforcement learning, with a critic network estimating the quality of an attempt. The researchers carried that critic between stages and recalibrated it before each refinement phase.
The paper’s motivation for stricter completion signals is reflected in its records: the source report says 748 failed runs declared a task complete within ten turns. That figure is described as a count from recorded training attempts, not as a general rate for terminal agents. The report also says the earlier reward analysis covered 4,352 recorded attempts; the available source material does not provide further details on the full distribution.
“The most interesting part is not the ‘self’ in the title. It is how the researchers stopped rewarding the agent for looking busy and started rewarding it for actually finishing.”
— Thorsten Meyer, ThorstenMeyerAI.com
Limits of the Reported Evidence
The results are from a preprint, and the material provided does not establish whether the work has undergone peer review. The source describes benchmark scores and a control experiment, but does not provide enough information here to assess how well the results generalize to other models, task sets or deployment conditions. The reported average across four Terminal-Bench suites also gives each suite equal weight and excludes the long-horizon and SWR100 results.
It remains unclear how much of the measured improvement carries over to tasks unlike those used in training, and whether the process can scale to substantially longer work without human decisions between stages. The small long-horizon result and the flat SWR100 score after the third stage limit what can be concluded about broad capability gains. The paper, as described in the source, does not show an agent independently choosing and executing its own training program without human oversight.
Further Tests of Agent Completion
The next evidence needed is independent evaluation of the reported results, including tests on tasks that were not used to shape the training pipeline and clear reporting of evaluation methods. Longer-horizon tests will also be needed to determine whether the observed gains extend beyond shorter benchmark tasks.
For now, the paper’s reported contribution is a human-directed training sequence and a reward design that gives verified success priority over partial progress. Whether that approach produces reliable performance in real operational settings, or meaningfully less human involvement in future training, remains an open question.
Key Questions
Did Terminal-RSI improve itself without human involvement?
No. The paper describes a pipeline whose stages, rewards, training settings and checkpoint choices were controlled by researchers. The system learned from verified examples it generated, but people designed and ran the process.
How much did the agent’s benchmark scores improve?
On Terminal-Bench 4.0, the reported score rose from 0.00% to 19.84%. The paper also gives an equal-weight average across four Terminal-Bench suites of 20.68% at the base and 50.58% at the final stage. Scores are means of three evaluation runs.
Could the final model complete long tasks?
Its reported average was 5 completed tasks out of 46 on the long-horizon benchmark, up from zero for the base model. That result indicates progress in the reported test, but also shows that most tasks were not completed.
How did the researchers decide whether a task was complete?
They used a protected verifier that checked the final task state. Under the paper’s definition, an agent’s declaration of success or a subset of passed checks was not enough; the complete verifier had to pass.
Source: ThorstenMeyerAI.com
