🔍 Read the full analysis: What To Consider Before Picking CUGA, Hermes, Or Deep Agents on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
IBM reports that CUGA completed 73 of 90 AppWorld tasks with GLM-5.3-flash, compared with 42 for Hermes and 27 for Deep Agents. The single-model, single-benchmark test suggests agent harness design can affect results, but does not establish which system performs best across other workloads or whether policies are enforced.
IBM reports that CUGA completed 73 of 90 AppWorld tasks using GLM-5.3-flash, compared with 42 for Hermes and 27 for Deep Agents. The comparison, discussed in the original analysis, suggests that the software coordinating an AI agent can influence performance on multi-step tasks, but it does not establish a general ranking among the three systems.
IBM says all three harnesses ran on the same 90 tasks across 30 AppWorld scenarios, using the same open-weight model. The evaluation used the Harbor framework and allowed one attempt per task. The reported task completion rates were 81.1% for CUGA, 46.7% for Hermes and 30.0% for Deep Agents.
IBM also reported results at the scenario level, counting a scenario only when all three of its variants passed. CUGA passed 20 of 30 scenarios, Hermes passed eight and Deep Agents passed two. The harness configurations differed: CUGA used task decomposition, app-focused tools, pre-execution verification and automatic reflection. Hermes used Harbor’s default setup with AppWorld tools over MCP. Deep Agents used AppWorld tools over MCP along with LangChain middleware for tool selection, context summarization, task lists, retries and call limits.
IBM says it applied two fixes to Deep Agents so malformed model responses would not end runs. The source identifies the experiment setup as publicly available in harbor-eval, IBM’s Harbor fork. The reported figures are IBM’s results; the supplied material does not describe an independent replication.
Why Harness Design Affects Results
The gap matters because an agent’s performance is shaped by more than the underlying model. A harness controls how tasks are divided, which tools are available and when actions or results are checked. In workflows that span apps, those design choices may affect whether an agent completes a task successfully.
For organizations comparing agents, the report is a reason to test the model, harness and intended workload together, rather than treating a model benchmark as a reliable forecast of deployment performance. But the scores do not show that CUGA will outperform Hermes or Deep Agents in personal assistance, research or other tasks for which those systems may be designed.
The experiment also says nothing about whether the systems reliably follow organizational policies during execution. IBM describes CUGA as having features for applying rules through its harness, but policy enforcement was not tested in these AppWorld results. That distinction matters for teams considering agents for work involving sensitive data or consequential actions.
Products Worth Considering
Bose QuietComfort Wireless Noise-Cancelling Over-Ear Headphones, Black
Apple Watch Series 8 GPS, 41mm, Midnight Aluminum Case with Midnight Sport Band (S/M)
Apple Watch Series 12 GPS 46mm Smartwatch
As an affiliate, we earn on qualifying purchases.
Three Harnesses, Different Uses
The source describes CUGA, short for ConfigUrable Generalist Agent, as IBM’s open-source harness for enterprise work across systems such as customer relationship management and payments. It presents Hermes from Nous Research as aimed at always-on personal assistants and Deep Agents from LangChain as suited to longer, multi-step work such as research. These descriptions frame the systems as tools with different intended uses, not interchangeable products in a universal contest.
AppWorld tests multi-step tasks across nine apps and 457 APIs, including email, payments and file storage. IBM characterizes the benchmark as a proxy for some abilities relevant to enterprise automation, such as reading API documentation, handling paginated results and changing state across apps. However, it uses consumer apps and does not directly establish how a system will perform on real enterprise services.
The source also cites a separate result from the best-of-Agent-Harnesses directory, where GLM-5.2’s SWE-bench Pro score reportedly rose from 23% to 52% after a harness change. That is a different comparison and should not be treated as evidence from the AppWorld test. The source says community directories track nearly 200 agent harnesses, underscoring that the choice of coordination software is one part of a crowded and changing field.
“This is not a verdict on Hermes or Deep Agents.”
— IBM
Limits of the AppWorld Evidence
The comparison covers one benchmark, one model and 90 tasks, with one attempt per task. IBM acknowledges that the harness setups were not identical, so the test does not isolate every implementation difference. The supplied material does not say how results would change with other models, different task sets, repeated attempts or production workloads.
It is also unclear whether the reported performance would hold on actual enterprise systems. AppWorld is a proxy built around consumer apps, not a direct test of company infrastructure. The experiment did not evaluate policy enforcement, access controls, approval steps or data boundaries, and the source provides no comparative measurements for cost, latency, large-scale reliability or deployment outcomes.
These limits do not negate the reported task counts, but they constrain what can be inferred from them. The results indicate a difference on this particular evaluation; they do not establish that CUGA is generally superior or that the same ranking would persist under different conditions.
How Teams Can Test Their Workflows
The experiment setup is available through harbor-eval, according to the source. Teams considering any of the three systems can use the report as a prompt to build evaluations around representative workflows, using the models and tool configurations they expect to run. They should measure not only task completion but also consistency across repeated attempts and variations in task conditions.
For enterprise use, an evaluation would also need to examine matters outside the AppWorld score, including access permissions, human approval points, data boundaries and behavior in the organization’s own environment. The supplied material gives no date for a broader comparison or an independent replication, so whether those will follow remains unknown.
Key Questions
Which system had the highest AppWorld task score?
CUGA had the highest reported score, completing 73 of 90 tasks. Hermes completed 42 and Deep Agents completed 27 in IBM’s comparison using GLM-5.3-flash.
Does the comparison prove CUGA is the best agent?
No. The result covers one model and one benchmark, and the harness configurations differed. It does not establish performance across other tasks or deployment settings.
What is AppWorld testing?
AppWorld uses multi-step tasks across nine apps and 457 APIs, including email, payments and file storage. It can test some tool-use abilities, but it does not directly represent real enterprise systems.
Did the test evaluate safety policies or access controls?
No. The reported experiment did not test policy enforcement, access controls, approval steps or data boundaries. Its task-completion scores are not evidence that those safeguards work.
What should teams do before choosing a harness?
Test the intended model, harness and tools on representative workflows. Measure repeatability as well as completion, and separately review permissions, approvals, data handling, cost and performance in the target environment.
Primary source: Hugging Face · via ThorstenMeyerAI.com
