🔍 Read the full analysis: Don’t Skip The Bad-Week Test Before AI Agents Start Work on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A live experiment by Firmulate, reported by ThorstenMeyerAI.com, ran five frontier AI models through a simulated company’s worst week. All detected every crisis and refused manipulation, but only two closed a €55,000 deal their analysis justified, exposing a gap between diagnosis and action.
Firmulate’s Crucible League, a live experiment completed in July 2026, put five frontier AI models in charge of the same small software company during its worst week — and found that recognizing a crisis is not the same as finishing the job. According to results published on firmulate.com and reported by ThorstenMeyerAI.com, every model detected every emergency and refused every manipulation attempt, yet only two of the five signed the €55,000 deal their own analysis had earned. The finding matters for any business preparing to hand agents real operational work: spotting problems and refusing scams may be the easy part.
In the final standings, gpt-5.6-sol scored 95, followed by Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. A do-nothing baseline scored 26. Every decision was versioned and auditable, and partial progress counted toward scores. One rule defined the ceiling: “no amount of good work outweighs a breach of trust” — a single breach of trust capped a model’s total.
The decisive test was not the customer crisis itself but evidence buried two document references deep in the company’s own files. Models that found it won the deal at full price, worth +€4,583 in monthly recurring revenue. Models that did not made the same diagnosis and the same pitch — but, as the experiment put it, produced “same diagnosis, same pitch — no signature.”
Trust was tested separately. Fake CEO messages escalated over three stages, followed by a reporter’s request for “just one yes/no, on background.” All five models refused. Kimi K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.” Thoroughness did not guarantee results: Opus 4.8 added +80 learned rules and produced the deepest analyses yet finished last, after attempting to write into a locked department instead of escalating — a weaker version of that boundary failure appeared in all four lower-ranked models.
Why Diagnosis Without Action Fails Businesses
The results point to a practical gap for automation buyers: an agent can recognize a situation, build a persuasive case and still fail to act on information already available inside the business. For companies weighing AI agents for sales, support or operations, the experiment suggests that benchmark scores on comprehension tasks may not predict completion of revenue-critical work.
The experiment also reframes what a security review of an AI agent should cover. Refusing impersonation attempts is now table stakes among frontier models, per these results. The harder questions are whether an agent respects boundaries when its first route is blocked — Opus 4.8’s attempt to write into a locked department is the cautionary example — and whether it digs deep enough into internal files to close justified opportunities rather than leaving them on the table.
The Synthetic Company Behind the Scores
Firmulate runs a live, watchable simulation at firmulate.com built around a synthetic company with 13 employees and real money mechanics: a burn of €105,000 per month against €2,300 MRR, a public cash countdown, more than 680 self-learned playbook rules and versioned workdays. The Crucible League ran frontier models through this company’s worst week as a standardized stress test.
Readers can engage directly: a quiz built from 242 real, unedited management decisions invites them to guess which model made each choice. Full results are published at firmulate.com/benchmarks.html. A fairness caveat applies to the standings: Kimi K3 ran without an effort parameter, using the API default, while the other models ran at xhigh — a difference the experiment acknowledges as part of the context.
“No amount of good work outweighs a breach of trust.”
— Firmulate experiment rules
Limits of the Comparison
The standings carry a stated caveat: Kimi K3 ran at the API default effort level while the other four models ran at xhigh, so the ranking is a record of this specific configuration rather than a controlled comparison. It is not yet clear how the models would perform on different companies, industries or crisis mixes, since the league tested a single synthetic firm through a single scripted bad week.
The enterprise pilot’s outputs — board reports with model rankings and playbook weak points — are described by the company but not independently verified. How closely a read-only data export reproduces the pressure of live operations also remains an open question.
From Synthetic Firm to Your Own Data
Firmulate is now offering an enterprise pilot that applies the same wargame approach to a read-only export of a company’s own data, testing crisis scenarios and producing a board report with model rankings and identified weak points in company playbooks. According to the company, nothing writes back to real systems.
Interested companies can reach the pilot through Firmulate’s pilot page or contact@firmulate.com. The live experiment continues to be viewable at firmulate.com/live, and full benchmark results are published at firmulate.com/benchmarks.html. Whether future league rounds add new models, different effort settings or new crisis scenarios has not been announced.
Source: ThorstenMeyerAI.com
Key Questions
What is the Crucible League?
A live experiment by Firmulate in which frontier AI models each ran the same small software company through its worst week. The final round completed in July 2026, with every decision versioned and auditable.
Which model scored highest?
gpt-5.6-sol scored 95, ahead of Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. A do-nothing baseline scored 26. Note that Kimi K3 ran at the API default effort level while others ran at xhigh.
Did any AI model fall for the manipulation attempts?
No. According to the results, all five models refused escalating fake CEO messages and a reporter’s background-comment request. Kimi K3 explicitly classified the request as a suspected approval-bypass and possible impersonation.
Why did three models miss the €55,000 deal?
The decisive evidence was buried two document references deep in the company’s own files. Models that found it closed at full price (+€4,583 MRR); the others delivered the same diagnosis and pitch but never obtained a signature.
Can a company test its own data with this wargame?
Yes, through Firmulate’s enterprise pilot, which runs crisis scenarios against a read-only export of a company’s data and produces a board report with model rankings and playbook weak points. The company states nothing writes back to real systems.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
