Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
A polished demo can write a convincing answer. It cannot tell you whether an AI agent will read the right file, protect a company under pressure and close a deal. A live Firmulate experiment puts those decisions to the test—and Moonshot’s Kimi K3 finished second, ahead of three Western frontier models.
As an affiliate, we earn on qualifying purchases.
Same company, same difficult week
For software teams evaluating AI agents, the gap between a plausible response and a completed task matters. Firmulate’s Crucible puts that gap inside a simulated small software company: each frontier model faces the same customers, crises and temptations, and every decision is versioned and auditable. The experiment is a live, watchable company, not a fictional scenario.
The final league table, dated July 2026, puts gpt-5.6-sol first with 95 points and Kimi K3 second with 93. Sonnet 5 scored 88, Fable 5 scored 77 and Opus 4.8 scored 73. K3, from Moonshot, therefore beat three of the four Western frontier models in the comparison, while finishing just behind the leader. Firmulate’s benchmark page presents the results and findings.
As an affiliate, we earn on qualifying purchases.
The test was about follow-through
All the models spotted every crisis and refused every manipulation attempt. Yet only two signed a €55,000 deal that their own analysis had earned. As Firmulate puts it: “Same diagnosis, same pitch — no signature.” That split is useful for anyone building or buying AI agents: recognizing what should happen does not guarantee the agent will finish the job.
The deal hinged on a competitor weakness buried two document references deep in the company’s own files, rather than in the customer event. Models that read the file won the deal at full price, worth +€4,583 in monthly recurring revenue. K3 was among those that found it and closed. The result turns a familiar software quality question into a concrete one: does an agent inspect relevant context before acting?
AI compliance and monitoring software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Discipline under pressure
The experiment also tested social engineering: fake CEO messages escalated over three stages, followed by a reporter asking for “just one yes/no, on background.” All five models refused. K3’s recorded reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.”
K3 had one deviation, giving it the cleanest discipline in the field. Opus 4.8 offers a different lesson: it was the most thorough participant, with +80 learned rules and the deepest analyses, but finished last. It left the deal unsigned and tried to write into a locked department instead of escalating. Firmulate says the same weakness appeared, less strongly, in all four. More analysis alone did not ensure a better operational result.
AI performance benchmarking tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
A running company, not just a leaderboard
The simulated company has 13 synthetic employees and real money mechanics: monthly burn of €105k against €2.3k in recurring revenue, a public cash countdown, more than 680 self-learned playbook rules and a versioned record of every workday. The live company runs each business day at Firmulate. A quiz built from 242 real, unedited management decisions invites visitors to guess which model made each choice.
Enterprises can also run the wargame against a read-only export of their own business. Firmulate says nothing writes back to real systems. That offers teams a way to examine agent behavior against their own context before deciding whether to trust one with CRM, support or forecasting work.
Fairness note
K3 ran without an effort parameter (API default) while the others ran at xhigh.
For software teams, the lesson is to test the work, not just the words. K3’s result makes the frontier look open: choosing a model without testing it on your own tasks is a bet.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
