🔍 Read the full analysis: FrontierCode: A Maintainer’s Checklist For Reviewing AI Pull Requests on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, testing 23 models on repository tasks graded for whether maintainers might accept the resulting change. Claude Opus 5.5 led with a 52.2% score and cleared all blocking checks in 57.7% of attempts; the results also show large cost differences and gaps between general intelligence rankings and repository performance.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark that evaluates whether an agent’s repository change meets maintainers’ review criteria, not only whether tests pass. Claude Opus 5.5 scored 52.2%, the highest result, but cleared every blocking check in 57.7% of attempts, indicating that even the leading model did not consistently produce changes meeting the benchmark’s required checks.
Artificial Analysis added the benchmark to its evaluations on October 12, 2026. Its runs used the same open-source agent harness, Stirrup, for every model, with a limit of 250 turns per attempt and three attempts per task. The evaluation covered the 100-task Main set; the full FrontierCode benchmark contains 150 tasks. Nine of the 100 Main-set tasks were not solved by any model on any attempt.
Claude Opus 5.5 (max with fallback) ranked first at 52.2%, followed by GPT-6.1 Sol (max) at 49.1% and Claude Fable 5.1 (max with fallback) at 49.0%. The leading model’s blocking-check pass rate was 57.7%; GPT-6.1 Sol’s was 54.7%. A failed blocking check makes an attempt score zero, so the headline score is not the same measure as the pass rate. Below the leading group, Grok 4.7 scored 33.8%, MiniMax-M3 23.0%, Inkling 15.8%, Muse Glimmer 13.9% and Nemotron 3 Ultra 10.5%.
FrontierCode assesses patches on six dimensions: hidden tests, command checks, reverse tests, adaptive tests, limits on files and diff size, and an LLM review against maintainers’ criteria. The review judge in Artificial Analysis’s evaluation was GPT-6 Luna (medium). The benchmark also penalizes overly broad changes and unhelpful tests; according to the source account, attempts flagged for looking up solutions receive a zero.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
The results matter to engineering teams considering AI agents for pull requests because correctness is only part of reviewability. A patch can pass tests yet still touch too many files, introduce weak tests or miss a maintainer’s expectations. FrontierCode makes those issues affect the score rather than treating a green test run as sufficient.
The benchmark’s leading pass rate also puts a limit on claims that coding agents can handle repository changes without human oversight. In this evaluation, Claude Opus 5.5 cleared every blocking check in 57.7% of attempts. That is a result on this benchmark, not a forecast of performance across all software teams or tasks, but it shows why maintainers remain part of the process.
Price adds another practical trade-off. Artificial Analysis reports costs ranging from $0.09 per task for GPT-6 Luna (max) to $10.15 for Claude Opus 5.5 (max with fallback), among the listed results. GPT-6.1 Sol scored 49.1% at $0.81 per task, while Opus scored 52.2% at $10.15. That is a 3.1-point score difference at a much higher listed cost, though benchmark scores and per-task costs alone do not establish which model is best for a particular codebase.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Repository Changes
FrontierCode was developed by Cognition and open-source maintainers, announced in June 2026 and updated to version 1.1 in July. More than 20 maintainers created tasks drawn from 36 repositories, including Celery, Budibase, uppy and Mattermost. The source says maintainers spent more than 40 hours on each task. The tasks are private, limiting the chance that models encountered their solutions during training.
Its scoring is designed to reflect review conditions. Reverse tests check whether an agent’s own tests fail against the original code; a test that passes before and after the patch may not demonstrate that the change works. Scope limits constrain touched files and diff size, while an LLM review checks work against maintainers’ criteria. Blocking checks determine whether an attempt scores at all; quality checks shape scores for attempts that pass.
The rankings also differ from general model rankings in Artificial Analysis’s Intelligence Index. Claude Sonnet 5.5, listed second among the compared models on that index, ranked fifth on FrontierCode at 45.8%. Grok 4.7 ranked 18th on FrontierCode at 33.8%, while DeepSeek V4.1 Flash placed seventh at 44.0%. These comparisons suggest that broad intelligence scores do not directly predict performance on this repository-review benchmark.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement, as quoted by ThorstenMeyerAI.com
software developer testing monitors
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What the Scores Cannot Establish
FrontierCode results measure performance on a defined set of tasks and under a particular harness, turn limit and review process. They do not establish how models will perform across every repository, team or production workflow. The benchmark’s use of an LLM reviewer, GPT-6 Luna (medium), is also part of the evaluation setup; the available source does not report how scores would change with a different reviewer.
Several prominent models had no reported FrontierCode result as of October 13, 2026, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the published ranking is not a comparison of every current model. The source also provides no independent results from maintainers deploying these systems in day-to-day pull-request workflows.
Cost figures are reported per task for the benchmark runs, but the supplied material does not describe them as a complete measure of operating costs in real projects. The results also do not identify one model as the right choice for every team: score, price, repository requirements and review practices may differ.
As an affiliate, we earn on qualifying purchases.
More Models and Real-World Tests
The next development to watch is whether the models without reported results are added to the leaderboard, particularly those named by the source as missing on October 13. Further entries would make comparisons more complete, though results would still need to be read alongside their evaluation conditions.
For maintainers, the immediate practical use is as a structured comparison of how agents handle review-oriented checks, not as a reason to remove human review. Teams considering these tools can compare the benchmark’s dimensions with their own acceptance criteria and test models on representative repository work. The published results do not specify a date for the next benchmark update or announce a deployment milestone.
AI model performance testing software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is FrontierCode?
FrontierCode is a coding benchmark developed by Cognition with open-source maintainers. It scores repository changes against tests, scope limits and review criteria, rather than relying only on whether tests pass.
Which model ranked highest?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’s results with a 52.2% FrontierCode score. It cleared every blocking check in 57.7% of attempts.
Does the top score mean the model can submit pull requests without review?
No. The result covers performance on FrontierCode’s tasks and setup, not all repositories or workflows. The top model failed to clear all blocking checks in a substantial share of attempts, and the benchmark does not establish that human review can be removed.
How does FrontierCode differ from a standard coding test?
It includes maintainer-oriented checks such as diff and file-scope limits, reverse tests and an LLM review against stated criteria. Failing a blocking check makes an attempt score zero even if other checks pass.
Were all major models included?
No. As of October 13, 2026, the source reported no FrontierCode results for Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their performance on the benchmark remained unknown.
Source: ThorstenMeyerAI.com
Columbus Day / Indigenous Peoples' Day Picks
long weekend sales
As an affiliate, we earn on qualifying purchases.
