My AI News Pipeline Test: The Top Configuration Isn’t In Production

🔍 Read the full analysis: My AI News Pipeline Test: The Top Configuration Isn’t In Production on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on monitors, keyboards and dev gear

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

GPT-6 Astra scored highest in Thorsten Meyer’s test of 71 AI configurations for writing news articles, but Meyer says his production pipeline uses GPT-6 Luna instead because it delivered consistently good results at roughly one-hundredth of Astra’s reported cost. In findings published October 13, Meyer also said two backup models were removed after producing fabricated quotes or invalid output.

The test ran from September 8 to October 13, 2026, using the same prompt as Meyer’s production workflow and two real news sources: a 6,340-character product filing and a 619-character political news item. Each configuration generated English and German articles. Two AI judges rated the output on source fidelity, fabrication, clarity about uncertainty, news value, structure and polish; German articles were also assessed for language quality.

Astra scored 4.00 out of 5 in both languages in the standard test, with a source-fidelity score of 5.00. Across a follow-up set of 12 articles, it averaged 3.93, with a reported 95% bootstrap interval of 3.83–4.02. Meyer put its cost at about $0.12 per article and its average generation time at around 50 seconds. Luna scored 3.67 in English and 3.68 in German in the standard test. In the robustness run it averaged 3.67, with source fidelity of 4.71; Meyer said every Luna article was valid for the pipeline.

Meyer reported Luna’s production cost over the previous week at about $0.0013 per article, with a median generation time of 17 seconds. The figures are his results for this workflow, not a general ranking of models across all tasks. He said the test included API, subscription, free-tier and locally run open-weight models, but the published account does not provide a full results table for all 71 configurations.

Why Luna Made the Production Cut

The results illustrate a practical trade-off for publishers using AI at scale: the highest score may not justify the price when a lower-cost model meets the workflow’s quality threshold. In Meyer’s test, Astra outperformed Luna, but its reported cost was about 100 times higher. For frequent article production, that difference could affect operating costs, while speed and consistency also matter.

The test also points to a separate editorial risk: fallback models can introduce errors when a primary model is unavailable. Meyer said the two backup systems produced invented quotations or output his pipeline could not use. Those findings make source fidelity and output validity relevant selection criteria alongside average quality. The results do not establish how the models would perform for other publishers, prompts or kinds of reporting.

Testing Models on News Work

Meyer’s report argues that broad model leaderboards do not answer the narrower question of which system should write a particular publisher’s articles. His evaluation focused on one production task: turning a supplied news source into a finished article using the site’s existing prompt. It tested two source lengths and two language editions, with the shorter source intended to reflect the material Meyer said his pipeline often receives.

For the most promising configurations, the test added 10 further articles, mostly based on short sources. Meyer treated that robustness run as a more reliable guide than the initial two-source comparison. He also said differences below about 0.3 points in the two-article test should be treated as noise. An article counted only if it yielded valid output after the repair step used in production.

The report says Luna was tested at six reasoning-effort settings. Meyer found no reliable quality gain from increasing effort: the highest setting cost about 12 times as much and took about 12 times as long as the no-effort setting, with scores still within the test’s stated uncertainty. He says the production route therefore remains on the no-effort setting.

“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, in the published report

Limits of the Model Comparison

The published results describe one publisher’s prompt, sources and production setup. They do not establish that Luna is the best choice for other newsrooms or that the same ranking would hold across different subjects, source quality or article formats. The main comparison began with only two sources, and the larger follow-up covered 12 articles for the strongest configurations.

Meyer reports that two AI judges rated the work and that results include 95% bootstrap intervals, but the account does not provide the full underlying outputs or detailed scores for every configuration. It is also unclear how the judges’ ratings compare with assessments by human editors, or how often the tested systems would make errors in routine production. The claims about performance and cost are reported by Meyer and are not independently verified in the source material.

The report says both fallback models were removed on October 7, but its description of the current routing ends with an incomplete statement about Luna running first and second. The full fallback order is therefore not clear from the published account.

Production Routing After the Test

Meyer says GPT-6 Luna remains the production model at the no-reasoning-effort setting, while GPT-6 Astra is not used for routine news because of its reported cost. The two problematic backups, GPT-4.1 nano and gpt-oss-120b, were removed from the news route on October 7. The report does not name a replacement fallback or describe a new failover plan.

The next useful evidence would be continued performance data from live production, including how frequently Luna produces invalid or inaccurate articles and whether its quality holds across a wider range of sources. Meyer’s test also leaves open whether further configurations or human editorial ratings would change the comparison. For now, its conclusion is limited to his measured workflow: Astra led the test, Luna was the operating choice, and the old backups did not meet the pipeline’s requirements.

Key Questions

Which model scored highest in the test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. It averaged 3.93 across the 12-article robustness run, according to Meyer.

Why does the production pipeline use GPT-6 Luna?

Meyer says Luna performed consistently and cost about $0.0013 per article in production, compared with about $0.12 per article for Astra. The choice reflects his workflow’s cost and quality requirements, not a universal model ranking.

Did higher reasoning effort improve Luna’s articles?

Meyer reported that differences across six reasoning-effort settings were within the test’s noise. The maximum setting took and cost about 12 times as much as the no-effort setting without a reliable quality gain.

Why were the fallback models removed?

Meyer said GPT-4.1 nano produced invented quotations and received a source-fidelity score of 1.00. He said locally run gpt-oss-120b produced zero valid articles out of four and generated named quotes that were not in the source.

Can these results be applied to other newsrooms?

Not directly. The test used Meyer’s prompt, sources and production workflow. Other publishers would need to evaluate models on their own material and editorial requirements.

Source: ThorstenMeyerAI.com