📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva, a European sovereign LLM trained from scratch on extensive Italian data, achieved surprisingly low performance on Italian academic benchmarks. This highlights challenges in scaling language models to meet country-specific knowledge needs and questions the assumption that larger data and parameters alone suffice.
Italy’s Minerva-3B language model, trained from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored only 4.9% on the INVALSI Italian school-exam benchmark, despite the country’s significant investment in infrastructure and data.
The Minerva project, led by Sapienza University of Rome and supported by Italy’s national research and AI strategy, built a large-scale language model with weights, training data, and code released openly. It utilized 128 GPUs on the Leonardo supercomputer, training on 2.5 trillion tokens, with a focus on Italian language data.
While Minerva-3B outperformed comparable multilingual models on Italian benchmarks, its performance on the INVALSI school exams was near chance, at just 4.9%. Researchers concluded that dataset size and parameters are more critical for complex language tasks than the proportion of native-language data alone. This result challenges the assumption that larger native-language datasets automatically lead to deeper country-specific knowledge in LLMs.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Building A large language model with Ai: A Practical Guide to Structuring LLM Systems from Scratch Using Reverse-Engineering Techniques
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
SLURM FOR AI AND DEEP LEARNING: GPU CLUSTER MANAGEMENT AND DISTRIBUTED TRAINING: SCHEDULE PYTORCH, TENSORFLOW, AND MULTI-NODE LLM WORKLOADS WITH JOB QUEUING AND RESOURCE OPTIMIZATION
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
SO-ARM101 Low-Cost AI Arm 3D Printed Skeleton
Open-source and low-cost: It is an open-source, low-cost robotic arm solution from TheRobotStudio.
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
AI Engineering: Building Applications with Foundation Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign LLM Strategies
The low performance of Minerva on academic benchmarks despite substantial investment suggests that simply scaling data and parameters may not be enough to achieve meaningful country-specific expertise in LLMs. This raises questions about the optimal allocation of resources and the true scale needed for national AI sovereignty.
It highlights the need for European projects to consider not only data volume but also the quality and diversity of training data, as well as the architectural choices necessary to develop models capable of understanding complex, domain-specific content.
European Sovereign LLM Development Approaches and Challenges
Italy’s Minerva project represents a different approach from Portugal’s AMÁLIA, which layered specialization onto a multilingual foundation. Minerva trained from scratch with a focus on Italian data, leveraging national infrastructure and open data policies. Despite this, the empirical results reveal limitations in current scaling strategies, emphasizing that larger datasets and more parameters alone do not guarantee high performance on complex tasks.
The broader European sovereign AI movement debates whether to focus on continuation pre-training or from-scratch development, with Minerva exemplifying the latter. Its results suggest that scaling efforts must be accompanied by considerations of data quality, diversity, and task-specific architecture.
“Despite large-scale training, the model’s performance on academic content remains near chance, indicating the need for deeper investment or different strategies.”
— Research team, Minerva project
Unresolved Questions About Scaling and Data Quality
It remains unclear what specific factors—such as data diversity, training methodologies, or architectural innovations—are necessary to improve performance on complex, domain-specific tasks within sovereign models. The results from Minerva suggest that larger scale alone may not suffice, but the precise thresholds and strategies are still under investigation.
Next Steps for European Sovereign AI Projects
Researchers and policymakers will likely reevaluate scaling strategies, emphasizing data quality and task-specific architectures. Ongoing iterations of Minerva and similar projects are expected to test alternative approaches, including targeted fine-tuning, architectural adjustments, and diversified data collection, to enhance country-specific knowledge depth.
Key Questions
Why did Minerva perform poorly on Italian academic tests despite large-scale training?
The evaluation suggests that dataset size and parameters are more critical than native-language data proportion, and current scaling may not address the complexity of academic content understanding.
Does this mean European sovereign LLMs are not worth the investment?
Not necessarily. It indicates that scale alone may not be sufficient, and strategic focus on data quality, diversity, and architecture is needed to realize meaningful country-specific expertise.
What are the implications for future LLM development in Europe?
European projects may need to prioritize targeted data collection, innovative architectures, and task-specific training to overcome current limitations demonstrated by Minerva’s results.
Is the low performance specific to Italian language or content?
The evaluation was on Italian academic tests, which are highly domain-specific. The results highlight broader challenges in scaling models for complex, specialized tasks regardless of language.
Source: ThorstenMeyerAI.com
