🔍 Read the full analysis: Which AI Setups Answered Our 15 Legal Questions? We Tested 42 on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Thorsten Meyer’s Lexassay test assessed 42 AI setups on 15 legal questions covering German, US and EU law. Claude Opus 5.5 led with 79 points out of 100, while the legal-focused GLM-5.3 FlashX Legal scored 8; the test author says citation mistakes were widespread. The results are a limited spot check, not a definitive ranking, and the author discloses that Claude agents graded the answers.
Lexassay, a legal AI test run by Thorsten Meyer, reports that 42 AI setups produced sharply different results on 15 questions about German, US and EU law. In the published scores, Claude Opus 5.5 ranked first with 79 out of 100, while the law-focused GLM-5.3 FlashX Legal received 8; the results highlight the risk of relying on chatbots for legal deadlines or citations. Meyer disclosed that his organization runs the test and that Claude agents graded the answers, a potential conflict given Claude’s top ranking.
The test included 10 published questions and five hidden questions. Tasks covered deadline calculations, recent legal changes, major court decisions and questions containing false assumptions. Models answered without internet access or tools, and the hidden-task scores were published without the underlying questions and answers. Meyer describes the exercise as a spot check rather than a strict benchmark because each model answered each task once.
Among the leading systems, GPT-6 Astra scored 72 and GPT-6.1 Sol scored 71, following Claude Opus 5.5’s 79. The report says the leader fully met every criterion on only eight of the 10 public tasks. The named legal model, GLM-5.3 FlashX Legal, scored 8, including zero on the hidden set; it fully solved none of the 15 tasks and gave no answer on two. Its base model scored 3, according to the results.
Mistral Large 4 scored 10, according to Meyer, with citation errors accounting for much of its lost score. Across all published runs, the test reports that reviewers verified 2,644 citations as real and identified 804 as invented or not supporting the claims made. The scoring method also deducted points for citations and for answers that omitted a note against treating the response as legal advice or a referral to a lawyer.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Legal Errors Can Change Outcomes
The results matter because legal questions can turn on details that a fluent answer may obscure. Meyer’s example concerns two people seeking to appeal civil judgments in Germany, served on the same day in 2027. He says the appeal deadline falls on different days in Munich and Berlin because Corpus Christi is a public holiday in Bavaria but not in Berlin. A chatbot that gives one date for both locations could lead a user to miss a filing deadline.
The test also illustrates that a plausible-looking citation is not proof that a legal answer is correct. Meyer says some systems cited real laws or rulings while misrepresenting what they said. That distinction can affect practical decisions, including whether someone pursues a claim or relies on a deadline. The findings do not establish how the systems would perform with web search, legal databases, supplied documents or professional review; they measure responses under the test’s restricted conditions.
Products Worth Considering
LG UltraGear 34GX90SA-W 34-inch WQHD Curved OLED Gaming Monitor
Apple MacBook Air 15-inch with M5 chip, 24GB RAM, 512GB SSD, Midnight
EcoFlow Delta 3 Portable Power Station – 1024 Wh, 1800 W
As an affiliate, we earn on qualifying purchases.
How Lexassay Scored the Models
Lexassay compares each answer with a prepared gold answer using three to six weighted criteria. The test author says citations are checked against official databases and classified as real, misattributed, invented or not findable. Each invented or misattributed citation reduces the task score by a quarter, while omitting a legal-advice disclaimer or referral to a lawyer costs 5 percent.
The test’s scope is narrower than a working legal assistant that can search current sources. Models had no internet access or tools, and each model-task pairing received one answer. That design can expose what a model may produce from its internal knowledge, but it does not provide repeated trials or show whether retrieval tools and human review would change the outcome. The article also notes that different effort settings were tested separately, so one model could appear more than once among the 42 setups.
Results may differ from other evaluations. Meyer says Vals AI’s legal-agent benchmark, which supplies documents to the models, placed Mistral Large 4 eighth, ahead of Claude Opus 5.5 in 21st and GPT-6 Astra in 24th. Those rankings are not directly comparable because the tests use different tasks and conditions.
““It is a spot check, not a benchmark in the strict sense: one answer per model and task, so a few points of difference are noise.””
— Thorsten Meyer, Lexassay test author
Limits of the Reported Scores
The results come from a test run by Meyer’s organization, which operates Lexassay. Meyer discloses that Claude agents graded the answers and that a Claude model ranked first. He says the method and public answers, gold answers and citation checks are available for review, but the disclosed grading arrangement remains relevant when interpreting the ranking. The supplied report does not include independent replication of the scores.
It is also unclear from the reported results how the models would perform across repeated attempts, changing prompts, live legal research or real cases reviewed by lawyers. The test’s one-answer-per-task design makes small score differences uncertain, and its no-tools format does not represent every product configuration available to users. The findings do not establish that any system is reliable or unreliable for all legal work.
Review the Answers and Citations
Meyer says the test’s method page and public answer materials allow readers to inspect how answers were graded and citations checked; hidden questions are not published, though their scores are. Readers seeking to compare the results should consult those materials and note the test conditions rather than treating the scores as a universal league table.
The report gives no announced date for another test or independent audit. Meyer advises using AI responses only as a starting point, checking cited laws and rulings in their original sources, and consulting a lawyer when a deadline, financial claim or communication with an authority is involved. The test is not legal advice, and the consequences of a missed deadline can be serious.
Key Questions
Which AI setup ranked highest?
Claude Opus 5.5 scored 79 out of 100, the highest score reported in the test. Meyer says it fully met every criterion on eight of the 10 public tasks.
How did the legal-focused model perform?
GLM-5.3 FlashX Legal, which Z.ai describes as designed for legal workflows, scored 8 out of 100 and zero on the hidden set. Meyer reports that it fully solved none of the 15 tasks and gave no answer on two.
Can these scores predict how a chatbot will handle my case?
Not reliably. Lexassay tested one answer per task without internet access or tools. Its author calls it a spot check, and the results do not establish how a system will perform with current legal sources, supplied documents or lawyer review.
Why does the test’s grading disclosure matter?
Meyer says Claude agents graded the answers, and a Claude model ranked first. He also discloses that his organization runs Lexassay. Those facts do not by themselves invalidate the results, but readers should consider them alongside the published method and answer materials.
Should I use a chatbot to calculate a legal deadline?
The test author advises readers not to rely on a chatbot for a deadline. Check the applicable law and official sources, and ask a lawyer when a filing deadline or other legal consequence is involved.
Source: ThorstenMeyerAI.com
