🔍 Read the full analysis: We Tried Weight-Space Programming At Home: Claude Edited A Local Model, GPT-6 Astra Checked The Work, And The Legal Score Stayed At Zero on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A home experiment used a Claude-written procedure to remove a looping direction from a local model’s weights. The edited model answered all 10 published Lexassay tasks, up from 4 on the comparison setup, but its legal score remained zero and its answers contained 55 wrong or invented citations. The result suggests the edit helped the model finish, not produce reliable legal analysis.
A home experiment reported by ThorstenMeyerAI.com found that a weight edit helped local model Ornith 1.5 35B-A3B finish more benchmark tasks, but did not improve its legal score. On the same experimental setup, the edited model answered 10 of 10 published Lexassay tasks, compared with 4 of 10 before the edit; its legal score stayed at zero, and the answers included 55 wrong or invented citations.
The experiment ran from Wednesday evening to Thursday morning and used a technique called directional ablation. A Claude Opus 5.5 session wrote code to run the model, measure an internal direction associated with looping, and remove that direction at half strength from the output weights of 27 of the model’s 40 layers. The report says there was no new training, no additional legal data, and no AI directly reading and editing the weights by reasoning over them.
The comparison is limited to 10 published Lexassay tasks, covering legal questions checked against primary sources. The edited model met 7 of 50 checked criteria, up from 3, but still received a legal score of zero. The report also says that, with the model card’s recommended sampling settings and no weight edit, Ornith answered 9 of 10 tasks on its original setup. That result complicates any claim that the weight edit alone caused the improvement.
Claude carried out the edit, and Claude agents graded answers in a blind review. OpenAI’s GPT-6 Astra, running in Codex, reviewed the code, figures and claims and identified faults, according to the report. The author describes this as one experiment and one independent reviewer, not two separate tests. Results and answers are said to be public on Lexassay.com.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
Finishing Answers Is Not Legal Accuracy
The experiment illustrates a distinction that matters as AI systems are adapted for specialized work: a model may have a generation or completion problem separate from a knowledge problem. If a system spends its output budget repeating itself, an intervention that reduces looping may let it produce a complete response. That does not establish that the response is correct.
Here, the model completed more tasks, but its score remained zero and the report counted 55 wrong or invented citations in the edited answers. For legal users, that gap is consequential: a fluent, finished answer can still misstate law or cite sources that do not support it. The report itself says the experiment is not legal advice.
The result is also a caution against treating a single benchmark change as proof of a general capability gain. The task set was small, only one run was performed per setting, and the no-edit result shifted substantially across setups. The evidence supports a narrow observation about task completion in this test, not a conclusion that the model became better at law.
Products Worth Considering
Amazon Kindle Paperwhite Signature Edition (12th Generation, 2024), 32 GB, Metallic Raspberry
Apple AirPods Pro (1st Generation) with MagSafe Charging Case
Apple Watch SE 3 GPS 40mm Smartwatch
As an affiliate, we earn on qualifying purchases.
How This Differs From MIT’s Method
The experiment was prompted by a report from four MIT CSAIL researchers—Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab—on what they call weight-space programming. In their reported Qwen example, GPT-6 Astra reasoned directly about another model’s weights and made an edit associated with a change in looping and mathematics performance. Their paper’s central caveat, as quoted by ThorstenMeyerAI.com, is that an edit may remove an obstacle hiding an ability, but cannot add an ability the model does not have.
The home test did not replicate that direct weight-reading approach. It used a more established method: run the model, record internal activations, calculate a direction, then project that direction out of selected weights. The report notes that researchers have used related single-direction edits before, including in work on refusal behavior. Its aim was to test a similar, narrower proposition: whether reducing a looping behavior could help a model complete tasks.
Lexassay asks models 15 questions about German, US and EU law without internet access or tools. Ten tasks are published and five are hidden, according to the report; the figures discussed here refer only to the published set. Ornith was selected because its benchmark runs showed looping, while other local models either appeared to lack the relevant knowledge or could not be tested under comparable conditions.
“Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked.”
— The MIT researchers, as quoted by ThorstenMeyerAI.com
Limits of the Benchmark Comparison
The results remain preliminary and limited to one run per setting. The report says the loop detector changed during the experiment, the development set was small, and three graders came from the same model family. It also acknowledges that the two local setups were not directly comparable: the unchanged model answered 8 of 10 tasks in its published run but only 4 of 10 on the setup used for the edited comparison.
Those differences make it difficult to isolate how much of the completion change came from the weight edit, sampling settings, or experimental setup. The report does not establish whether the result will repeat across additional runs, other models, or Lexassay’s five hidden tasks. Nor does it show that the intervention improves legal reliability; the reported legal score remained zero.
Further Runs Must Test Reliability
The immediate next step is replication: rerun the edited and unedited model under the same settings, with the same loop detector and evaluation process, and report variation across multiple trials. Testing the hidden Lexassay tasks could also help show whether the approach generalizes beyond the published questions, provided those tasks remain unavailable for tuning.
Until such results are available, the public benchmark records offer a way to inspect the reported answers and citation checks, but they do not settle whether the edit caused the improvement. The experiment’s current finding is narrower: the edited model completed more published tasks in one comparison, while its legal score did not improve.
Key Questions
What did the experiment change?
A Claude Opus 5.5 session wrote code that measured a direction associated with looping and removed it at half strength from the output weights of 27 of Ornith 1.5 35B-A3B’s 40 layers. The report says the procedure involved no new training or legal data.
Did the edit make the model better at law?
The report does not show a higher legal score. The edited model met 7 of 50 checked criteria, compared with 3 before, but its legal score remained zero; its answers included 55 wrong or invented citations.
How many tasks did the edited model answer?
It answered all 10 published Lexassay tasks on the comparison setup, compared with 4 of 10 before the edit on that setup. The report also says the unchanged model answered 9 of 10 with model-card sampling settings on its original setup, so the setups are not directly comparable.
Was this the same method used in the MIT research?
No. The home experiment used directional ablation based on measured model activations. The MIT example described in the report involved GPT-6 Astra reasoning directly about another model’s weights.
Can the result be treated as proof that the edit works?
No. It was one experiment with one run per setting, a small published task set, and acknowledged differences between setups. Replication under matched conditions is needed to establish whether the completion change is repeatable or attributable to the edit.
Source: ThorstenMeyerAI.com
