MIT CSAIL's Sparks of Weight-Space Programming (Takehi, Teoh, Zhang, Khattab, 2 Nov 2026) says a frontier LLM, GPT-6 Astra, can read and write neural-network weights directly, with no optimizer and no gradient descent. Several people said the claims were hard to check. So I checked all of them against the released logs, and reran what I could on my own hardware with my own code.
I'm not affiliated with the authors. All code and results are in Ciatel-Arlanta/weight-space-lab, and the raw generation outputs are attached to its v1 release.
Disclosure: This audit was run with AI coding agents: Claude Opus 5.5 led it and delegated smaller tasks to GPT-6 Luna and Gemini 3.8 Flash. Every number was checked against the scripts' output. See "Astra vs Opus" below for who did what.
The short answer
The numbers are real. The framing is stronger than the evidence.
Every number I traced appears in the released logs. Whenever enough was released to recompute a result, my independent code reproduced it. That includes the headline result: on my GPU, their edited Qwen3.5-0.8B scores 51% on MATH-500, against 9% for the unedited model. It also beats the best decoding-only fix I found by 25 points on both math sets.
What doesn't hold up as well is how the results are described. Most results come from single runs, with selection that is partly undisclosed. Some baselines are weak or missing. A few edits are described as gentler or smaller than they are. And four results rest on models or opponents the authors didn't release, so nobody outside the lab can check them.
| Claim groups audited | 9 of 9 (Exps 1–8 plus the general method claims) |
| Results independently reproduced with my own code | 6, in Exps 2, 4, 5, 6 and 8 |
| Fabricated numbers or hidden tool calls found | 0 |
| Edit vs the best decoding-only fix, on both math sets | +25 points (paired p = 1e-11) |
Scorecard
Verdicts: Confirmed means the logs and my recomputation both support the claim. With caveat means it holds as worded but needs context the post leaves out. Contradicted means the logs show something different. Not verifiable means the needed material isn't released.
| Exp | Headline claim | Independent check | Verdict | Main caveat |
|---|---|---|---|---|
| 1 | Names the right game for 4/4 PPO policies from weights as text | Network shapes and weight text match the released files (17,253 params) | With caveat | One attempt each. "Frogger" was scored as Freeway. The prompt labels each input channel's role. |
| 2 | Names the shape each of 3 diffusion models draws, no forward pass | My own DDPM sampler draws filled circles, squares and triangles | Confirmed | Astra described outlines, not filled shapes. The post says so itself. |
| 3 | Right language 6/6; finds every copying head and no false ones | Answers match the logs; all 48 Python runs obey the no-forward-pass rule | Confirmed vs logs; ground truth not verifiable | Embeddings and output layer are withheld. |
| 4 | One reply, no tools: Kinetix car goes from 0% to 100% success, changing 3.5% of weights | Re-simulated in public Kinetix: base 0/64, edited 64/64; the 51 edit statements reproduce the network bit for bit | With caveat | One deterministic trajectory. The change is about 26× the size of the original weights. |
| 5 | Networks written from zero beat 100M-step PPO in three MinAtar games | My 1,000-episode re-score: 503 / 51.8 / 52.8 (post: 517 / 53.1 / 53.2) | Confirmed | Weak PPO baselines; checkpoints not released. |
| 6 | 9×9 Go from scratch; 50 passes beats an equal-size PPO | My own Go engine replays all 6 released head-to-heads within noise | Confirmed; main opponents not released | The equal-size PPO is a weak player. |
| 7 | Spots the sycophantic Gemma in 3/3 pairs with an attention tool | Behaviour rates recomputed; Figure 2 reproduced exactly | With caveat | One attempt; 1-in-8 chance level; one pick at 55%. |
| 8 | Qwen3.5-0.8B: MATH-500 7.9 → 48.1 by editing 131,072 weights | Tensor diff, re-derivation, and my own 200-problem eval against a decoding baseline | Reproduced; wording contradicted | Two directions, flipped and amplified 5×. One run, never repeated. |
| 8b | Could not improve Gemma 3 270M | Scores and Astra's "not established" statements match the logs | Confirmed | Lenient grading not disclosed. |
| General | Tool-free, no training loops, complete logs | 1,125 turns tool-free; no optimizer or gradient code | With caveat | The number of excluded trial runs is unknown. |
Method
The authors released a logs repository (RikiyaT/lm_understand-logs), with every prompt, reply, code run, network and evaluation file, and the edited model on Hugging Face. Their harness is not released.
I first wrote the post's claims down verbatim: 6 general claims and 42 experiment claims. Then I traced every number to the file it came from. Wherever the released material allowed it, I recomputed the result with code written from the experiment's own description.
Ground rules:
- The logs were treated as untrusted data. Nothing in them was executed, imported or unpickled, and every recomputation used my own scripts in isolated Python.
- Reruns of Exps 1–7 ran on CPU. Qwen generation ran on the GPU (vLLM 0.33, bf16, an RTX 4060 Laptop with 8 GB).
- Decoding baselines were chosen on practice problems only, never on the test set.
Exp 8: fixing Qwen3.5-0.8B's thinking loops
This is the post's biggest claim. With thinking on, Qwen3.5-0.8B mostly loops until it hits its 32,768-token limit. Astra traced the looping to "one direction written by the MLPs of layers 15 and 19" and "subtracted that direction from 64 output columns in each (131,072 weights)". With 50 forward passes, MATH-500 went from 7.9 to 48.1 and GSM8K from 13.8 to 67.1.
The post never answers the obvious question: would a decoding change, like a repetition penalty, do the same job? I ran that test.
My generation run: 200 test problems, full length
| Config | MATH-500 | Hit limit | GSM8K | Hit limit | Cut answers ending in a verbatim loop |
|---|---|---|---|---|---|
| Base, the post's settings | 9.0% ± 2.9 | 91% | 19.0% ± 3.9 | 83% | 97% / 98% |
| Decoding baseline: repetition penalty 1.1 | 26.0% ± 4.4 | 73% | 38.0% ± 4.9 | 53% | 90% / 96% |
| Their edit (released model) | 51.0% ± 5.0 | 36% | 63.0% ± 4.9 | 7% | 11% / 29% |
| Post, same problem ids (base → edit) | 8.8 → 50.2 | 10.0 → 65.0 |
100 MATH-500 and 100 GSM8K test problems, a random subset, one sample each. Sampling as in the post: temperature 0.6, top-p 0.95, top-k 20, presence penalty 1.5, 32,768 new tokens, thinking on. ± is one standard error. Answers are graded by the last boxed answer after the thinking, using math-verify. My grader agrees with all 3,200 of the logs' own grades.
Head to head on the same problems
| Comparison (all 200) | Only the first right | Only the second right | McNemar p |
|---|---|---|---|
| Edit vs decoding baseline | 55 | 5 | 1e-11 |
| Decoding baseline vs base | 44 | 8 | 4e-7 |
The decoding fix is real: one sampling flag roughly triples the base score on MATH-500. It closes about 40% of the gap to the edit (17 of 42 points on MATH-500, 19 of 44 on GSM8K). It mostly delays the loops rather than removing them: most of its answers that hit the limit still end in a word-for-word repeat. The edit's remaining limit hits are mostly long reasoning that doesn't repeat.
How the baseline was picked
| Candidate, base model | Practice accuracy | Cut |
|---|---|---|
| The post's settings | 14.1% | 85.9% |
| Presence penalty 2.0 (vLLM's max) | 13.3% | 87.5% |
| Temperature 1.0 (the model card's setting) | 28.9% | 70.3% |
| Repetition penalty 1.1 | 32.8% | 61.7% |
The candidates ran on 128 practice problems (64 GSM8K-train, 64 MATH-train), the pool the post's own practice problems come from. A fixed rule picked the winner before any test run. I tried three single changes only, so combinations or other samplers might do better.
What the edit actually is
I diffed the released model against the base model tensor by tensor, and rebuilt the edit from Astra's own code in the logs. For each layer L in 15 and 19:
W_L = W0_L − 6 · u_L (m_L ⊙ u_Lᵀ W0_L) u_L: a unit vector in the 1,024-d residual stream m_L: a mask selecting 64 of the 3,584 MLP neurons
| Claim in the post | What the weights and logs show | Verdict |
|---|---|---|
| 131,072 weights, layers 15 and 19, 64 columns each | Exactly 2 of 488 tensors differ; 64 × 1,024 × 2 entries; rank 1 per tensor | Confirmed |
| "one direction" | Two different vectors, with an absolute cosine of 0.849 between them | Contradicted |
| "subtracted that direction" | The strength is 6, not 1. Each selected neuron's write along u goes from c to −5c: flipped and amplified, not removed. | Contradicted in wording |
| Direction found from the loops | u comes from 4 pairs of short texts Astra wrote (a productive continuation vs a hand-written loop). Rebuilding it gives cosine 0.99998 and the same 64 of 64 columns. | Confirmed |
| The results table | Matches the logs exactly. Each row is a different edit from one run. About 11 Qwen runs exist across prompt versions v2–v5. The 50-pass run is the only one with an evaluation. | Misleading framing |
| "raising IFEval by 3×" and "six other benchmarks" | 21.0 → 62.3 (2.97×). The logs also score AMC23, Minerva, OlympiadBench, MMLU-Pro, GPQA and AIME, but the post's table shows only three benchmarks. | Under-reported |
| Neutral task setup | The task prompt named the repetition mechanism as a hint | With caveat |
| Gemma: 8.5% vs 93% hit the token limit | Not like for like: 8,192 tokens without thinking vs 32,768 with thinking. The Gemma scores use lenient grading (strict: 2.8 → 2.6–3.8). | With caveat |
I also rebuilt the edit from the base model by following Astra's logged steps, and compared it with the published tensors: 99.0% of entries are identical after bf16 rounding, and the change differs from theirs by 0.6–0.9%. Their published model is exactly what their logged procedure produces.
Verdict on Exp 8: the released edit does what the post says, and it beats a fair decoding baseline by a wide margin. But the post describes the edit more gently than it is, and the editing procedure behind it has been shown once. Whether Astra would find a 48-point fix again on a second run is unknown.
Exp 1: which game is this policy for?
Four PPO policies (500 → 32 → 32 → 5, tanh) trained on 10×10 grid games are given to Astra as text, with no game named.
| Trained on | Astra's answer | Confidence |
|---|---|---|
| Breakout | Breakout | 99% |
| Pong | Pong | 96% |
| Pac-Man | Pac-Man | 96% |
| Freeway | "Frogger" (scored correct by a keyword rule) | 97% |
- The weight text matches the released files, and no game names or candidate lists appear in the prompt.
- The prompt does name each input channel's role ("agent", "moving objects", "reward targets", "obstacles"). That turns much of the task into reading a spatial layout.
- One attempt per network, and only the best of 3 PPO seeds per game is released. The run paths mention an earlier prompt version that is not released.
Exp 2: what does this diffusion model draw?
Three small DDPMs (12×12 images, 23,824 weights each). Astra named all three shapes without a forward pass, at 84%, 77% and 88% confidence. I wrote my own DDPM sampler from the experiment's spec and sampled the released weights:
| Circle model, 64/64 filled | Square model, 64/64 filled | Triangle model, 63/64 filled |
|---|---|---|
 |  |  |
The models draw filled shapes, but Astra described hollow outlines for all three, as the post itself says. Each identification is a single attempt.
Exp 3: which language, and which heads copy?
Three character-level transformers trained on Python, romaji or English. Astra gets only the four middle blocks, with no embeddings and no output layer, plus up to 100 Python runs that may not execute the network.
| Trained on | Answers (2 attempts) | Copying heads named |
|---|---|---|
| Python | 0.56 / 0.52 | 2.1, 3.1, 3.5 |
| Romaji | 0.55 / 0.62 | 3.3 |
| English | 0.58 / 0.48 | 3.1 |
- All answers and named heads match the logs. None of the 48 code runs execute the network on any input.
- The ground-truth copying scores need the withheld embeddings and output layer, so nobody outside can recompute them.
- The post counts a head as copying above 0.2, while the logs' own scoring file uses 0.3. At 0.3, three of the five named heads drop out.
- "Romaji never uses q, l, v and x" can't be checked without the corpus.
Exp 4: fixing a Kinetix car that stalls on a ramp
| Check | Result |
|---|---|
| Base policy, my rerun in public Kinetix | 0 / 64 |
| Edited policy, my rerun | 64 / 64 |
| Astra's 51 edit statements → released network | Bit-for-bit match |
| Weights changed | 6,073 / 171,664 (3.54%) |
| Size of change vs original weights | 25.7× |
| Largest new bias (original max 0.094) | 876 |
| Hint words in the prompt ("reverse", "run-up", …) | None |
- The fix is genuine. Astra worked out from the geometry alone that the car needs to back up for a run-up.
- "3.5% of the weights" understates the change. It is a hard-coded controller with saturated units and logit margins around 600.
- "All 1,024 episodes" is one deterministic 73-step trajectory, repeated. One PPO seed was trained.
Exp 5: MinAtar policies written from zero
| Game | PPO 100M (best of 10) | PPO 500M (best of 3) | Astra, post | Astra, my re-score |
|---|---|---|---|---|
| Breakout | 15.8 | 25.3 | 517.4 ± 7.4 | 503.0 ± 7.7 |
| Seaquest | 5.6 | 88.9 | 53.1 ± 1.4 | 51.8 ± 1.3 |
| Asterix | 25.8 | 55.2 | 53.2 ± 1.0 | 52.8 ± 1.0 |
- The networks I rebuilt from Astra's replies match the released tensors exactly, and my re-score agrees with the post within 1.4 standard errors.
- The PPO baselines look weak. The best 100M Breakout seed never scores above 17 in 1,000 episodes, and the 500M Seaquest seeds score 88.9, 7.5 and 4.4.
- The PPO checkpoints are not released, so the baselines can't be re-scored.
Exp 6: a 9×9 Go network from scratch
I wrote my own Go engine from the rules in Astra's prompt and replayed every released head-to-head, 2,000 games each.
| Network | vs small PPO s0 | vs small PPO s2 |
|---|---|---|
| Astra, 0 passes | 54.2% | 54.9% |
| Astra, 50 passes | 84.0% | 80.8% |
| Same-size PPO, lr 3e-5 (the baseline) | 52–73% | 48–76% |
| Same-size PPO, first version (lr 3e-4) | 5–7% | 5–7% |
- Every replayed win rate is within sampling noise of the logged one, and game lengths, margins and how games ended all match too.
- The "equal-size PPO trained on 20 million games" is only about as strong as the 32-wide nets, and no real Go program is compared. The authors re-tuned it after the first version lost almost every game, and they disclosed this.
- The 0-pass network loses to every equal-size seed (26–43%). The small scored opponents are not released.
Exp 7: which Gemma is the sycophant?
| Pair | Astra's pick | P(correct) |
|---|---|---|
| 100 | Correct | 0.90 |
| 101 (Figure 2) | Correct | 0.97 |
| 102 | Correct | 0.55 |
- Behaviour rates check out (sycophant 100%, honest model 0.7–2.8%), and Figure 2 recomputes exactly from the logged attention arrays.
- There was one attention-tool attempt, and 3/3 binary picks happens by chance 1 time in 8.
- Weights-only attempts 1–3 had no embeddings or tokenizer. Only attempt 4 matched the successful run's information, and it was stopped unfinished.
Across all experiments
- Tool-free: all 1,125 turns are marked tool-free, and the raw event streams contain no tool-call items.
- No training: no optimizer, gradient or least-squares weight fit appears anywhere in Astra's code.
- "Everything": overstated. The logs README itself lists omissions, including intermediate files, draft networks, the Exp 3 corpus and the Exp 7 weights.
- Selection: trial runs of earlier prompt versions are left out, and their number isn't given. This is the largest risk I couldn't quantify.
Where the post says more than the evidence
| The post says | What was measured |
|---|---|
| "subtracted that direction" (Exp 8) | Two directions, each flipped and amplified 5× |
| "changing 3.5% of the weights" (Exp 4) | The change is about 26× the size of the original weights |
| "succeed in all 1,024 episodes" (Exp 4) | One deterministic trajectory, repeated |
| "beats the equal-sized PPO network" (Exp 6) | True for all three seeds, but the baseline is a weak player |
| "all three sets of pairs" (Exp 7) | One attempt; 1-in-8 chance level; last pick at 55% |
| "48.1 with 50 forward passes" (Exp 8) | One run, never repeated, and no decoding baseline |
| "MATH-500 6.8 → 7.0–7.7" (Exp 8b) | Lenient grading (strict: 2.8 → 2.6–3.8) |
| "everything we sent Astra" (logs) | Earlier trial runs, in an unknown number, are left out |
The omissions mostly cut the other way. The Exp 8 benchmarks left out of the table mostly favour the edit. The omitted Gemma runs strengthen the negative result. In Exp 6, the baseline was made stronger. This looks like under-reporting, not cherry-picking.
Astra vs Opus: how each side did the work
The research was done by GPT-6 Astra, according to the authors' logs. This audit was led by Claude Opus 5.5 in Claude Code, which delegated smaller verification tasks to two helper models: GPT-6 Luna through Codex, and Gemini 3.8 Flash through Antigravity. The jobs are different, so this is not a capability contest. Astra had to produce weight edits. The audit had to check them. Astra's numbers come from the run.json files in their logs, and Opus's from this session's own transcript.
The two setups
| Astra (the research) | Opus (this audit) | |
|---|---|---|
| Model and harness | gpt-6-astra via codex exec | claude-opus-5-5 via Claude Code |
| Job | Read and write network weights for 8 experiments | Check 48 claims against the logs; rerun what was released |
| Tools | Codex built-in tools disabled. Weights as text (Exps 1, 2, 4, 5); a metered Python sandbox (Exps 3, 6, 7, 8) | Shell, file editing, a local GPU and helper models. Banned from executing anything in the authors' logs |
| Model calls | 1,125 turns | 314 calls (lead) + 2 web-fetch helpers |
| Input tokens | 981M | 69.4M, 97.5% of them cache reads |
| Output tokens | 33.8M (16.9M of it reasoning) | 0.30M (lead) + 0.02M (web-fetch helpers) |
| Model time | 24.5 h, summed over turns | 2.7 h active, across a 23 h session |
| Helper models | None | 13 delegated tasks: 8 to GPT-6 Luna (11.9 h of wall time, mostly three long runs in parallel), 5 to Gemini 3.8 Flash (1.0 h) |
| Compute | CPU sandbox; GPU generation for Exp 8's forward passes | 13.2 GPU-hours of vLLM generation on a laptop RTX 4060, plus CPU reruns |
Token counts for the delegated tasks aren't exposed by the orchestrator, so only their wall time is listed. The lead's output count includes its reasoning.
Who did what, per experiment
| Exp | Astra: turns / output tokens | Audit: who did the work | What the audit did |
|---|---|---|---|
| 1 | 4 / 7,118 | Gemini (13 min, with Exp 2); Opus review | Weight-text check, prompt scan for leaks |
| 2 | 3 / 7,468 | Gemini; Opus review | DDPM sampler written from the spec |
| 3 | 54 / 512,515 | Luna (2 h 43 min); Opus review | Audit of all 48 code runs |
| 4 | 1 / 10,134 | Gemini (21 min); Opus addendum | Kinetix re-simulation and edit replay. Opus added the rounding and change-size findings |
| 5 | 3 / 73,454 | Luna (3 h 24 min); Opus review | MinAtar re-score, 1,000 episodes per game |
| 6 | 176 / 7,635,663 | Luna (2 h 42 min, logs check); Opus wrote the Go engine | Replay of all 6 head-to-heads. Luna's two replay attempts didn't finish |
| 7 | 322 / 15,141,765 | Gemini (9 min); Opus addendum | Behaviour rates, Figure 2 recompute, access-level comparison |
| 8 | 562 / 10,426,128 | Opus; Luna fact-checked the notes; Gemini wrote two analysis scripts | Tensor diff, edit re-derivation, the 200-problem eval and the decoding baseline |
| General | – | Luna (2 h 43 min) | Tool-use and training-code audit of all 1,125 turns |
Mistakes made along the way, and how they were caught
- Opus, Exp 4: a first diff showed 99.9% of weights changed. The cause was comparing against unrounded weights, while the prompt rounds to 3 significant digits. After matching by name against the rounded weights, the changed count came out at 6,073.
- Opus, Exp 7: misread the key order of a predictions file, and briefly recorded one pick as wrong. It corrected this to 3/3.
- Luna, Exp 6: twice failed to finish a Go replay (too slow, and wrong end-of-game handling). Opus wrote the engine instead, and it matched the logs.
- Luna, Exp 5: caught two of its own mistakes (a channel mapping and an unseeded RNG) and discarded those runs before reporting.
- Infrastructure: another app on the GPU slowed generation to under 5 tokens/s for about 75 minutes, until it was closed.
Every delegated report was read by the lead, and the corrections are recorded in the per-experiment reports.
What would settle the rest
| Open point | What would settle it |
|---|---|
| Is the Exp 8 result repeatable? | Repeat runs of the editing procedure. I'm building an open harness to try this. |
| Is there a stronger decoding baseline? | Combined penalties, DRY sampling, more samples per problem |
| Exp 3 copying heads | Release of the embeddings and output layers |
| Exp 5 PPO baselines | Release of the PPO checkpoints |
| Exp 6 main win rates | Release of the two small PPO opponents |
| Exp 7 models | Rebuilding them from the released data and script |
| Selection everywhere | The count and outcomes of the excluded trial runs |
Limits of this audit: the Exp 8 eval uses one sample per problem on 200 problems, so each accuracy is uncertain by 3–5 points either way. I verified the released artifacts but couldn't rerun Astra itself. The code scans can't rule out anything the release doesn't contain.
Reproduce
Every number here comes from a script in the repository, run against an unmodified clone of the authors' logs.
# Exp 8 generation eval (vLLM in WSL, one 8 GB GPU, about 11 hours) bash scripts/run_plan_b.sh python -I scripts/paired.py runs/test/test_edit.jsonl runs/test/test_dec_rep11.jsonl # Exp 8 weights python -I scripts/diff_models.py python -I scripts/rederive_edit.py python -I scripts/check_grader.py # Exps 1-7 and the general audit (CPU only) python -I scripts/verify/exp2_verify.py python -I scripts/verify/exp4_resimulate.py python -I scripts/verify/exp5_score.py python -I scripts/verify/exp6_replay.py --games 2000 --seed 0 python -I scripts/verify/exp7_recompute_fig2.py python -I scripts/verify/general_audit.py