Weight-Space Programming: MIT CSAIL says an LLM edited 0.016% of Qwen3.5-0.8B's weights, MATH-500 7.9% to 48.1%

MIT CSAIL researchers say a general-purpose AI model can read and rewrite another neural network's raw weights, with no training run and no gradient descent. In a blog post published on 2 November 2026, Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab describe what they call Weight-Space Programming, and report that edits to 0.016% of the weights of the small open model Qwen3.5-0.8B lifted its MATH-500 score from 7.9% to 48.1%. The result is early, comes from a blog post rather than a peer-reviewed paper, and has an important catch: most of the gain came from stopping the model from looping, not from teaching it new maths.

Key points

• The authors (MIT CSAIL) say GPT-6 Astra, a frontier LLM, edited 131,072 of Qwen3.5-0.8B's roughly 0.8 billion weights (0.016%) without any training, and MATH-500 rose from 7.9% to 48.1% (blog post, 2 November 2026).
• The same edit raised GSM8K from 13.8% to 67.1% and IFEval from 21.0% to 62.3%, and cut the share of MATH-500 answers that hit the 32,768-token limit from 93.0% to 33.1%.
• With only 10 or 25 forward passes the loops barely went away; the big gain needed 50 forward passes.
• On Gemma 3 270M, which rarely loops, the same approach did not help: no edit moved MATH-500 by more than about a point.
• The work is a blog post with logs released on GitHub, not a peer-reviewed paper, and the authors call the results "very early".

What happened

On 2 November the four authors, all listed at MIT CSAIL, published "Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights". The claim is simple to state: give a frontier language model the raw numbers of a small network, as text or through a Python sandbox, and it can work out what the network does, fix a failure, or write a working network from scratch, without ever running an optimiser such as Adam or training with gradients.

The authors use a model they call GPT-6 Astra throughout; we could not check that model independently. They ran eight experiments, which we summarise from their post.

• Reading weights. Given the 17,253 weights of a small policy network, Astra named the game it was trained on (Breakout, Pong, Pac-Man or Freeway) for all four networks, without being told the options. It identified what three small diffusion models were trained to draw, though it drew every shape as an outline when the real models produce filled shapes. Given only the four middle transformer blocks of three 7.2-million-weight character models (no input or output layers), it named the training text (Python code, romaji Japanese or English) in all three cases, and found the "copying" attention heads that match those measured by running the models.
• Fixing weights. A reinforcement-learning network stuck in a local minimum on a car-and-ramp task succeeded in all 1,024 test episodes after Astra edited 3.5% of its 171,664 weights in one try. Before the edit the car never succeeded in 256 episodes.
• Writing weights from scratch. For three MinAtar games, Astra wrote the weights in one reply with no data, loss, gradients or tools. The authors report that these networks beat PPO agents trained for 100 million steps in all three games.
• Scaling up. For 9x9 Go with a 1,480,274-weight network, Astra wrote the network layer by layer in Python. With 50 allowed forward passes it beat an equal-sized PPO network trained on 20 million games, according to the authors.
• A failure. Asked to tell which of two Gemma 3 270M copies had been fine-tuned to be sycophantic, using weights alone, Astra could not, in four attempts. Allowed to see only attention patterns on texts it wrote, it picked the sycophant in all three pairs.

The headline experiment is the eighth.

Background

Until now, neural weights were written by optimisation, by networks trained to produce other networks' weights, or by compilers from formal programs, as the authors put it. Researchers have used language models to improve prompts, whole agent harnesses and training code. Editing weights directly is a different step: the model acts as the optimiser. The authors also point to mechanistic interpretability, where humans trace circuits and edit weights by hand, as the field this touches.

Their related-work section notes one tiny precedent, AdderBoard, in which coding agents set the weights of a very small transformer by hand, and argues that what is new here is a general frontier model, used as it is, that reads and writes the weights themselves.

The numbers

The Qwen3.5-0.8B experiment is the one most likely to be quoted. The authors set a budget of forward passes (10, 25 or 50) for Astra to test the model on practice problems from the GSM8K and MATH training sets (7,469 and 7,275 problems; the test sets were never shown to it). Evaluation used thinking mode, a 32,768-token limit and the sampling settings the authors list.

Base7.9

10 passes12.3

25 passes14.1

50 passes48.1

Qwen3.5-0.8B on MATH-500 after weight edits, by forward-pass budget (% correct)

	% correct
Base	7.9
10 passes	12.3
25 passes	14.1
50 passes	48.1

Source: Takehi, Teoh, Zhang and Khattab, Sparks of Weight-Space Programming (2 November 2026)

Base93.0

10 passes87.2

25 passes86.4

50 passes33.1

Share of MATH-500 answers that hit the 32,768-token limit (% of answers)

	% of answers
Base	93.0
10 passes	87.2
25 passes	86.4
50 passes	33.1

Source: Takehi, Teoh, Zhang and Khattab, Sparks of Weight-Space Programming (2 November 2026)

The two charts tell the same story. With the base model, 93.0% of MATH-500 answers ran to the token limit, which means the model was usually stuck repeating itself and never reached an answer. The authors say 85 of its first 96 practice answers ran to the limit. Astra traced the looping to one direction written by the MLPs of layers 15 and 19, and subtracted that direction from 64 output columns in each of those layers, which is the 131,072 weights (0.016%).

The authors are explicit about what this does and does not show: "Fixing loops, however, is not the same as making a model better at math", in their words, since it lets the model finish answers it could already work out. The Gemma 3 270M control makes the point. Only 8.5% of its answers reach the limit, its base MATH-500 score is 6.8%, and the edited versions scored between 7.0% and 7.7% across 5 to 50 forward passes. Astra itself reported the gains as not established.

Full results table from the post for the Qwen model:

Qwen3.5-0.8B	MATH-500	GSM8K	IFEval	Hit token limit
Base	7.9%	13.8%	21.0%	93.0%
Edited, 10 passes	12.3%	25.7%	49.0%	87.2%
Edited, 25 passes	14.1%	25.1%	39.9%	86.4%
Edited, 50 passes	48.1%	67.1%	62.3%	33.1%

Note the jump from 25 to 50 passes. The results are not a smooth curve, and each score is an average of a few samples per problem (the authors list 4 on MATH-500 and 2 on GSM8K and IFEval), so small gaps between rows should not be over-read.

How it works

In the weight-editing set-up, Astra is given a Python session with the model's weights loaded and a limited number of forward passes. It cannot train, and it cannot see the benchmark test sets. It reads statistics of the weights, runs the model on practice prompts, forms a hypothesis about which component causes the failure, and writes the change into the tensors.

The loop fix resembles a known technique in interpretability: removing a behaviour by deleting a single direction from a model's weights, which the authors cite. What is different is who finds the direction. Here a language model locates it and applies it, guided by the symptoms it sees.

For the from-scratch tasks, the authors describe Astra writing the network as a small program in neural form. In Breakout, the first layer computes where the ball will land and the second moves the paddle there. In Go, early layers mark stones and liberties, middle layers pass information along chains of stones to find groups in danger, and the last layer scores each move. With no forward passes, the authors say, it wrote this in one Python run of 336 lines.

What it means

For researchers. The authors suggest uses such as auditing a model without running benchmarks (for example looking for backdoors), cheap capability steering and unlearning, and model merging across different architectures. These are their proposals, not demonstrated results. The only real-model result is the Qwen loop fix, and the sycophancy experiment shows the reading side still fails on weights alone.

For developers running small models. A concrete lesson: a small reasoning model that loops can look far worse at maths than it is. The authors note that Qwen's model card describes this model as prone to thinking loops. We checked the model card: it lists recommended sampling settings (including a presence penalty) and says the model runs in non-thinking mode by default, while the authors evaluated with thinking on. Your own results may differ from theirs if you use other settings.

For the skeptic. A 0.8-billion-parameter model that scored 7.9% on MATH-500 was in a degenerate state. Lifting it to 48.1% is real on their set-up, but it says more about a fixable failure than about general reasoning gains. The same approach did nothing for a model without that failure.

What we don't know yet

• Whether others can reproduce the results. The authors released complete logs at github.com/RikiyaT/lm_understand-logs and an edited Qwen model on Hugging Face, and thank Hugging Face's Ahsen Khaliq for rebuilding it from the logs and re-running its evaluation. We have not run it ourselves.
• How well it scales beyond a 0.8-billion-parameter model, and whether it avoids the forgetting that fine-tuning can cause. The authors say it may, because so few weights change, but do not report a test of that.
• Whether other frontier models can do this. Every experiment uses GPT-6 Astra.
• Cost. The post counts forward passes but, in the parts we read, gives no dollar or compute cost for the Astra sessions.
• Peer review. This is a blog post. The authors say a benchmark and harness for evaluating the method are still being built.

What to watch next

The benchmark and harness the authors say they are building, independent replications of the Qwen edit, and whether anyone shows a gain on a model that does not loop. Also watch whether interpretability groups test the "audit without evaluation" idea on models with known backdoors.

Sources

• Sparks of Weight-Space Programming (Takehi, Teoh, Zhang, Khattab, MIT CSAIL, 2 November 2026), primary source
• The Register: Forget retraining, AI can now tweak another AI's weights (3 November 2026)
• Qwen3.5-0.8B model card on Hugging Face
• Authors' experiment logs on GitHub, linked from the post; we did not open the logs