Weight ablation: edit the model, not the run

Note

Generated from notebooks/applications/weight_ablation.ipynb, which you can run yourself. The outputs below are the ones it produced.

Every other intervention in Murano is applied to activations, at run time, by a hook. Remove the hook and the model is exactly as it was.

Weight ablation is permanent. It projects a direction out of the weight matrices themselves, so every matrix that reads from the residual stream can no longer see that direction, and every matrix that writes to it can no longer produce it. The result is an ordinary transformer that has simply lost a concept, and you can save it and hand it to someone with no interpretability tooling at all.

Key questions

• Can a concept be removed from the weights rather than suppressed per-run?
• How many matrices does that touch?
• Does the edited model still produce fluent text?

Structure

1. Set up
2. Derive the direction
3. Project it out of the weights
4. Compare and score
5. Save the edited model

Model and data. Weight ablation needs a Llama-family layout (separate q/k/v/o projections and a gated MLP), so this is the one notebook that does not run on GPT-2. It uses Llama-3.2-1B in float32 and needs a few GB of GPU memory. We load it from an ungated mirror so the notebook runs without a license prompt; meta-llama/Llama-3.2-1B-Instruct is the same weights if you have access. Contrastive sentences come from murano.tasks.

Requirements. No extras: pip install murano-interp

1. Set up

import torch

from murano import MuranoModel, Pipeline, keys
from murano.artifacts import PromptBatch
from murano.dataset import MuranoDataset
from murano.steps import (
    GenerationMetric,
    Load,
    Record,
    SteeringVector,
    WeightAblation,
)
from murano.tasks import positive_word_rate, sentiment

OUTPUT_DIR = "murano_outputs/weight_ablation"

# GPT-2 is small enough that bfloat16 rounding degrades its predictions.
model = MuranoModel("unsloth/Llama-3.2-1B-Instruct", dtype=torch.float32)
print(f"{model.model_id}: {model.n_layers} layers, d_model={model.d_model}")

unsloth/Llama-3.2-1B-Instruct: 16 layers, d_model=2048

2. Derive the direction

Exactly as in steering.ipynb: contrastive prompts, record the last token, take the difference of class means.

positive, negative = sentiment(n_per_class=8)

results = Pipeline(
    [
        Load(MuranoDataset.contrastive(positive=positive, negative=negative)),
        Record(model, layers="all", position="last", batch_size=4),
        SteeringVector(normalize=True),
    ]
).run()

print("best separating layer:", results[keys.STEERING].best_layer)

best separating layer: L12.resid_post

3. Project it out of the weights

WeightAblation reads the steering key, builds a projection operator from the best layer’s direction, and rewrites the token embedding, every attention q_proj / k_proj / v_proj (which read the residual stream), every o_proj (which writes to it), and every MLP gate_proj / up_proj / down_proj.

It validates the architecture before touching anything, so an unsupported model fails cleanly rather than ending up half-edited. It also generates from the clean and the ablated model, so the comparison comes for free.

EVAL_PROMPTS = [
    "The movie was",
    "I think you are",
    "My honest opinion of this restaurant is that it",
]

results[keys.PROMPTS] = PromptBatch(prompts=EVAL_PROMPTS)
results = WeightAblation(model, gen_kwargs={"max_new_tokens": 20, "do_sample": False})(
    results
)

print("weight matrices modified:", results[keys.WEIGHT_ABLATION].metadata["n_modified"])

weight matrices modified: 113

4. Compare and score

comparison = results[keys.INTERVENE]
for i, prompt in enumerate(comparison.prompts):
    print(f"{prompt!r}")
    print(f"    clean  : {comparison.clean_generations[i]!r}")
    print(f"    ablated: {comparison.modified_generations[i]!r}")
    print()

'The movie was'
    clean  : ' released in 2008 and directed by the renowned director, Christopher Nolan. The film is based on'
    ablated: ' released in 2008 and directed by Peter Jackson. The film is based on the novel of the'

'I think you are'
    clean  : ' referring to the popular video game series "The Legend of Zelda" by Nintendo. The series has been'
    ablated: ' referring to the popular video game series "The Legend of Zelda" by Nintendo. The series has been'

'My honest opinion of this restaurant is that it'
    clean  : "'s a bit of a mixed bag. On the one hand, the food is consistently good and the"
    ablated: "'s a bit of a mixed bag. On the one hand, the service was top-notch. Our"

The model is still fluent. Projecting one direction out of a hundred-odd weight matrices did not turn it into a noise generator, which is the whole reason the technique is interesting.

Look closely, though. Some completions change a lot, and at least one may not change at all. A single sentiment direction estimated from eight sentence pairs is a blunt instrument, and these prompts are not strongly sentiment-bearing to begin with. Score it rather than trusting the eye.

metric = GenerationMetric(
    metric_name="positive_word_rate", score_fn=positive_word_rate
)(results)[keys.METRIC]

print(f"{metric.baseline_label}: {metric.baseline_score:.2f}")
print(f"{metric.modified_label}: {metric.modified_score:.2f}")

clean: 0.33
modified: 0.00

The positive-word rate drops after ablation, which is the direction we expected. On three prompts that is an anecdote, not a measurement: the point of the metric here is to show how you would establish the claim, not to have established it. A real study needs an evaluation set and a trained sentiment classifier.

5. Save the edited model

save_ablated_model writes weights and tokenizer in HuggingFace format. The result loads with plain transformers, with no Murano dependency.

Note that this writes the entire model, not a patch: around 4.7 GB for a 1B model in float32. Skip this cell if you only wanted the comparison above.

from murano import save_ablated_model

path = save_ablated_model(model, f"{OUTPUT_DIR}/ablated_model")
print("saved to:", path)

saved to: murano_outputs/weight_ablation/ablated_model

Weight ablation versus activation steering

	Intervene	WeightAblation
what changes	activations, per forward	the weight matrices
persistence	only while the hook is set	permanent
architectures	any	Llama-family layout only
exportable	no	yes, as a HuggingFace model

Use Intervene to ask questions. Use WeightAblation when you want to hand someone a model with the concept genuinely gone. The canonical application is ablating a refusal direction to study how refusal is implemented.

What next

• steering.ipynb applies the same direction at run time instead, which is reversible and works on any architecture.