Steering: find a direction and change what the model writes

Note

Generated from notebooks/applications/steering.ipynb, which you can run yourself. The outputs below are the ones it produced.

A steering vector is a single direction in the residual stream that encodes a concept. Take the mean activation of positive sentences, subtract the mean of negative ones, and the difference points from “negative” toward “positive”. Add that direction back during generation and the model’s output shifts.

Key questions

• Is sentiment stored as one linear direction, and if so, at which layer?
• Does adding that direction actually change what the model generates?
• Can we measure the change instead of eyeballing it?

Structure

1. Set up
2. Derive the direction
3. Plot the direction
4. Steer generation
5. Score the effect
6. Save

Model and data. GPT-2 small in float32. It is small enough that bfloat16 rounding visibly degrades its predictions, so the notebooks load it explicitly in float32. 25 positive and 25 negative sentences from murano.tasks.

Requirements. pip install murano-interp[plot]

1. Set up

import torch

from murano import MuranoModel, Pipeline, keys
from murano.dataset import MuranoDataset
from murano.steps import (
    GenerationMetric,
    Intervene,
    Load,
    LoadPrompts,
    Plot,
    Record,
    Save,
    SteeringVector,
)
from murano.tasks import positive_word_rate, sentiment

OUTPUT_DIR = "murano_outputs/steering"

# GPT-2 is small enough that bfloat16 rounding degrades its predictions.
model = MuranoModel("gpt2", dtype=torch.float32)
print(f"{model.model_id}: {model.n_layers} layers, {model.n_heads} heads")

gpt2: 12 layers, 12 heads

2. Derive the direction

Record captures the residual stream at the last token of each sentence, which is where a decoder-only model has seen the whole thing. SteeringVector then takes the difference of class means at every layer.

separation_scores measures how cleanly that direction splits the two classes: higher is better.

positive, negative = sentiment(n_per_class=25)
dataset = MuranoDataset.contrastive(positive=positive, negative=negative)

results = Pipeline(
    [
        Load(dataset),
        Record(model, layers="all", position="last", batch_size=8),
        SteeringVector(normalize=True),
    ]
).run()

steering = results[keys.STEERING]
print("best layer:", steering.best_layer, "\n")
for layer, score in sorted(steering.separation_scores.items()):
    marker = "  <-- best" if layer == steering.best_layer else ""
    print(f"  {layer}: {score:.3f}{marker}")

best layer: L8.resid_post

  L0.resid_post: 0.927
  L1.resid_post: 0.876
  L2.resid_post: 0.765
  L3.resid_post: 0.822
  L4.resid_post: 1.207
  L5.resid_post: 1.294
  L6.resid_post: 1.204
  L7.resid_post: 1.366
  L8.resid_post: 1.461  <-- best
  L9.resid_post: 1.424
  L10.resid_post: 1.338
  L11.resid_post: 0.725

3. Plot the direction

Plot recognizes a SteeringResult and draws two figures: the per-layer separation scores, and the cosine similarity between the directions found at each pair of layers. A bright block in the second plot means those layers agree on where “positive” points, which is evidence the concept is one stable direction rather than a different one per layer.

Plot(output_dir=OUTPUT_DIR)(results);

4. Steer generation

Two knobs decide whether anything happens: where the direction is applied (direction_layers) and how hard (alpha). We will get both wrong first, on purpose.

EVAL_PROMPTS = [
    "I think that the food was",
    "The movie was",
    "My friend is a",
    "Today I feel",
]


def show(comparison):
    """Print each prompt with its clean and steered completion, then score both."""
    for i, prompt in enumerate(comparison.prompts):
        print(f"{prompt!r}")
        print(f"    clean  : {comparison.clean_generations[i]!r}")
        print(f"    steered: {comparison.modified_generations[i]!r}")
        print()
    clean = positive_word_rate(comparison.clean_generations)
    steered = positive_word_rate(comparison.modified_generations)
    print(f"positive-word rate: clean {clean:.2f} -> steered {steered:.2f}")

Attempt 1: steer at the best-separating layer

The obvious move is to intervene where the concept separates best. Here that is layer 0.

show(
    Pipeline(
        [
            LoadPrompts(EVAL_PROMPTS),
            Intervene(
                model,
                direction_key=keys.STEERING,
                mode="steer",
                alpha=6.0,
                direction_layers="best",
                gen_kwargs={
                    "max_new_tokens": 15,
                    "do_sample": False,
                },
            ),
        ]
    ).run(results.copy())[keys.INTERVENE]
)

'I think that the food was'
    clean  : " good, but I don't think it was good enough for me. I"
    steered: " good, but I don't think it was good enough for me. I"

'The movie was'
    clean  : ' released in Japan on May 7, 2016.\n\nThe movie is set'
    steered: ' released in Japan on May 7, 2016.\n\nThe movie is based'

'My friend is a'
    clean  : " very good person, and I'm very happy with her. I'm very"
    steered: " very good person, and I'm very happy with her. I'm very"

'Today I feel'
    clean  : " like I'm in a different place. I'm not in a place where"
    steered: " like I'm in a different place. I'm not in a place where"

positive-word rate: clean 0.50 -> steered 0.50

The wording shifts a little, but the sentiment does not: the positive-word rate is unchanged. Nudging layer 0 rearranges the sentence without making it any warmer.

This is the single most useful lesson in this notebook. Separation tells you where a concept is readable, not where it is effective. Layer 0’s residual stream is dominated by token embeddings, and the words “love” and “hate” are lexically obvious there, which is exactly why the separation score is highest. But eleven more layers of computation sit downstream, and they overwrite whatever we nudge at layer 0.

A probe would have been delighted with layer 0. A causal intervention is not.

Attempt 2: steer at every layer

Applying the direction at every layer keeps it present all the way to the unembedding.

comparison = Pipeline(
    [
        LoadPrompts(EVAL_PROMPTS),
        Intervene(
            model,
            direction_key=keys.STEERING,
            mode="steer",
            alpha=4.0,
            direction_layers="all",
            gen_kwargs={"max_new_tokens": 15, "do_sample": False},
        ),
    ]
).run(results.copy())[keys.INTERVENE]

show(comparison)

'I think that the food was'
    clean  : " good, but I don't think it was good enough for me. I"
    steered: ' delicious and the service was great. I would definitely recommend this place to anyone'

'The movie was'
    clean  : ' released in Japan on May 7, 2016.\n\nThe movie is set'
    steered: ' shot in the summer of 2013 and was shot in the summer of 2014.'

'My friend is a'
    clean  : " very good person, and I'm very happy with her. I'm very"
    steered: ' very nice person and I am very happy with the way she is doing.'

'Today I feel'
    clean  : " like I'm in a different place. I'm not in a place where"
    steered: ' that the new year is a great time to celebrate the new year with a'

positive-word rate: clean 0.50 -> steered 0.75

Now the sentiment moves, and the rate moves with it. The completions turn approving without falling apart.

Attempt 3: push too hard

alpha is not free. Turn it up and the direction swamps the rest of the residual stream, and the model stops producing language at all.

show(
    Pipeline(
        [
            LoadPrompts(EVAL_PROMPTS),
            Intervene(
                model,
                direction_key=keys.STEERING,
                mode="steer",
                alpha=16.0,
                direction_layers="all",
                gen_kwargs={
                    "max_new_tokens": 12,
                    "do_sample": False,
                },
            ),
        ]
    ).run(results.copy())[keys.INTERVENE]
)

'I think that the food was'
    clean  : " good, but I don't think it was good enough for"
    steered: ' the and the and the\n\n\n\n\n\n\n'

'The movie was'
    clean  : ' released in Japan on May 7, 2016.\n\nThe'
    steered: ' a happy and happy.\n\n\n\n\n\n\n'

'My friend is a'
    clean  : " very good person, and I'm very happy with her."
    steered: ' happy and and and\n\n\n\n\n\n\n\n'

'Today I feel'
    clean  : " like I'm in a different place. I'm not in"
    steered: ' the same and and and\n\n\n\n\n\n\n'

positive-word rate: clean 0.50 -> steered 0.50

That degenerate output is what over-steering looks like. Somewhere between these extremes is a usable range, and finding it is empirical: there is no substitute for looking at the text.

We keep the alpha=4.0, every-layer result from attempt 2 as comparison.

5. Score the effect

Reading four generations is not evidence. GenerationMetric applies a scoring function to both sides of the comparison, so the shift becomes a number.

positive_word_rate is deliberately crude: a keyword count over a handful of prompts. It is enough to show the direction of the effect and far too small to quantify it. Swap in a sentiment classifier and a real evaluation set before reporting anything.

results[keys.INTERVENE] = comparison
scored = GenerationMetric(
    metric_name="positive_word_rate", score_fn=positive_word_rate
)(results)

metric = scored[keys.METRIC]
print(metric.metric_name)
print(f"  {metric.baseline_label}: {metric.baseline_score:.2f}")
print(f"  {metric.modified_label}: {metric.modified_score:.2f}")

positive_word_rate
  clean: 0.50
  modified: 0.75

Had we scored attempt 1 instead, both numbers would be identical. The metric is what turns “the text looks different” into a claim you can defend.

6. Save

Every recognized artifact lands under one tree.

run_dir = Save(
    output_dir="murano_outputs", run_name="steering", model_id=model.model_id
)(scored)[keys.OUTPUT_DIR]

print("saved to:", run_dir)

saved to: murano_outputs/steering

What next

• probing.ipynb asks the complementary question: can we read the concept off the activations?
• weight_ablation.ipynb bakes a direction into the weights permanently.
• custom_pipeline.ipynb measures readability and steerability layer by layer, and shows they disagree.