SAE steering: writing with a feature, and where it breaks

Once you can read SAE features you can try to write with them: add a feature’s decoder direction to the residual stream during generation and see whether the model bends toward that concept. This is the mechanism behind Anthropic’s Golden Gate Claude.

We find a California feature, confirm it really is the California feature, and then steer with it. Most of the notebook is about the two things that decide whether that works: which feature you picked, and how hard you push. The second one does not have a happy ending here, and the reason is worth more than a working demo would have been.

Key questions

How do I find the feature for a concept I care about?
Why is the feature that fires hardest on a word often the wrong one?
How hard should I push, and what if no strength works?

Structure

Set up
Shortlist the features that fire on the concept
Pick the feature by what it promotes
Calibrate the strength against the residual stream
Steer, and watch the window close
Save

Model and data. Gemma 2 2B Instruct with the 16k gemma-scope SAE at layer 20. The model is gated on the Hub: accept its licence once, then hf auth login. Unlike the GPT-2 notebooks this one loads a 2B model, so expect a few GB of GPU memory. Six probe sentences about California, and four unrelated questions.

Requirements. pip install murano-interp[sae]

1. Set up

from murano import MuranoModel, Pipeline, keys
from murano.steps import (
    SAEEncode,
    SAEFeatureLabel,
    Save,
    sae_steer,
    top_sae_features_for_tokens,
)
from murano.steps.prompts import LoadPrompts

OUTPUT_DIR = "murano_outputs/sae_steering"

MODEL_ID = "google/gemma-2-2b-it"
SAE_RELEASE = "gemma-scope-2b-pt-res-canonical"
SAE_ID = "layer_20/width_16k/canonical"

model = MuranoModel(MODEL_ID)
print(f"{model.model_id}: {model.n_layers} layers")

google/gemma-2-2b-it: 26 layers

2. Shortlist the features that fire on the concept

top_sae_features_for_tokens returns the features whose activation is largest on the concept’s own tokens.

PROBE_PROMPTS = [
    "California is the most populous state in the United States.",
    "Many technology companies are headquartered in California.",
    "The coast of California stretches for hundreds of miles.",
    "California is famous for its beaches, mountains, and deserts.",
    "Hollywood, in southern California, is the center of the film industry.",
    "Wine from northern California is exported all over the world.",
]
CONCEPT_TOKENS = {"California"}

encode = SAEEncode(model, release=SAE_RELEASE, sae_id=SAE_ID)
results = Pipeline([LoadPrompts(PROBE_PROMPTS), encode]).run()

candidates = top_sae_features_for_tokens(
    results[keys.SAE_RECORD], model, CONCEPT_TOKENS, n=8
)
print("fires hardest on the 'California' tokens:", candidates)

fires hardest on the 'California' tokens: [6631, 11552, 1466, 4919, 5859, 1692, 9768, 2436]

3. Pick the feature by what it promotes

Firing on a token is not the same as meaning it. Broad features, high-frequency features and tokenization artifacts all fire on " California", and steering with one of those injects noise rather than the concept.

So rank the shortlist by what each candidate promotes through the unembedding, the same logit lens sae_features.ipynb used, and keep the first that actually says California.

results = Pipeline([SAEFeatureLabel(model, feat_ids=candidates, k_tokens=3)]).run(
    results
)
labels = results["feature_labels"]

california = [
    fid
    for fid in candidates
    if any("california" in token.lower() for token in labels.tokens[fid])
]

print("what each candidate promotes:\n")
for fid in candidates:
    promoted = [token.strip() for token in labels.tokens[fid]]
    mark = "   <-- means California" if fid in california else ""
    print(f"  #{fid:<6} {promoted}{mark}")

FEATURE_ID = california[0]
print(f"\nsteering with #{FEATURE_ID}. The candidates ranked above it fire on the")
print("token without carrying the concept: one promotes gibberish, one promotes")
print("'State' in the sense of StatefulWidget, one is a bare 'Cal' fragment.")

what each candidate promotes:

  #6631   ['tartalomajánló', 'ⓧ', 'ignty']
  #11552  ['State', 'statewide', 'StatefulWidget']
  #1466   ['California', 'California', 'CALIFORNIA']   <-- means California
  #4919   ['CAL', 'Cal', 'Cal']
  #5859   ['cities', 'natale', 'where']
  #1692   ['expandindo', 'kaarangay', 'cauſe']
  #9768   ['[…]', '', '...']
  #2436   ['nationals', 'national', 'voisin']

steering with #1466. The candidates ranked above it fire on the
token without carrying the concept: one promotes gibberish, one promotes
'State' in the sense of StatefulWidget, one is a bare 'Cal' fragment.

That is the notebook’s first real lesson, and it is cheap to check: activation ranks candidates, the logit lens picks one. Skipping the second step is the usual reason SAE steering “does nothing”.

4. Calibrate the strength against the residual stream

sae_steer normalizes the decoder direction to unit length and adds alpha * direction to the residual stream, at every decoded token, at the one site the SAE was trained on. So alpha is an absolute magnitude, and an absolute magnitude means nothing until you know what it is competing with. Measure that first.

store = model.record(
    ["What is your favorite color?", "Tell me an interesting fact about whales."],
    layers=[20],
    position="none",
)
residual_norm = float(store.positive[(20, "residual")].float().norm(dim=-1).mean())

print(f"mean residual norm at the SAE's site: {residual_norm:.0f}")
print(f"alpha=150  is {150 / residual_norm:.2f} x that")
print(f"alpha=2000 is {2000 / residual_norm:.2f} x that")

mean residual norm at the SAE's site: 795
alpha=150  is 0.19 x that
alpha=2000 is 2.52 x that

5. Steer, and watch the window close

Three strengths, each a fixed fraction of that norm. Read the generations, not a metric: the failure modes here are lexical, and no scalar would show them to you.

PROMPTS = [
    "What is your favorite color?",
    "How should I spend a weekend in Berlin?",
    "Write a short recipe for pasta carbonara.",
    "Tell me an interesting fact about whales.",
]


def report(comparison, alpha):
    print(f"alpha = {alpha}  ({alpha / residual_norm:.2f} x residual norm)\n")
    for prompt, clean, modified in zip(
        comparison.prompts,
        comparison.baseline_generations,
        comparison.modified_generations,
    ):
        print(f"  {prompt}")
        print(f"    clean   : {clean.strip()[:110]!r}")
        print(f"    steered : {modified.strip()[:110]!r}\n")

Attempt 1: a fifth of the residual norm

report(
    Pipeline(
        [
            LoadPrompts(PROMPTS),
            sae_steer(
                model,
                encode.sae_model,
                FEATURE_ID,
                alpha=150.0,
                gen_kwargs={"max_new_tokens": 60, "do_sample": False},
            ),
        ]
    ).run()[keys.INTERVENE],
    alpha=150.0,
)

alpha = 150.0  (0.19 x residual norm)

  What is your favorite color?
    clean   : "As an AI, I don't have personal preferences like a favorite color.  \n\nBut I can tell you that the color blue i"
    steered : "As an AI, I don't have personal preferences like favorite colors.  \n\nBut I can tell you that the most popular "

  How should I spend a weekend in Berlin?
    clean   : "Berlin is a vibrant city with a rich history and culture. There's something for everyone, from history buffs t"
    steered : "##  A Weekend in Berlin: A Mix of Culture, History, and Fun\n\nHere's a possible itinerary for a fun-filled week"

  Write a short recipe for pasta carbonara.
    clean   : '**Ingredients:**\n\n* 1 pound spaghetti\n* 1/2 cup pancetta, diced\n* 2 large eggs\n* 1/2 cup grated Pecorino Roman'
    steered : '**Ingredients:**\n\n* 1 pound spaghetti\n* 1/2 cup pancetta, diced\n* 4 large eggs\n* 1/2 cup grated Parmesan chees'

  Tell me an interesting fact about whales.
    clean   : "Did you know that whales are mammals? \n\nThat's right! They breathe air, give birth to live young, and nurse th"
    steered : "Here's a fun fact: \n\n**Some whales can sing!** \n\nNot just any song, but complex songs that can last for hours."

Nothing. The answers are fluent and the concept is absent. At a fifth of the residual norm the direction is a rounding error against everything else written into that site.

Attempt 2: half the residual norm

report(
    Pipeline(
        [
            LoadPrompts(PROMPTS),
            sae_steer(
                model,
                encode.sae_model,
                FEATURE_ID,
                alpha=420.0,
                gen_kwargs={"max_new_tokens": 60, "do_sample": False},
            ),
        ]
    ).run()[keys.INTERVENE],
    alpha=420.0,
)

alpha = 420.0  (0.53 x residual norm)

  What is your favorite color?
    clean   : "As an AI, I don't have personal preferences like a favorite color.  \n\nBut I can tell you that the color blue i"
    steered : 'The color of the ocean is a favorite of many.'

  How should I spend a weekend in Berlin?
    clean   : "Berlin is a vibrant city with a rich history and culture. There's something for everyone, from history buffs t"
    steered : "It's a fun city with lots to offer, but it can be overwhelming to figure out how to spend your time.  \n\nTo hel"

  Write a short recipe for pasta carbonara.
    clean   : '**Ingredients:**\n\n* 1 pound spaghetti\n* 1/2 cup pancetta, diced\n* 2 large eggs\n* 1/2 cup grated Pecorino Roman'
    steered : '**Ingredients:**\n\n* 1 pound pasta (Monter Monterey, or other firm, low-water-release pasta)\n* 1-2 garlic, slic'

  Tell me an interesting fact about whales.
    clean   : "Did you know that whales are mammals? \n\nThat's right! They breathe air, give birth to live young, and nurse th"
    steered : "Here's a fun one:\n\n**The largest known species of whale, the humb Humb Humb, is named after the color of its s"

Now the feature is doing something, and it is not what we wanted. California leaks into the carbonara recipe as Monter Monterey, a broken lexical intrusion in an ingredient list, and the whale fact dissolves into the humb Humb Humb. The model is not thinking about California; individual token choices are being overridden by a vector that grows stronger than the local context.

The text is already degrading before the concept has become content.

Attempt 3: two and a half times the residual norm

report(
    Pipeline(
        [
            LoadPrompts(PROMPTS),
            sae_steer(
                model,
                encode.sae_model,
                FEATURE_ID,
                alpha=2000.0,
                gen_kwargs={"max_new_tokens": 60, "do_sample": False},
            ),
        ]
    ).run()[keys.INTERVENE],
    alpha=2000.0,
)

alpha = 2000.0  (2.52 x residual norm)

  What is your favorite color?
    clean   : "As an AI, I don't have personal preferences like a favorite color.  \n\nBut I can tell you that the color blue i"
    steered : 'California California California California California California California California California California '

  How should I spend a weekend in Berlin?
    clean   : "Berlin is a vibrant city with a rich history and culture. There's something for everyone, from history buffs t"
    steered : 'California California California California California California California California California California '

  Write a short recipe for pasta carbonara.
    clean   : '**Ingredients:**\n\n* 1 pound spaghetti\n* 1/2 cup pancetta, diced\n* 2 large eggs\n* 1/2 cup grated Pecorino Roman'
    steered : 'California California California California California California California California California California '

  Tell me an interesting fact about whales.
    clean   : "Did you know that whales are mammals? \n\nThat's right! They breathe air, give birth to live young, and nurse th"
    steered : 'California California California California California California California California California California '

Total collapse. Every prompt returns the same word forever.

What this notebook found

There is no strength at which this feature makes Gemma-2 talk about California. Below about half the residual norm it does nothing; above it, the text breaks before the concept arrives. The window is empty, and the arithmetic says why: the direction is added at every decoded token, so to make " California" win the unembedding it must grow comparable to the residual stream, and at that point the residual stream is the direction. The model has nothing left to condition on.

Two things follow, and they are the reason this notebook does not end in a demo.

Read the generations. A positive-word rate or a logit difference would have scored attempt 2 as a success: the concept is measurably present. It is present as a corrupted ingredient list.

Adding a fixed magnitude is not the only way to steer. Scaling Monosemanticity does not add a constant vector; it clamps the feature’s own activation to a high value and lets the decoder write whatever that implies, which leaves the rest of the residual stream intact. sae_steer implements the additive form, which is standard, and which is enough to show a feature is causally live. Reproducing Golden Gate Claude needs the clamping form.

6. Save

run_dir = Save(
    output_dir="murano_outputs", run_name="sae_steering", model_id=model.model_id
)(results)[keys.OUTPUT_DIR]

print("saved to:", run_dir)

saved to: murano_outputs/sae_steering

What next

sae_features.ipynb covers the reading half, and the logit-lens trick used here.
steering.ipynb steers with a contrastive direction instead, on a model small enough that a usable strength exists.