A custom pipeline: readable is not the same as steerable

Note

Generated from notebooks/applications/custom_pipeline.ipynb, which you can run yourself. The outputs below are the ones it produced.

Everything so far used the steps as they ship. This notebook answers a question no single step answers, by mixing several and writing the one that is missing.

The question comes from two earlier notebooks that seem to disagree. probing.ipynb finds sentiment is linearly readable at almost every layer, including layer 0. steering.ipynb finds that steering at layer 0 does nothing at all. So: at which layer is sentiment both readable and causally effective? No single step answers that. Three of them, composed, do.

Key questions

• How readable is sentiment at each layer, and how effective is intervening there?
• Do the two profiles peak in the same place?
• How do I write a step when the built-in ones do not fit?

Structure

1. Set up
2. Watch the pipeline run
3. Readability: probe every layer
4. Effectiveness: intervene during a forward pass
5. Put the two profiles side by side
6. Package the experiment as a step
7. Save and reload

Model and data. GPT-2 small in float32. It is small enough that bfloat16 rounding visibly degrades its predictions, so the notebooks load it explicitly in float32. 20 positive and 20 negative sentences from murano.tasks, and five held-out prompts.

Requirements. pip install murano-interp[probe,plot]

1. Set up

import torch

from murano import MuranoModel, Pipeline, keys
from murano.dataset import LabeledDataset, MuranoDataset
from murano.plotting import plot_line_chart, save_figure
from murano.results import Results
from murano.steps import (
    Load,
    LoadPrompts,
    LogitDiffStep,
    Logits,
    Probe,
    Record,
    Save,
    SteeringVector,
    Sweep,
)
from murano.steps.base import Step
from murano.steps.intervene import steer_direction
from murano.tasks import sentiment

OUTPUT_DIR = "murano_outputs/custom_pipeline"

# GPT-2 is small enough that bfloat16 rounding degrades its predictions.
model = MuranoModel("gpt2", dtype=torch.float32)
print(f"{model.model_id}: {model.n_layers} layers, {model.n_heads} heads")

gpt2: 12 layers, 12 heads

2. Watch the pipeline run

setup_logging turns on Murano’s logger, so every step announces itself as it runs. Useful when a pipeline is long enough that you want to see where the time goes.

import logging

from murano import setup_logging

setup_logging(logging.INFO)

3. Readability: probe every layer

The first half is a standard probing pipeline. What we keep is the accuracy at every layer, not just the best one: that curve is the readability profile.

positive, negative = sentiment(n_per_class=20)

probe_results = Pipeline(
    [
        Load(
            LabeledDataset.from_lists(
                texts=positive + negative,
                labels=[0] * len(positive) + [1] * len(negative),
                label_names=["positive", "negative"],
            )
        ),
        Record(model, layers="all", position="last", batch_size=8),
        Probe(cv=5, refit=False),
    ]
).run()

readability = {
    node.layer: accuracy
    for node, accuracy in probe_results[keys.PROBE].accuracy_per_layer.items()
}
print(
    "\nprobe accuracy per layer:",
    {k: round(v, 3) for k, v in sorted(readability.items())},
)

probe accuracy per layer: {0: 0.825, 1: 0.85, 2: 0.775, 3: 0.85, 4: 0.85, 5: 0.925, 6: 0.825, 7: 0.9, 8: 0.925, 9: 0.925, 10: 0.9, 11: 0.9}

We also need the steering direction itself, from the same sentences.

steering_results = Pipeline(
    [
        Load(MuranoDataset.contrastive(positive=positive, negative=negative)),
        Record(model, layers="all", position="last", batch_size=8),
        SteeringVector(normalize=True),
    ]
).run()

steering = steering_results[keys.STEERING]
print("\nbest separating layer:", steering.best_layer)

best separating layer: L8.resid_post

4. Effectiveness: intervene during a forward pass

We need, one layer at a time: apply the steering direction during a forward pass and read the resulting logits. Intervene applies exactly that edit, but it generates text, which is slow and scores coarsely. Logits runs a forward pass, and it takes the same fn that Intervene does. So a twelve-layer study costs twelve forward passes rather than twelve decodes.

Sweep then runs that pair of steps once per layer.

The score is a logit difference between two ordinary words, " good" and " bad", at the last prompt position. That is continuous, unlike counting positive words in a generation, which is what makes a twelve-point curve legible.

EVAL_PROMPTS = [
    "I think that the food was",
    "The movie was",
    "My friend is a",
    "Today I feel",
    "The service was",
]

setup_logging(logging.WARNING)  # twelve forward passes would flood the output

clean = (
    Pipeline(
        [
            LoadPrompts(EVAL_PROMPTS),
            Logits(model),
            LogitDiffStep(" good", " bad", output_key="valence", model=model),
        ]
    )
    .run()["valence"]
    .value
)
print(f"clean logit(' good') - logit(' bad') = {clean:+.4f}")

clean logit(' good') - logit(' bad') = +2.4945

direction = steering.direction_per_layer

swept = Pipeline(
    [
        LoadPrompts(EVAL_PROMPTS),
        Sweep(
            over=list(range(model.n_layers)),
            steps=lambda layer: [
                Logits(
                    model,
                    fn=steer_direction(direction, alpha=8.0),
                    layers=[layer],
                    modules="residual",
                    logits_key="steered_logits",
                    targets=None,
                ),
                LogitDiffStep(
                    " good",
                    " bad",
                    logits_key="steered_logits",
                    output_key="valence",
                    model=model,
                ),
            ],
            read="valence",
        ),
    ]
).run()

effectiveness = {
    layer: value - clean for layer, value in swept[keys.SWEEP].scores.items()
}

print(f"{'layer':>5} {'probe accuracy':>16} {'steering effect':>17}")
print("-" * 41)
for layer in range(model.n_layers):
    print(f"{layer:>5} {readability[layer]:>16.3f} {effectiveness[layer]:>+17.4f}")

layer   probe accuracy   steering effect
-----------------------------------------
    0            0.825           +0.5777
    1            0.850           +0.5443
    2            0.775           +0.9168
    3            0.850           +0.5388
    4            0.850           +0.6724
    5            0.925           +0.9020
    6            0.825           +1.1579
    7            0.900           +0.9707
    8            0.925           +1.0385
    9            0.925           +0.7725
   10            0.900           +0.5771
   11            0.900           +0.4133

5. Put the two profiles side by side

plot_line_chart takes one x-axis and a named series per line.

layers = list(range(model.n_layers))
fig = plot_line_chart(
    x_data=layers,
    y_series={
        "probe accuracy (readable)": [readability[i] for i in layers],
        "steering effect (effective)": [effectiveness[i] for i in layers],
    },
    title="Where sentiment is readable, and where it is effective",
    x_label="layer",
    y_label="score",
)
save_figure(fig, f"{OUTPUT_DIR}/plots/readable_vs_effective.png")
fig

most_readable = max(readability, key=readability.get)
most_effective = max(effectiveness, key=effectiveness.get)

print(
    f"most readable layer : {most_readable}  (accuracy {readability[most_readable]:.3f})"
)
print(
    f"most effective layer: {most_effective}  (effect {effectiveness[most_effective]:+.4f})"
)
print(
    f"layer 0 is readable at {readability[0]:.3f} but only {effectiveness[0]:+.4f} effective"
)

most readable layer : 5  (accuracy 0.925)
most effective layer: 6  (effect +1.1579)
layer 0 is readable at 0.825 but only +0.5777 effective

The two curves have different shapes, and that is the answer.

Readability is nearly flat. A probe reads sentiment off every layer at roughly the same accuracy, including layer 0, because words like “love” carry the label in their token embeddings before any attention or MLP has run.

Effectiveness is not flat. It varies by about a factor of three across the network. It climbs through the early layers, peaks in the middle, and then falls away, reaching its lowest value at the final layer. That last point is the cleanest case: the final layer is among the most readable and the least effective, because nothing downstream is left to act on the change we make there.

So a layer being readable tells you almost nothing about whether intervening there will work. A probe answers where the information is. Only an intervention answers where the model uses it.

One loose end worth naming. steering.ipynb concluded that steering at layer 0 does “nothing at all”, and here layer 0 scores about half the peak. Both are honest readings of their own metric: a positive-word count over four generations cannot see a modest shift in the logits, while this continuous score can. The metric decided the conclusion, which is the lesson of metrics.ipynb.

6. Package the experiment as a step

Once the analysis works, wrap it so the next person runs it in one line.

from murano.artifacts import MetricScore


class ReadableVsEffective(Step):
    """Summarize where a concept is readable against where it is effective."""

    reads = [keys.PROBE]
    writes = ["profile"]

    def __init__(self, effectiveness: dict[int, float]):
        self.effectiveness = effectiveness

    def __call__(self, results: Results) -> Results:
        accuracy = {
            node.layer: value
            for node, value in results[keys.PROBE].accuracy_per_layer.items()
        }
        readable = max(accuracy, key=accuracy.get)
        effective = max(self.effectiveness, key=self.effectiveness.get)
        results["profile"] = {
            "most_readable_layer": readable,
            "most_effective_layer": effective,
            "they_agree": readable == effective,
        }
        results["gap"] = MetricScore(
            metric_name="readable_minus_effective_layer",
            value=float(readable - effective),
        )
        return results


probe_results = ReadableVsEffective(effectiveness)(probe_results)
print(probe_results["profile"])

{'most_readable_layer': 5, 'most_effective_layer': 6, 'they_agree': False}

7. Save and reload

Save writes every artifact it recognizes, and the load_* functions read them back, so an analysis can be split across sessions or shared as files.

from murano import load_metric_score

run_dir = Save(
    output_dir="murano_outputs", run_name="custom_pipeline", model_id=model.model_id
)(probe_results)[keys.OUTPUT_DIR]
print("saved to:", run_dir)

reloaded = load_metric_score(f"{run_dir}/metrics/gap.json")
print("reloaded:", reloaded.metric_name, "=", reloaded.value)

saved to: murano_outputs/custom_pipeline
reloaded: readable_minus_effective_layer = -1.0

What next

• probing.ipynb and steering.ipynb are the two halves this notebook joins.
• circuit_discovery.ipynb builds a different custom experiment, around attribution and patching.