Skip to content
Murano

Tasks

Small canonical tasks shared by the notebooks, tests, and reproductions.

Interpretability work needs a task before it needs a method, and the same two toy tasks keep reappearing: a circuit-analysis task with a matched clean and corrupt prompt (:func:ioi), and a contrastive concept task for probing and steering (:func:sentiment). Defining them once here keeps every tutorial from carrying its own copy, and gives the fixtures a place to be tested.

Both are deliberately tiny and hand-written: large enough to demonstrate a method, far too small to measure one. Swap in a real dataset before drawing a conclusion.

Attributes:

  • IOI_TEMPLATE: The sentence frame :func:ioi fills in.
  • POSITIVE_WORDS: Vocabulary behind :func:positive_word_rate.

ioi

Section titled “ioi”
def ioi(n: int = 8) -> CleanCorruptDataset:

Build the indirect-object-identification task.

Each clean prompt names two people and then has the subject give a drink, so the next token should be the other name, the indirect object. The corrupt prompt swaps who gives, which flips the answer. That pairing is what activation patching and path patching resample across.

Equivalent to calling :meth:~murano.dataset.CleanCorruptDataset.from_pairs with the four lists this function assembles.

Args:

  • n: Number of prompt pairs, up to the 12 name pairs available.

Returns:

  • A: class:~murano.dataset.CleanCorruptDataset whose correct answers
  • are the indirect objects and whose incorrect answers are the
  • subjects, each with the leading space the tokenizer expects.

Raises:

  • ValueError: If n is not between 1 and the number of name pairs.

Example:

  • task = ioi(n=2)

  • task.clean[0]

  • ‘When Mary and John went to the store, John gave a drink to’

  • task.correct[0], task.incorrect[0]

  • (’ Mary’, ’ John’)

sentiment

Section titled “sentiment”
def sentiment(n_per_class: int = 25) -> tuple[list[str], list[str]]:

Return contrastive positive and negative sentences.

The two lists are matched only in size and register, not word for word: the concept, not the wording, is what a probe or a steering vector should pick up. Feed them to :meth:~murano.dataset.MuranoDataset.contrastive for steering, or to :meth:~murano.dataset.LabeledDataset.from_lists for probing.

Args:

  • n_per_class: Sentences to take from each class, up to 25.

Returns:

  • A (positive, negative) pair of equal-length sentence lists.

Raises:

  • ValueError: If n_per_class is not between 1 and 25.

positive_word_rate

Section titled “positive_word_rate”
def positive_word_rate(generations: Sequence[str]) -> float:

Fraction of generations containing at least one positive word.

A deliberately crude scorer, kept because it makes the shape of an evaluation obvious in a tutorial: a metric turns “the text looks different” into a number you can defend. It is a keyword count, not a sentiment classifier, and it says nothing on a handful of prompts. Replace it before reporting anything.

Args:

  • generations: Model completions to score.

Returns:

  • The fraction in [0, 1].

Raises:

  • ValueError: If generations is empty.