Steps — Evaluate

Generic evaluation steps for generation comparisons.

GenerationMetric

Score baseline vs modified generations with a user-provided metric.

Reads from results:

• results['intervene']: GenerationComparison

Writes to results:

• results['metric']: MetricComparison

Args:

• metric_name: Identifier written to MetricComparison.metric_name.
• score_fn: Aggregate scoring function applied to a list of generations.
• item_score_fn: Optional per-item scorer; results populate
• MetricComparison.baseline_scores / modified_scores.
• input_key: Results key to read the GenerationComparison from.
• output_key: Results key under which to write the MetricComparison.

__init__

def __init__(self, metric_name: str, score_fn: Callable[[list[str]], float], item_score_fn: Callable[[str], float] | None = None, input_key: str = keys.INTERVENE, output_key: str = keys.METRIC):

expected_read_types

def expected_read_types(self, results = None, available_types = None):

Return {input_key: GenerationComparison}.

expected_write_types

def expected_write_types(self, results = None, available_types = None):

Return {output_key: MetricComparison}.