Value-like role
Tied to relatively context-agnostic information
A feature associated with Paris may remain relevant across different prompts because its role is tied to the same underlying information.
UKP Lab · Technical University of Darmstadt
Sparse Autoencoders Encode Both Concepts and Functions:The Downstream Geometry of Feature Effects
How comparable are a feature’s downstream effects when that feature is intervened on across contexts?
* Equal contribution
UKP Lab, Technical University of Darmstadt
Department of Electrical Engineering, Indian Institute of Technology Delhi
Loading RAVEL feature cards…
TL;DR
Across SAE variants, consistent one-dimensional effects are rare: few features behave like reusable directions. Value-like features more often exhibit structured, low-dimensional effects, while pointer-like features predominantly exhibit diffuse effects.
Two feature roles
A value says what. A pointer says where or how.
What remains invariant can be the information itself or the function that the feature supports.
Pointer-like role
Supports a function over context-supplied values
In a copying or rule-following setting, a feature may support the same operation while acting on values supplied by the context.
The method
Analyze an effect cloud, not a single steering object.
FEGA removes the same active SAE feature across contexts and analyzes the resulting cloud of logit changes.
Feature Effect Geometry Analysis (FEGA)
Effect of ablating feature in context
Compare directional change across contexts
Effect of ablating feature in context
Choose valid contexts
For each context, trace the original activation at the task-specific analysis position to layer ℓ.
Locate one shared feature
Encode each layer-ℓ activation with the SAE and identify the same active feature j.
Run paired interventions
Use the SAE reconstruction as the baseline, then zero only feature j and propagate both paths through the frozen model tail.
Form the effect cloud
Subtract the baseline logit readout from the feature-ablated readout; the retained effects across valid contexts form the cloud.
Choose paired contexts
Collect contexts where the candidate SAE feature is active and preserve their separate model trajectories.
Locate the shared feature
At layer ℓ, encode both residuals and identify the same active feature j in the SAE representation.
Run paired interventions
Propagate the reconstructed baseline and feature-ablated reconstruction through the frozen model tail.
Compare effect geometry
Subtract the paired logit readouts and compare the resulting directional changes across contexts.
RAVEL-style city–country contextsLiteral Sequence Copying contexts
hℓ originalh̄ℓ reconstructedĥℓ,j↓ ablated
Sample contexts establish where a feature is active.The blue residual hℓ reaches the SAE site, where one latent is isolated.Green h̄ℓ retains the reconstruction; red ĥℓ,j↓ removes only feature j.Each context contributes one downstream effect vector.
Choose paired contexts
For each task, collect valid contexts where the candidate latent is active. The illustration alternates between a value-like city-country task and a pointer-like copying task.
Select one shared latent
Encode each activation with the SAE, then identify feature j in the shared latent representation.
Remove only that latent
Compare the SAE reconstruction with the same reconstruction after setting feature j to zero. The frozen model tail receives both paths.
Map the effect cloud
Subtract downstream logit readouts across every valid context. FEGA then studies the cloud’s directions, subspaces, modes, and residual structure.
Value-like features · RAVEL results
Value-like effects more often show low-dimensional organization.
Value-like candidates exhibit low-dimensional structure more often than pointer-like candidates. Their structured effects usually span multiple directions, while directed rays remain rare.
Interactive value-like geometry atlas
Inspect the reported feature map.
Select an SAE architecture or geometry label, then hover or select a point to inspect its recorded label and metrics.
Loading atlas…
each dot = one featurered ring = rendered example
UMAP positions and geometry labels are read from FEGA’s per-feature reporting artifacts.
Takeaways
What FEGA reveals.
Our results show that a feature can be interpretable and causally relevant without providing a stable direction for steering.
Value-like candidates exhibit low-dimensional structure more often than pointer-like candidates.
Their structured effects usually span multiple directions, while directed rays remain rare.
Mapped pointer-like effects are overwhelmingly diffuse.
Among pointer-like candidates with a mapped geometry, stable low-dimensional structures are rare, consistent with their effects changing as the prompt-local target changes.
A shared functional role need not produce a shared direction in logit space.
A feature may support a similar operation across prompts, but copying “apple” and copying “car”, for instance, require effects on different output logits. As the relevant value changes, the downstream effect can change with it.
Selected pointer-like feature sets contribute causally to task performance.
Jointly ablating them produces consistently larger drops in accuracy compared to same-sized matched random controls.
Citation
BibTeX
@misc{hoang2026sparseautoencodersencodeconcepts,
title = {Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effects},
author = {Hoang, Phu Gia and Chatterjee, Anwoy and Chakraborty, Tanmoy and Gurevych, Iryna and Dutta, Subhabrata},
year = {2026}, eprint = {2607.24645}, archivePrefix = {arXiv},
primaryClass = {cs.LG}, url = {https://arxiv.org/abs/2607.24645}
}