From Leaky Thoughts to Private Reasoning

Controlling What LRMs Say to Themselves

Haritz Puerto1, Haonan Li2, Xudong Han3,2, Timothy Baldwin2,3, Iryna Gurevych1,2
1UKP Lab, TU Darmstadt & ATHENE, Germany  ·  2MBZUAI, UAE  ·  3LibrAI
Paper arXiv Code Models Data

TL;DR

Reasoning traces leak private data even when the model is explicitly told not to. We treat this as a controllability problem: privacy directives are just instructions, so we teach models to follow instructions inside the reasoning trace and decode the trace and the answer with two different LoRA adapters (Staged Decoding). This yields up to +20.9 points in instruction following and +51.9 points in privacy.

A reasoning model reproduces the confidential information and the password in its reasoning trace despite an explicit privacy directive; a controllable model does not.

LRMs leak sensitive context into their reasoning trace despite instructions. Our method enforces instruction following throughout reasoning, reducing trace leakage while preserving final-answer compliance.

The reasoning trace is an attack surface

The reasoning trace (RT) is usually treated as a hidden scratchpad and therefore assumed safe. It is not: a simple prompt injection can force the model to reproduce its RT in the visible answer (Green et al., 2025). And LRMs routinely ignore explicit privacy directives while reasoning, with reported violation rates of 19–78%(Green et al., 2025).

Prior work on instruction following (IF) in reasoning models looks almost exclusively at the final answer (FA). Nobody had asked whether IF can be improved within the reasoning process, or whether such gains transfer to privacy. That is the gap we fill.

Method

1. An SFT dataset for reasoning-trace instructions. We rewrite DeepSeek-R1 reasoning traces on GSM8K so that they obey a sampled instruction of three privacy-free kinds: formatting (e.g., write the RT as a bullet plan), style (e.g., reason in the voice of a given character), and reasoning type (e.g., use deductive reasoning). We build three nested datasets (1k / 2k / 3k examples) targeting the RT, the FA, or both, and train LoRA adapters on each. Because the training instructions contain no privacy content, any privacy gain must come from better general instruction following.

2. Staged Decoding. The checkpoint that is best at IF in the RT is generally not the one best at IF in the FA. Staged Decoding sidesteps that tension: generate the RT with the IF-RT adapter, swap adapters at the end-of-thinking token, and generate the answer with the IF-FA adapter. The swap is cheap (+0.10% of end-to-end decoding time) and requires no retraining.

Staged Decoding generates the reasoning trace with one LoRA adapter and the final answer with another, swapping at the end-of-thinking token.

Staged Decoding generates the reasoning trace and the final answer with different LoRA adapters.

Results

We evaluate six models from two families (Qwen 3 and Phi 4, 1.7B–14B) on two IF benchmarks (IFEval, MathIF) and two privacy benchmarks (PasswordEval, PEEP), with two seeds each.

Staged Decoding gets the best of both worlds. It keeps the IF-RT performance of the RT-specialized checkpoint while recovering its lost IF-FA, reaching the best average IF in 9 of 12 cases, with gains over the untrained baseline of up to 20.9 points (+6.66 on IFEval and +10.74 on MathIF on average).

Better instruction following means better privacy. Staged Decoding gives the best privacy in 10 of 12 setups, averaging +21.7 points on PasswordEval and +22.7 on PEEP, with a maximum gain of 51.9 points (Qwen 3 14B on PasswordEval). A per-instance paired analysis restricted to well-formed outputs (paired bootstrap 95% CI, B = 20,000, and a two-sided paired permutation test with Holm correction) confirms a significant privacy improvement in 11 of 12 settings; the sole exception is Qwen 3 1.7B on PasswordEval, the smallest model.

Total privacy score (higher is better), baseline vs. Staged Decoding.
Model PasswordEval PEEP
Base Staged Δ Base Staged Δ
Qwen 3 1.7B 42.122.6−19.5 35.143.4+8.3
Qwen 3 4B 41.254.1+13.0 46.167.0+20.9
Qwen 3 8B 41.185.1+44.0 52.565.7+13.2
Qwen 3 14B 41.293.1+51.9 56.487.9+31.5
Phi 4 3.8B 27.151.7+24.5 40.074.0+34.0
Phi 4 14B 74.490.4+16.0 48.576.8+28.3

The trade-off is real. Consistent with prior work, stronger instruction following can cost reasoning utility: on MathIF the baseline beats Staged Decoding, and IF-RT correlates with utility at −0.65. On the privacy benchmarks the picture is milder, with weak and non-significant correlations (−0.33 on PasswordEval, −0.24 on PEEP). Against RANA, the post-hoc anonymization upper bound, Staged Decoding loses less utility in five of six models on PasswordEval and matches or recovers the baseline utility in four. We attribute the remaining drop largely to the small, GSM8K-only training set.

Takeaways

  • Privacy leakage in reasoning traces is, at root, an instruction-following failure. Fixing IF inside the trace fixes the cause, not the symptom.
  • Training on privacy-free instructions transfers to privacy benchmarks, so the gains are not overfitting to one leakage formulation.
  • Staged Decoding resolves the tension between IF in the trace and IF in the answer at negligible cost.
  • Our approach is complementary to post-hoc anonymization, hiding traces, and prompt-injection defenses, and targets model providers rather than API consumers.

BibTeX

@misc{puerto2026leakythoughts,
      title={From Leaky Thoughts to Private Reasoning: Controlling What LRMs Say to Themselves},
      author={Haritz Puerto and Haonan Li and Xudong Han and Timothy Baldwin and Iryna Gurevych},
      year={2026},
      eprint={2602.24210},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.24210},
}