STaR bootstrap loop (and how shortcuts sneak in)
the STaR loop
1. sample (rationale, answer)
base model, T > 0
2. filter: keep if answer == label
discards incorrect rationales entirely
3. SFT on the kept rationales
teacher forcing on (problem, rationale, answer)
4. rationalize failures
inject answer, re-ask for the rationale
repeat
what the loop reinforces
sound rationale · right answer
shortcut rationale · right answer
both get kept. filter cannot tell them apart.
three variants, three signal shapes
STaR (Zelikman 2022)
keep-if-correct filter on generator.
repeat SFT rounds.
GPT-J + STaR ≈ fine-tuned 30x larger model
on GSM8K (no new human labels).
V-STaR (Hosseini 2024)
DPO-train a verifier on correct AND
incorrect rationales (the STaR waste).
inference: sample N, pick verifier's top.
+4 to +17 pp over prior self-improve.
Quiet-STaR (Zelikman 2024)
per-token internal rationale.
end-of-thought token + mixing head.
Mistral 7B GSM8K 5.9% -> 10.9%
no task-specific fine-tune.
shared weakness
all three use answer-conditioned gradient.
shortcut-reasoning reaches the label.
fix: process reward models + OOD eval.