Policy Alignment Telemetry (30 Preference Pairs)
60-tok · 4-ctx · tanh-24
Reward Accuracy (r_w > r_l)
10/30
33.3%
Mean Margin Δr:
-0.012
KL Divergence D_KL(πθ ‖ πref)
TETHER: β=0.10
0.0002
nats
Mean Loss L_DPO:
0.6991
Known DPO Effect: Both Log-Probs Fall While Margin Grows
Sequence Log-Probabilities
Mean log πθ(y_w|x)
-5.642
ref: -5.638
Mean log πθ(y_l|x)
-5.581
ref: -5.562
Log-Ratio Margin m
-0.077
Δlog π_w − Δlog π_l
Exact Single-Pair Loss Inspector
Pair #01 (Click any row)
Chosen Δlog π_w = log πθ − log πref
-5.6120 − (-5.6100) = -0.0020
Rejected Δlog π_l = log πθ − log πref
-5.4810 − (-5.4900) = +0.0090
Margin m = Δlog π_w − Δlog π_l
m = -0.0110 | β·m = -0.0011
L_DPO = −log σ(β · m)
EXACT 4-DEC
0.6937
Grad Wt σ(−β · m)
ACTIVE
0.5003
Live Decoding Comparison: Frozen Reference vs. Policy
◀ Prev
Next ▶
<user>
help me now
<asst>
Prompt #01 / 30
Frozen Reference (π_ref)
SFT Copy
no go do it <end>
log πref(y_w): -5.61
ranks: rude > polite
Live Policy (π_θ)
Unaligned
no go do it <end>
r_w: +0.00 | r_l: +0.00
Δr: +0.000
● Demo Scenario (20s Loop)
◼ Still (Peak Separation)
⏸ Pause Training
Step +1
↺ Reset πθ ← πref
KL Beta (β):
0.10
Learning Rate (η):
0.055
Step: 0000