DPO, animated: teach a tiny model to prefer

Explain direct preference optimisation by running it on a tiny model in the browser: a frozen reference, a policy, preference pairs, and the implicit reward moving in front of the viewer.

Build

• Reuse the same setup as a small SFT demo: a vocabulary of about 60 word tokens with user, assistant and end, a from-scratch next-token model (embedding, 4-token context, one tanh hidden layer, softmax) with hand-written gradients, pretrained on about 200 generic sentences and fine-tuned on about 30 instruction pairs on load. Freeze a copy as the reference model; the policy starts as another copy.
• Preference dataset: about 30 prompts, each with a chosen response and a rejected response from the vocabulary. Make the preference obvious and consistent, for example chosen answers are polite and short, rejected ones are rude or rambling.
• Implement the DPO loss exactly: for each pair compute the sequence log-probabilities under policy and reference, the implicit rewards beta times (log pi minus log pi ref) for chosen and rejected, and the loss as negative log sigmoid of the reward margin. A beta slider from 0.01 to 1 and a learning rate control.
• Hero: every pair as a row with two markers on a horizontal axis, chosen in blue and rejected in red, showing their implicit rewards; training makes the blue markers drift right and the red left, the margin between them shaded. A histogram of margins across pairs sits at the right edge and slides positive.
• Gauges: KL divergence of the policy from the reference measured on the prompts, and the fraction of pairs where the policy already ranks chosen above rejected (reward accuracy). Big number: reward accuracy.
• A chat box that shows the reference model's answer and the policy's answer to the same prompt side by side.
• Demo mode: pairs appear with markers overlapping; training runs and the markers separate while KL rises slowly and accuracy climbs to near 100 percent; beta is then raised to 1 and the markers pull back together, captioned as the KL tether; beta returns to 0.1 and the chat box shows the policy giving the polite answer where the reference gave the rude one. Still: the fully separated marker rows with reward accuracy at its peak and the KL value printed beside it.

What separates great from passable

• The loss for one pair is displayed with its inputs, and the printed value equals negative log sigmoid of beta times the margin to four decimals; a per-pair gradient weight sigmoid of negative margin is shown fading for pairs already ordered correctly.
• Raising beta with everything else fixed reduces the final KL and the final margins; lowering it does the opposite. If beta has no visible effect the reference term is missing.
• The known DPO effect where both chosen and rejected log-probabilities fall while the margin grows is visible in a small readout of mean chosen and rejected log-probabilities, not hidden.