Michele LaurelliBlog

DPO (Direct Preference Optimization)

DPO (Direct Preference Optimization): A method that directly optimizes a policy on preferred and dispreferred response pairs without explicit RL. It uses a reference model and a contrastive preference loss, removing the separate reward model typical of RLHF. Representative data and control of the reference coefficient remain critical.

In italiano: DPO (Direct Preference Optimization)Legge III · Come una rete impara dai dati

It uses a reference model and a contrastive preference loss, removing the separate reward model typical of RLHF. Representative data and control of the reference coefficient remain critical.

Examples

  • Chosen/rejected pairs
  • Alignment without PPO
  • Policy constrained to a reference

Fonti