Michele LaurelliBlog

Reward model

Reward model: A model trained to score responses according to human preferences or learned criteria. It provides a scalar signal for RLHF or candidate selection. It can inherit judge bias and be exploited by the policy, so out-of-distribution evaluation is necessary.

In italiano: Reward ModelLegge III · Come una rete impara dai dati

It provides a scalar signal for RLHF or candidate selection. It can inherit judge bias and be exploited by the policy, so out-of-distribution evaluation is necessary.

Examples

  • Helpfulness score
  • Best-of-N ranking
  • Reward for PPO optimization