Self-Supervised Learning

Self-Supervised Learning: Learning paradigm where models create supervision signal from unlabeled data. Predicts parts of input from other parts (masked LM, image inpainting, next frame). Enables pre-training on massive unlabeled datasets. Foundation of BERT, GPT. Examples include BERT masked language modeling, GPT next token prediction, Image rotation prediction.

In italiano: Self-Supervised LearningLegge III · Come una rete impara dai dati

Predicts parts of input from other parts (masked LM, image inpainting, next frame). Enables pre-training on massive unlabeled datasets. Foundation of BERT, GPT.

Examples

  • BERT masked language modeling
  • GPT next token prediction
  • Image rotation prediction