Masked Language Modeling

Masked Language Modeling: Masked language modeling trains a model to reconstruct selected tokens from their surrounding context. It is the main pretraining objective used by BERT-style encoder models. Each masked prediction can use tokens on both sides, so the representation is bidirectional rather than restricted to previous tokens.

In italiano: Masked Language ModelingLegge III · Come una rete impara dai dati

Each masked prediction can use tokens on both sides, so the representation is bidirectional rather than restricted to previous tokens. BERT selects 15% of tokens and, among those, replaces 80% with [MASK], 10% with a random token and leaves 10% unchanged. Causal language modeling instead predicts the next token using only the left context.

Formula

L = −Σᵢ∈M log p(xᵢ | x∖M), where M is the set of masked positions

Numerical example

import torch
import torch.nn.functional as F

# Vocabulary scores for one masked position; true token ID = 2
logits = torch.tensor([[0.1, 0.2, 2.0, 0.3]])
target = torch.tensor([2])
loss = F.cross_entropy(logits, target)
print(round(loss.item(), 4))  # 0.4039

When to use it

Use masked language modeling to pretrain bidirectional encoders for classification, token labeling, retrieval and other understanding tasks that benefit from full-sequence context.

Limitations

The [MASK] token creates a mismatch when it is absent during fine-tuning or inference, and the objective does not directly teach left-to-right generation. Training also computes losses only on selected positions.

Examples

  • BERT pre-training
  • Cloze task
  • Bidirectional language understanding