Masked Language Modeling
Masked Language Modeling: Masked language modeling trains a model to reconstruct selected tokens from their surrounding context. It is the main pretraining objective used by BERT-style encoder models. Each masked prediction can use tokens on both sides, so the representation is bidirectional rather than restricted to previous tokens.
In italiano: Masked Language ModelingLegge III · Come una rete impara dai datiEach masked prediction can use tokens on both sides, so the representation is bidirectional rather than restricted to previous tokens. BERT selects 15% of tokens and, among those, replaces 80% with [MASK], 10% with a random token and leaves 10% unchanged. Causal language modeling instead predicts the next token using only the left context.
Formula
L = −Σᵢ∈M log p(xᵢ | x∖M), where M is the set of masked positions
Numerical example
import torch
import torch.nn.functional as F
# Vocabulary scores for one masked position; true token ID = 2
logits = torch.tensor([[0.1, 0.2, 2.0, 0.3]])
target = torch.tensor([2])
loss = F.cross_entropy(logits, target)
print(round(loss.item(), 4)) # 0.4039When to use it
Use masked language modeling to pretrain bidirectional encoders for classification, token labeling, retrieval and other understanding tasks that benefit from full-sequence context.
Limitations
The [MASK] token creates a mismatch when it is absent during fine-tuning or inference, and the objective does not directly teach left-to-right generation. Training also computes losses only on selected positions.
Examples
- BERT pre-training
- Cloze task
- Bidirectional language understanding