Causal Masking
Causal Masking: Masking technique preventing attention to future positions in autoregressive models. Ensures token at position i can only attend to positions ≤ i. Essential for GPT-style models. Implemented as triangular mask. Examples include GPT autoregressive generation, Decoder self-attention, Preventing future leakage.
In italiano: Causal MaskingLegge III · Come una rete impara dai datiEnsures token at position i can only attend to positions ≤ i. Essential for GPT-style models. Implemented as triangular mask.
Examples
- GPT autoregressive generation
- Decoder self-attention
- Preventing future leakage