Causal Masking

Causal Masking: Masking technique preventing attention to future positions in autoregressive models. Ensures token at position i can only attend to positions ≤ i. Essential for GPT-style models. Implemented as triangular mask. Examples include GPT autoregressive generation, Decoder self-attention, Preventing future leakage.

In italiano: Causal MaskingLegge III · Come una rete impara dai dati

Ensures token at position i can only attend to positions ≤ i. Essential for GPT-style models. Implemented as triangular mask.

Examples

  • GPT autoregressive generation
  • Decoder self-attention
  • Preventing future leakage