Attention Mechanism
/əˈtɛnʃən ˈmɛkənɪzəm/
A technique allowing models to focus on specific parts of the input when producing output.
Attention assigns importance weights to different input elements. Used in machine translation, image captioning, and transformers. Enables models to handle long sequences effectively.
Examples
- Focusing on relevant words in translation
- Highlighting important image regions
- Multi-head attention in transformers