Multimodal AI

/ˌmʌltɪˈmoʊdəl/

Multimodal AI: AI systems that can process and relate information from multiple modalities like text, images, audio, and video. Multimodal models learn joint representations across modalities. Examples include CLIP (vision-language), Flamingo (visual question answering), and GPT-4V (vision understanding). Examples include CLIP matching images to text, GPT-4V describing images, Image captioning systems.

In italiano: AI MultimodaleLegge III · Come una rete impara dai dati

Multimodal models learn joint representations across modalities. Examples include CLIP (vision-language), Flamingo (visual question answering), and GPT-4V (vision understanding).

Examples

  • CLIP matching images to text
  • GPT-4V describing images
  • Image captioning systems