LoRA (Low-Rank Adaptation)

LoRA (Low-Rank Adaptation): Parameter-efficient fine-tuning adding trainable low-rank matrices to frozen weights. Adds small trainable matrices A and B where ΔW = BA. Reduces trainable parameters to 0.1-1% while maintaining performance. Enables efficient adapter-based fine-tuning. Examples include Fine-tuning LLMs with LoRA, Adapter modules, Multi-task learning.

In italiano: LoRA (Low-Rank Adaptation)Legge III · Come una rete impara dai dati

Adds small trainable matrices A and B where ΔW = BA. Reduces trainable parameters to 0.1-1% while maintaining performance. Enables efficient adapter-based fine-tuning.

Examples

  • Fine-tuning LLMs with LoRA
  • Adapter modules
  • Multi-task learning

Fonti