New challenges in high-dimensional statistics / Statistique mathématique

Collection New challenges in high-dimensional statistics / Statistique mathématique

Organisateur(s) Klopp, Olga ; Pouet, Christophe ; Rakhlin, Alexander
Date(s) 16/12/2024 - 20/12/2024
URL associée https://conferences.cirm-math.fr/3055.html
00:00:00 / 00:00:00
5 5

Attention layers provably solve single-location regression

De Claire Boyer

Attention-based models, such as Transformer, excel across various tasks but lack a comprehensive theoretical understanding, especially regarding token-wise sparsity and internal linear representations. To address this gap, we introduce the single-location regression task, where only one token in a sequence determines the output, and its position is a latent random variable, retrievable via a linear projection of the input. To solve this task, we propose a dedicated predictor, which turns out to be a simplified version of a non-linear self-attention layer. We study its theoretical properties, by showing its asymptotic Bayes optimality and analyzing its training dynamics. In particular, despite the non-convex nature of the problem, the predictor effectively learns the underlying structure. This work highlights the capacity of attention mechanisms to handle sparse token information and internal linear structures.

Informations sur la vidéo

Données de citation

  • DOI 10.24350/CIRM.V.20279403
  • Citer cette vidéo Boyer, Claire (19/12/2024). Attention layers provably solve single-location regression. CIRM. Audiovisual resource. DOI: 10.24350/CIRM.V.20279403
  • URL https://dx.doi.org/10.24350/CIRM.V.20279403

Domaine(s)

Bibliographie

Dernières questions liées sur MathOverflow

Pour poser une question, votre compte Carmin.tv doit être connecté à mathoverflow

Poser une question sur MathOverflow




Inscrivez-vous

  • Mettez des vidéos en favori
  • Ajoutez des vidéos à regarder plus tard &
    conservez votre historique de consultation
  • Commentez avec la communauté
    scientifique
  • Recevez des notifications de mise à jour
    de vos sujets favoris
Donner son avis