The mean-field dynamics of transformers | Vidéo | Carmin.tv

00:00:00 / 00:00:00

The mean-field dynamics of transformers

De Philippe Rigollet

Apparaît dans la collection : New challenges in high-dimensional statistics / Statistique mathématique 2025

We develop a mathematical framework that interprets Transformer attention as an interacting particle system and studies its continuum (mean-field) limits. By idealizing attention on the sphere, we connect Transformer dynamics to Wasserstein gradient flows, synchronization models (Kuramoto), and mean-shift clustering. Central to our results is a global clustering phenomenon whereby tokens cluster asymptotically after long metastable states where they are arranged into multiple clusters. We further analyze a tractable equiangular reduction to obtain exact clustering rates, show how commonly used normalization schemes alter contraction speeds, and identify a phase transition for long-context attention. The results highlight both the mechanisms that drive representation collapse and the regimes that preserve expressive, multi-cluster structure in deep attention architectures.

Informations sur la vidéo

Date de captation 16/12/2025
Date de publication 12/01/2026
Institut CIRM
Licence CC BY NC ND
Langue Anglais
Audience Chercheurs, Doctorants
Réalisateur(s) Guillaume Hennenfent
Format MP4

Données de citation

DOI 10.24350/CIRM.V.20425603
Citer cette vidéo Rigollet, Philippe (16/12/2025). The mean-field dynamics of transformers. CIRM. Audiovisual resource. DOI: 10.24350/CIRM.V.20425603
URL https://dx.doi.org/10.24350/CIRM.V.20425603

Domaine(s)

Bibliographie

RIGOLLET, Philippe. The Mean-Field Dynamics of Transformers. arXiv preprint arXiv:2512.01868, 2025. - https://doi.org/10.48550/arXiv.2512.01868

Codes MSC

Dernières questions liées sur MathOverflow

Pour poser une question, votre compte Carmin.tv doit être connecté à mathoverflow

Poser une question sur MathOverflow

Copyright Carmin.tv 2026

Donner son avis