The mean-field dynamics of transformers | Vidéo | Carmin.tv

00:00:00 / 00:00:00

The mean-field dynamics of transformers

By Philippe Rigollet

Appears in collection : New challenges in high-dimensional statistics / Statistique mathématique 2025

We develop a mathematical framework that interprets Transformer attention as an interacting particle system and studies its continuum (mean-field) limits. By idealizing attention on the sphere, we connect Transformer dynamics to Wasserstein gradient flows, synchronization models (Kuramoto), and mean-shift clustering. Central to our results is a global clustering phenomenon whereby tokens cluster asymptotically after long metastable states where they are arranged into multiple clusters. We further analyze a tractable equiangular reduction to obtain exact clustering rates, show how commonly used normalization schemes alter contraction speeds, and identify a phase transition for long-context attention. The results highlight both the mechanisms that drive representation collapse and the regimes that preserve expressive, multi-cluster structure in deep attention architectures.

Information about the video

Date of recording 16/12/2025
Date of publication 12/01/2026
Institution CIRM
Licence CC BY NC ND
Language English
Audience Researchers, Graduate Students
Director(s) Guillaume Hennenfent
Format MP4

Citation data

DOI 10.24350/CIRM.V.20425603
Cite this video Rigollet, Philippe (16/12/2025). The mean-field dynamics of transformers. CIRM. Audiovisual resource. DOI: 10.24350/CIRM.V.20425603
URL https://dx.doi.org/10.24350/CIRM.V.20425603

Domain(s)

Bibliography

RIGOLLET, Philippe. The Mean-Field Dynamics of Transformers. arXiv preprint arXiv:2512.01868, 2025. - https://doi.org/10.48550/arXiv.2512.01868

MSC codes

Last related questions on MathOverflow

You have to connect your Carmin.tv account with mathoverflow to add question

Ask a question on MathOverflow

Copyright Carmin.tv 2026

Give feedback