Collection 2022 - T3 - WS1 - Non-Linear and High Dimensional Inference

Organisateur(s) Aamari, Eddie ; Aaron, Catherine ; Chazal, Frédéric ; Fischer, Aurélie ; Hoffmann, Marc ; Le Brigant, Alice ; Levrard, Clément ; Michel, Bertrand

Date(s) 03/10/2022 - 07/10/2022

URL associée https://indico.math.cnrs.fr/event/7545/

9 21

Convergence of Sharpness-Aware Minimization

De Peter Bartlett

We consider Sharpness-Aware Minimization (SAM), a gradient-based optimization method for deep networks that has exhibited performance improvements on image and language prediction problems. We show that when SAM is applied with a convex quadratic objective, for most random initializations it converges to a cycle that oscillates between either side of the minimum in the direction with the largest curvature, and we provide bounds on the rate of convergence. In the non-quadratic case, we show that such oscillations effectively perform gradient descent, with a smaller step-size, on the spectral norm of the Hessian. In such cases, SAM's update may be regarded as a third derivative---the derivative of the Hessian in the leading eigenvector direction---that encourages drift toward wider minima.

Informations sur la vidéo

Date de captation 06/10/2022
Date de publication 03/05/2024
Institut IHP
Langue Anglais
Audience Chercheurs, Doctorants
Réalisateur(s) Alexandre Duplessis, Marco Perez
Format MP4
Lieu Amphitheater Hermite, IHP

Données de citation

DOI 10.57987/IHP.2022.T3.WS1.009
Citer cette vidéo Bartlett, Peter (06/10/2022). Convergence of Sharpness-Aware Minimization. IHP. Audiovisual resource. DOI: 10.57987/IHP.2022.T3.WS1.009
URL https://dx.doi.org/10.57987/IHP.2022.T3.WS1.009