2022 - T3 - WS1 - Non-Linear and High Dimensional Inference

Collection 2022 - T3 - WS1 - Non-Linear and High Dimensional Inference

Organizer(s) Aamari, Eddie ; Aaron, Catherine ; Chazal, Frédéric ; Fischer, Aurélie ; Hoffmann, Marc ; Le Brigant, Alice ; Levrard, Clément ; Michel, Bertrand

Date(s) 03/10/2022 - 07/10/2022

linked URL https://indico.math.cnrs.fr/event/7545/

17 21

Neural networks, wide and deep, singular kernels and Bayes optimality

By Mikhail Belkin

Wide and deep neural networks are used in many important practical settings. In this talk, I will discuss some aspects of width and depth related to optimization and generalization. I will first discuss what happens when neural networks become infinitely wide, giving a general result for the transition to linearity (i.e., showing that neural networks become linear functions of parameters) for a broad class of wide neural networks corresponding to directed graphs. I will then proceed to the ques- tion of depth, showing equivalence between infinitely wide and deep fully connected networks trained with gradient descent and Nadaraya-Watson predictors based on certain singular kernels. Using this connection we show that for certain activation functions these wide and deep networks are (asymptotically) optimal for classifica- tion but, interestingly, never for regression. (Based on joint work with Chaoyue Liu, Adit Radhakrishnan, Caroline Uhler and Libin Zhu.)

Information about the video

Date of publication 05/04/2024
Institution IHP
Language English
Format MP4

Last related questions on MathOverflow

You have to connect your Carmin.tv account with mathoverflow to add question

Ask a question on MathOverflow

All the collection videos

Scaling ResNets in the Large-depth Regime

By Adeline Fermanian

Overcoming the curse of dimensionality with deep neural networks

By Sophie Langer

Learning on and near Low-Dimensional Subsets of the Wasserstein Manifold

By Alex Cloninger

Extrinsic and Intrinsic Operator Estimations for Manifold Learning

By John Harlim

A graph coupling view of dimension reduction

By Franck Picard

Topologically penalized regression on manifolds

By Wolfgang Polonik

Bayesian nonparametric estimation of a density living near an unknown manifold

By Judith Rousseau

Linear methods for non-linear inverse problems

By Botond Szabo

Convergence of Sharpness-Aware Minimization

By Peter Bartlett

Dimensionality reduction in reinforcement learning by randomisation

By Denis Belomestny

Stein effect for estimating many vector means: a "blessing of dimensionality" phenomenon

By Gilles Blanchard

On the use of overfitting for estimator selection

By Claire Lacour

Optimal Permutation estimation in crowdsourcing problems

By Nicolas Verzelen

What does LIME really see in images?

By Damien Garreau

A statistical analysis of an image classification problem

By Johannes Schmidt-Hieber

Learning a partial correlation graph using only a few covariance queries

By Vasiliki Velona

published on April 5, 2024

Neural networks, wide and deep, singular kernels and Bayes optimality

By Mikhail Belkin

published on April 5, 2024

Understanding the geometry of high-dimensional data through the reach

By Clément Bérenfeld

published on April 5, 2024

Manifold Learning, Explanations and Eigenflows - Part 1

By Marina Meila

published on April 5, 2024

Manifold Learning, Explanations and Eigenflows - Part 2

By Cécile Mailler

published on April 5, 2024

On high-dimensional Lévy-driven Ornstein-Uhlenbeck processes

By Claudia Strauch

Copyright Carmin.tv 2024

Give feedback