Regularization techniques originally developed to solve linear inverse problems can be extended to derive nonparametric machine learning methods. These methods perform well in practice and can be shown to have optimal statistical guarantees, however, computational requirements can prevent application to large scale scenarios. In this talk, we will describe recent attempts to tackle this challenge. Our presentation will be divided in two parts. In the first part, we will discuss so called iterative regularization, aka early stopping regularization. In particular, we will discuss accelerated and stochastic variants of this method and show how they allow to control at the same time the statistical and time complexities of the obtained solutions. In the second part, we will discuss novel regularization schemes obtained combining regularization with stochastic projections. These latter methods allow to control not only the statistical and time complexities of the obtained solutions but also the memory requirements.