Optimization has always played an important role in data science but with the recent and rapid development of machine learning methods with very successful real-world applications, the need for fast optimization methods for training high-dimensional models over large datasets has grown immensely. Although, stochastic optimization methods are widely used in these problems, their mathematical analysis is far from being complete. In this course, we will introduce state-of-the-art rst and second-order stochastic gradient methods for solving large-scale optimization problems and review their theoretical background on convergence rate analysis. We will also see some applications to observe these powerful methods at work. The course is aimed at graduate students with some background in probability and calculus. No knowledge of machine learning methods is required.