Bayesian optimization (BO) is a model-based approach for gradient-free black-box function optimization, such as hyperparameter optimization. Typically, BO relies on conventional Gaussian process (GP) regression, whose algorithmic complexity is cubic in the number of evaluations. As a result, GP-based BO cannot leverage large numbers of past function evaluations, for example, to warm-start related BO runs. We propose a multi-task adaptive Bayesian linear regression model for transfer learning in BO, whose complexity is linear in the function evaluations: one Bayesian linear regression model is associated to each black-box function optimization problem (or task), while transfer learning is achieved by coupling the models through a shared neural network. A first set of experiments show that the neural network learns a representation suitable for warm-starting the black-box optimization problems and that BO runs can be accelerated when the target black-box function (e.g., validation loss) is learned together with other related signals (e.g., training loss). The proposed method was found to be at least one order of magnitude faster that methods recently published in the literature. A second set of experiments show that our approach can further be combined with Hyperband, replacing the uniform random sampling of hyperparameter candidates by an adaptive non-uniform sampling procedure. Our extension not only improves the precision resolution of Hyperband but also supports transfer learning, both, within a Hyperband run and across previous hyperparameter tuning tasks. This is joint work with R. Jenatton, L. Valkov, F. Winkelmolen, C. Archambeau and M. Seeger.