Date: 25 June
Hour: 1 p.m.
Venue: QA1.1 amphitheatre, Alameda campus and online
Speaker: Laura Balzano (University of Michigan)
Title: “Efficient Low-Dimensional Compression for Deep Overparameterized Learning”
Abstract:
While overparameterization in machine learning models offers great benefits in terms of optimization and generalization, it also leads to increased computational requirements as model sizes grow. In this work, we show that by leveraging inherent low-dimensional structure within the model parameter updates, we can reap the benefits of overparameterization without the computational burden. In practice, we demonstrate the effectiveness of this approach for deep low-rank matrix completion as well as fine-tuning language models. For theory of deep overparameterized low-rank matrix recovery, we show that the learning dynamics of each weight matrix are confined to an invariant low-dimensional subspace. Consequently, we can construct and train compact, highly compressed factorizations possessing the same benefits as their overparameterized counterparts. For language model fine-tuning, we introduce a method called “Deep LoRA”, which improves the existing low-rank adaptation (LoRA) technique, leading to reduced overfitting and a simplified hyperparameter setup, all while maintaining comparable efficiency. The effectiveness of Deep LoRA is validated through its performance on natural language understanding tasks, particularly when fine-tuning with a limited number of samples.
Speaker bio:
Laura Balzano is an associate professor of Electrical Engineering and Computer Science, and of Statistics by courtesy, at the University of Michigan. She is recipient of the NSF Career Award, ARO Young Investigator Award, AFOSR Young Investigator Award, and faculty fellowships from Intel and 3M. She received an MLK Spirit Award and the Vulcans Education Excellence Award at the University of Michigan. Her expertise is in statistical signal processing, matrix factorization, and optimization. Laura received a BS from Rice University, MS from UCLA, and PhD from the University of Wisconsin Madison in Electrical and Computer Engineering.
Priberam is a member of the IST Spin-Off Community®.
The Priberam Machine Learning Lunch Seminars are free of charge. Prior registration is required.
