Constrained deep learning
I develop scalable methods for training neural networks that satisfy explicit requirements such as fairness, sparsity, and safety.
My research focuses on constrained optimization, feasible learning, and Lagrangian methods. I am supervised by Simon Lacoste-Julien and co-develop Cooper, an open-source PyTorch library for constrained optimization in deep learning.
Perspective
This paper argues that fixed penalty terms are the wrong default for enforcing explicit requirements in deep learning. Instead, when a problem naturally has targets to satisfy, we should solve it as a constrained optimization problem with tailored methods rather than hope that penalty tuning recovers the right trade-off.
Open-source software
Cooper is an open-source package for solving constrained optimization problems in deep learning. It implements Lagrangian-based first-order update schemes and makes it easy to combine constrained optimization algorithms with PyTorch models, autograd, and modern training pipelines. See the code and docs.
Learning paradigm
Feasible Learning trains models by solving a feasibility problem that bounds the loss on every training example, rather than optimizing for average performance. It is a sample-centric alternative to ERM for settings where tail behavior and per-example reliability matter.
Theory
This paper shows that dual optimistic ascent on the dual variables of the Lagrangian is equivalent to gradient descent-ascent on the Augmented Lagrangian in the single-step, first-order regime commonly used in constrained deep learning. The result helps explain the practical success of dual optimism and establish its limits.
* denotes equal contribution. ^ denotes equal supervision.