DMS Colloquium: Yixi Xu

Speaker: Yixi Xu,  Ph. D. candidate in Purdue University

Title: Weight normalized deep neural networks

Abstract: : Deep neural networks (DNNs) have recently demonstrated an amazing performance on many challenging artificial intelligence tasks.  DNNs have become popular due to their predictive power and flexibility in model fitting. One of the central questions about DNNs is to explain their generalization ability, even when the number of unknown parameters is much larger than the sample size. In this talk, we study a general framework of norm-based capacity control for \(L_{p,q}\) weight normalized deep neural networks and further propose a sparse neural network. We establish the upper bound on the Rademacher complexities of the \(L_{p,q}\) weight normalized deep neural networks. Especially, with an \(L_{1,\infty}\) normalization,  we discuss properties of a width-independent capacity control, where the sample complexity only depends on the depth by a square root term. In addition, for an \(L_{1,\infty}\) weight normalized network with ReLU, the approximation error can be sufficiently controlled by the \(L_1\) norm of the output layer. These results provide theoretical justifications on the usage of such weight normalization to reduce the generalization error. Finally, an easily implemented projected gradient descent algorithm is introduced to practically obtain a sparse neural network via \(L_{1,\infty}\)-weight normalization. Va​rious experiments are performed to validate the theory and demonstrate the effectiveness of the resulting approach.