Abstract: At a basic level Deep Neural Networks (DNNs) are frameworks for representing functions. The network architecture (number of layers, and widths of layers) and design choices (types of activation functions at nodes) determine the complexity (expressive power) of function that can be represented by that network. From linear interpolation to Multilinear Algebra the geometry of tensors play an important role in understanding the expressive power of DNNs. I’ll explain this connection, and describe some open questions.