Share Email Print

Proceedings Paper

Memory-optimal neural network approximation
Author(s): Helmut Bölcskei; Philipp Grohs; Gitta Kutyniok; Philipp Petersen
Format Member Price Non-Member Price
PDF $17.00 $21.00

Paper Abstract

We summarize the main results of a recent theory—developed by the authors—establishing fundamental lower bounds on the connectivity and memory requirements of deep neural networks as a function of the complexity of the function class to be approximated by the network. These bounds are shown to be achievable. Specifically, all function classes that are optimally approximated by a general class of representation systems—so-called affine systems—can be approximated by deep neural networks with minimal connectivity and memory requirements. Affine systems encompass a wealth of representation systems from applied harmonic analysis such as wavelets, shearlets, ridgelets, α-shearlets, and more generally α-molecules. This result elucidates a remarkable universality property of deep neural networks and shows that they achieve the optimum approximation properties of all affine systems combined. Finally, we present numerical experiments demonstrating that the standard stochastic gradient descent algorithm generates deep neural networks which provide close-to-optimal approximation rates at minimal connectivity. Moreover, stochastic gradient descent is found to actually learn approximations that are sparse in the representation system optimally sparsifying the function class the network is trained on.

Paper Details

Date Published: 24 August 2017
PDF: 12 pages
Proc. SPIE 10394, Wavelets and Sparsity XVII, 103940Q (24 August 2017); doi: 10.1117/12.2272490
Show Author Affiliations
Helmut Bölcskei, ETH Zurich (Switzerland)
Philipp Grohs, Univ. of Vienna (Austria)
Gitta Kutyniok, Technische Univ. Berlin (Germany)
Philipp Petersen, Technische Univ. Berlin (Germany)

Published in SPIE Proceedings Vol. 10394:
Wavelets and Sparsity XVII
Yue M. Lu; Dimitri Van De Ville; Manos Papadakis, Editor(s)

© SPIE. Terms of Use
Back to Top
Sign in to read the full article
Create a free SPIE account to get access to
premium articles and original research
Forgot your username?