Share Email Print

Proceedings Paper

Smoothing of cost function leads to faster convergence of neural network learning
Author(s): Li-Qun Xu; Trevor J. Hall
Format Member Price Non-Member Price
PDF $14.40 $18.00
cover GOOD NEWS! Your organization subscribes to the SPIE Digital Library. You may be able to download this paper for free. Check Access

Paper Abstract

One of the major problems in supervised learning of neural networks is the inevitable local minima inherent in the cost function f(W,D). This often makes classic gradient-descent-based learning algorithms that calculate the weight updates for each iteration according to (Delta) W(t) equals -(eta) (DOT)$DELwf(W,D) powerless. In this paper we describe a new strategy to solve this problem, which, adaptively, changes the learning rate and manipulates the gradient estimator simultaneously. The idea is to implicitly convert the local- minima-laden cost function f((DOT)) into a sequence of its smoothed versions {f(beta t)}Ttequals1, which, subject to the parameter (beta) t, bears less details at time t equals 1 and gradually more later on, the learning is actually performed on this sequence of functionals. The corresponding smoothed global minima obtained in this way, {Wt}Ttequals1, thus progressively approximate W--the desired global minimum. Experimental results on a nonconvex function minimization problem and a typical neural network learning task are given, analyses and discussions of some important issues are provided.

Paper Details

Date Published: 2 March 1994
PDF: 9 pages
Proc. SPIE 2243, Applications of Artificial Neural Networks V, (2 March 1994); doi: 10.1117/12.169959
Show Author Affiliations
Li-Qun Xu, King's College London (United Kingdom)
Trevor J. Hall, King's College London (United Kingdom)

Published in SPIE Proceedings Vol. 2243:
Applications of Artificial Neural Networks V
Steven K. Rogers; Dennis W. Ruck, Editor(s)

© SPIE. Terms of Use
Back to Top