Factor Analysis Using Delta-Rule Wake-Sleep Learning

Radford M. Neal, Dept. of Statistics and Dept. of Computer Science, University of Toronto
Peter Dayan, Dept. of Brain and Cognitive Sciences, Massachusetts Institute of Technology

We describe a linear network that models correlations between real-valued visible variables using one or more real-valued hidden variables - a factor analysis model. This model can be seen as a linear version of the ``Helmholtz machine'', and its parameters can be learned using the ``wake-sleep'' method, in which learning of the primary ``generative'' model is assisted by a ``recognition'' model, whose role is to fill in the values of hidden variables based on the values of visible variables. The generative and recognition models are jointly learned in ``wake'' and ``sleep'' phases, using just the delta rule. This learning procedure is comparable in simplicity to Hebbian learning, which produces a somewhat different representation of correlations in terms of principal components. We argue that the simplicity of wake-sleep learning makes factor analysis a plausible alternative to Hebbian learning as a model of activity-dependent cortical plasticity.

Neural Computation, vol. 9, pp. 1781-1803.

There is software available on-line that implements the method described.

Associated references: An earlier version of this paper was issued as a technical report:
Neal, R. M. and Dayan, P. (1996) ``Factor analysis using delta-rule wake-sleep learning'', Technical Report No. 9607, Dept. of Statistics, University of Toronto, 23 pages: abstract, postscript, pdf, associated references, associated software.

The ``wake-sleep'' algorithm is described in the following paper:

Hinton, G. E., Dayan, P., Frey, B. J., and Neal, R. M. (1995) ``The ``wake-sleep'' algorithm for unsupervised neural networks'', Science, vol. 268, pp. 1158-1161: abstract, associated references.
The wake-sleep algorithm is a way of learning ``Helmholtz Machines'', which are discussed in the following paper:
Dayan, P., Hinton, G. E., Neal, R. M., and Zemel, R. S. (1995) ``The Helmholtz machine'', Neural Computation, vol. 7, pp. 1022-1037: abstract, associated reference.