Factor Analysis Using Delta-Rule Wake-Sleep Learning

We describe a linear network that models correlations between real-valued visible variables using one or more real-valued hidden variables - a factor analysis model. This model can be seen as a linear version of the ``Helmholtz machine'', and its parameters can be learned using the ``wake-sleep'' method, in which learning of the primary ``generative'' model is assisted by a ``recognition'' model, whose role is to fill in the values of hidden variables based on the values of visible variables. The generative and recognition models are jointly learned in ``wake'' and ``sleep'' phases, using just the delta rule. This learning procedure is comparable in simplicity to Oja's version of Hebbian learning, which produces a somewhat different representation of correlations in terms of principal components. We argue that the simplicity of wake-sleep learning makes factor analysis a plausible alternative to Hebbian learning as a model of activity-dependent cortical plasticity.

Technical Report No. 9607, Dept. of Statistics, University of Toronto (July 1996), 23 pages: postscript, pdf, associated software.

There is software available on-line that implements the method described.

Neal, R. M. and Dayan, P. (1997) ``Factor analysis using delta-rule wake-sleep learning'', Neural Computation, vol. 9, pp. 1781-1803: abstract, associated references, associated software.

The ``wake-sleep'' algorithm is described in the following paper:

Hinton, G. E., Dayan, P., Frey, B. J., and Neal, R. M. (1995) ``The ``wake-sleep'' algorithm for unsupervised neural networks'', Science, vol. 268, pp. 1158-1161: abstract, associated references.

Dayan, P., Hinton, G. E., Neal, R. M., and Zemel, R. S. (1995) ``The Helmholtz machine'', Neural Computation, vol. 7, pp. 1022-1037: abstract, associated reference.