## Factor Analysis Using Delta-Rule Wake-Sleep Learning

**Radford M. Neal,
Dept. of Statistics and Dept. of Computer Science, University of Toronto**

Peter Dayan,
Dept. of Brain and Cognitive Sciences, Massachusetts Institute of Technology
We describe a linear network that models correlations between
real-valued visible variables using one or more real-valued hidden
variables - a *factor analysis* model. This model can be seen
as a linear version of the ``Helmholtz machine'', and its parameters
can be learned using the ``wake-sleep'' method, in which learning of
the primary ``generative'' model is assisted by a ``recognition''
model, whose role is to fill in the values of hidden variables based
on the values of visible variables. The generative and recognition
models are jointly learned in ``wake'' and ``sleep'' phases, using just
the delta rule. This learning procedure is comparable in simplicity to
Oja's version of Hebbian learning, which produces a somewhat different
representation of correlations in terms of principal components. We
argue that the simplicity of wake-sleep learning makes factor analysis
a plausible alternative to Hebbian learning as a model of
activity-dependent cortical plasticity.

Technical Report No. 9607, Dept. of Statistics, University of Toronto
(July 1996), 23 pages:
postscript,
pdf,
associated software.

There is software available on-line
that implements the method described.

**Associated references:**
A revised version of this paper has been published:
Neal, R. M. and Dayan, P. (1997) ``Factor analysis using delta-rule wake-sleep
learning'', *Neural Computation*, vol. 9, pp. 1781-1803:
abstract,
associated references,
associated software.

The ``wake-sleep'' algorithm is described in the following paper:

Hinton, G. E., Dayan, P., Frey, B. J., and Neal, R. M. (1995)
``The ``wake-sleep'' algorithm for unsupervised neural networks'',
*Science*, vol. 268, pp. 1158-1161:
abstract,
associated references.

The wake-sleep algorithm is a way of learning ``Helmholtz Machines'', which
are discussed in the following paper:
Dayan, P., Hinton, G. E., Neal, R. M., and Zemel, R. S. (1995) ``The
Helmholtz machine'', *Neural Computation*, vol. 7, pp. 1022-1037:
abstract,
associated reference.