## The ``Wake-Sleep'' Algorithm for Unsupervised Neural Networks

**Geoffrey E. Hinton,
Peter Dayan,
Brendan J. Frey, and
Radford M. Neal**

Dept. of Computer Science, University of Toronto
An unsupervised learning algorithm for a multilayer network of
stochastic neurons is described. Bottom-up ``recognition''
connections convert the input into representations in successive
hidden layers, and top-down ``generative'' connections reconstruct the
representation in one layer from the representation in the layer
above. In the ``wake'' phase, neurons are driven by recognition
connections, and generative connections are adapted to increase the
probability that they would reconstruct the correct activity vector in
the layer below. In the ``sleep'' phase, neurons are driven by
generative connections, and recognition connections are adapted to
increase the probability that they would produce the correct activity
vector in the layer above.

*Science*, vol. 268, pp. 1158-1161 (1995).

**Associated references:**
The wake-sleep algorithm is a variation on the ``Helmholtz Machine'',
discussed in the following paper:
Dayan, P., Hinton, G. E., Neal, R. M., and Zemel, R. S. (1995) ``The
Helmholtz machine'', *Neural Computation*, vol. 7, pp. 1022-1037:
abstract.

The wake-sleep algorithm is used to learn factor analysis models in
the following technical report:
Neal, R. M. and Dayan, P. (1996) ``Factor analysis using delta-rule wake-sleep
learning'', Technical Report No. 9607, Dept. of Statistics, University of
Toronto, 23 pages:
abstract,
postscript, pdf,
associated references,
associated software.

The wake-sleep algorithm is used to model temporal sequences in the
following conference paper:
Hinton, G. E., Dayan, P., To, A., and Neal, R. (1995) ``The Helmholtz
machine through time'', in F. Fogelman-Soulie and R. Gallinari
(editors) *ICANN-95*, pp. 483-490:
abstract.