NET: Bayesian inference for neural networks using Markov chain Monte Carlo.
The 'net' programs implement Bayesian inference for models based on
multilayer perceptron networks using Markov chain Monte Carlo methods.
For full details, see the thesis, Bayesian Learning for Neural Networks,
by Radford M. Neal, Dept. of Computer Science, University of Toronto.
The networks handled have connections from a set of real-valued input
units to each of zero or more layers of real-valued hidden units.
Each hidden layer (except the last) has connections to the next hidden
layer. The output layer has connections from the input layer and from
the hidden layers.
This architecture is diagramed below, for a network with three hidden
layers:
-----------------------
| Input Units |
-----------------------
| |
---------------------------------------- |
| | | |
v | | |
------------------ | | |
| Hidden layer 0 | | | |
------------------ | | |
| | | | |
| --------- | | |
| | | | |
| v v | |
| ------------------ | |
| | Hidden layer 1 | | |
| ------------------ | |
| | | | |
| | ---------- | |
| | | | |
| | v v |
| | ------------------ |
| | | Hidden layer 2 | |
| | ------------------ |
| | | |
| | --------------- |
| ----------------------------- | |
----------------------------------------- | | |
| | | |
v v v v
-----------------------
| Output Units |
-----------------------
Any of the connection groups shown above may be absent, which is the
same as their weights all being zero.
The hidden units use the 'tanh' activation function. Nominally, the
output units are real-valued and use a linear activation function, but
discrete outputs and non-linearities may be obtained in effect with
some data models (see below).
Each hidden and output unit has a "bias" that is added to its other
inputs before the activation function is applied. Each input and
hidden unit has an "offset" that is added to its output after the
activation function is applied (or just to the specified input value,
for input units). Like connections, biases and offsets may also be
absent if desired.
A hierarchical scheme of prior distributions is used for the weights,
biases, offsets and gains in a network, in which the priors for all
parameters of one class can be coupled. These priors can also be
scaled in accord with the number of units in the source layer, in a
way that is intended to produce a reasonable limit as the number of
units in each hidden layer goes to infinity. Networks with this
architecture can also be defined that behave reasonably as the number
of hidden layers goes to infinity.
A data model may be defined that relates the values of the output
units for given inputs to the probability distribution of the data
observed in conjunction with these inputs in a training or test case.
Targets may be missing for some cases (written as "?"), in which case
they are ignored when computing the likelihood (as is appropriate if
they are "missing at random").
Copyright (c) 1995-2004 by Radford M. Neal