NET-MC:  Do Markov chain simulation to sample networks.

The net-mc program is the specialization of xxx-mc to the task of
sampling from the posterior distribution for a neural network model,
or from the prior distribution, if no training set is specified.  See
xxx-mc.doc for the generic features of this program.

The following applications-specific sampling procedures are implemented:

   sample-hyper       

       Does Gibbs sampling for the hyperparameters controlling the 
       distributions of parameters (weights, biases, etc.).

   sample-noise       

       Does Gibbs sampling for the noise variances.

   sample-lower-hyper 

       Does Gibbs sampling for all lower-level hyperparameters.

   rgrid--upper-hyper [ stepsize ]

       Does random-grid Metropolis updates (one at a time) for 
       the logs of all upper-level hyperparameters, in "precision" 
       form.  The default stepsize is 0.1.

   sample-lower-noise

       Does Gibbs sampling for all lower-level noise variances.

   rgrid-upper-noise [ stepsize ]

       Does random-grid Metropolis updates (one at a time) for the 
       logs of all upper-level hyperameters controlling noise
       variances, in "precison" form.  The default stepsize is 0.1.

   sample-sigmas  

       Does the equivalent of both sample-hyper and sample-noise.

   sample-lower-sigmas

       Does the equivalent of both sample-lower-hyper and 
       sample-lower-noise.

   sample-upper-sigmas

       Does the equivalent of both sample-upper-hyper and 
       sample-upper-noise.

An "upper-level" hyperparameter is one that controls the distribution
of lower-level hyperparameters or noise variances (which may be either
explicit or implicit).  The "lower-level" hyperparameters directly
control the distributions of weights.  Looked at another way, the
lower-level hyperparameters are the ones at the bottom level of the
hierarchy, or for which all lower-level hyperparameters have
degenerate distributions concentrated on the value of the higher-level
hyperparameter.

The random grid metropolis updates done with the above commands record
information (eg, rejection rate) for later display in the same way as
generic rgrid-met-1 updates.  

When coupling is being done, upper-level hyperparameters should be
updated only with random-grid updates; Gibbs sampling for these
upper-level hyperparamters will desynchronize the random number
streams (because of the way it is implemented using ARS), preventing
coalescence.  Lower-level hyperparameters can be updated with Gibbs
sampling, and they will exactly coalesce once the parameters they
control and the upper-level hyperparameters that control them have
exactly coalesced.

Default stepsizes for updates of parameters (weights, biases, etc.) by
the generic Markov chain operations are set by a complicated heuristic
procedure that is described in Appendix A of the thesis.

Tempering methods and Annealed Importance sampling are supported.  The
effect of running at an inverse temperature other than one is to
multiply the likelihood part of the energy by that amount.  At inverse
temperature zero, the distribution is simply the prior for the
hyperparameters and weights.  The marginal likelihood for a model can
be found using Annealed Importance Sampling, since the log likelihood
part of the energy has all the appropriate normalizing constants.

            Copyright (c) 1995-2004 by Radford M. Neal