EXAMPLES OF LEARNING WITH GRADIENT DESCENT, EARLY STOPPING & ENSEMBLES
Although this software is intended primarily to support research in
Bayesian methods, I have also implemented traditional gradient-descent
learning for neural networks. This allows easy comparisons of
traditional and Bayesian methods, and supports research into
variations of traditional methods that may work better.
In particular, the software supports the "early stopping" technique.
When a network is trained to a minimum of the error on the training
set (minus the log likelihood), performance on test data is often bad,
since the training data has been "overfit". To avoid this, many
people use one of the networks from earlier in the training process,
selected based on the error on a separate "validation set".
To do early stopping, the available training data must be partitioned
into an "estimation set" and a "validation set". The network
parameters (weights and biases) are randomly initialized to values
close to zero, and then gradually changed (by gradient descent) so as
to minimize error on the estimation set. The error on the validation
set is computed for the networks found during this process, and the
single network with minimum validation error is used to make
predictions for future test cases.
The split into estimation and validation sets in this procedure seems
a bit arbitrary and wasteful. To alleviate this problem, one can
train several networks using early stopping, based on different
splits, and on different random initializations of the weights.
Predictions for training cases are then be made by averaging the
predictions from the networks selected from each of these runs, a
process somewhat analogous to the averaging done when making
predictions for a Bayesian model based on a sample from the posterior
distribution.
Neural network training using early stopping and ensembles is
described by Carl Rasmussen in his thesis (available in Postscript
from http://www.cs.utoronto.ca/~carl), and in a paper of mine on
``Assessing relevance determination methods using DELVE'' (see my web
page).
To demonstrate how the software can be used to do gradient descent
learning, early stopping, and prediction using ensembles, I will show
here how these methods can be applied to the binary response problem
used as an example earlier (see Ex-netgp-b.doc).
The data and command files for these examples are in the "ex-gdes"
directory.
Gradient descent learning for the binary response problem.
First, we will see how to use the software to set the network
parameters so as to minimize error (minus the log likelihood) on the
entire training set, or at least to get as close to the minimum error
as we can using gradient descent optimization. We can then see how
well this network performs on test data.
To start, we need to specify the network architecture. This is done
using the same sort of command as is used for Bayesian networks,
except that the prior specifications are simply "+" or "-", indicating
whether the corresponding sets of weights are present or absent. The
following command creates a network with two inputs, one layer of 15
hidden units, and one output unit, with input-hidden weights, hidden
biases, hidden-output weights, and an output bias:
> net-spec blog.gd 2 15 1 / - + + - + - +
The "+" is actually translated to a very large number, with the result
that the "prior" for these parameters has virtually no effect.
Instead of a "+", one can put a positive number, s, which produces the
effect of "weight decay" with penalty equal to the sum of the squares
of these weights times 1/(2*s^2). More elaborate hierarchical priors
are meaningless if training is to be done by gradient descent.
Next, we must specify the data model. For this problem, the response
is binary, and we use a model in which the probability of a response
of 1 is found by passing the output of the network through the
logistic function. The following command specifies this:
> model-spec blog.gd binary
When the response is real, a noise standard deviation of 1 would
conventionally be used, which causes minus the log likelihood to be
half the squared error.
The location of the data is specified as for Bayesian networks:
> data-spec blog.gd 2 1 2 / bdata.train .
For these examples, the 300 training cases are stored in bdata.train
(one per line, with the two inputs coming first). The 200 test cases
are in bdata.test, but this is not mentioned above. The reason for
doing things this way (rather than putting all the data in one file)
will be apparent when we get to the examples using early stopping.
Finally, we can train the network using gradient descent, with the
command:
> net-gd blog.gd 100000 1000 / 0.4 batch
This does 100000 iterations of "batch" gradient descent (ie, with each
update based on all training cases), with networks being saved in the
log file every 1000 iterations. The software also supports "on-line"
gradient descent, which is often faster to converge initially, but
does not reach the exact optimum. See net-gd.doc for details.
The stepsize to use for gradient descent learning is specified as
well; here it is 0.4. If learning is unstable (ie, the error
sometimes goes up rather than down), the stepsize will have to be
reduced. Net-gd does not try to determine relative stepsizes itself,
but stepsizes for groups of parameters can be set individually.
The above command takes 200 seconds on the system used (see
Ex-system.doc). While waiting, you can monitor progress using
net-plt. For example, the progress of the training error can be
viewed with the command:
> net-plt t l blog.gd | plot
Individual networks can be displayed using net-display. For example,
> net-display -w blog.gd 1000
displays the network at iteration 1000. The "-w" option suppresses
the hyperparameter values, which are meaningless with this model.
Once training has finished, we can make predictions based on the last
network, which should have the lowest training error. The following
command prints a summary of performance at predicting the cases in
bdata.test, both in terms of the log probability assigned to the
correct target value and in terms of the error rate when guessing the
target:
> net-pred mpa blog.gd 100000 / bdata.test .
Number of iterations used: 1
Number of test cases: 200
Average log probability of targets: -0.364+-0.066
Fraction of guesses that were wrong: 0.1450+-0.0250
Performance is substantially worse than that obtained using Bayesian
training. This is due to "overfitting". The following command
illustrates the problem by plotting the change during the run of the
training error and the error on the test cases:
> net-plt t lL blog.gd / bdata.test . | plot
From this plot, it is clear that we would have been better off to stop
training earlier than 100000 iterations. Of course, we can't stop
training based on the test error plotted above, since we don't know
the test targets when training.
Gradient descent with early stopping for the binary response problem.
We can try to prevent overfitting by choosing one of the networks
found during training according to performance on a subset of the
available training cases that we have excluded from the set used for
the gradient descent training. This training scheme can be
implemented using the following commands:
> net-spec blog.gdes 2 15 1 / - + + - + - +
> model-spec blog.gdes binary
> data-spec blog.gdes 2 1 2 / bdata.train@1:225 . bdata.train@226:300 .
> net-gd blog.gdes 100 5 / 0.4 batch
> net-gd blog.gdes 1000 50 / 0.4 batch
> net-gd blog.gdes 20000 500 / 0.4 batch
> net-plt t L blog.gdes | find-min
Of the 300 available training cases, the first three-quarters are used
for the gradient descent training, while the last quarter are used to
choose the a network from those found during training. These 75
validation cases are listed as "test" cases in the data-spec command
above, though they are not true test cases. This allows the best of
the networks according to error on the validation set to be found
using the net-plt command, in conjunction with find-min (documented in
find-min.doc). Note that to save computer time, one might wish to
actually stop the training once it becomes apparent that further
training is unlikely to find a better network, but that is not
attempted here.
Three net-gd commands are used above so that networks can be saved to
the log file more frequently early in training. It sometimes happens
that the best network according to validation error is from very early
in the training process. We would not wish to miss it as a result of
saving too few networks in the early stages of training.
The final find-min command above outputs "3000" as the iteration that
gives the best validation error. We can use this network to make
predictions for test cases, as below:
> net-pred mpa blog.gdes 3000 / bdata.test .
Number of iterations used: 1
Number of test cases: 200
Average log probability of targets: -0.263+-0.043
Fraction of guesses that were wrong: 0.1150+-0.0226
As can be seen, performance is considerably better than that obtained
in the previous section by training for 100000 iterations.
Using an ensemble of networks trained by early stopping.
In the early stopping procedure just described, the use of the first
three-quarters of the training data for estimation and the last
one-quarter for validation is arbitrary. Whenever a training
procedure involves arbitrary or random choices, it is generally better
(on average) to repeat the procedure several times with different
choices, and then make predictions by averaging the predictions made
by the networks in this "ensemble". The following commands implement
this idea for early stopping, by averaging over both the choice of
which quarter of the data to use for validation, and over the random
choice of initial weights:
> net-spec blog.gdese1 2 15 1 / - + + - + - +
> model-spec blog.gdese1 binary
> data-spec blog.gdese1 2 1 2 / bdata.train@-226:300 . bdata.train@226:300 .
> rand-seed blog.gdese1 1
> net-gd blog.gdese1 100 5 / 0.4 batch
> net-gd blog.gdese1 1000 50 / 0.4 batch
> net-gd blog.gdese1 20000 500 / 0.4 batch
> net-plt t L blog.gdese1 | find-min
> net-spec blog.gdese2 2 15 1 / - + + - + - +
> model-spec blog.gdese2 binary
> data-spec blog.gdese2 2 1 2 / bdata.train@-151:225 . bdata.train@151:225 .
> rand-seed blog.gdese2 2
> net-gd blog.gdese2 100 5 / 0.4 batch
> net-gd blog.gdese2 1000 50 / 0.4 batch
> net-gd blog.gdese2 20000 500 / 0.4 batch
> net-plt t L blog.gdese2 | find-min
> net-spec blog.gdese3 2 15 1 / - + + - + - +
> model-spec blog.gdese3 binary
> data-spec blog.gdese3 2 1 2 / bdata.train@-76:150 . bdata.train@75:150 .
> rand-seed blog.gdese3 3
> net-gd blog.gdese3 100 5 / 0.4 batch
> net-gd blog.gdese3 1000 50 / 0.4 batch
> net-gd blog.gdese3 20000 500 / 0.4 batch
> net-plt t L blog.gdese3 | find-min
> net-spec blog.gdese4 2 15 1 / - + + - + - +
> model-spec blog.gdese4 binary
> data-spec blog.gdese4 2 1 2 / bdata.train@-1:75 . bdata.train@1:75 .
> rand-seed blog.gdese4 4
> net-gd blog.gdese4 100 5 / 0.4 batch
> net-gd blog.gdese4 1000 50 / 0.4 batch
> net-gd blog.gdese4 20000 500 / 0.4 batch
> net-plt t L blog.gdese4 | find-min
The networks selected from the four training runs above as having
lowest validation error are at iterations 3000, 950, 3500, and 11000.
We can now make predictions for test cases using these four networks,
as follows:
> net-pred mpa blog.gdese1 3000 blog.gdese2 950 \
blog.gdese3 3500 blog.gdese4 11000 / bdata.test .
Number of iterations used: 4
Number of test cases: 200
Average log probability of targets: -0.263+-0.040
Fraction of guesses that were wrong: 0.1050+-0.0217
The resulting classification performance is slightly better than was
found using just the first of the four networks.