A REGRESSION PROBLEM WITH OUTLIERS

Finally, we will go back to the simple regression problem we started
with, but now some of the cases will be "outliers", for which the
noise is much greater than for normal cases.

In this synthetic data, the input variable, x, again had a standard
Gaussian distribution and the corresponding target value came from a
distribution with mean given by

         0.3 + 0.4*x + 0.5*sin(2.7*x) + 1.1/(1+x^2)

For most cases, the distribution about this mean was Gaussian with
standard deviation 0.1.  However, with probability 0.05, a case is an
"outlier", for which the standard deviation was 1.0 instead.

I generated 200 cases in total, stored in the file 'odata'.  The first
100 of these cases are meant for training, the second 100 for testing.
It is also possible to test on 'rdata', to see how well the function
learned predicts data that is never corrupted by high noise.


A neural network model for regression with outliers.

One way to model data with "outliers" is to let the noise level vary
from one case to another.  If the noise for the outlier cases is set
to be higher, they will end up having less influence on the function
learned, as is desirable.  The software allows the noise variance for
a case to vary according to an inverse gamma distribution.  This is
effectively the same as letting the noise have a t-distribution rather
than a Gaussian distribution.

The commands used to do this are as follows:

    > net-spec olog.net 1 8 1 / ih=0.05:0.5 bh=0.05:0.5 ho=x0.05:0.5 bo=100 
    > model-spec olog.net real 0.05:0.5::4
    > data-spec olog.net 1 1 / odata@1:100 . odata@101:200 .

    > net-gen olog.net fix 0.5
    > mc-spec olog.net repeat 10 sample-noise heatbath hybrid 100:10 0.2
    > net-mc olog.net 1

    > mc-spec olog.net sample-sigmas heatbath hybrid 1000:10 0.4
    > net-mc olog.net 400

The crucial difference is in the 'model-spec' command, where the noise
prior of 0.05:0.5::4 specifies that the per-case noise precision
(inverse variance) follows a gamma distribution with shape parameter
of 4.  When this is integrated over, a t-distribution with 4 degrees
of freedom results.  This t-distribution is by no means an exact model
of the way the noise was actually generated, but its fairly heavy
tails are enough to prevent the model from paying undue attention to
the outliers.

The above commands take 165 seconds on the system used (see
Ex-system.doc).  The resulting model can be tested on data from the
same source using net-pred:

    > net-pred na olog.net 101:

    Number of iterations used: 300

    Number of test cases: 100

    Average squared error guessing mean:   0.01917+-0.01057

One can also see how well the model does on the uncorrupted data that
was used originally:

    > net-pred na olog.net 101: / rdata@101:200 .

    Number of iterations used: 300

    Number of test cases: 100

    Average squared error guessing mean:   0.00927+-0.00124

This is similar to the results obtained earlier with the model trained
on uncorrupted data (slightly better, in fact, but that's just luck).

In contrast, the results are substantially worse when the data with
outliers is used to train a standard model where the noise is
Gaussian, with the same variance for each case.


A Gaussian process model for regression with outliers.

Gaussian process regression can also use a t-distribution for the
noise, specified using 'model-spec', as above.  Implementation of this
model requires sampling for function values in training cases, so a
small amount of "jitter" will almost always have to be included in the
covariance function.  A "sample-variances" operation must also be
specified in 'mc-spec', to allow the case-by-case noise variances to
be sampled.  The following commands illustrate how this is done:

    > gp-spec olog.gpt 1 1 1 - 0.001 / 0.05:0.5 0.05:0.5
    > model-spec olog.gpt real 0.05:0.5::4
    > data-spec olog.gpt 1 1 / odata@1:100 . odata@101:200 .

    > gp-gen olog.gpt fix 0.5 0.1
    > mc-spec olog.gpt sample-variances heatbath hybrid 20:4 0.5
    > gp-mc olog.gpt 200

This takes 30 seconds on the system used (see Ex-system.doc).  The
progress of the run can be monitored by examining the case-by-case
noise standard deviations in (say) the first 8 training cases, as
follows:

    > gp-plt t v@0:7 olog.gpt | plot

Once the run has converged, a few of these standard deviations (for
cases that are outliers) should be much bigger than the others.  The
noise standard deviations can also be examined using the "-n" option
of 'gp-display'.

Predictions can be made using 'gp-pred':

    > gp-pred na olog.gpt 101:%5

    Number of iterations used: 20

    Number of test cases: 100

    Average squared error guessing mean:   0.01939+-0.01134

This performance is very similar to that of the network model.