A PROBABILITY ESTIMATION PROBLEM WITH BINARY DATA
As a first illustration of the mixture model software, I generated a
dataset in which each case was composed of ten binary attributes. The
distribution of these binary vectors was a mixture of four component
distributions, in each of which the ten attributes were independent.
The four components each had probability 0.25 of being chosen to
generate a case. The probabilities for each component for each binary
attributes were as shown below:
1 2 3 4 5 6 7 8 9 10
1 0.1 0.2 0.2 0.2 0.2 0.8 0.8 0.8 0.8 0.7
2 0.1 0.8 0.8 0.8 0.8 0.2 0.2 0.2 0.2 0.7
3 0.9 0.2 0.2 0.8 0.8 0.2 0.2 0.8 0.8 0.7
4 0.9 0.8 0.8 0.2 0.2 0.8 0.8 0.2 0.2 0.7
Each row gives the probabilities of each of the attributes being '1'
for one component of the mixture. The columns are for the ten binary
attributes in each case. The vectors generated in this way can be
seen as coming from one of four "patterns": 0000011111, 0111100001,
1001100111, and 1110011001, but with each bit of the chosen pattern
having a small probability of being switched (ranging from 0.1 to 0.3)
in any particular case.
I generated 1000 cases from this distribution, which are stored in the
file 'bdata'. The first 500 are used for training (the others are not
used at all at the moment).
A finite mixture model for the binary data.
We will first see what happens when we model this data as a mixture of
four components, which is the true number of components for this
problem. Each component of the mixture will give each target
attribute a certain probability of being '1' rather than '0'. These
probabilities are determined from the "offset" parameters for each
target, by means of the "logistic" function:
Pr(target=1) = 1 / (1 + exp(-offset))
The offset parameters for each attribute for each component are given
Gaussian prior distributions, with means and standard deviations that
are hyperparameters. These hyperparameters could be fixed, but here
we will make them variable, separately for each target attribute, but
with each of them linked to a top-level hyperparameter, common to all
targets.
We set up this mixture model with the 'mix-spec' command, as follows:
> mix-spec blog.4 0 10 4 / x1 0.05:0.5:0.2 10
Here, "blog.4" is the name of the log file used for this run, which is
created by the 'mix-spec' command. The arguments of 'mix-spec' that
follow the name of the log file are the number of input attributes
(always 0 for mixture models at present), the number of target
attributes (10 for this problem), and the number of mixture components
(set to the true value of 4 in the command above). The specifications
for priors are given following these arguments, after a "/" separator.
The first prior specification gives the concentration parameter of the
Dirichlet distribution for the mixing proportions for the components.
If this specification starts with "x", this value is automatically
divided by the number of components, which is the scaling required for
the model to reach a sensible limit as the number of components goes
to infinity. The specification of "x1" above mildly favours unequal
mixing proportions (recall that the true proportions are equal).
The second argument after the "/" specifies the prior for the standard
deviations of the "offsets" for components (which determine the
probabilities for the binary target attributes). In the hierarchical
scheme specified here (described in general in prior.doc), there is a
top-level standard deviation hyperparameter, given a prior in which
the corresponding "precision" (standard deviation to the power -2) has
a gamma distribution with mean 0.05 and shape parameter 0.5. There is
a lower-level standard deviation hyperparameter for each target; for
each of these, the corresponding precision has a gamma distribution
with mean given by the higher-level precision, and shape parameter of
0.2. Both these priors are rather vague (but the shape parameter of
0.2 is vaguer than 0.5), so these hyperparameters can adapt to the
structure of the data.
The last argument of 'mix-spec' is the standard deviation for the
means of the offsets for each of the targets, here set to 10.
Currently, this standard deviation for the mean offset is the same for
all targets, and is fixed in the 'mix-spec' command (it cannot be a
variable hyperparameter).
We can look at these specifications by calling 'mix-spec' with just
the log file as an argument:
> mix-spec blog.4
Number of inputs: 0
Number of targets: 10
Number of components: 4
Dirichlet concentration parameter: x1.000
Prior for SD hyperparameters: 0.050:0.50:0.20
Prior for mean component offsets: 10.000
After the 'mix-spec' command, we need to specify that the targets are
binary using the 'model-spec' command, as follows:
> model-spec blog.4 binary
We can then specify where the data comes from using 'data-spec':
> data-spec blog.4 0 10 2 / bdata@1:500 .
The number of inputs is here specified to be 0 (as it must be at
present for mixture models), and the number of targets is specified to
be 10. These must match the values given to 'mix-spec'. We also
specify that the targets must be the integers 0 or 1 putting in a
following argument of 2 (meaning that there are 2 possible values,
which start at 0). After a slash, we say where the inputs and targets
come from. Note that we have to say where the inputs come from even
though there are 0 of them, but fortunately, we can then say that the
targets come from the same place by just using the "." argument. In
the specification above, the data comes from the first 500 lines of
the file 'bdata', with one case per line. See data-spec.doc for more
details.
Finally, we need to specify how Markov chain sampling is to be done
for this model. At present, none of the standard Markov chain methods
are allowed for mixture models, only the specialized procedures
documented in mix-mc.doc. Each of these procedures updates one of the
three parts of the state - the values of the hyperparameters, the
values of the parameters for the various mixture components, and the
values of the indicator variables saying which mixture component is
currently associated with each training case. The values in each of
these parts of the state can be updated by Gibbs sampling, using the
corresponding procedure. The following call of 'mc-spec' sets this
up:
> mc-spec blog.4 repeat 20 gibbs-indicators gibbs-params gibbs-hypers
Here, the three Gibbs sampling operations are repeated 20 times each
iteration, just to cut down on the number of states saved in the log
file.
We can now run the Markov chain simulation for 100 iterations with the
following command:
> mix-mc blog.4 100
The simulation starts with a state in which all the training cases are
associated with the same component, whose parameters are set to their
means, as specified by hyperparameter values that are also set to
their prior means. The 'mix-gen' program could be used to initialize
things differently (see mix-gen.doc).
The above run takes about half a minute on our 550MHz Pentium III. It
can be run in the background, in which case progress can be monitored
with the 'mix-display' and 'mix-plt' programs. For example, after a
while, we might see the following:
> mix-display blog.4
MIXTURE MODEL IN FILE "blog.4" WITH INDEX 50
HYPERPARAMETERS
Standard deviations for component offsets:
0.326: 1.407 1.159 1.213 3.331 1.417
1.557 3.433 2.203 4.990 0.192
Means for component offsets:
-0.370 -0.126 -0.538 +0.521 -0.620
-1.642 +1.510 -1.206 -1.502 +0.941
PARAMETERS AND FREQUENCIES FOR COMPONENTS OF THE MIXTURE
1: 0.286 -1.799 +0.931 +1.753 +1.481 +0.853
-1.503 -1.433 -0.942 -1.245 +1.165
2: 0.252 +1.905 +1.054 +1.749 -2.588 -1.594
+1.062 +1.625 -1.532 -1.227 +1.252
3: 0.250 -2.109 -1.505 -2.220 -0.961 -1.402
+2.265 +1.635 +1.463 +2.015 +1.082
4: 0.212 +1.743 -1.050 -1.470 +1.502 +1.553
-1.593 -1.376 +1.569 +1.025 +0.864
This displays the state at the last iteration computed so far (here
iteration 50). The hyperparameter values are shown first, with
top-level hyperparameters on the left, before the colon. The
lower-level hyperparameters follow, ordered by the target attribute
that they pertain to. Values for the component parameters follow,
sorted by the fraction of training cases that they are currently
associated with. These fractions are the first numbers after 1:, 2:,
etc.; note that the actual mixing probabilities are not explicitly
represented, but are instead always integrated over. Any components
that are not associated with any training case are omitted from the
output, and are not explicitly represented in the state. The other
numbers shown for each component are the "offset" parameters for each
target attribute.
In this example, the simulation appears to have learned the essence of
the distribution by the iteration shown above. Recall that a positive
offset corresponds to the probability of a 1 being greater than 1/2, a
negative offset to the probability of a 1 being less than 1/2. The
four components shown above can thus be seen to correspond to the four
patterns used to generate the cases, as described earlier. Note that
the last of the offset standard deviation hyperparameters (for the
last target attribute) is quite small. This indicates that the model
has "learned" that the probabilities for the last target are the same
for all components, and hence can be modeled most efficiently at the
hyperparameter level, by the mean for that offset, rather than
separately for each component.
The indicators of which components are associated with each training
case can also be examined with 'mix-display', and of course iterations
other than the last can be viewed. See mix-display.doc for details.
One can also look at the progress of the simulation using 'mix-plt'.
For example, the following will produce a time-plot of the cumulative
proportions of the training cases associated with the four components
(as sorted by decreasing frequency):
> mix-plt t C1C2C3C4 blog.4 | plot
Here, 'plot' is some suitable plotting program. One can also just let
the output of 'mix-plt' go to standard output, and then examine the
numbers manually. Other quantities can also be plotted, as described
in mix-quantities.doc.
As well as examining quantities with 'mix-display' and 'mix-plt', one
can also produce a sample of cases from the predictive distribution
using 'mix-cases'. Unfortunately, these are about the only things one
can do with the results at the moment. There are no programs that
find predictive probabilities, fill in missing values in partial
cases, or do any of the other useful things that are quite possible
with these models, but which haven't been implemented. See
mix-extensions.doc for a wish list of such facilities.
A countably infinite mixture model for the binary data.
Even though the true distribution for this example is a mixture of
four components, good results can nevertheless be obtained using a
mixture with a countably infinite number of components. The prior for
mixture proportions used with such a model is designed so that a few
of the infinite number of components have substantial probability, so
the model does not result in an "overfitted" solution, in which every
training case is "explained" as coming from a different component.
We specify a countably infinite mixture model by simply omitting the
argument to 'mix-spec' that gives the number of components. For this
example, we change the 'mix-spec' command used above to the following:
> mix-spec blog.inf 0 10 / x1 0.05:0.5:0.2 10
The Dirichlet prior specification for mixing proportions of "x1" is
the same as for the mixture model with four components. The "x"
specifies scaling downward with the number of components, which
produces a sensible limit as the number of components goes to
infinity, as here. The "x" is therefore required for infinite
mixtures.
The other arguments of 'mix-spec' are as described for the finite
mixture model. The 'model-spec' and 'data-spec' commands used are
also the same:
> model-spec blog.inf binary
> data-spec blog.inf 0 10 2 / bdata@1:500 .
We must change the 'mc-spec' command, however, since it is impossible
to do Gibbs sampling for the component indicators associated with
training cases - since their number is infinite, we can't compute all
the required conditional probabilities. However, we can instead use a
specialized Metropolis-Hastings update, in which a new component to go
with a training case is proposed with probability determined by the
frequencies with which components are associated with other training
cases. The proposal is accepted or rejected based on the resulting
change in likelihood. The process can then be repeated any desired
number of times, to produce an approximation to Gibbs sampling (the
"approximation" is only with respect to convergence speed, the answer
is exact, asymptotically). With this change, we can use the following
'mc-spec' command for this model:
> mc-spec blog.inf repeat 20 met-indicators 10 gibbs-params gibbs-hypers
This is the same as for the finite mixture model, except that 10
repetitions of the Metropolis-Hastings update for the indicators are
specified using 'met-indicators 10', in place of 'gibbs-indicators'.
As before, we can now run the simulation with a command such as:
> mix-mc blog.inf 100
We can examine the states with 'mix-display'. For example, once the
'mix-mc' command completes, the state should be something like the
following:
> mix-display blog.inf
MIXTURE MODEL IN FILE "blog.inf" WITH INDEX 100
HYPERPARAMETERS
Standard deviations for component offsets:
0.359: 2.747 4.294 1.280 3.530 1.353
2.461 5.661 2.928 1.506 0.080
Means for component offsets:
+0.023 +0.452 +0.792 +0.507 -0.084
-0.634 -4.794 +4.560 -0.293 +1.124
PARAMETERS AND FREQUENCIES FOR COMPONENTS OF THE MIXTURE
1: 0.310 -1.565 +0.617 +1.467 +1.191 +0.468
-0.569 -1.192 -0.397 -0.749 +1.003
2: 0.254 -1.374 -1.134 -2.009 -1.008 -1.825
+2.384 +2.377 +1.579 +2.553 +1.080
3: 0.238 +2.949 +0.979 +2.419 -2.694 -1.331
+1.597 +1.590 -1.689 -1.190 +1.124
4: 0.168 +1.565 -1.687 -1.958 +1.762 +1.633
-1.365 -6.917 +1.001 +1.180 +0.919
5: 0.012 +1.780 -3.038 -0.664 +5.257 -3.499
-8.315 -5.443 +0.816 -0.774 +1.143
6: 0.012 +1.892 -0.991 +3.387 +0.513 +1.287
-4.129 +1.283 +8.346 +1.452 +0.882
7: 0.004 +5.850 +1.412 +0.295 -5.867 +0.701
-0.403 -9.392 +7.748 -2.211 +1.163
8: 0.002 +1.028 -3.102 +2.383 -1.070 +1.691
+0.497 +0.062 +0.505 -0.762 +1.226
The output is similar to that seen for the mixture model with four
components, except that eight components are associated with at least
one training case at iteration 100, as shown above. (The infinite
number of remaining components are not explicitly represented, and are
not displayed by 'mix-display'.) Four of these components are
associated with very few training cases, however (as seen from the
fractions of the training set after the component number). This is to
be expected when the true distribution can be expressed using only
four components. The number of such low-frequency components will
vary from iteration to iteration, as will other aspects of the state,
in accordance with the variability in the posterior distribution.
A number of other ways of sampling for the indicators for countably
infinite mixture models are also available. Here are two:
> mc-spec blog.inf2 repeat 20 \
gibbs-ext-indicators 2 gibbs-params gibbs-hypers
> mc-spec blog.inf3 repeat 20 \
met1-indicators gibbs1-indicators gibbs-params gibbs-hypers
These methods may be more efficient than using the "met-indicators"
operation for some models. See mix-mc.doc for descriptions of these
and other Markov chain operations for mixture models.