A PROBABILITY ESTIMATION PROBLEM WITH BINARY DATA
As a first illustration of the mixture model and Dirichlet diffusion
tree software, I generated a dataset in which each case was composed
of ten binary attributes. The distribution of these binary vectors
was a mixture of four component distributions, in each of which the
ten attributes were independent. The four components each had
probability 0.25 of being chosen to generate a case. The
probabilities for each component for each binary attributes were as
shown below:
1 2 3 4 5 6 7 8 9 10
1 0.1 0.2 0.2 0.2 0.2 0.8 0.8 0.8 0.8 0.7
2 0.1 0.8 0.8 0.8 0.8 0.2 0.2 0.2 0.2 0.7
3 0.9 0.2 0.2 0.8 0.8 0.2 0.2 0.8 0.8 0.7
4 0.9 0.8 0.8 0.2 0.2 0.8 0.8 0.2 0.2 0.7
Each row gives the probabilities of each of the attributes being '1'
for one component of the mixture. The columns are for the ten binary
attributes in each case. The vectors generated in this way can be
seen as coming from one of four "patterns": 0000011111, 0111100001,
1001100111, and 1110011001, but with each bit of the chosen pattern
having a small probability of being switched (ranging from 0.1 to 0.3)
in any particular case.
I generated 1000 cases from this distribution, which are stored in the
file 'bdata'. The first 500 are used for training, the rest for
testing.
A finite mixture model for the binary data.
We will first see what happens when we model this data as a mixture of
four components, which is the true number of components for this
problem. Each component of the mixture will give each target
attribute a certain probability of being '1' rather than '0'. These
probabilities are determined from the "offset" parameters for each
target, by means of the "logistic" function:
Pr(target=1) = 1 / (1 + exp(-offset))
The offset parameters for each attribute for each component are given
Gaussian prior distributions, with means and standard deviations that
are hyperparameters. These hyperparameters could be fixed, but here
we will make them variable, separately for each target attribute, but
with each of them linked to a top-level hyperparameter, common to all
targets.
We set up this mixture model with the 'mix-spec' command, as follows:
> mix-spec blog.4 0 10 4 / x1 0.05:0.5:0.2 10
Here, "blog.4" is the name of the log file used for this run, which is
created by the 'mix-spec' command. The arguments of 'mix-spec' that
follow the name of the log file are the number of input attributes
(always 0 for mixture models at present), the number of target
attributes (10 for this problem), and the number of mixture components
(set to the true value of 4 in the command above). The specifications
for priors are given following these arguments, after a "/" separator.
The first prior specification gives the concentration parameter of the
Dirichlet distribution for the mixing proportions for the components.
If this specification starts with "x", this value is automatically
divided by the number of components, which is the scaling required for
the model to reach a sensible limit as the number of components goes
to infinity. The specification of "x1" above mildly favours unequal
mixing proportions (recall that the true proportions are equal).
The second argument after the "/" specifies the prior for the standard
deviations of the "offsets" for components (which determine the
probabilities for the binary target attributes). In the hierarchical
scheme specified here (described in general in prior.doc), there is a
top-level standard deviation hyperparameter, given a prior in which
the corresponding "precision" (standard deviation to the power -2) has
a gamma distribution with mean 0.05 and shape parameter 0.5. There is
a lower-level standard deviation hyperparameter for each target; for
each of these, the corresponding precision has a gamma distribution
with mean given by the higher-level precision, and shape parameter of
0.2. Both these priors are rather vague (but the shape parameter of
0.2 is vaguer than 0.5), so these hyperparameters can adapt to the
structure of the data.
The last argument of 'mix-spec' is the standard deviation for the
means of the offsets for each of the targets, here set to 10.
Currently, this standard deviation for the mean offset is the same for
all targets, and is fixed in the 'mix-spec' command (it cannot be a
variable hyperparameter).
We can look at these specifications by calling 'mix-spec' with just
the log file as an argument:
> mix-spec blog.4
Number of inputs: 0
Number of targets: 10
Number of components: 4
Dirichlet concentration parameter: x1.000
Prior for SD hyperparameters: 0.050:0.50:0.20
Prior for mean component offsets: 10.000
After the 'mix-spec' command, we need to specify that the targets are
binary using the 'model-spec' command, as follows:
> model-spec blog.4 binary
We can then specify where the data comes from using 'data-spec':
> data-spec blog.4 0 10 2 / bdata@1:500 . bdata@501:1000 .
The number of inputs is here specified to be 0 (as it must be at
present for mixture models), and the number of targets is specified to
be 10. These must match the values given to 'mix-spec'. We also
specify that the targets must be the integers 0 or 1 putting in a
following argument of 2 (meaning that there are 2 possible values,
which start at 0). After a slash, we say where the inputs and targets
come from. Note that we have to say where the inputs come from even
though there are 0 of them, but fortunately, we can then say that the
targets come from the same place by just using the "." argument. In
the specification above, the data comes from the first 500 lines of
the file 'bdata', with one case per line. The remaining 500 lines are
specified to be used for testing. See data-spec.doc for more details.
Finally, we need to specify how Markov chain sampling is to be done
for this model. At present, none of the standard Markov chain methods
are allowed for mixture models, only the specialized procedures
documented in mix-mc.doc. Each of these procedures updates one of the
three parts of the state - the values of the hyperparameters, the
values of the parameters for the various mixture components, and the
values of the indicator variables saying which mixture component is
currently associated with each training case. The values in each of
these parts of the state can be updated by Gibbs sampling, using the
corresponding procedure. The following call of 'mc-spec' sets this
up:
> mc-spec blog.4 repeat 20 gibbs-indicators gibbs-params gibbs-hypers
Here, the three Gibbs sampling operations are repeated 20 times each
iteration, just to cut down on the number of states saved in the log
file.
We can now run the Markov chain simulation for 100 iterations with the
following command:
> mix-mc blog.4 100
The simulation starts with a state in which all the training cases are
associated with the same component, whose parameters are set to their
means, as specified by hyperparameter values that are also set to
their prior means. The 'mix-gen' program could be used to initialize
things differently (see mix-gen.doc).
The above run takes 0.75 seconds on the system used (see
Ex-test-system.doc). If it took longer, you could monitor its
progress with the 'mix-display' and 'mix-plt' programs. For example,
if you're quick, you might see the following:
> mix-display blog.4
MIXTURE MODEL IN FILE "blog.4" WITH INDEX 50
HYPERPARAMETERS
Standard deviations for component offsets:
0.342: 1.815 4.273 3.556 3.410 1.767
4.219 2.153 1.191 4.576 0.196
Means for component offsets:
+0.924 +1.093 -0.684 +1.170 -1.183
+0.949 -1.397 +0.131 -1.657 +1.074
PARAMETERS AND FREQUENCIES FOR COMPONENTS OF THE MIXTURE
1: 0.274 -1.693 +0.815 +1.113 +1.512 +1.024
-1.555 -0.914 -0.761 -1.035 +1.163
2: 0.270 +1.995 +1.008 +1.841 -2.204 -1.222
+2.118 +1.706 -1.676 -1.341 +0.990
3: 0.252 -1.307 -1.388 -2.530 -1.468 -1.207
+1.603 +2.047 +1.941 +2.373 +1.032
4: 0.204 +2.098 -1.174 -1.541 +1.916 +1.663
-1.531 -2.646 +1.010 +1.317 +1.349
This displays the state at the last iteration computed so far (here
iteration 50). The hyperparameter values are shown first, with
top-level hyperparameters on the left, before the colon. The
lower-level hyperparameters follow, ordered by the target attribute
that they pertain to. Values for the component parameters follow,
sorted by the fraction of training cases that they are currently
associated with. These fractions are the first numbers after 1:, 2:,
etc.; note that the actual mixing probabilities are not explicitly
represented, but are instead always integrated over. Any components
that are not associated with any training case are omitted from the
output, and are not explicitly represented in the state. The other
numbers shown for each component are the "offset" parameters for each
target attribute.
In this example, the simulation appears to have learned the essence of
the distribution by the iteration shown above. Recall that a positive
offset corresponds to the probability of a 1 being greater than 1/2, a
negative offset to the probability of a 1 being less than 1/2. The
four components shown above can thus be seen to correspond to the four
patterns used to generate the cases, as described earlier. Note that
the last of the offset standard deviation hyperparameters (for the
last target attribute) is quite small. This indicates that the model
has "learned" that the probabilities for the last target are the same
for all components, and hence can be modeled most efficiently at the
hyperparameter level, by the mean for that offset, rather than
separately for each component.
The indicators of which components are associated with each training
case can also be examined with 'mix-display', and of course iterations
other than the last can be viewed. See mix-display.doc for details.
One can also look at the progress of the simulation using 'mix-plt'.
For example, the following will produce a time-plot of the cumulative
proportions of the training cases associated with the four components
(as sorted by decreasing frequency):
> mix-plt t C1C2C3C4 blog.4 | plot
Here, 'plot' is some suitable plotting program. One can also just let
the output of 'mix-plt' go to standard output, and then examine the
numbers manually. Other quantities can also be plotted, as described
in mix-quantities.doc.
As well as examining quantities with 'mix-display' and 'mix-plt', one
can also produce a sample of cases from the predictive distribution
using 'mix-cases'. Predictive probabilities for test cases can be
found using 'mix-pred'. The following command gives the average log
probability for all the test cases, using the last 80 iterations:
> mix-pred pa blog.4 21:
Number of iterations used: 80
Number of test cases: 500
Average log probability of targets: -6.099+-0.069
For comparison, the average log probability for these test cases
using the true mixture distribution is -6.034.
A countably infinite mixture model for the binary data.
Even though the true distribution for this example is a mixture of
four components, good results can nevertheless be obtained using a
mixture with a countably infinite number of components. The prior for
mixture proportions used with such a model is designed so that a few
of the infinite number of components have substantial probability, so
the model does not result in an "overfitted" solution, in which every
training case is "explained" as coming from a different component.
We specify a countably infinite mixture model by simply omitting the
argument to 'mix-spec' that gives the number of components. For this
example, we change the 'mix-spec' command used above to the following:
> mix-spec blog.inf 0 10 / x1 0.05:0.5:0.2 10
The Dirichlet prior specification for mixing proportions of "x1" is
the same as for the mixture model with four components. The "x"
specifies scaling downward with the number of components, which
produces a sensible limit as the number of components goes to
infinity, as here. The "x" is therefore required for infinite
mixtures.
The other arguments of 'mix-spec' are as described for the finite
mixture model. The 'model-spec' and 'data-spec' commands used are
also the same:
> model-spec blog.inf binary
> data-spec blog.inf 0 10 2 / bdata@1:500 . bdata@501:1000 .
We must change the 'mc-spec' command, however, since it is impossible
to do Gibbs sampling for the component indicators associated with
training cases - since their number is infinite, we can't compute all
the required conditional probabilities. However, we can instead use a
specialized Metropolis-Hastings update, in which a new component to go
with a training case is proposed with probability determined by the
frequencies with which components are associated with other training
cases. The proposal is accepted or rejected based on the resulting
change in likelihood. The process can then be repeated any desired
number of times, to produce an approximation to Gibbs sampling (the
"approximation" is only with respect to convergence speed, the answer
is exact, asymptotically). With this change, we can use the following
'mc-spec' command for this model:
> mc-spec blog.inf repeat 20 met-indicators 10 gibbs-params gibbs-hypers
This is the same as for the finite mixture model, except that 10
repetitions of the Metropolis-Hastings update for the indicators are
specified using 'met-indicators 10', in place of 'gibbs-indicators'.
As before, we can now run the simulation with a command such as:
> mix-mc blog.inf 100
We can examine the states with 'mix-display'. For example, once the
'mix-mc' command completes, which takes 2.8 seconds on the system used
(see Ex-test-system.doc), the state should be something like the
following:
> mix-display blog.inf
MIXTURE MODEL IN FILE "blog.inf" WITH INDEX 100
HYPERPARAMETERS
Standard deviations for component offsets:
0.652: 3.090 1.238 1.875 1.723 2.406
3.148 5.278 1.911 2.290 0.277
Means for component offsets:
+0.322 +0.008 -1.202 -0.365 +0.911
-1.460 +0.081 +1.179 +0.458 +1.089
PARAMETERS AND FREQUENCIES FOR COMPONENTS OF THE MIXTURE
1: 0.292 -1.781 +0.988 +1.278 +1.351 +0.670
-1.254 -1.021 -0.865 -1.030 +1.493
2: 0.252 -1.900 -1.600 -2.164 -1.237 -1.618
+1.506 +1.906 +1.830 +1.822 +0.981
3: 0.248 +2.619 +0.830 +2.038 -2.486 -1.930
+1.892 +2.153 -1.425 -1.929 +0.947
4: 0.190 +2.327 -1.354 -1.353 +1.696 +1.724
-1.741 -3.573 +0.967 +1.332 +0.943
5: 0.014 +1.957 -2.620 +1.094 +2.526 +3.792
-2.249 +5.191 +0.864 +4.884 +0.975
6: 0.002 -5.484 -0.456 -2.541 +0.298 +3.347
+1.349 -4.903 +1.000 -0.070 +1.466
7: 0.002 -1.659 -0.611 -3.036 -3.022 -0.820
-3.068 -1.275 +1.828 +2.542 +0.534
The output is similar to that seen for the mixture model with four
components, except that seven components are associated with at least
one training case at iteration 100, as shown above. (The infinite
number of remaining components are not explicitly represented, and are
not displayed by 'mix-display'.) Three of these components are
associated with very few training cases, however (as seen from the
fractions of the training set after the component number). This is to
be expected when the true distribution can be expressed using only
four components. The number of such low-frequency components will
vary from iteration to iteration, as will other aspects of the state,
in accordance with the variability in the posterior distribution.
Since there are exactly four components in the real distribution, one
would expect that the model with four components would perform better
than a model with an infinite number of components. Any penalty is
quite small in this example, however, as can be seen by looking at the
predictive probabilities:
> mix-pred pa blog.inf 21:
Number of iterations used: 80
Number of test cases: 500
Average log probability of targets: -6.116+-0.070
This is almost the same as the average log probability for the four
component model.
A number of other ways of sampling for the indicators for countably
infinite mixture models are also available. Here are two:
> mc-spec blog.inf2 repeat 20 \
gibbs-ext-indicators 2 gibbs-params gibbs-hypers
> mc-spec blog.inf3 repeat 20 \
met1-indicators gibbs1-indicators gibbs-params gibbs-hypers
These methods may be more efficient than using the "met-indicators"
operation for some models. See mix-mc.doc for descriptions of these
and other Markov chain operations for mixture models.
A Dirichlet diffusion tree model for the binary data.
Binary data can also be modeled using Dirichlet diffusion trees. This
is not the simplest example of how Dirichlet diffusion trees can be
used, since the trees naturally produce real valued data. To model
binary data, these real values are treated as latent values, which are
put through the logistic function to produce the probabilities for the
binary data to be "1". The binary data example here does not have the
sort of hierarchical structure that would make use of a Dirichlet
diffusion tree model desirable. For both these reasons, the following
example of how to model binary data with a Dirichlet diffusion tree
may not be the best introduction to such models - you might want to
read the real-valued example in Ex-mixdft-r.doc first.
Nevertheless, this example does demonstrate how binary data can be
modeled, and shows that even when there is no hierarchical structure
to the data, a Dirichlet diffusion tree model can do quite well.
We start by specifying a Dirichlet diffusion tree model, as follows:
> dft-spec blog.dft 0 10 / 2 - 1
As for a mixture model specification, this says that there are 0
"input" variables (as there must be at present), and 10 "target"
variables. The argument of "2" is the standard deviation for the
diffusion process that produces the latent variables used to model the
binary variables (this could have been a prior specification rather
than a fixed value). The last two arguments are the parameters of the
divergence function used to determine (stochastically) when the
diffusion paths for different cases diverge. The general form of the
divergence function is
a(t) = c0 + c1/(1-t) + c2/(1-t)^2
The parameters c0, c1, c2 are given at the end of the 'dft-spec'
command, with "-" being equivalent to zero. Any zero parameters at
the end can just be omitted. The specification above therefore
corresponds to the divergence function a(t) = 1/(1-t). Any of c0, c1,
and c2 can be a prior specification, instead of a constant. See
dft-spec.doc for details, and Ex-mixdft-r.doc for an example.
We next specify the data model, which is "binary", indicating binary
data modeled using the logistic function:
> model-spec blog.dft binary
We go on to specify the source of the data, as for the mixture example
above:
> data-spec blog.dft 0 10 2 / bdata@1:500 . bdata@501:1000 .
Finally, we need to specify how to do Markov chain sampling for the
posterior distribution of the trees and latent values that underlie
the data.
> mc-spec blog.dft repeat 10 gibbs-latent slice-positions met-terminals
Since the hyperparameters are all fixed for this model, there is no
need for Markov chain operations to update them. The operations above
will lead to the state of the Markov chain being only the latent
values and the tree structure, with divergence times, but without the
locations of non-terminal nodes (which will be integrated over). The
'gibbs-latent' operation does a Gibbs sampling scan over the latent
values associated with the targets in training cases. (Initially, the
latent values are taken to be +1 for targets of "1", and -1 for
targets of "0".) The 'slice-positions' and 'met-terminals' operations
both update the tree structure and divergence times. See dft-mc.doc
for details.
We can run the Markov chain for 100 iterations with the following
command:
> dft-mc blog.dft 100
We can see the progress of the Markov chain by examining various
quantities, such as the latent values associated with training cases.
The following command will plot the latent values for the first target
for the first six training cases:
> dft-plt t o1@1:6 blog.dft | plot
The first four of these cases have "0" for their first target value,
the last two have "1". The plot should show that the latent values
for the first four cases tend to be negative, whereas the latent
values for the last two cases tend to be positive, at least once the
Markov chain has approached convergence.
The 'dft-mc' command above takes 19 seconds on the system used (see
Ex-test-system.doc). Once it is done, we can see how well the model
predicts new data using the following command, which looks at the
Dirichlet diffusion tree models produced by the last 60 iterations of
the Markov chain:
> dft-pred pa blog.dft 41:
Number of iterations used: 60
Number of test cases: 500
Average log probability of targets: -6.110+-0.066
This takes 9.1 seconds on the system used. Performance is not much
worse than for the four-component mixture model above, which is based
on the true structure of the data.