Bayesian Mixture Modeling by Monte Carlo Simulation

Radford M. Neal, Dept. of Computer Science, University of Toronto

It is shown that Bayesian inference from data modeled by a mixture distribution can feasibly be performed via Monte Carlo simulation. This method exhibits the true Bayesian predictive distribution, implicitly integrating over the entire underlying parameter space. An infinite number of mixture components can be accommodated without difficulty, using a prior distribution for mixing proportions that selects a reasonable subset of components to explain any finite training set. The need to decide on a "correct" number of components is thereby avoided. The feasibility of the method is shown empirically for a simple classification task.

Technical Report CRG-TR-91-2 (June 1991), 23 pages: postscript, pdf.

Associated reference: A condensed version of ``Bayesian mixture modeling by Monte Carlo simulation'' appeared as the following workshop paper:
Neal, R. M. (1992) ``Bayesian mixture modeling'', in C. R. Smith, G. J. Erickson, and P. O. Neudorfer (editors) Maximum Entropy and Bayesian Methods: Proceedings of the 11th International Workshop on Maximum Entropy and Bayesian Methods of Statistical Analysis, Seattle, 1991, pp. 197-211, Dordrecht: Kluwer Academic Publishers: abstract.
The models in this paper with a countably-infinite number of components are equivalent to what are called Dirichlet process mixture models. Newer work on these is described in the following technical reports:
Neal, R. M. (1998) ``Markov chain sampling methods for Dirichlet process mixture models'', Technical Report No. 9815, Dept. of Statistics, University of Toronto, 17 pages: abstract, postscript, pdf, associated references, associated software.

Jain, S. and Neal, R. M. (2000) ``A Split-Merge Markov Chain Monte Carlo Procedure for the Dirichlet Process Mixture Model'', Technical Report No. 2003, Dept. of Statistics (July 2000), 32 pages: abstract, postscript, pdf, associated references.