Note: The test on December 8 at 3pm will be held in BA B024, not the usual lecture/tutorial room.
Instructor:
Radford Neal, Office: SS6016A, Phone: (416) 978-4970, Email: radford@cs.utoronto.ca, Office hours: Tuesdays 2:10-3:00.
Lectures:
Mondays and Wednesdays, 3:10pm to 4:00pm, from September 11 to December 6, except for Thanksgiving (October 9). Held in BA1220.
Tutorials:
Fridays, 3:10pm to 4:00pm, from September 22 to December 8. Held in BA1220.TA: Renqiang Min
NOTE: Tests will be held during the tutorial time, so you must not have a time conflict with the tutorial.
Evaluation:
Three in-class tests, worth 20% each. Tentatively scheduled for October 13, November 17, and December 8.Four assignments, worth 10% each. Tentatively due on October 11, November 1, November 15, and November 29 (handed out two weeks earlier).
Textbook:
There is no textbook for this course. Lecture slides will be available from this web page. I will also post links to various on-line references.
Computing:
The assignments will involve writing programs in Matlab/Octave or R. You can download Octave and R for your home computer for free from their web sites. You can also obtain an account on the CS department's CDF computer system, where R and Matlab are available - just type "R" or "matlab" (or "matlab -nosplash -nojvm" for a less GUI version).
Lecture slides:
The slides are available in Postscript (ps) and Portable Document Format (pdf), in both one-per-page and four-per-page (4up) versions.Introduction: ps, pdf, ps-4up, pdf-4up
Nearest-neighbor and linear regression: ps, pdf, ps-4up, pdf-4up
Probability and loss functions: ps, pdf, ps-4up, pdf-4up
Naive Bayes classifiers: ps, pdf, ps-4up, pdf-4up
Logistic regression (revised and extended): ps, pdf, ps-4up, pdf-4up
Decision trees (revised and extended): ps, pdf, ps-4up, pdf-4up
Cross validation: ps, pdf, ps-4up, pdf-4up
Neural networks: ps, pdf, ps-4up, pdf-4up
Maximum likelihood estimation for neural networks: ps, pdf, ps-4up, pdf-4up
Training neural networks with early stopping: ps, pdf, ps-4up, pdf-4up
Bayesian learning: ps, pdf, ps-4up, pdf-4up
Bayesian neural networks: ps, pdf, ps-4up, pdf-4up
Markov chain Monte Carlo: ps, pdf, ps-4up, pdf-4up
Clustering: ps, pdf, ps-4up, pdf-4up
Mixture models (revised and extended): ps, pdf, ps-4up, pdf-4up
Principal Component Analysis (revised): ps, pdf, ps-4up, pdf-4up
Factor Analysis (extended): ps, pdf, ps-4up, pdf-4up
Nonlinear dimensionality reduction: ps, pdf, ps-4up, pdf-4up
Assignments
Assignment 1:Handout: ps, pdf.Assignment 2:
Due date has been extended to October 18.
Data set 1: train x, train y, test x, test y.
Data set 2: train x, train y, test x, test y.
Maximum likelihood logistic regression in R: all functions.
Maximum likelihood logistic regression in Matlab: estimation, likelihood, prediction.
Fudged functions needed for the old Matlab on CDF: fudged estimation, fudged minus log likeihood.
Solution: R functions for ordinary LR, R functions for bounded LR, R test script, output of script, plots produced by script.Handout: ps, pdf.Assignment 3:
Data files: train x, train y, test x, test y.
Solution: MLP functions, script, output, plots.Handout: ps, pdf.Assignment 4:
Here are some notes on how to do some things you'll need to do.
Here is the data. Note: I mistakenly put only 49 cases in the data file. That's OK. Just use 49 rather than 50.
Solution: Main functions, script, output plots.Handout: ps, pdf.
Here is the gene expression data and the cancer indicators.
Demonstration programs
Demo of K-NN and linear regression in R using prostate cancer data: knn program, script, training data, test data.
Demo of Naive Bayes in Matlab/Octave: nbayes program, script, training inputs, training targets, test inputs, test targets.
Demo of decision trees using the R "rpart" function: R script, output of script.
K means function and example: R function for K means, script to try it out.
Q learning program and demo: Q learning functions, Demo 1, Demo 2, Demo 2m.
These programs are also available on CDF in /u/radford/411/demo.
Some useful on-line references
Proceedings of the annual conference on Neural Information Processing Systems (NIPS)
Information Theory, Inference, and Learning Algorithms, by David MacKay
Reinforcement Learning: An Introduction, by Richard S. Sutton and Andrew G. Barto
My tutorial on Bayesian methods for machine learning: Postscript or PDF.
Web pages for past related courses:
STA 414 (Spring 2006)
CSC 321 (Spring 2006, Geoffrey Hinton)
CSC 411 (Fall 2005, Anthony Bonner)
CSC 411 (Fall 2004, Richard Zemel)