Syllabus and Course Schedule

Some of the following slides will be updated soon.






Week 1

Introduction to ML


What is machine learning?

What are the basic elements of machine learning?

Introduction to decision trees and instance-based learning.

Book: Pattern Recognition and Machine Learning, Bishop [link]

Book: Probabilistic Reasoning in Intelligent Systems, Judea Pearl [link]

Book: Probabilistic Graphical Models, D. Koller et al [link]


Week 2

MLE, Bayesian and MAP Estimation


What is MLE?

What is Bayesian inference? How to choose the priors?

What is MAP estimation?


Week 3

Generative and Discriminative Learning


What are generative models?

What are discriminative models?

Naïve Bayes and logistic regression.

Theoretical analysis of generative and discriminative models.

Hw1 is out.

Week 4

Learning Theory


Bias-Variance Tradeoff

Model complexity and its influence on generalization performance

VC theory


Week 5

Support vector machines I


Geometric view of margin and the basic principle

Hard margin and soft margin support vector machines

Multi-class support vector machines

Basics of constrained optimization

Hw2 is out.

Week 6

Support Vector Machines II


The kernel trick

Structured Support Vector Machines

Book: The nature of statistical learning theory, V. N. Vapnik [link]

A Tutorial on Support Vector Machines for Pattern Recognition, Christopher J.C. Burges [link]

An Introduction to Support Vector Machines, S. John et al. [link]


Week 7

Online Learning and the Perceptron Algorithm


Online learning algorithms

The perceptron algorithm


Week 8

Clustering and EM



Gaussian Mixture Models

EM algorithms

A Gentle Tutorial on the EM Algorithm and its Application to Parameter Estimation for Gaussian Mixture and Hidden Markov Models, J. A. Bilmes [link]


Week 9

HMM and Graphical Models I


HMM and general graphical model representations

Exact inference algorithms

Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data, J. Lafferty et al. [link]

A maximum entropy approach to natural language processing, A. L. Berger et al. [link]

Maximum Entropy Markov Models for Information Extraction and Segmentation, A. McCallum et al. [link]

Hw3 is out.

Week 10

Graphical Models II


Parameter learning and structure learning

Inference with latent variables

Variational Methods

Book: Graphical Models, Exponential Families, and Variational Inference, M. J. Wainwright et al. [link]


Week 11

Basics of Nonparametric Bayesian Methods and DP Mixtures


Basics of nonparametric methods and nonparametric Bayesian methods

Finite Bayesian Gaussian mixture models

Dirichlet process mixture models

Markov chain sampling methods for Dirichlet process mixture models, R. M. Neal [link]

Variational inference for Dirichlet process mixtures, D. M. Blei et al. [link]

Hierarchical Dirichlet processes Y. W. Teh et al. [link]

Hw4 is out.

Week 12

Monte Carlo Methods


Basic Monte Carlo sampling methods

Markov chain Monte Carlo methods

Chap. 11 of Pattern Recognition and Machine Learning, Bishop [link]

Introduction to Monte Carlo Methods. D.J.C. [link]


Week 13

Topic Models


Latent semantic analysis and probabilistic latent semantic analysis

Hierarchical Bayesian topic models

Latent Dirichlet Allocation, JMLR, D. M. Blei et al. [link]

Finding Scientific Topics, T. L. Griffiths et al. [link]

Estimating a Dirichlet Distribution, T. P. Minka [link]


Week 14

Feature Selection and Sparse Learning in High Dimensions


Basic methods for feature selection

Sparse learning with L1 norms

Learning with structured sparsity

Regression Shrinkage and Selection via the Lasso, R. Tibshiranit [link]

Optimization with Sparsity-Inducing Penalties, F. Bach et al. [link]


Week 15

Advanced topics and Summary


Emergence of simple-cell receptive field properties by learning a sparse code for natural images, B. A. Olshausen [link]

Online Learning for Matrix Factorization and Sparse Coding, J. Mairal et al. [link]


Week 16

Project Presentations