Syllabus and Course Schedule

Some of the following slides will be updated soon.

Week

Lecture

Topics

References

Announcements

Week 1

Introduction to ML

[slides]

What is machine learning?

What are the basic elements of machine learning?

Introduction to decision trees and instance-based learning.

Book: Pattern Recognition and Machine Learning, Bishop [link]

Book: Probabilistic Reasoning in Intelligent Systems, Judea Pearl [link]

Book: Probabilistic Graphical Models, D. Koller et al [link]

 

Week 2

MLE, Bayesian and MAP Estimation

[slides]

What is MLE?

What is Bayesian inference? How to choose the priors?

What is MAP estimation?



 

Week 3

Generative and Discriminative Learning

[slides]

What are generative models?

What are discriminative models?

Naïve Bayes and logistic regression.

Theoretical analysis of generative and discriminative models.


Hw1 is out.

Week 4

Learning Theory

[slides]

Bias-Variance Tradeoff

Model complexity and its influence on generalization performance

VC theory


 

Week 5

Support vector machines I

[slides]

Geometric view of margin and the basic principle

Hard margin and soft margin support vector machines

Multi-class support vector machines

Basics of constrained optimization



Hw2 is out.

Week 6

Support Vector Machines II

[slides]

The kernel trick

Structured Support Vector Machines

Book: The nature of statistical learning theory, V. N. Vapnik [link]

A Tutorial on Support Vector Machines for Pattern Recognition, Christopher J.C. Burges [link]

An Introduction to Support Vector Machines, S. John et al. [link]

 

Week 7

Online Learning and the Perceptron Algorithm

[slides]

Online learning algorithms

The perceptron algorithm


 

Week 8

Clustering and EM

[slides]

K-means

Gaussian Mixture Models

EM algorithms


A Gentle Tutorial on the EM Algorithm and its Application to Parameter Estimation for Gaussian Mixture and Hidden Markov Models, J. A. Bilmes [link]

 

Week 9

HMM and Graphical Models I

[slides]

HMM and general graphical model representations

Exact inference algorithms


Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data, J. Lafferty et al. [link]

A maximum entropy approach to natural language processing, A. L. Berger et al. [link]

Maximum Entropy Markov Models for Information Extraction and Segmentation, A. McCallum et al. [link]

Hw3 is out.

Week 10

Graphical Models II

[slides]

Parameter learning and structure learning

Inference with latent variables

Variational Methods

Book: Graphical Models, Exponential Families, and Variational Inference, M. J. Wainwright et al. [link]

 

Week 11

Basics of Nonparametric Bayesian Methods and DP Mixtures

[slides]

Basics of nonparametric methods and nonparametric Bayesian methods

Finite Bayesian Gaussian mixture models

Dirichlet process mixture models


Markov chain sampling methods for Dirichlet process mixture models, R. M. Neal [link]

Variational inference for Dirichlet process mixtures, D. M. Blei et al. [link]

Hierarchical Dirichlet processes Y. W. Teh et al. [link]

Hw4 is out.

Week 12

Monte Carlo Methods

[slides]

Basic Monte Carlo sampling methods

Markov chain Monte Carlo methods


Chap. 11 of Pattern Recognition and Machine Learning, Bishop [link]

Introduction to Monte Carlo Methods. D.J.C. [link]

 

Week 13

Topic Models

[slides]

Latent semantic analysis and probabilistic latent semantic analysis

Hierarchical Bayesian topic models

Latent Dirichlet Allocation, JMLR, D. M. Blei et al. [link]

Finding Scientific Topics, T. L. Griffiths et al. [link]

Estimating a Dirichlet Distribution, T. P. Minka [link]

 

Week 14

Feature Selection and Sparse Learning in High Dimensions

[slides]

Basic methods for feature selection

Sparse learning with L1 norms

Learning with structured sparsity

Regression Shrinkage and Selection via the Lasso, R. Tibshiranit [link]

Optimization with Sparsity-Inducing Penalties, F. Bach et al. [link]

 

Week 15

Advanced topics and Summary

[slides]


Emergence of simple-cell receptive field properties by learning a sparse code for natural images, B. A. Olshausen [link]

Online Learning for Matrix Factorization and Sparse Coding, J. Mairal et al. [link]

 

Week 16

Project Presentations