Machine Learning with Exponential Families


Machine learning is an exciting new subject dealing with automatic recognition of patterns (e.g. automatic recognition of faces, postal codes on envelopes, speech recognition), prediction (e.g. predicting the stock market), datamining (e.g. finding good customers, fraudulent transactions), applications in bioinformatics (finding relevant genes, annotating the genome, processing DNA Microarray images), or internet-related problems (spam filtering, searching, sorting, network security). It is becoming a key area for technical advance and people skilled in this area are worldwide in high demand.

Formal Description

This unit introduces the fundamentals of machine learning, based on the unifying framework of exponential families. The course requires mathematical and computer skills. It will cover linear algebra and numerical mathematics techniques, linear classification, regression, mathematical programming, the Perceptron, Graphical Models, Boosting, density estimation, conditional random fields, Regression methods, Kernels and Regularization.


Prof. Stephane Canu
Dr. Alexander J. Smola
Dr. Vishy Vishwanathan
RSISE, Australian National University

Canberra, ACT 0200

Phone : (+61) 2 6125-8652
Cellular: (+61) 410 457 686
Fax: (+61) 2 6125-8651

[email protected]

Dates of the Course

The course will be held on May 4 - July 2, with lectures every Tuesday and Thursday at 2pm-4pm in the video conferencing facility of the Crisp lecture theater. Completion of the assignment sheets is optional, however I strongly encourage students too solve the problems.

The assessment will be in the form of oral exams, probably one week after the end of the lectures.


  • Fourier Transforms
  • Linear Algebra (eigenvalues, matrix inverses, etc.)
  • programming skills if you want to apply the methods The first week will be a quick refresher course in linear algebra and statistics.



  • Week 1: Introduction and Mathematical Basics ( Lecture 1 and Lecture 2 )
  • Week 2: Density Estimation and Exponential Families ( Lecture 1 and Lecture 2 )
  • Week 3: Kernels ( Lecture 1 and 2 )
  • Week 4: Classification and Regression ( Lecture 1 and 2 )
  • Week 5: Optimization ( Lecture 1 and 2 )
  • Week 6: Graphical Models
  • Week 7: Graphical Models
  • Week 8: Conditional Random Fields
  • Week 9: Boosting


  • Assignment 1: Introduction to conditional probabilities (Assignment 1)
  • Assignment 2: Classical pattern recognition algorithms (Assignment 2)
  • Assignment 3: Prior knowledge (Assignment 3)

    Scribe Notes

  • Each student is expected to scribe one or two lectures of vishy
  • The LaTeX template to use is here
  • Use this file for definitions Definitions.tex. Unless you really need it do *not* define new definitions. Please discuss with vishy before making changes to this file!
  • Scribe notes are due 2 weeks after the lecture and are a part of your assesment.


  • MATLAB. Here's a quick start guide.
  • SVLab Toolbox
  • GCC. See Brian Kernigham's tutorial or a list of other C tutorials.
    Last modified May 17, 2004