Machine Learning with Exponential Families

Australian National University, 2004

Overview

Machine learning is an exciting new subject dealing with automatic recognition of patterns (e.g. automatic recognition of faces, postal codes on envelopes, speech recognition), prediction (e.g. predicting the stock market), datamining (e.g. finding good customers, fraudulent transactions), applications in bioinformatics (finding relevant genes, annotating the genome, processing DNA Microarray images), or internet-related problems (spam filtering, searching, sorting, network security). It is becoming a key area for technical advance and people skilled in this area are worldwide in high demand.

Formal Description

This unit introduces the fundamentals of machine learning, based on the unifying framework of exponential families. The course requires mathematical and computer skills. It will cover linear algebra and numerical mathematics techniques, linear classification, regression, mathematical programming, the Perceptron, Graphical Models, Boosting, density estimation, conditional random fields, Regression methods, Kernels and Regularization.

Presenters

  • Prof. Stephane Canu
  • Dr. Alexander J. Smola
  • Dr. Vishy Vishwanathan

RSISE, Australian National University, Canberra, ACT 0200

Dates of the Course

The course will be held on May 4 – July 2, with lectures every Tuesday and Thursday at 2pm–4pm in the video conferencing facility of the Crisp lecture theater. Completion of the assignment sheets is optional, however I strongly encourage students to solve the problems.

The assessment will be in the form of oral exams, probably one week after the end of the lectures.

Prerequisites

  • Fourier Transforms
  • Linear Algebra (eigenvalues, matrix inverses, etc.)
  • Programming skills if you want to apply the methods

The first week will be a quick refresher course in linear algebra and statistics.

Contents

Lectures

  • Week 1: Introduction and Mathematical Basics (Lecture 1 and Lecture 2)
  • Week 2: Density Estimation and Exponential Families (Lecture 3 and Lecture 4)
  • Week 3: Kernels (Lecture 5)
  • Week 4: Classification and Regression (Lecture 6)
  • Week 5: Optimization (Lecture 7)
  • Week 6: Graphical Models
  • Week 7: Graphical Models
  • Week 8: Conditional Random Fields
  • Week 9: Boosting

Assignments

  • Assignment 1: Introduction to conditional probabilities (Assignment 1)
  • Assignment 2: Classical pattern recognition algorithms (Assignment 2)
  • Assignment 3: Prior knowledge (Assignment 3)

Scribe Notes

  • Each student is expected to scribe one or two lectures of Vishy.
  • The LaTeX template to use is here.
  • Use this file for definitions: Definitions.tex. Unless you really need it, do not define new definitions. Please discuss with Vishy before making changes to this file.
  • Scribe notes are due 2 weeks after the lecture and are a part of your assessment.

Software