Exponential Families and Kernels

MLSS 2004

Abstract

In this course I will discuss how exponential families, a standard tool in statistics, can be used with great success in machine learning to unify many existing algorithms and to invent novel ones quite effortlessly. In particular, I will show how they can be used in feature space to recover Gaussian Process classification for multiclass discrimination, sequence annotation (via Conditional Random Fields), and how they can lead to Gaussian Process Regression with heteroscedastic noise assumptions.

Prerequisites

• Elementary Linear Algebra
• Calculus
• Experience with Bayesian Methods is beneficial, however not required
• Experience with Kernel Methods is likewise beneficial, but not required
• A background in statistics would be useful, however it is not required

Contents

• Unit 1: Exponential Families
• Unit 2: Conditioning and Feature Spaces
• Unit 3: Applications 1 (Classification, Regression, Novelty Detection)
• Unit 4: Applications 2 (Graphical Models and Conditional Random Fields)