Machine Learning
Home
FAQ
Syllabus
Resources
Contact
Google Group
Lectures
1 Introduction
2 Basic Statistics
3 Instance based methods
4 Perceptron
5 Support Vectors
6 Kernels
7 Kernels
8 Stochastic Convergence
9 Tail Bounds
10 Risk Minimization
11 Learning Theory
12 Gaussian Processes
13 Exponential Families
14 PCA
15 Graphical Models
16 Dynamic Programming
17 Latent Variables
18 Sampling
19 Information Theory
20 Trees
21 Neural Networks
22 Boosting
23 Kalman Filter
24 Reinforcement Learning
25 Scalability
Projects
Recitations
1 Statistics
2 Linear Algebra
3 Optimization
4 Model Selection
5 Convergence Bounds
6 Kernels
CMU
Alex Smola
Barnabas Poczos
Ina Fiterau
Junier Oliva
Leila Wehbe
Mu Li
Xuezhi Wang
MLD
CMU

Information Theory

Introduction to Machine Learning - 10-701/15-781

Content

  • Entropy

    • Basic properties, decomposition

    • Kullback-Leibler Divergence (with properties)

    • Mutual Information

  • Examples

    • Exponential families

    • Gaussian mutual information as special case

  • Application - cocktail party problem

    • Independent Component Analysis

    • JADE, Radical, InfoMax, and FastICA (time permitting)

Supplementary material

Slides in PDF and Keynote coming soon. If you want to extract the equations from the slides you can do so by using LaTeXit, simply by dragging the equation images into it.

Videos

This is unedited video straight from a Lumix GF2 with a 20mm lens which should explain the sound (it doesn't have a dedicated audio input) … But it should help as a supplement with the slides (YouTube typically makes the 1080i version available within 1 week of the upload).