Linear Methods

Slides in Keynote and PDF.

Content

• Perceptron

  • Neurons and Learning
  • Perceptron learning rule
  • Convergence Analysis

• Nonlinearity and Kernels

  • Preprocessing
  • Perceptron on features
  • Kernels and Kernel Perceptron

• Support Vector machines

  • Large Margin Classification
  • Soft Margin Loss
  • Kernel Trick
  • Regression
  • Novelty Detection

• Efficient Implementation

  • Optimization and active sets
  • Hash kernels for high-dimensional data

Supplementary material

• F. Rosenblatt, The perceptron: A probabilistic model for information storage and organization in the brain, Psychological Review, Vol 65(6), Nov 1958, 386-408.
• Y. Freund and R. Schapire, Large margin classification using the perceptron algorithm, In Machine Learning 37(3):277-296, 1999.
• N. Aronszajn, Theory of reproducing Kernels, Trans. Amer. Math. Soc. 68 (1950), 337-404
• G. Wahba, Spline Models for Observational Data, SIAM 1990
• B. Scholkopf, A. Smola, K. Muller, Nonlinear Component Analysis as Kernel Eigenvalue Problem, Neural Computation, 10:1299-1319, 1998
• S. Mika et al, Kernel-PCA and Denoising in Feature Spaces, NIPS 1999
• B. Scholkopf et al, Support Vector Method for Novelty Detection, Neural Computation, 2000
• A. Smola, L. Song, C.H. Teo, Relative Novelty Detection, NIPS 2009
• B. Scholkopf et al., Shrinking the Tube: A New Support Vector Regression Algorithm, Neural Computation, 1999

Videos