Deep Networks

Slides in Keynote and PDF.

Content

• Overview

  • CPU performance, data, and the sweet spot for algorithms

  • Perceptron and going nonlinear (wide or deep)

  • Backpropagation

• Layers

  • Fully connected

  • Convolutions

  • Invariances for images

  • Whole system training

• Objective functions

  • Classification and SoftMax

  • Regression

  • Autoencoder

  • Contrastive Estimation

  • Invariances

• Optimization

  • Stochastic Gradient Descent

  • Learning rates, AdaGrad, Minibatches

  • Momentum

  • Dropout and DropConnect for regularization

• Memory

  • Recurrent Networks

  • Hidden Markov Models

  • Long Short Term Memory

  • Memory Networks

Supplementary material

• Nair and Hinton, 2010, Rectified Linear Unites

• Simonyan and Zisserman, 2014, Narrow and deep beats wide and shallow

• Szegedy et al., 2014 Inception Layer in GoogLeNet

• Le Cun, Bottou, Bengio, Haffner, 2001 Whole system training

• Grefenstette et al, 2014 Autoencoder between domains

• Senior, Heigold, Ranzato and Yang, 2013 Learning Rate Comparison

• Duchi, Hazan, Singer, 2010 AdaGrad

• Srivastava, Hinton, Krizhevski, Sutskever, Salakhutdinov, 2015 Dropout

• Graves, 2013 LSTM Tutorial

• Graves, Wayne, Danihelka, 2014, Neural Turing Machine

• Weston, Chopra, Bordes, 2014, Memory Networks

Video