Math and Optimization

Slides in Keynote and PDF.

Content

• Linear Algebra

  • Vectors

  • Norms & inner products

  • Linear Independence

  • Rotations

  • Matrices

• Unconstrained Optimization

  • Convexity 101

  • Gradient Descent

  • Distributed Implementation

  • Newton's Method

  • Broyden-Fletcher-Goldfarb-Shanno

• Constrained Optimization

  • Basic Convexity

  • Linear & Quadratic Programs

  • Interior Point Solvers

  • Bundle Methods

• Online Optimization

  • Stochastic Gradient Descent

• Discrete Problems

Supplementary material

• Nesterov and Vial (expected convergence) http://dl.acm.org/citation.cfm?id=1377347

• Bartlett, Hazan, Rakhlin (strong convexity SGD) http://books.nips.cc/papers/files/nips20/NIPS2007_0699.pdf

• TAO (toolkit for advanced optimization) http://www.mcs.anl.gov/research/projects/tao/

• Ratliff, Bagnell, Zinkevich http:martin.zinkevich.orgpublicationsratliff_nathan_2007_3.pdf]

• Shalev-Shwartz, Srebro, Singer (Pegasos paper) http://dl.acm.org/citation.cfm?id=1273598

• Langford, Smola, Zinkevich (slow learners are fast) http://arxiv.org/abs/0911.0491

• Hogwild (Recht, Wright, Re) http://pages.cs.wisc.edu/~brecht/papers/hogwildTR.pdf

Videos

The content of chapters 1 and 2 was covered in the recitations.