Maximum a Posteriori (MAP) Estimation

Maximizes $\hat{w}\in \arg {\max }_w\{P(w|D)\}$

Given our data, which model $w$ is the best model?

This is connected to MLE through Bayes’ Rule:

$P(w|D)=\frac{P(D|w)P(w)}{P(D)}\propto P(D|w)P(w)$

Intuitively, $P(w)$ is accounting for how ‘likely’ this model is. We can also treat this as a regularizer.

$$\begin{array}{rl}\hat{w}\in \arg \underset{w}{\max }\{P(w|D)\}& \equiv \arg \underset{w}{\max }\{\prod _{i=1}^{n}P(D_i|w)P(w)\}\\ & \equiv \arg \underset{w}{\min }\{-\sum _{i=1}^n\log (P(D_i|w))-\log (P(w))\}\end{array}$$

Where $-\log (P(w))$ acts like the regularizing term. In fact, many regularizers are equivalent to negative log-priors.

Relation between regularized loss functions

L2-Regularized Least Squares

If we assume a Gaussian likelihood and a Gaussian prior, then MAP estimation is equivalent to minimizing $f(w)=\frac{1}{2}\Vert Xw-y{\Vert }^2+\frac{\lambda }{2}\Vert w{\Vert }^2$

L2-Regularized Robust Regression

If we assume a Laplace likelihood and a Gaussian prior, then MAP estimation is equivalent to minimizing $f(w)=\Vert Xw-y{\Vert }_1+\frac{\lambda }{2}\Vert w{\Vert }^2$