Calculus

Partial Derivatives

Linear

Functions with more than one variable. e.g. $f(x)$ where $x\in {\mathbb{R}}^3$ the following multivariate linear

$$\begin{array}{c}\begin{array}{rl}f(x_1,x_2,x_3)& =a_1{x}_1+a_2{x}_2+a_3{x}_3+b\\ & =\sum _{i=1}^3{a}_i{x}_i+b\\ & =a^{T}x+b\end{array}\end{array}$$

The gradient is then the partial derivative with respect to each variable

$\nabla f(x)=\left[\begin{array}{c}\frac{\partial f}{\partial x_1}\\ \frac{\partial f}{\partial x_2}\\ \frac{\partial f}{\partial x_3}\end{array}\right]=\left[\begin{array}{c}{a}_1\\ a_2\\ a_3\end{array}\right]$

Quadratic

e.g. $f(x)$ where $x\in {\mathbb{R}}^2$ and $A=\left[\begin{array}{cc}2& -1\\ -1& 1\end{array}\right]$

$$\begin{array}{c}\begin{array}{rl}f(x)& =\frac{1}{2}{x}^{T}Ax+b^{T}x+c\\ & =\sum _{i=1}^2\sum _{j=1}^2{a}_{ij}{x}_i{x}_j+\sum _{i=1}^2{b}_i{x}_i+c\end{array}\end{array}$$

If $A$ is symmetric, $\nabla f(x)=Ax+b$. In the non-symmetric case, $\nabla f(x)=\frac{1}{2}(A+A^T)x+b$

Generalizations of gradients for $d$ dimensions given (expressed as matrices and vectors):

1. $A=X^{T}X$
2. $b$ = $X^{T}y$
3. $c=\frac{1}{2}{y}^{T}y$

So

1. $\nabla \frac{1}{2}{w}^{T}Aw=Aw$ (if A is symmetric)
2. $\nabla w^{T}b=b$
3. $\nabla c=0$