Gaussian RBFs

Non-parametric basis. Can think about it as a sum of gaussian ‘bumps’.

Replace $x_i=(x_{i1},x_{i2},\dots ,x_{in})$ with

$z_i=\underset{\text{n features}}{\underbrace{(g(\Vert x_i-x_1\Vert ),g(\Vert x_i-x_2\Vert ),\dots ,g(\Vert x_i-x_n\Vert ))}}$

where $g(x)=\exp (-\frac{x^2}{2{\sigma }^2})$

Gaussian RBFs are universal approximators

• Enough bumps can approximate any continuous function to arbitrary precision.
• Achieve optimal test error as ‘n’ goes to infinity.