Fundamental Tradeoff

The fundamental tradeoff has two parts:

1. How small you can make the training error
2. How well training error approximates the test error.

$\begin{array}{rl}{E}_{\text{test}}& =E_{\text{approx}}+E_{\text{train}}\\ & =(E_{\text{test}}-E_{\text{train}})+E_{\text{train}}\end{array}$

Generally,

• Training error goes down when the model gets more complicated, however
• Approximation error goes up when the model gets more complicated