Example of an Ensemble method. They are non-parametric
They work by taking a vote from a set of deep decision trees. Two key ingredients to help ensure the deep decision trees make independent errors
- Bootstrap sampling: generate different “versions” of your dataset
- Usually done by sampling with replacement times, this creates a bootstrap sample
- On average, this maintains roughly the same distribution as the original
- Random Trees: grow decision trees that incorporates some randomness
- Randomly sample a small number of possible features (typically )
- Only consider these random features when searching for the optimal rule so splits will tend to use different features in different trees