Generalization

To measure the generalization, we define a generalization error¹,

$$\mathcal{G}={\mathcal{L}}_P(\hat{f})-{\mathcal{L}}_E(\hat{f}),$$

where ${\mathcal{L}}_P$ is the population loss, ${\mathcal{L}}_E$ is the empirical loss, and $\hat{f}$ is our model by minimizing the empirical loss.

However, we do not know the actual joint probability $p(x,y)$ of our dataset ${x}_i,y_i$. Thus the population loss is not known. In machine learning, we usually use cross validation where we split our dataset into train and test dataset. We approximate the population loss using the test dataset.

---

1. Roelofs R. Measuring generalization and overfitting in machine learning. 2019.https://escholarship.org/uc/item/6j01x9mz. ↩

---

Contributors: LM