KL Divergence¶
In one minute
The Kullback-Leibler divergence compares two probability distributions. It is not symmetric, so it is not a proper distance; the page shows this with a Gaussian and a bimodal Gaussian mixture. It is a special case of the f-divergence.
The Kullback–Leibler (KL) divergence is defined as
Suppose \(p\) is a Gaussian distribution and \(q\) is a bimodal Gaussian mixture, the KL divergence \(\operatorname{D}_\mathrm{KL}(p \parallel q )\) and \(\operatorname{D}_\mathrm{KL}(q \parallel p )\) are different as KL divergence is not necessarily symmetric. Thus the KL divergence is not a proper distance definition.

KL divergence is a special case of f-divergence.
Try It: The Two Directions of the KL Divergence¶
As in the figure above, \(p\) is a Gaussian and \(q\) is a mixture with two modes. Move \(p\), or the modes of \(q\), and compare \(\operatorname{D}_\mathrm{KL}(p \parallel q)\) with \(\operatorname{D}_\mathrm{KL}(q \parallel p)\). The lower chart shows where each of the two integrals collects its value.
Related: f-Divergence · Entropy · Mutual Information