Skip to content

Mutual Information

In one minute

Mutual information of two random variables is the expected log ratio of their joint distribution to the product of their marginals. That makes it the KL divergence between the joint distribution and the product of the marginals.

Read first: KL Divergence

Mutual information is

\[ I(X;Y) = \mathbb E_{p_{XY}} \ln \frac{P_{XY}}{P_X P_Y}. \]

Mutual information is closely related to KL divergence,

\[ I(X;Y) = D_{\mathrm{KL}} \left( P_{XY}(x,y) \parallel P_X(x) P_{Y}(y) \right). \]

Related: InfoGAN · Deep Infomax