Mutual Information¶
In one minute
Mutual information of two random variables is the expected log ratio of their joint distribution to the product of their marginals. That makes it the KL divergence between the joint distribution and the product of the marginals.
Read first: KL Divergence
Mutual information is
\[
I(X;Y) = \mathbb E_{p_{XY}} \ln \frac{P_{XY}}{P_X P_Y}.
\]
Mutual information is closely related to KL divergence,
\[
I(X;Y) = D_{\mathrm{KL}} \left( P_{XY}(x,y) \parallel P_X(x) P_{Y}(y) \right).
\]
Related: InfoGAN · Deep Infomax