Kullback–Leibler divergence

The terms Kullback -Leibler divergence ( KL- divergence short ), Kullback -Leibler entropy, Kullback -Leibler information or the Kullback -Leibler distance ( to Solomon and Richard Kullback Liebler ) describe a measure of the difference between two probability distributions. Typically, one of the distributions represents empirical observations, or a precise probability distribution, while the other represents a model or an approximation.

Note: The KL divergence is also called the relative entropy, the relative entropy term is sometimes used also for the mutual information.

Formally, the KL divergence of the probability functions and discrete values ​​determined as follows:

The distributions of continuous values ​​and the probability density functions and illustrated, however, an integral is calculated:

The Kullback -Leibler divergence from information theory point of view indicates how much space is wasted per character on average when a is applied to coding based on an information source that follows the actual distribution. Thus there is a link to the channel capacity. Mathematically, this is compatible with the statement that the KL divergence is, and equality holds only if P and Q are identical.

The specific choice of the base of the logarithm in the calculation depends upon should be expected in any information unit. In practice, it is the KL divergence often in bits and Shannon and it uses the base 2, rare (base 10) are also Nit (base) and Ban needed.

Instead of the Kullback -Leibler divergence of the cross-entropy is also often used. This delivers quality comparable values, but can be estimated without exact knowledge of. In practical applications this is advantageous, as the actual background distribution of the observation data is mostly unknown.

Documents

  • S. Kullback, RA Leibler: On information and sufficiency. In: Annals of Mathematical Statistics. 22, No. 1, March 1951, pp. 79-86.
  • S. Kullback, John Wiley & Sons ( eds.): Information theory and statistics. In 1959.
  • Template: Internet resource / maintenance / access date not in ISO format Springer Online Reference Works. eom.springer.de, accessed on 31 March 2008 (English).
  • Random variable
  • Information Theory
Probability mass function Ban (information) Cross entropy
478245
de