Cross entropy

The cross-entropy is a measure of the quality of a model for a probability distribution of information theory and mathematical statistics. Closely related to the cross-entropy is the perplexity, which may deliver with the same explanatory power more illustrative numerical values.

Definition

Let be a random variable with target amount that is distributed according to. It should further be a distribution on the same sample space.

Then the cross-entropy is defined by:

Here denotes the entropy of and the Kullback -Leibler divergence of two distributions.

Equivalent formulation

By inserting the two defining equations obtained after simplification in the discrete case

And in the continuous case ( with density functions and )

Estimate

Although the cross-entropy has a similar significance as the pure Kullback -Leibler divergence, the former can, however, even without precise knowledge of the estimate. In practical application, therefore, is usually an approximation of an unknown distribution.

According to the above equation is valid:

Where the expectation value call.

Now, are realizations of, that is, an independent and identically distributed according to sample, is therefore

An unbiased estimator for the cross-entropy.

Books and links

• Rubinstein, Reuven Y. / Kroese, Dirk P.: The Cross - Entropy Method - A Unified Approach to Combinatorial Optimization, Monte - Carlo Simulation and Machine Learning. Springer 2004, ISBN 978-0-387-21240-1.
• Entropy script Heidelberg University
• Statistical Language Models University of Munich (PDF, 531 kB)

• Information Theory
• Random variable
• Descriptive Statistics