Proof — Shannon entropy (information theory)

Statement

The Shannon entropy H(X) = -Σ p(x) log p(x) measures the average information content of a random variable X. It quantifies uncertainty and the minimum bits needed to encode a source. Shannon's 1948 paper founded information theory, establishing channel capacity, source coding, and the mathematical framework for communication and data compression.

Formula

H(X) = -Σ p(x) log₂ p(x)

Equation

H(X) = -Σᵢ p(xᵢ) log p(xᵢ) (bits for binary log)

Proof sketch

Shannon's "A Mathematical Theory of Communication" (1948) built on Kolmogorov's probability axioms. He defined entropy as the expected information content, proved the source coding theorem (optimal compression limit), and the channel coding theorem (reliable transmission up to capacity C).