entropy of a partition

Let $(X,ℬ,\mu )$ be a probability space. A measurable partition of $X$ is a partition such that each of itselements is a measurable set (i.e. an element of $ℬ$).

Given a finite measurable partition $𝒫$,its is

$$H_{\mu }(𝒫)=\sum _{P\in 𝒫}-\mu (P)\log \mu (P),$$

where we assume $0\log 0=0$ for convenience.

Remarks.

1. Entropy can be interpreted as a measure of the a priori uncertainity about theoutcome of the measurement an experiment, assuming that we are measuring it through the given partition(i.e., we are going to be told in which atom of the partition the result is).Thus, the finer a partition is, the higher the resulting entropy. In particular, the trivialpartition $\{X\}$ has entropy $0$, since there is only one possible outcome, so there is nouncertainity at all. On the other hand, the measurement gives no information at allabout the “real” outcome of the experiment, which reflects the complementary intepretation of entropy:as the information gained from the measurement.This is because of the intuitive fact that moreuncertainity about the outcome of the measurement means that more information will be obtained fromknowing it about the “real” outcome.

2. Equally intuitive is the fact that among all measurable partitions of $X$ into $n$ atoms, the maximumpossibleentropy is attained at those in which the atoms are equally likely (i.e., all atoms have equalmeasure $1/n$).This can be proved by means of standard calculus, anda direct computation shows that the maximum value is $\log n$.

3. Since the definition of entropy involves only the measure of atoms of the given partition, twopartitions which are equal modulo measure zero have the same entropy.

4. There is a natural correspondence between finite measurable partitions and finite sub-$\sigma$-algebras of $ℬ$. For this reason,to each finite sub-$\sigma$-algebra $𝒫$ we can define its entropy by $H_{\mu }(𝒫)$ where $𝒫$ is the (unique) partition which generates $𝒫$. For short, we denote this entropy by $H_{\mu }(𝒫)$.