entropy encoding
An entropy encoding is a coding scheme that involves assigning codes to symbols so as to match code lengths with the probabilities of the symbols. Typically, entropy encoders are used to compress data by replacing symbols represented by equal-length codes with symbols represented by codes proportional to the negative logarithm of the probability. Therefore, the most common symbols use the shortest codes.
According to Shannon’s theorem, the optimal code length for a symbol is
where is the number of symbols used to make output codes and is the probability of the input symbol.
Two of the most common entropy encoding techniques are Huffman encoding and arithmetic encoding.