Bregman divergence

A Bregman divergence, or Bregman distance, $B_F$ on a space $𝒳\subseteq {ℝ}^d$ is defined for a strictly convex and differentiable function $F:𝒳\to ℝ$ as

$$B_F(p,q)=F(p)-F(q)-⟨p-q,\nabla F(q)⟩,$$(1)

where

$$⟨p,q⟩=p^{T}q$$

denotes the inner product, and

$$\nabla F(x)={[\frac{\partial F}{\partial x_1},\mathrm{\cdots },\frac{\partial F}{\partial x_d}]}^T$$

the partial derivatives.

Choosing $F(x)={\sum }_{i=1}^d{x}_i^2$ yields the squared Euclidean distance $B_{x^2}(p,q)={||p-q||}^2$, and choosing $F(x)={\sum }_{i=1}^d{x}_i\log x_i$ yields the relative entropy, called the Kullback-Leibler divergence.