Berezin integral

In information theory, Pinsker's inequality, named after its inventor Mark Semenovich Pinsker, is an inequality that bounds the total variation distance (or statistical distance) in terms of the Kullback-Leibler divergence. The inequality is tight up to constant factors.

Pinsker's inequality states that, if P and Q are two probability distributions, then

$\delta (P,Q)\le \sqrt{\frac{1}{2}{D}_{\mathrm{K}\mathrm{L}}(P\Vert Q)}$

where

$\delta (P,Q)=\sup \{|P(A)-Q(A)|:A\text{ is an event to which probabilities are assigned.}\}$

is the total variation distance (or statistical distance) between P and Q and

$D_{\mathrm{K}\mathrm{L}}(P\Vert Q)=\sum _i\ln \left(\frac{P(i)}{Q(i)}\right)P(i)$

is the Kullback-Leibler divergence in nats.

Pinsker first proved the inequality with a worse constant. The inequality in the above form was proved independently by Kullback, Csiszár, and Kemperman.

An inverse of the inequality cannot hold: for every $ϵ>0$, there are distributions with $\delta (P,Q)\le ϵ$ but $D_{\mathrm{K}\mathrm{L}}(P\Vert Q)=\mathrm{\infty }$.

References

43 year old Petroleum Engineer Harry from Deep River, usually spends time with hobbies and interests like renting movies, property developers in singapore new condominium and vehicle racing. Constantly enjoys going to destinations like Camino Real de Tierra Adentro.

Additional reading

• Thomas M. Cover and Joy A. Thomas: Elements of Information Theory, 2nd edition, Willey-Interscience, 2006
• Nicolo Cesa-Bianchi and Gábor Lugosi: Prediction, Learning, and Games, Cambridge University Press, 2006