|
|
| Line 1: |
Line 1: |
| In [[probability theory]], '''Hoeffding's inequality''' provides an [[upper bound]] on the [[probability]] that the sum of [[random variables]] deviates from its [[expected value]].
| | Marvella is what you can call her but it's not the most feminine name out there. Body developing is 1 of the issues I love most. Her husband and her live in Puerto Rico but she will have to move one working day or an additional. My working day job is a meter reader.<br><br>My site; diet meal delivery ([http://r.fbtest.net/dietmealdelivery74619 source website]) |
| Hoeffding's inequality was proved by [[Wassily Hoeffding]] in 1963.<ref>{{harvtxt|Hoeffding|1963}}</ref>
| |
| | |
| Hoeffding's inequality is a special case of the [[Azuma–Hoeffding inequality]], and it is more general than the [[Bernstein inequalities in probability theory|Bernstein inequality]], proved by [[Sergei Bernstein]] in 1923. They are also special cases of [[McDiarmid's inequality]].
| |
| | |
| == Special case of Bernoulli random variables ==
| |
| | |
| Hoeffding's inequality can be applied to the important special case of identically distributed [[Bernoulli trial|Bernoulli random variables]], and this is how the inequality is often used in [[combinatorics]] and [[computer science]].
| |
| We consider a coin that shows heads with probability <math>p</math> and tails with probability <math>1-p</math>.
| |
| We toss the coin <math>n</math> times.
| |
| The [[expected value|expected]] number of times the coin comes up heads is <math>p\cdot n</math>.
| |
| Furthermore, the probability that the coin comes up heads at most <math>k</math> times can be exactly quantified by the following expression:
| |
| | |
| :<math>\Pr\Big(n \text{ coin tosses yield heads at most } k \text{ times}\Big)= \sum_{i=0}^{k} \binom{n}{i} p^i (1-p)^{n-i}\,.</math>
| |
| | |
| In the case that <math>k=(p-\epsilon) n</math> for some <math>\epsilon > 0</math>,
| |
| Hoeffding's inequality bounds this probability by a term that is exponentially small in <math>\epsilon^2 \cdot n</math>:
| |
| :<math>\Pr\Big(n \text{ coin tosses yield heads at most } (p-\epsilon) n \text{ times}\Big)\leq\exp\big(-2\epsilon^2 n\big)\,.</math>
| |
| | |
| Similarly, in the case that <math>k=(p+\epsilon) n</math> for some <math>\epsilon > 0</math>,
| |
| Hoeffding's inequality bounds the probability that we see at least <math>\epsilon n</math> more tosses that show heads than we would expect:
| |
| :<math>\Pr\Big(n \text{ coin tosses yield heads at least } (p+\epsilon) n \text{ times}\Big)\leq\exp\big(-2\epsilon^2 n\big)\,.</math>
| |
| | |
| Hence Hoeffding's inequality implies that the number of heads that we see is concentrated around its mean, with exponentially small tail.
| |
| :<math>\Pr\Big(n \text{ coin tosses yield heads between } (p-\epsilon)n \text{ and } (p+\epsilon)n \text{ times}\Big)\geq 1-2\exp\big(-2\epsilon^2 n\big)\,.</math>
| |
| | |
| == General case ==
| |
| | |
| Let
| |
| | |
| :<math>X_1, \dots, X_n \!</math>
| |
| | |
| be [[independent random variables]].
| |
| Assume that the <math>X_i</math> are [[almost sure]]ly bounded; that is, assume for <math>1 \leq i \leq n</math> that
| |
| | |
| :<math>\Pr(X_i \in [a_i, b_i]) = 1. \!</math>
| |
| | |
| We define the empirical mean of these variables
| |
| | |
| :<math>\overline X = \frac{1}{n}(X_1 + \cdots + X_n).</math> | |
| | |
| Theorem 2 of {{harvtxt|Hoeffding|1963}} proves the inequalities
| |
| | |
| :<math>\Pr(\overline X - \mathrm{E}[\overline X] \geq t) \leq \exp \left( - \frac{2n^2t^2}{\sum_{i=1}^n (b_i - a_i)^2} \right),\!</math>
| |
| :<math>\Pr(|\overline X - \mathrm{E}[\overline X]| \geq t) \leq 2\exp \left( - \frac{2n^2t^2}{\sum_{i=1}^n (b_i - a_i)^2} \right),\!</math>
| |
| | |
| which are valid for positive values of ''t''. Here <math>\mathrm{E}[\overline X]</math> is the [[expected value]] of <math>\overline X</math>.
| |
| The inequalities can be also stated in terms of the sum
| |
| | |
| :<math>S = X_1 + \cdots + X_n</math>
| |
| | |
| of the random variables:
| |
| | |
| :<math>\Pr(S - \mathrm{E}[S] \geq t) \leq \exp \left( - \frac{2t^2}{\sum_{i=1}^n (b_i - a_i)^2} \right),\!</math>
| |
| :<math>\Pr(|S - \mathrm{E}[S]| \geq t) \leq 2\exp \left( - \frac{2t^2}{\sum_{i=1}^n (b_i - a_i)^2} \right).\!</math>
| |
| | |
| Note that the inequalities also hold when the <math>X_i</math> have been obtained using sampling without replacement; in this case the random variables are not independent anymore. A proof of this statement can be found in Hoeffding's paper. For slightly better bounds in the case of sampling without replacement, see for instance the paper by {{harvtxt|Serfling|1974}}.
| |
| | |
| ==See also==
| |
| *[[Bennett's inequality]]
| |
| *[[Chebyshev's inequality]]
| |
| *[[Markov's inequality]]
| |
| *[[Chernoff bounds]]
| |
| *[[Hoeffding's lemma]]
| |
| | |
| ==Notes==
| |
| {{reflist}}
| |
| | |
| ==References==
| |
| {{refbegin}}
| |
| * {{cite journal
| |
| | first1=Robert J. | last1=Serfling
| |
| | title=Probability Inequalities for the Sum in Sampling without Replacement
| |
| | journal=The Annals of Statistics
| |
| | pages=39–48
| |
| | year=1974
| |
| | ref=harv
| |
| | volume=2
| |
| | number=1
| |
| | doi=10.1214/aos/1176342611}}
| |
| * {{cite journal
| |
| | first1=Wassily | last1=Hoeffding
| |
| | title=Probability inequalities for sums of bounded random variables
| |
| | journal=Journal of the American Statistical Association
| |
| | pages=13–30
| |
| |date=March 1963
| |
| | ref=harv
| |
| | volume=58
| |
| | number=301
| |
| | jstor=2282952}}
| |
| | |
| {{refend}}
| |
| | |
| [[Category:Probabilistic inequalities]]
| |
Marvella is what you can call her but it's not the most feminine name out there. Body developing is 1 of the issues I love most. Her husband and her live in Puerto Rico but she will have to move one working day or an additional. My working day job is a meter reader.
My site; diet meal delivery (source website)