Convenience yield

2013-04-28T00:55:21Z

68.193.96.140:

'''Differential entropy''' (also referred to as '''continuous entropy''') is a concept in [[information theory]] that extends the idea of (Shannon) [[information entropy|entropy]], a measure of average [[surprisal]] of a [[random variable]], to continuous [[probability distribution]]s.

==Definition==
Let ''X'' be a random variable with a [[probability density function]] ''f'' whose [[support (mathematics)|support]] is a set <math>\mathbb X</math>. The ''differential entropy'' ''h''(''X'') or ''h''(''f'') is defined as

:<math>h(X) = -\int_\mathbb{X} f(x)\log f(x)\,dx</math>.

For probability distributions which don't have an explicit density function expression, but have an explicit [[quantile function]] expression, ''Q''(''p''), then ''h''(''Q'') can be defined in terms of the derivative of ''Q''(''p'') i.e. the quantile density function ''Q'''(''p'') as <ref>{{Citation |last1=Vasicek |first1=Oldrich |year=1976 |title=A Test for Normality Based on Sample Entropy |journal=Journal of the Royal Statistical Society, Series B |volume=38 |issue=1 |pages=54–59 |postscript=. }}</ref>

:<math>h(Q) = \int_0^1 \log Q'(p)\,dp</math>.

As with its discrete analog, the units of differential entropy depend on the base of the [[logarithm]], which is usually 2 (i.e., the units are [[bit]]s). See [[logarithmic units]] for logarithms taken in different bases. Related concepts such as [[joint entropy|joint]], [[conditional entropy|conditional]] differential entropy, and [[Kullback-Leibler divergence|relative entropy]] are defined in a similar fashion. Unlike the discrete analog, the differential entropy has an offset that depends on the units used to measure ''X''.<ref name="gibbs">Pages 183-184, {{cite book |last=Gibbs |first=Josiah Willard |authorlink=Josiah Willard Gibbs |title=[[Elementary Principles in Statistical Mechanics|Elementary Principles in Statistical Mechanics, developed with especial reference to the rational foundation of thermodynamics]] |year=1902 |publisher=[[Charles Scribner's Sons]] |location=New York}}</ref> For example, the differential entropy of a quantity in measured millimeters will be log(1000) more than the same quantity measured in meters; a dimensionless quantity will have differential entropy of log(1000) more than the same quantity divided by 1000.

One must take care in trying to apply properties of discrete entropy to differential entropy, since probability density functions can be greater than 1. For example, [[Uniform distribution (continuous)|Uniform]](0,1/2) has ''negative'' differential entropy

:<math>\int_0^\frac{1}{2} -2\log(2)\,dx=-\log(2)\,</math>.

Thus, differential entropy does not share all properties of discrete entropy.

Note that the continuous [[mutual information]] ''I''(''X'';''Y'') has the distinction of retaining its fundamental significance as a measure of discrete information since it is actually the limit of the discrete mutual information of ''partitions'' of ''X'' and ''Y'' as these partitions become finer and finer. Thus it is invariant under non-linear [[homeomorphisms]] (continuous and uniquely invertible maps)
,<ref>{{cite journal
| first = Alexander
| last = Kraskov
| coauthors = Stögbauer, Grassberger
| year = 2004
| title = Estimating mutual information
| journal = Phys. Rev. E
| volume = 60
| pages = 066138
| doi =10.1103/PhysRevE.69.066138
|arxiv = cond-mat/0305641 |bibcode = 2004PhRvE..69f6138K }}</ref> including linear <ref name = Reza>{{ cite book | title = An Introduction to Information Theory | author = Fazlollah M. Reza | publisher = Dover Publications, Inc., New York | year = 1961, 1994 | isbn = 0-486-68210-2 | url = http://books.google.com/books?id=RtzpRAiX6OgC&pg=PA8&dq=intitle:%22An+Introduction+to+Information+Theory%22++%22entropy+of+a+simple+source%22&as_brr=0&ei=zP79Ro7UBovqoQK4g_nCCw&sig=j3lPgyYrC3-bvn1Td42TZgTzj0Q }}</ref> transformations of ''X'' and ''Y'', and still represents the amount of discrete information that can be transmitted over a channel that admits a continuous space of values.

==Properties of differential entropy==
* For two densities ''f'' and ''g'', the [[Kullback-Leibler divergence]] ''D''(''f''||''g'') is nonnegative with equality if ''f'' = ''g'' [[almost everywhere]]. Similarly, for two random variables ''X'' and ''Y'', ''I''(''X'';''Y'') ≥ 0 and ''h''(''X''|''Y'') ≤ ''h''(''X'') with equality [[if and only if]] ''X'' and ''Y'' are [[Statistical independence|independent]].
* The chain rule for differential entropy holds as in the discrete case
::<math>h(X_1, \ldots, X_n) = \sum_{i=1}^{n} h(X_i|X_1, \ldots, X_{i-1}) \leq \sum h(X_i)</math>.
* Differential entropy is translation invariant, i.e., ''h''(''X'' + ''c'') = ''h''(''X'') for a constant ''c''.
* Differential entropy is in general not invariant under arbitrary invertible maps. In particular, for a constant ''a'', ''h''(''aX'') = ''h''(''X'') + log|''a''|. For a vector valued random variable '''X''' and a matrix ''A'', ''h''(''A'' '''X''') = ''h''('''X''') + log|det(''A'')|.
* In general, for a transformation from a random vector to another random vector with same dimension '''Y''' = ''m''('''X'''), the corresponding entropies are related via
::<math>h(\mathbf{Y}) \leq h(\mathbf{X}) + \int f(x) \log \left\vert \frac{\partial m}{\partial x} \right\vert dx</math>
:where <math>\left\vert \frac{\partial m}{\partial x} \right\vert</math> is the [[Jacobian matrix and determinant|Jacobian]] of the transformation ''m''. Equality is achieved if the transform is a bijection. When ''m'' is a rigid rotation, translation, or combination thereof, the Jacobian determinant is always 1, and ''h''(''Y'') = ''h''(''X'').
* If a random vector '''X''' in '''R'''''n'' has mean zero and [[covariance]] matrix ''K'', <math>h(\mathbf{X}) \leq \frac{1}{2} \log[(2\pi e)^n \det{K}]</math> with equality if and only if '''X''' is [[jointly gaussian]] (see [[#Maximization in the normal distribution|below]]).

However, differential entropy does not have other desirable properties:
* It is not invariant under [[change of variables]], and is therefore most useful with dimensionless variables.
* It can be negative.
A modification of differential entropy that addresses these drawbacks is the '''relative information entropy''', also known as the [[Kullback–Leibler divergence]], which includes an [[invariant measure]] factor (see [[limiting density of discrete points]]).

== Maximization in the normal distribution ==
With a [[normal distribution]], differential entropy is maximized for a given variance. The following is a proof that a Gaussian variable has the largest entropy amongst all random variables of equal variance.

Let ''g''(''x'') be a [[Normal distribution|Gaussian]] [[Probability density function|PDF]] with mean μ and variance σ2 and ''f''(''x'') an arbitrary [[Probability density function|PDF]] with the same variance. Since differential entropy is translation invariant we can assume that ''f''(''x'') has the same mean of μ as ''g''(''x'').

Consider the [[Kullback-Leibler divergence]] between the two distributions
:<math> 0 \leq D_{KL}(f || g) = \int_{-\infty}^\infty f(x) \log \left( \frac{f(x)}{g(x)} \right) dx = -h(f) - \int_{-\infty}^\infty f(x)\log(g(x)) dx.</math>
Now note that
:<math>\begin{align}
\int_{-\infty}^\infty f(x)\log(g(x)) dx &= \int_{-\infty}^\infty f(x)\log\left( \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}\right) dx \\
&= \int_{-\infty}^\infty f(x) \log\frac{1}{\sqrt{2\pi\sigma^2}} dx + \log(e)\int_{-\infty}^\infty f(x)\left( -\frac{(x-\mu)^2}{2\sigma^2}\right) dx \\
&= -\tfrac{1}{2}\log(2\pi\sigma^2) - \log(e)\frac{\sigma^2}{2\sigma^2} \\
&= -\tfrac{1}{2}\left(\log(2\pi\sigma^2) + \log(e)\right) \\
&= -\tfrac{1}{2}\log(2\pi e \sigma^2) \\
&= -h(g)
\end{align}</math>
because the result does not depend on ''f''(''x'') other than through the variance. Combining the two results yields
:<math> h(g) - h(f) \geq 0 \!</math>
with equality when ''g''(''x'') = ''f''(''x'') following from the properties of [[Kullback-Leibler divergence]].

This result may also be demonstrated using the [[variational calculus]]. A Lagrangian function with two Lagrangian multipliers may be defined as:

:<math>L=\int_{-\infty}^\infty g(x)\ln(g(x))\,dx-\lambda_0\left(1-\int_{-\infty}^\infty g(x)\,dx\right)-\lambda\left(\sigma^2-\int_{-\infty}^\infty g(x)(x-\mu)^2\,dx\right)</math>

where ''g(x)'' is some function with mean μ. When the entropy of ''g(x)'' is at a maximum and the constraint equations, which consist of the normalization condition <math>\left(1=\int_{-\infty}^\infty g(x)\,dx\right)</math> and the requirement of fixed variance <math>\left(\sigma^2=\int_{-\infty}^\infty g(x)(x-\mu)^2\,dx\right)</math>, are both satisfied, then a small variation δ''g''(''x'') about ''g(x)'' will produce a variation δ''L'' about ''L'' which is equal to zero:

:<math>0=\delta L=\int_{-\infty}^\infty \delta g(x)\left (\ln(g(x))+1+\lambda_0+\lambda(x-\mu)^2\right )\,dx</math>

Since this must hold for any small δ''g''(''x''), the term in brackets must be zero, and solving for ''g(x)'' yields:

:<math>g(x)=e^{-\lambda_0-1-\lambda(x-\mu)^2}</math>

Using the constraint equations to solve for λ0 and λ yields the normal distribution:

:<math>g(x)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}</math>

==Example: Exponential distribution==
Let ''X'' be an [[exponential distribution|exponentially distributed]] random variable with parameter λ, that is, with probability density function

:<math>f(x) = \lambda e^{-\lambda x} \mbox{ for } x \geq 0.</math>

Its differential entropy is then
{|
|-
| <math>h_e(X)\,</math>
| <math>=-\int_0^\infty \lambda e^{-\lambda x} \log (\lambda e^{-\lambda x})\,dx</math>
|-
|
| <math>= -\left(\int_0^\infty (\log \lambda)\lambda e^{-\lambda x}\,dx + \int_0^\infty (-\lambda x) \lambda e^{-\lambda x}\,dx\right) </math>
|-
|
| <math>= -\log \lambda \int_0^\infty f(x)\,dx + \lambda E[X]</math>
|-
|
| <math>= -\log\lambda + 1\,.</math>
|}

Here, <math>h_e(X)</math> was used rather than <math>h(X)</math> to make it explicit that the logarithm was taken to base ''e'', to simplify the calculation.

==Differential entropies for various distributions==
In the table below <math>\Gamma(x) = \int_0^{\infty} e^{-t} t^{x-1} dt</math> is the [[gamma function]], <math>\psi(x) = \frac{d}{dx} \ln\Gamma(x)=\frac{\Gamma'(x)}{\Gamma(x)}</math> is the [[digamma function]], <math>B(p,q) = \frac{\Gamma(p)\Gamma(q)}{\Gamma(p+q)}</math> is the [[beta function]], and γ''E'' is [[Euler-Mascheroni constant|Euler's constant]]. Each distribution maximizes the entropy for a particular set of functional constraints listed in the fourth column, and the constraint that x be included in the support of the probability density, which is listed in the fifth column.<ref>{{cite journal |last1=Park |first1=Sung Y. |last2=Bera |first2=Anil K. |year=2009 |title=Maximum entropy autoregressive conditional heteroskedasticity model |journal=Journal of Econometrics |volume= |issue= |pages=219–230 |publisher=Elsevier |doi= |url=http://www.wise.xmu.edu.cn/Master/Download/..%5C..%5CUploadFiles%5Cpaper-masterdownload%5C2009519932327055475115776.pdf |accessdate=2011-06-02 }}</ref>
{| class="wikitable" style="background:white"
|+ Table of differential entropies and corresponding maximum entropy constraints
|-
! Distribution Name !! Probability density function (pdf) !! Entropy in [[Nat (information)|nats]] !! Maximum Entropy Constraint || Support
|-
| [[Uniform distribution (continuous)|Uniform]] || <math>f(x) = \frac{1}{b-a}</math> || <math>\ln(b - a) \,</math> ||None||<math>[a,b]\,</math>
|-
| [[Normal distribution|Normal]] || <math>f(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)</math> || <math>\ln\left(\sigma\sqrt{2\,\pi\,e}\right) </math>||<math>E(x)=\mu,\,E((x-\mu)^2)=\sigma^2</math>||<math>(-\infty,\infty)\,</math>
|-
| [[Exponential distribution|Exponential]] || <math>f(x) = \lambda \exp\left(-\lambda x\right)</math> || <math>1 - \ln \lambda \, </math>||<math>E(x)=1/\lambda\,</math>||<math>[0,\infty)\,</math>
|-
| [[Rayleigh distribution|Rayleigh]] || <math>f(x) = \frac{x}{\sigma^2} \exp\left(-\frac{x^2}{2\sigma^2}\right)</math> || <math>1 + \ln \frac{\sigma}{\sqrt{2}} + \frac{\gamma_E}{2}</math>||<math>E(x^2)=2\sigma^2, E(\ln(x))=\frac{\ln(2\sigma^2)-\gamma_E}{2}\,</math>||<math>[0,\infty)\,</math>
|-
| [[Beta distribution|Beta]] || <math>f(x) = \frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha,\beta)}</math> for <math>0 \leq x \leq 1</math> || <math> \ln B(\alpha,\beta) - (\alpha-1)[\psi(\alpha) - \psi(\alpha +\beta)]\,</math> <math>- (\beta-1)[\psi(\beta) - \psi(\alpha + \beta)] \, </math>||<math>E(\ln(x))=\psi(\alpha)-\psi(\alpha+\beta)\,</math> <math>E(\ln(1-x))=\psi(\beta )-\psi(\alpha+\beta)\,</math>||<math>[0,1]\,</math>
|-
| [[Cauchy distribution|Cauchy]] || <math>f(x) = \frac{\gamma}{\pi} \frac{1}{\gamma^2 + x^2}</math> || <math>\ln(4\pi\gamma) \, </math>||<math>E(\ln(x^2+\gamma^2))=\ln(4\gamma^2)\,</math>||<math>(-\infty,\infty)\,</math>
|-
| [[Chi distribution|Chi]] || <math>f(x) = \frac{2}{2^{k/2} \Gamma(k/2)} x^{k-1} \exp\left(-\frac{x^2}{2}\right)</math> || <math>\ln{\frac{\Gamma(k/2)}{\sqrt{2}}} - \frac{k-1}{2} \psi\left(\frac{k}{2}\right) + \frac{k}{2}</math>||<math>E(x^2)=k,\,E(\ln(x))=\frac{1}{2}\left[\psi\left(\frac{k}{2}\right)\!+\!\ln(2)\right]</math>||<math>[0,\infty)\,</math>
|-
| [[Chi-squared distribution|Chi-squared]] || <math>f(x) = \frac{1}{2^{k/2} \Gamma(k/2)} x^{\frac{k}{2}\!-\!1} \exp\left(-\frac{x}{2}\right)</math> || <math>\ln 2\Gamma\left(\frac{k}{2}\right) - \left(1 - \frac{k}{2}\right)\psi\left(\frac{k}{2}\right) + \frac{k}{2}</math>||<math>E(x)=k,\,E(\ln(x))=\psi\left(\frac{k}{2}\right)+\ln(2)</math>||<math>[0,\infty)\,</math>
|-
| [[Erlang distribution|Erlang]] || <math>f(x) = \frac{\lambda^k}{(k-1)!} x^{k-1} \exp(-\lambda x)</math> || <math>(1-k)\psi(k) + \ln \frac{\Gamma(k)}{\lambda} + k</math>||<math>E(x)=k/\lambda,\,E(\ln(x))=\psi(k)-\ln(\lambda)</math>||<math>[0,\infty)\,</math>
|-
| [[F distribution|F]] || <math>f(x) = \frac{n_1^{\frac{n_1}{2}} n_2^{\frac{n_2}{2}}}{B(\frac{n_1}{2},\frac{n_2}{2})} \frac{x^{\frac{n_1}{2} - 1}}{(n_2 + n_1 x)^{\frac{n_1 + n2}{2}}}</math> || <math>\ln \frac{n_1}{n_2} B\left(\frac{n_1}{2},\frac{n_2}{2}\right) + \left(1 - \frac{n_1}{2}\right) \psi\left(\frac{n_1}{2}\right) -</math> <math>\left(1 + \frac{n_2}{2}\right)\psi\left(\frac{n_2}{2}\right) + \frac{n_1 + n_2}{2} \psi\left(\frac{n_1\!+\!n_2}{2}\right)</math>||<math>\,</math>||<math>[0,\infty)\,</math>
|-
| [[Gamma distribution|Gamma]] || <math>f(x) = \frac{x^{k - 1} \exp(-\frac{x}{\theta})}{\theta^k \Gamma(k)}</math> || <math>\ln(\theta \Gamma(k)) + (1 - k)\psi(k) + k \, </math>||<math>E(x)=k\theta,\,E(\ln(x))=\psi(k)+\ln(\theta)</math>||<math>[0,\infty)\,</math>
|-
| [[Laplace distribution|Laplace]] || <math>f(x) = \frac{1}{2b} \exp\left(-\frac{|x - \mu|}{b}\right)</math> || <math>1 + \ln(2b) \, </math>||<math>E(|x-\mu|)=b\,</math>||<math>(-\infty,\infty)\,</math>
|-
| [[Logistic distribution|Logistic]] || <math>f(x) = \frac{e^{-x}}{(1 + e^{-x})^2}</math> || <math>2 \, </math>||<math>\,</math>||<math>(-\infty,\infty)\,</math>
|-
| [[Log-normal distribution|Lognormal]] || <math>f(x) = \frac{1}{\sigma x \sqrt{2\pi}} \exp\left(-\frac{(\ln x - \mu)^2}{2\sigma^2}\right)</math> || <math>\mu + \frac{1}{2} \ln(2\pi e \sigma^2)</math>||<math>E(\ln(x))=\mu,E((\ln(x) - \mu)^2)=\sigma^2\,</math>||<math>[0,\infty)\,</math>
|-
| [[Maxwell-Boltzmann distribution|Maxwell-Boltzmann]] || <math>f(x) = \frac{1}{a^3}\sqrt{\frac{2}{\pi}}\,x^{2}\exp\left(-\frac{x^2}{2a^2}\right)</math> || <math>\frac{1}{2}-\gamma_E-\ln(a\sqrt{2\pi})</math>||<math>E(x^2)=3a^2,\,E(\ln(x))\!=\!1\!+\!\ln\left(\frac{a}{\sqrt{2}}\right)\!-\!\frac{\gamma_E}{2}</math>||<math>[0,\infty)\,</math>
|-
| [[Generalized Gaussian distribution|Generalized normal]] || <math>f(x) = \frac{2 \beta^{\frac{\alpha}{2}}}{\Gamma(\frac{\alpha}{2})} x^{\alpha - 1} \exp(-\beta x^2)</math> || <math>\ln{\frac{\Gamma(\alpha/2)}{2\beta^{\frac{1}{2}}}} - \frac{\alpha - 1}{2} \psi\left(\frac{\alpha}{2}\right) + \frac{\alpha}{2}</math>||<math>\,</math>||<math>(-\infty,\infty)\,</math>
|-
| [[Pareto distribution|Pareto]] || <math>f(x) = \frac{\alpha x_m^\alpha}{x^{\alpha+1}}</math> || <math>\ln \frac{x_m}{\alpha} + 1 + \frac{1}{\alpha}</math>||<math>E(\ln(x))=\frac{1}{\alpha}+\ln(x_m)\,</math>||<math>[x_m,\infty)\,</math>
|-
| [[Student's t-distribution|Student's t]] || <math>f(x) = \frac{(1 + x^2/\nu)^{-\frac{\nu+1}{2}}}{\sqrt{\nu}B(\frac{1}{2},\frac{\nu}{2})}</math> || <math>\frac{\nu\!+\!1}{2}\left(\psi\left(\frac{\nu\!+\!1}{2}\right)\!-\!\psi\left(\frac{\nu}{2}\right)\right)\!+\!\ln \sqrt{\nu} B\left(\frac{1}{2},\frac{\nu}{2}\right)</math>||<math>E(\ln(x^2\!+\!\nu))=\log \left(\nu\right)\!-\!\psi \left(\frac{\nu}{2}\right)\!+\!\psi\left(\frac{\nu\!+\!1}{2} \right)\,</math>||<math>(-\infty,\infty)\,</math>
|-
| [[Triangular distribution|Triangular]] || <math> f(x) = \begin{cases}
\frac{2(x-a)}{(b-a)(c-a)} & \mathrm{for\ } a \le x \leq c, \\[4pt]
\frac{2(b-x)}{(b-a)(b-c)} & \mathrm{for\ } c < x \le b, \\[4pt]
\end{cases}</math> || <math>\frac{1}{2} + \ln \frac{b-a}{2}</math>||<math>\,</math>||<math>[0,1]\,</math>
|-
| [[Weibull distribution|Weibull]] || <math>f(x) = \frac{k}{\lambda^k} x^{k-1} \exp\left(-\frac{x^k}{\lambda^k}\right)</math> || <math>\frac{(k-1)\gamma_E}{k} + \ln \frac{\lambda}{k} + 1</math>||<math>E(x^k)=\lambda^k,E(\ln(x))=\ln(\lambda)-\frac{\gamma_E}{k}\,</math>||<math>[0,\infty)\,</math>
|-
| [[Multivariate normal distribution|Multivariate normal]] || <math>
f_X(\vec{x}) =</math> <math> \frac{\exp \left( -\frac{1}{2} ( \vec{x} - \vec{\mu})^\top \Sigma^{-1}\cdot(\vec{x} - \vec{\mu}) \right)} {(2\pi)^{N/2} \left|\Sigma\right|^{1/2}}</math> || <math>\frac{1}{2}\ln\{(2\pi e)^{N} \det(\Sigma)\}</math>||<math>E(\vec{x})=\vec{\mu},\,E((\vec{x}-\vec{\mu})(\vec{x}-\vec{\mu})^T)=\Sigma\,</math>||<math>(-\vec{\infty},\vec{\infty})\,</math>
|-
|}

(Many of the differential entropies are from.<ref name="lazorathie">{{cite journal|author=Lazo, A. and P. Rathie|title=On the entropy of continuous probability distributions|journal=Information Theory, IEEE Transactions on|year=1978|volume=24(1)|pages=120-122|doi=10.1109/TIT.1978.1055832}}</ref>

==Variants==
As described above, differential entropy does not share all properties of discrete entropy. For example, the differential entropy can be negative; also it is not invariant under continuous coordinate transformations. [[Edwin Thompson Jaynes]] showed in fact that the expression above is not the correct limit of the expression for a finite set of probabilities.<ref>{{cite journal |author=Jaynes, E.T. |authorlink=Edwin Thompson Jaynes |title=Information Theory And Statistical Mechanics |journal=Brandeis University Summer Institute Lectures In Theoretical Physics |volume=3 |issue=sect. 4b |pages=181–218 |year=1963 |url=http://bayes.wustl.edu/etj/articles/brandeis.pdf |format=PDF}}</ref>

A modification of differential entropy adds an [[invariant measure]] factor to correct this, (see [[limiting density of discrete points]]). If ''m(x)'' is further constrained to be a probability density, the resulting notion is called [[relative entropy]] in information theory:

:<math>D(p||m) = \int p(x)\log\frac{p(x)}{m(x)}\,dx.</math>

The definition of differential entropy above can be obtained by partitioning the range of ''X'' into bins of length ''h'' with associated sample points ''ih'' within the bins, for ''X'' Riemann integrable. This gives a [[Quantization (signal processing)|quantized]] version of ''X'', defined by ''Xh'' = ''ih'' if ''ih'' ≤ ''X'' ≤ (''i''+1)''h''. Then the entropy of ''Xh'' is

:<math>H_h=-\sum_i hf(ih)\log (f(ih)) - \sum hf(ih)\log(h).</math>

The first term on the right approximates the differential entropy, while the second term is approximately −log(''h''). Note that this procedure suggests that the entropy in the discrete sense of a continuous random variable should be ∞.

==See also==
*[[Information entropy]]
*[[Information theory]]
*[[Limiting density of discrete points]]
*[[Self-information]]
*[[Kullback-Leibler divergence]]
*[[Entropy estimation]]

== References ==
{{reflist}}
* Thomas M. Cover, Joy A. Thomas. ''Elements of Information Theory'' New York: Wiley, 1991. ISBN 0-471-06259-6

==External links==
* {{springer|title=Differential entropy|id=p/d031890}}
* {{planetmath reference|id=1915|title=Differential entropy}}

[[Category:Entropy and information]]
[[Category:Information theory]]
[[Category:Statistical randomness]]
[[Category:Randomness]]

Taxicab number

2012-08-29T17:22:17Z

68.193.2.168: The year corrected as 1938 (in place of 1954). G.H. Hardy died in 1947 itself. The THIRD Edition of the publication jointly by both the authors was issued in 1954.

Hello beneficial friend. Let me introduce my family. I am Merlin. Guam has for ages been his home but now he is considering other options. Collecting kites is one of the things Truly like most. Administering databases is my day job now but I've already applied for the next one. Check out my website here: https://www.youtube.com/watch?v=pns2W-mSPU8 my homepage [https://www.youtube.com/watch?v=pns2W-mSPU8 Scalar Energy Pendant Review]

Ecological Debt Day

2012-08-24T17:45:59Z

68.193.145.214: Undid revision 508964655 by 68.193.145.214 (talk)

by Nas, is very fitting and the film agrees with it. The next step is to visit your Word - Press blog dashboard. I thought about what would happen by placing a text widget in the sidebar beneath my banner ad, and so it went. Transforming your designs to Word - Press blogs is not that easy because of the simplified way in creating your very own themes. After activating, you will find their website link and get the activation code from their website. Generally, for my private income-making market websites, I will thoroughly research and discover the leading 10 most worthwhile niches to venture into. If you are a positive thinker businessman then today you have to put your business online. Our Daily Deal Software plugin brings the simplicity of setting up a Word - Press blog to the daily deal space. t need to use the back button or the URL to get to your home page. Word - Press makes it possible to successfully and manage your website. Usually, Wordpress owners selling the ad space on monthly basis and this means a residual income source. As of now, Pin2Press is getting ready to hit the market. This platform can be customizedaccording to the requirements of the business. Newer programs allow website owners and internet marketers to automatically and dynamically change words in their content to match the keywords entered by their web visitors in their search queries'a feat that they cannot easily achieve with older software. Premium vs Customised Word - Press Themes - Premium themes are a lot like customised themes but without the customised price and without the wait. It has become a more prevalent cause of infertility and the fertility clinic are having more and more couples with infertility problems. And, make no mistake,India's Fertility Clinics and IVF specialists are amongst the best in the world,and have been for some time. Normally, the Word - Press developers make a thorough research on your website goals and then ingrain the most suitable graphical design elements to your website. There are many advantages of hiring Wordpress developers for Wordpress project development:. Now all you have to do is log into your Word - Press site making use of the very same username and password that you initially had in your previous site. Every single module contains published data and guidelines, usually a lot more than 1 video, and when pertinent, incentive links and PDF files to assist you out. An ease of use which pertains to both internet site back-end and front-end users alike. However, you must also manually approve or reject comments so that your website does not promote parasitic behavior. ) Remote Login: With the process of PSD to Wordpress conversion comes the advantage of flexibility. For your information, it is an open source web content management system If you have any inquiries regarding wherever and how to use [http://cyurl.biz/wordpress_dropbox_backup_180599 wordpress backup plugin], you can make contact with us at our web-site. .

153 (number)

2012-08-08T00:21:21Z

68.193.2.168:

Gabrielle is what her [http://search.about.com/?q=husband+loves husband loves] to call a though she doesn't totally like being called like that. As a woman what your girl really likes is mah jongg but she hasn't ever made a dime to barefoot jogging. Software creating a is where her primary income comes from but soon her husband and consequently her will start their own business. For a while she's been in Massachusetts. Go to her website to come out more: http://prometeu.net Look at my web blog hack clash of clans ([http://prometeu.net just click the up coming document])

formulasearchengine - User contributions [en]

Convenience yield

Taxicab number

Ecological Debt Day

153 (number)