en>Paradoctor: /* References */ added See also section

2013-11-01T16:05:30Z

References: added See also section

← Older revision		Revision as of 18:05, 1 November 2013
Line 1:		Line 1:
	~~They contact me Emilia~~. ~~Bookkeeping~~ is ~~what I do~~. ~~Puerto Rico~~ is ~~exactly~~ where he and ~~his spouse reside~~. The ~~favorite hobby~~ for ~~my kids~~ and me is to ~~play baseball~~ but I ~~haven~~'~~t produced~~ a ~~dime with it~~.<br><br>~~Check out my website~~; [http://www.~~adosphere~~.~~com~~/~~poyocum home std test kit~~]		This article discusses how [[information theory]] (a branch of mathematics studying the transmission, processing and storage of [[information]]) is related to [[measure theory]] (a branch of mathematics related to [[integral\|integration]] and [[probability]]).

			== Measures in information theory ==
			Many of the concepts in information theory have separate definitions and formulas for [[continuous probability distribution\|continuous]] and [[discrete probability distribution\|discrete]] cases. For example, [[entropy]] <math>H(X)</math> is usually defined for discrete random variables, whereas for continuous random variables the related concept of [[differential entropy]], written <math>h(X)</math>, is used (see Cover and Thomas, 2006, chapter 8). Both these concepts are mathematical [[expected value\|expectations]], but the expectation is defined with an [[integral]] for the continuous case, and a sum for the discrete case.

			These separate definitions can be more closely related in terms of [[measure theory]]. For discrete random variables, probability mass functions can be considered density functions with respect to the counting measure, thus requiring only basic [[discrete mathematics]] to perform the integration, in the measure-theoretic sense. Because the same integration expression is used for the continuous case, which uses basic [[calculus]], the same concepts and expressions can be used for both discrete and continuous cases. Consider the formula for the differential entropy of a continuous [[random variable]] <math>X</math> with [[probability density function]] <math>f(x)</math>:
			:<math> h(X) = -\int_X f(x) \log f(x) \,dx. </math>
			This can usually be interpreted as the following [[Riemann-Stieltjes integral]]:
			:<math> h(X) = -\int_X f(x) \log f(x) \,d\mu(x), </math>
			where <math>\mu</math> is the [[Lebesgue measure]]. But if instead, <math>X</math> is discrete, <math>f</math> is a probability mass function, and <math>\nu</math> is the [[counting measure]], we can write:
			:<math> H(X) = -\int_X f(x) \log f(x) \,d\nu(x) = -\sum_{x\in X} f(x) \log f(x). </math>
			The integral expression and the general concept is identical to the continuous case; the only difference is the measure used. In both cases the probability density function <math>f</math> is the [[Radon–Nikodym theorem\|Radon–Nikodym derivative]] of the [[probability measure]] with respect to the measure against which the integral is taken.

			If <math> \mathbb P </math> is the probability measure on <math>X</math>, then the integral can also be taken directly with respect to <math> \mathbb P </math>:
			:<math> h(X) = -\int_X \log \frac{\mathrm d \mathbb P}{\mathrm d\mu} \,d\mathbb P, </math>

			If instead of the underlying measure μ we take another probability measure <math> \mathbb Q </math>, we are led to the [[Kullback–Leibler divergence]]: let <math>\mathbb P</math> and <math>\mathbb Q</math> be probability measures over the same space. Then if <math>\mathbb P</math> is [[absolute continuity\|absolutely continuous]] with respect to <math>\mathbb Q</math>, written <math>\mathbb P << \mathbb Q,</math> the [[Radon–Nikodym theorem\|Radon–Nikodym derivative]] <math>\frac{\mathrm d\mathbb P}{\mathrm d\mathbb Q}</math> exists and the Kullback–Leibler divergence can be expressed in its full generality:

			:<math>D_\mathrm{KL}(\mathbb P \\| \mathbb Q)
			= \int_{\mathrm{supp}\mathbb P}
			\frac{\mathrm d\mathbb P}{\mathrm d\mathbb Q}
			\log \frac{\mathrm d\mathbb P}{\mathrm d\mathbb Q}
			\,d \mathbb Q
			= \int_{\mathrm{supp}\mathbb P}
			\log \frac{\mathrm d\mathbb P}{\mathrm d\mathbb Q}
			\,d \mathbb P,
			</math>
			where the integral runs over the [[support (mathematics)\|support]] of <math>\mathbb P.</math> Note that we have dropped the negative sign: the Kullback–Leibler divergence is always non-negative due to [[Gibbs' inequality]].

			==Entropy as a "measure"==

			[[Image:Entropy-mutual-information-relative-entropy-relation-diagram.svg\|thumb\|256px\|right\|Individual (H(X),H(Y)), joint (H(X,Y)), and conditional entropies for a pair of correlated subsystems X,Y with mutual information I(X; Y).]]

			There is an analogy between [[Claude Elwood Shannon\|Shannon]]'s basic "[[Quantities of information\|measures]]" of the [[information entropy\|information]] content of random variables and a [[measure theory\|measure]] over sets. Namely the [[joint entropy]], [[conditional entropy]], and [[mutual information]] can be considered as the measure of a [[union (set theory)\|set union]], [[set difference]], and [[intersection (set theory)\|set intersection]], respectively (Reza pp. 106–108).

			If we associate the existence of abstract [[Set (mathematics)\|sets]] <math>\tilde X</math> and <math>\tilde Y</math> to arbitrary [[discrete mathematics\|discrete]] [[random variable]]s ''X'' and ''Y'', somehow representing the [[information entropy\|information]] borne by ''X'' and ''Y'', respectively, such that:
			* <math>\mu(\tilde X \cap \tilde Y) = 0 </math> whenever ''X'' and ''Y'' are unconditionally [[statistical independence\|independent]], and
			* <math>\tilde X = \tilde Y</math> whenever ''X'' and ''Y'' are such that either one is completely determined by the other (i.e. by a bijection);

			where <math>\mu</math> is a [[signed measure]] over these sets, and we set:

			: <math>H(X) = \mu(\tilde X),</math>
			: <math>H(Y) = \mu(\tilde Y),</math>
			: <math>H(X,Y) = \mu(\tilde X \cup \tilde Y),</math>
			: <math>H(X\|Y) = \mu(\tilde X \,\backslash\, \tilde Y),</math>
			: <math>I(X;Y) = \mu(\tilde X \cap \tilde Y);</math>

			we find that [[Claude Elwood Shannon\|Shannon]]'s "measure" of information content satisfies all the postulates and basic properties of a formal measure over sets, as commonly illustrated in an '''[[information diagram]]'''. This can be a handy [[mnemonic device]] in some situations, e.g.

			:{\| border=0 cellspacing=5 cellpadding=0
			\| <math>H(X,Y)=H(X)+H(Y\|X)\,</math>
			\| <math>\mu(\tilde X \cup \tilde Y)=\mu(\tilde X)+\mu(\tilde Y \,\backslash\, \tilde X)</math>
			\|-
			\| <math>I(X;Y)=H(X)-H(X\|Y)\,</math>
			\| <math>\mu(\tilde X \cap \tilde Y)=\mu(\tilde X)-\mu(\tilde X \,\backslash\, \tilde Y)</math>
			\|}

			Because the entropy, joint entropy, conditional entropy, and bivariate mutual information of discrete random variables are all nonnegative, many basic [[inequalities in information theory]] (among no more than two random variables) can be derived from this formulation by considering the measure μ to be nonnegative.

			==Multivariate mutual information==
			{{main\|Multivariate mutual information}}
			Certain extensions to the definitions of Shannon's basic measures of information are necessary to deal with the [[Sigma-algebra\|σ-algebra]] generated by the sets that would be associated to three or more arbitrary random variables. (See Reza pp. 106–108 for an informal but rather complete discussion.) Namely <math>H(X,Y,Z,\cdots)</math> needs to be defined in the obvious way as the entropy of a joint distribution, and a multivariate [[mutual information]] <math>I(X;Y;Z;\cdots)</math> defined in a suitable manner so that we can set:

			: <math>H(X,Y,Z,\cdots) = \mu(\tilde X \cup \tilde Y \cup \tilde Z \cup \cdots),</math>
			: <math>I(X;Y;Z;\cdots) = \mu(\tilde X \cap \tilde Y \cap \tilde Z \cap \cdots);</math>

			in order to define the (signed) measure over the whole σ-algebra. There is no single universally accepted definition for the mutivariate mutual information, but the one that corresponds here to the measure of a set intersection is due to Fano (Srinivasa). The definition is recursive. As a base case the mutual information of a single random variable is defined to be its entropy: <math>I(X)=H(X)</math>. Then for <math>n\geq 2</math> we set
			: <math>I(X_1; \cdots; X_n) = I(X_1; \cdots; X_{n-1}) - I(X_1; \cdots; X_{n-1}\|X_n),</math>
			where the [[conditional mutual information]] is defined as
			: <math>I(X_1; \cdots; X_{n-1}\|X_n) = \mathbb E_{X_n}\big(I(X_1; \cdots; X_{n-1})\|X_n\big).</math>
			The first step in the recursion yields Shannon's definition <math>I(X_1;X_2)=H(X_1)-H(X_1\|X_2).</math> It is interesting to note that the mutual information (same as [[interaction information]] but for a change in sign) of three or more random variables can be negative as well as positive: Let ''X'' and ''Y'' be two independent fair coin flips, and let ''Z'' be their [[exclusive or]]. Then <math>I(X;Y;Z) = - 1</math> bit.

			Many other variations are possible for three or more random variables: for example, <math>I(X,Y;Z)</math> is the mutual information of the joint distribution of ''X'' and ''Y'' relative to ''Z'', and can be interpreted as <math>\mu((\tilde X \cup \tilde Y) \cap \tilde Z).</math> Many more complicated expressions can be built this way, and still have meaning, e.g. <math> I(X,Y;Z\|W),</math> or <math> H(X,Z\|W,Y).</math>

			==References==
			* Thomas M. Cover and Joy A. Thomas. ''Elements of Information Theory'', second edition, 2006. New Jersey: Wiley and Sons. ISBN 978-0-471-24195-9.
			* Fazlollah M. Reza. ''An Introduction to Information Theory''. New York: McGraw-Hill 1961. New York: Dover 1994. ISBN 0-486-68210-2
			* Sunil Srinivasa. ''A Review on Multivariate Mutual Information''. Notre Dame EE-80653 Information Theory Tutorials, Fall 2005. [http://www.nd.edu/~jnl/ee80653/Fall2005/tutorials/sunil.pdf PDF].
			(contains errors in formulas (9) and (23) )
			* R. W. Yeung, "On entropy, information inequalities, and Groups." [http://user-www.ie.cuhk.edu.hk/~ITIP/online/tutorial.ps PS]

			==See also==
			*[[Information theory]]
			*[[Measure theory]]
			*[[Set theory]]

			[[Category:Information theory]]
			[[Category:Measure theory]]

en>Kyoakoa: −Category:Linguistics; +Category:Linguistic morphology using HotCat

2011-08-05T03:17:47Z

−Category:Linguistics; +Category:Linguistic morphology using HotCat

New page

They contact me Emilia. Bookkeeping is what I do. Puerto Rico is exactly where he and his spouse reside. The favorite hobby for my kids and me is to play baseball but I haven't produced a dime with it.<br><br>Check out my website; [http://www.adosphere.com/poyocum home std test kit]

Bracketing paradox - Revision history

en>Paradoctor: /* References */ added See also section

en>Kyoakoa: −Category:Linguistics; +Category:Linguistic morphology using HotCat