|
|
| (One intermediate revision by one other user not shown) |
| Line 1: |
Line 1: |
| [[Image:Entropy-mutual-information-relative-entropy-relation-diagram.svg|thumb|256px|right|Individual (H(X),H(Y)), joint (H(X,Y)), and conditional entropies for a pair of correlated subsystems X,Y with mutual information I(X; Y).]]
| |
|
| |
|
| In [[probability theory]] and [[information theory]], the '''mutual information''' or (formerly) '''transinformation''' of two [[random variable]]s is a measure of the variables' mutual dependence. The most common [[unit of measurement]] of mutual information is the [[bit]].
| |
|
| |
|
| == Definition of mutual information ==
| | In front of playing a new video clip clip game, read the more indulgent book. Most dvds have a book people can purchase separately. You may want to consider doing this and reading it before you play, or even while you are playing. This way, you can possibly get the most in of your game run.<br><br>Go online for help. Virtually every game has its actually own legion of devoted devotees, lots of which spend countless hours crafting comprehensive maps and guides. Additionally there are newsgroups where you are qualified to speak one on an with other players. Benefit from this found diamond and it is possible to eventually get past that level you have been trapped in on forever.<br><br>Interweaving social trends form a net in which many people trapped. When Each Tygers of Pan Tang sang 'It's lonely towards the top. Everyones trying to do most people in', these people coppied much from clash of clans chop tool no survey. A society without [http://prometeu.net clash of clans hack tool] no survey is for being a society with no knowledge, in that it very good.<br><br>Principally clash of clans identify tool no survey puts together believe in among the people. Society is probably definitely [http://Browse.deviantart.com/?qh=§ion=&global=1&q=powered powered] by expert pressure, one of the most powerful forces on the planet. Like long as peer make utilizes its power as for good, clash of clans hack tool no survey will have its set in community.<br><br>Be certain to may not let online games take over your daily life. Game titles can be quite additive, time also have to make totally sure you moderate the time that you investing guitar playing such games. Merchandise in your articles invest an excessive level of time playing video game, your actual life could begin to falter.<br><br>Individual world can be forced by supply and demand. We shall look over the Greek-Roman model. Using special care to highlight the role because of clash of clans get into tool no survey within the vast framework of which usually this provides.<br><br>You don''t necessarily really need one of the highly developed troops to [http://Www.Encyclopedia.com/searchresults.aspx?q=win+victories win victories]. A mass volume of barbarians, your first-level troop, also can totally destroy an opponent village, and strangely it''s quite enjoyable to from the virtual carnage. |
| Formally, the mutual information of two discrete random variables ''X'' and ''Y'' can be defined as:
| |
| | |
| :<math> I(X;Y) = \sum_{y \in Y} \sum_{x \in X}
| |
| p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)}
| |
| \right) }, \,\!
| |
| </math>
| |
| | |
| where ''p''(''x'',''y'') is the [[joint distribution|joint probability distribution function]] of ''X'' and ''Y'', and <math>p(x)</math> and <math>p(y)</math> are the [[marginal probability]] distribution functions of ''X'' and ''Y'' respectively.
| |
| | |
| In the case of [[continuous function|continuous random variables]], the summation is replaced by a definite [[double integral]]: | |
| | |
| :<math> I(X;Y) = \int_Y \int_X
| |
| p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)}
| |
| \right) } \; dx \,dy,
| |
| | |
| </math>
| |
| | |
| where ''p''(''x'',''y'') is now the joint probability ''density'' function of ''X'' and ''Y'', and <math>p(x)</math> and <math>p(y)</math> are the marginal probability density functions of ''X'' and ''Y'' respectively.
| |
| | |
| If the log base 2 is used, the units of mutual information are the [[bit]].
| |
| | |
| Intuitively, mutual information measures the information that ''X'' and ''Y'' share: it measures how much knowing one of these variables reduces uncertainty about the other. For example, if ''X'' and ''Y'' are independent, then knowing ''X'' does not give any information about ''Y'' and vice versa, so their mutual information is zero. At the other extreme, if X is a deterministic function of Y and Y is a deterministic function of X then all information conveyed by ''X'' is shared with ''Y'': knowing ''X'' determines the value of ''Y'' and vice versa. As a result, in this case the mutual information is the same as the uncertainty contained in ''Y'' (or ''X'') alone, namely the [[information entropy|entropy]] of ''Y'' (or ''X''). Moreover, this mutual information is the same as the entropy of X and as the entropy of Y. (A very special case of this is when X and Y are the same random variable.)
| |
| | |
| Mutual information is a measure of the inherent dependence expressed in the [[joint distribution]] of ''X'' and ''Y'' relative to the joint distribution of ''X'' and ''Y'' under the assumption of independence.
| |
| Mutual information therefore measures dependence in the following sense: ''I''(''X''; ''Y'') = 0 [[if and only if]] ''X'' and ''Y'' are independent random variables. This is easy to see in one direction: if ''X'' and ''Y'' are independent, then ''p''(''x'',''y'') = ''p''(''x'') ''p''(''y''), and therefore:
| |
| | |
| :<math> \log{ \left( \frac{p(x,y)}{p(x)\,p(y)} \right) } = \log 1 = 0. \,\!
| |
| </math>
| |
| | |
| Moreover, mutual information is nonnegative (i.e. ''I''(''X'';''Y'') ≥ 0; see below) and [[Symmetric function|symmetric]] (i.e. ''I''(''X'';''Y'') = ''I''(''Y'';''X'')).
| |
| | |
| ==Relation to other quantities==
| |
| Mutual information can be equivalently expressed as
| |
| | |
| :<math>
| |
| \begin{align}
| |
| I(X;Y) & {} = H(X) - H(X|Y) \\
| |
| & {} = H(Y) - H(Y|X) \\
| |
| & {} = H(X) + H(Y) - H(X,Y) \\
| |
| & {} = H(X,Y) - H(X|Y) - H(Y|X)
| |
| \end{align}
| |
| </math>
| |
| | |
| where <math>\ H(X)</math> and <math>\ H(Y)</math> are the marginal [[information entropy|entropies]], ''H''(''X''|''Y'') and ''H''(''Y''|''X'') are the [[conditional entropy|conditional entropies]], and ''H''(''X'',''Y'') is the [[joint entropy]] of ''X'' and ''Y''. Using [[Jensen's inequality]] on the definition of mutual information we can show that ''I''(''X'';''Y'') is non-negative, consequently, <math>\ H(X) \ge H(X|Y)</math>.
| |
| | |
| Intuitively, if entropy ''H''(''X'') is regarded as a measure of uncertainty about a random variable, then ''H''(''X''|''Y'') is a measure of what ''Y'' does ''not'' say about ''X''. This is "the amount of uncertainty remaining about ''X'' after ''Y'' is known", and thus the right side of the first of these equalities can be read as "the amount of uncertainty in ''X'', minus the amount of uncertainty in ''X'' which remains after ''Y'' is known", which is equivalent to "the amount of uncertainty in ''X'' which is removed by knowing ''Y''". This corroborates the intuitive meaning of mutual information as the amount of information (that is, reduction in uncertainty) that knowing either variable provides about the other.
| |
| | |
| Note that in the discrete case ''H''(''X''|''X'') = 0 and therefore ''H''(''X'') = ''I''(''X'';''X''). Thus ''I''(''X'';''X'') ≥ ''I''(''X'';''Y''), and one can formulate the basic principle that a variable contains at least as much information about itself as any other variable can provide.
| |
| | |
| Mutual information can also be expressed as a [[Kullback-Leibler divergence]], of the product ''p''(''x'') × ''p''(''y'') of the [[marginal distribution]]s of the two random variables ''X'' and ''Y'', from ''p''(''x'',''y'') the random variables' [[joint distribution]]:
| |
| | |
| :<math> I(X;Y) = D_{\mathrm{KL}}(p(x,y)\|p(x)p(y)). </math>
| |
| | |
| Furthermore, let ''p''(''x''|''y'') = ''p''(''x'', ''y'') / ''p''(''y''). Then
| |
| | |
| :<math>
| |
| \begin{align}
| |
| I(X;Y) & {} = \sum_y p(y) \sum_x p(x|y) \log_2 \frac{p(x|y)}{p(x)} \\
| |
| & {} = \sum_y p(y) \; D_{\mathrm{KL}}(p(x|y)\|p(x)) \\
| |
| & {} = \mathbb{E}_Y\{D_{\mathrm{KL}}(p(x|y)\|p(x))\}.
| |
| \end{align}
| |
| </math>
| |
| | |
| Thus mutual information can also be understood as the [[expected value|expectation]] of the Kullback-Leibler divergence of the univariate distribution ''p''(''x'') of ''X'' from the [[conditional distribution]] ''p''(''x''|''y'') of ''X'' given ''Y'': the more different the distributions ''p''(''x''|''y'') and ''p''(''x''), the greater the [[Kullback-Leibler divergence|information gain]].
| |
| | |
| == Variations of mutual information ==
| |
| Several variations on mutual information have been proposed to suit various needs. Among these are normalized variants and generalizations to more than two variables.
| |
| | |
| === Metric ===
| |
| Many applications require a [[metric (mathematics)|metric]], that is, a distance measure between points. The quantity
| |
| | |
| :<math>d(X,Y) = H(X,Y) - I(X;Y) = H(X) + H(Y) - 2I(X;Y) = H(X|Y) + H(Y|X)</math>
| |
| | |
| satisfies the properties of a metric ([[triangle inequality]], [[non-negative|non-negativity]], [[identity of indiscernibles|indiscernability]] and symmetry). This distance metric is also known as the [[Variation of information]].
| |
| | |
| Since one has <math>d(X,Y) \le H(X,Y)</math>, a natural normalized variant is
| |
| | |
| :<math>D(X,Y) = d(X,Y)/H(X,Y) \le 1.</math>
| |
| | |
| The metric ''D'' is a [[universal metric]], in that if any other distance measure places ''X'' and ''Y'' close-by, then the ''D'' will also judge them close.<ref>Alexander Kraskov, Harald Stögbauer, Ralph G. Andrzejak, and [[Peter Grassberger]], "Hierarchical Clustering Based on Mutual Information", (2003) ''[http://arxiv.org/abs/q-bio/0311039 ArXiv q-bio/0311039]''</ref>
| |
| | |
| A set-theoretic interpretation of information (see the figure for [[Conditional entropy]]) shows that
| |
| | |
| :<math>D(X,Y) = 1 - I(X;Y)/H(X,Y)</math>
| |
| | |
| which is effectively the [[Jaccard index|Jaccard distance]] between ''X'' and ''Y''.
| |
| | |
| === Conditional mutual information ===
| |
| {{Main|Conditional mutual information}}
| |
| Sometimes it is useful to express the mutual information of two random variables conditioned on a third.
| |
| :<math>I(X;Y|Z) = \mathbb E_Z \big(I(X;Y)|Z\big)
| |
| = \sum_{z\in Z} \sum_{y\in Y} \sum_{x\in X}
| |
| p_Z(z) p_{X,Y|Z}(x,y|z) \log \frac{p_{X,Y|Z}(x,y|z)}{p_{X|Z}(x|z)p_{Y|Z}(y|z)},</math>
| |
| which can be simplified as
| |
| :<math>I(X;Y|Z) = \sum_{z\in Z} \sum_{y\in Y} \sum_{x\in X}
| |
| p_{X,Y,Z}(x,y,z) \log \frac{p_Z(z)p_{X,Y,Z}(x,y,z)}{p_{X,Z}(x,z)p_{Y,Z}(y,z)}.</math>
| |
| Conditioning on a third random variable may either increase or decrease the mutual information, but it is always true that
| |
| :<math>I(X;Y|Z) \ge 0</math>
| |
| for discrete, jointly distributed random variables ''X'', ''Y'', ''Z''. This result has been used as a basic building block for proving other [[inequalities in information theory]].
| |
| | |
| === Multivariate mutual information ===
| |
| {{main|Multivariate mutual information}}
| |
| Several generalizations of mutual information to more than two random variables have been proposed, such as [[total correlation]] and [[interaction information]]. If Shannon entropy is viewed as a [[signed measure]] in the context of [[information diagram]]s, as explained in the article ''[[Information theory and measure theory]]'', then the only definition of multivariate mutual information that makes sense{{Citation needed|date=January 2009}} is as follows:
| |
| :<math>I(X_1;X_1) = H(X_1)</math>
| |
| and for <math>n > 1,</math>
| |
| :<math>I(X_1;\,...\,;X_n) = I(X_1;\,...\,;X_{n-1}) - I(X_1;\,...\,;X_{n-1}|X_n),</math>
| |
| where (as above) we define
| |
| :<math>I(X_1;\,...\,;X_{n-1}|X_n) = \mathbb E_{X_n} \big(I(X_1;\,...\,;X_{n-1})|X_n\big).</math>
| |
| (This definition of multivariate mutual information is identical to that of [[interaction information]] except for a change in sign when the number of random variables is odd.)
| |
| | |
| If <math> A </math> and <math> B </math> are two sets of variables, then the mutual information between them is:
| |
| :<math>I(A,B) = H(A\cup B)+H(A\cap B) - H(A) - H(B),</math>
| |
| | |
| | |
| ====Applications====
| |
| Applying information diagrams blindly to derive the above definition{{Citation needed|date=July 2009}} has been criticised, and indeed it has found rather limited practical application, since it is difficult to visualize or grasp the significance of this quantity for a large number of random variables. It can be zero, positive, or negative for any <math>n \ge 3.</math>
| |
| | |
| One high-dimensional generalization scheme which maximizes the mutual information between the joint distribution and other target variables is found to be useful in [[feature selection]].<ref>{{cite book
| |
| | author = Christopher D. Manning, Prabhakar Raghavan, Hinrich Schütze
| |
| | title = An Introduction to Information Retrieval
| |
| | publisher = [[Cambridge University Press]]
| |
| | year = 2008
| |
| | isbn = 0-521-86571-9 }}</ref>
| |
| | |
| Mutual information is also used in the area of signal processing as a measure of similarity between two signals. For example, FMI metric<ref>Haghighat, M. B. A., Aghagolzadeh, A., & Seyedarabi, H. (2011). [http://dx.doi.org/10.1016/j.compeleceng.2011.07.012 A non-reference image fusion metric based on mutual information of image features]. Computers & Electrical Engineering, 37(5), 744-756.</ref> is an image fusion performance measure that makes use of mutual information in order to measure the amount of information that the fused image contains about the source images.
| |
| | |
| === Normalized variants ===
| |
| Normalized variants of the mutual information are provided by the ''coefficients of constraint'' <ref>{{cite book|author=Coombs, C. H., Dawes, R. M. & Tversky, A.|year=1970|title=Mathematical Psychology: An Elementary Introduction|publisher=Prentice-Hall, Englewood Cliffs, NJ}}</ref> or [[uncertainty coefficient]] <ref name=pressflannery>{{Cite book | last1=Press | first1=WH | last2=Teukolsky | first2=SA | last3=Vetterling | first3=WT | last4=Flannery | first4=BP | year=2007 | title=Numerical Recipes: The Art of Scientific Computing | edition=3rd | publisher=Cambridge University Press | publication-place=New York | isbn=978-0-521-88068-8 | chapter=Section 14.7.3. Conditional Entropy and Mutual Information | chapter-url=http://apps.nrbook.com/empanel/index.html#pg=758 | ref=harv | postscript=<!-- Bot inserted parameter. Either remove it; or change its value to "." for the cite to end in a ".", as necessary. -->{{inconsistent citations}}}}
| |
| </ref>
| |
| | |
| :<math>
| |
| C_{XY}=\frac{I(X;Y)}{H(Y)} ~~~~\mbox{and}~~~~ C_{YX}=\frac{I(X;Y)}{H(X)}.
| |
| </math>
| |
| | |
| The two coefficients are not necessarily equal. In some cases a symmetric measure may be desired, such as the following ''redundancy''{{Citation needed|date=July 2008}} measure:
| |
| | |
| :<math>R= \frac{I(X;Y)}{H(X)+H(Y)}</math>
| |
| | |
| which attains a minimum of zero when the variables are independent and a maximum value of
| |
| | |
| :<math>R_{\max }=\frac{\min (H(X),H(Y))}{H(X)+H(Y)} </math>
| |
| | |
| when one variable becomes completely redundant with the knowledge of the other. See also ''[[Redundancy (information theory)]]''. Another symmetrical measure is the ''symmetric uncertainty'' (Witten & Frank 2005), given by
| |
| | |
| :<math>U(X,Y) = 2R = 2\frac{I(X;Y)}{H(X)+H(Y)}</math>
| |
| | |
| which represents a weighted average of the two uncertainty coefficients.<ref name=pressflannery/>
| |
| | |
| If we consider mutual information as a special case of the [[total correlation]] or [[dual total correlation]], the normalized versions are respectively,
| |
| :<math>\frac{I(X;Y)}{\min\left[ H(X),H(Y)\right]}</math> and <math>\frac{I(X;Y)}{H(X,Y)} \; .</math>
| |
| | |
| Other normalized versions are provided by the following expressions.<ref>{{cite book|url=http://www.cse.ohio-state.edu/~weit/Information-Theoretic%20Measures%20for%20Knowledge%20Discovery%20and%20Data%20Mining.pdf|author=Yao, Y. Y.|year=2003|chapter=Information-theoretic measures for knowledge discovery and data mining|title=Entropy Measures, Maximum Entropy Principle and Emerging Applications|editor=Karmeshu|publisher=Springer|pages=115–136}}</ref><ref>
| |
| {{cite journal
| |
| | last = [[Alexander Strehl|Strehl]]
| |
| | first = Alexander
| |
| | coauthors = Joydeep Ghosh
| |
| | title = Cluster ensembles – a knowledge reuse framework for combining multiple partitions
| |
| | journal = Journal of Machine Learning Research
| |
| | volume = 3
| |
| | pages = 583–617
| |
| | year = 2002
| |
| | url = http://strehl.com/download/strehl-jmlr02.pdf
| |
| | format = [[PDF]]
| |
| | doi = 10.1162/153244303321897735
| |
| | ref = harv
| |
| }}</ref>
| |
| | |
| :<math>
| |
| \frac{I(X;Y)}{\min\left[ H(X),H(Y)\right]}, ~~~~~~~ \frac{I(X;Y)}{H(X,Y)}, ~~~~~~~ \frac{I(X;Y)}{\sqrt{H(X)H(Y)}}
| |
| </math>
| |
| | |
| The quantity
| |
| | |
| :<math>D^\prime(X,Y)=1-\frac{I(X;Y)}{\max(H(X),H(Y))}</math>
| |
| | |
| is a [[metric (mathematics)|metric]], ''i.e.'' satisfies the triangle inequality, ''etc.''
| |
| | |
| === Weighted variants ===
| |
| In the traditional formulation of the mutual information,
| |
| | |
| :<math> I(X;Y) = \sum_{y \in Y} \sum_{x \in X} p(x,y) \log \frac{p(x,y)}{p(x)\,p(y)}, </math>
| |
| | |
| each ''event'' or ''object'' specified by <math>(x,y)</math> is weighted by the corresponding probability <math>p(x,y)</math>. This assumes that all objects or events are equivalent ''apart from'' their probability of occurrence. However, in some applications it may be the case that certain objects or events are more ''significant'' than others, or that certain patterns of association are more semantically important than others.
| |
| | |
| For example, the deterministic mapping <math>\{(1,1),(2,2),(3,3)\}</math> may be viewed as stronger than the deterministic mapping <math>\{(1,3),(2,1),(3,2)\}</math>, although these relationships would yield the same mutual information. This is because the mutual information is not sensitive at all to any inherent ordering in the variable values (Cronbach 1954, Coombs & Dawes 1970, Lockhead 1970), and is therefore not sensitive at all to the '''form''' of the relational mapping between the associated variables. If it is desired that the former relation — showing agreement on all variable values — be judged stronger than the later relation, then it is possible to use the following ''weighted mutual information'' (Guiasu 1977)
| |
| | |
| :<math> I(X;Y) = \sum_{y \in Y} \sum_{x \in X} w(x,y) p(x,y) \log \frac{p(x,y)}{p(x)\,p(y)}, </math>
| |
| | |
| which places a weight <math>w(x,y)</math> on the probability of each variable value co-occurrence, <math>p(x,y)</math>. This allows that certain probabilities may carry more or less significance than others, thereby allowing the quantification of relevant ''holistic'' or ''prägnanz'' factors. In the above example, using larger relative weights for <math>w(1,1)</math>, <math>w(2,2)</math>, and <math>w(3,3)</math> would have the effect of assessing greater ''informativeness'' for the relation <math>\{(1,1),(2,2),(3,3)\}</math> than for the relation <math>\{(1,3),(2,1),(3,2)\}</math>, which may be desirable in some cases of pattern recognition, and the like. There has been little mathematical work done on the weighted mutual information and its properties, however.
| |
| | |
| === Adjusted mutual information ===
| |
| {{main|adjusted mutual information}}
| |
| | |
| A probability distribution can be viewed as a [[partition of a set]]. One may then ask: if a set were partitioned randomly, what would the distribution of probabilities be? What would the expectation value of the mutual information be? The [[adjusted mutual information]] or AMI subtracts the expectation value of the MI, so that the AMI is zero when two different distributions are random, and one when two distributions are identical. The AMI is defined in analogy to the [[adjusted Rand index]] of two different partitions of a set.
| |
| | |
| === Absolute mutual information ===<!-- This section is linked from [[Kolmogorov complexity]] -->
| |
| Using the ideas of [[Kolmogorov complexity]], one can consider the mutual information of two sequences independent of any probability distribution:
| |
| | |
| :<math>
| |
| I_K(X;Y) = K(X) - K(X|Y).
| |
| </math> | |
| | |
| To establish that this quantity is symmetric up to a logarithmic factor (<math>I_K(X;Y) \approx I_K(Y;X)</math>) requires the [[chain rule for Kolmogorov complexity]] {{Harvard citation|Li|1997}}.
| |
| Approximations of this quantity via [[Data compression|compression]] can be used to define a [[Metric (mathematics)|distance measure]] to perform a [[hierarchical clustering]] of sequences without having any [[domain knowledge]] of the sequences {{Harvard citation|Cilibrasi|2005}}.
| |
| | |
| === Mutual information for discrete data===
| |
| | |
| When ''X'' and ''Y'' are limited to be in a discrete number of states, observation data is summarized
| |
| in a [[contingency table]], with row variable ''X'' (or i) and column variable ''Y'' (or j).
| |
| Mutual information is one of the measures of
| |
| [[association (statistics)|association]] or [[correlation and dependence|correlation]]
| |
| between the row and column variables. Other measures of association include
| |
| [[Pearson's chi-squared test]] statistics, [[G-test]] statistics, etc. In fact,
| |
| mutual information is equal to [[G-test]] statistics divided by 2N where N is
| |
| the sample size. | |
| | |
| In the special case where the number of states for both row and column variables
| |
| is 2 (i,j=1,2), the [[degrees of freedom (statistics)|degrees of freedom]] of
| |
| the [[Pearson's chi-squared test]] is 1. Out of the four terms in the summation:
| |
| | |
| :<math> \sum_{i,j } p_{ij} \log \frac{p_{ij}}{p_i p_j }</math>
| |
| | |
| only one is independent. It is the reason that mutual information function has an
| |
| exact relationship with the [[correlation function]] <math> p_{X=1, Y=1}-p_{X=1}p_{Y=1}</math> for
| |
| [[binary number|binary]] sequences
| |
| .<ref>
| |
| {{cite journal | author = Wentian Li | title = Mutual information functions versus correlation functions | journal = J. Stat. Phys. | volume = 60 | issue = 5-6 | pages = 823–837 | year = 1990 | doi = 10.1007/BF01025996| url = }}
| |
| </ref>
| |
| | |
| ==Applications of mutual information==
| |
| In many applications, one wants to maximize mutual information (thus increasing dependencies), which is often equivalent to minimizing [[conditional entropy]]. Examples include:
| |
| * In [[search engine technology]], mutual information between phrases and contexts is used as a feature for k-means clustering to discover semantic clusters (concepts).<ref name=magerman>[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.78.4178&rep=rep1&type=pdf ] Parsing a Natural Language Using Mutual Information Statistics by David M. Magerman and Mitchell P. Marcus</ref>
| |
| * In [[telecommunications]], the [[channel capacity]] is equal to the mutual information, maximized over all input distributions.
| |
| * [[Discriminative training]] procedures for [[hidden Markov model]]s have been proposed based on the [[maximum mutual information]] (MMI) criterion.
| |
| * [[Nucleic acid secondary structure|RNA secondary structure]] prediction from a [[multiple sequence alignment]].
| |
| * [[Phylogenetic profiling]] prediction from pairwise present and disappearance of functionally link [[gene]]s.
| |
| * Mutual information has been used as a criterion for [[feature selection]] and feature transformations in [[machine learning]]. It can be used to characterize both the relevance and redundancy of variables, such as the [[minimum redundancy feature selection]].
| |
| * Mutual information is used in determining the similarity of two different [[cluster analysis|clustering]]s of a dataset. As such, it provides some advantages over the traditional [[Rand index]].
| |
| * Mutual information of words is often used as a significance function for the computation of [[collocation]]s in [[corpus linguistics]]. This has the added complexity that no word-instance is an instance to two different words; rather, one counts instances where 2 words occur adjacent or in close proximity; this slightly complicates the calculation, since the expected probability of one word occurring within N words of another, goes up with N.
| |
| * Mutual information is used in [[medical imaging]] for [[image registration]]. Given a reference image (for example, a brain scan), and a second image which needs to be put into the same [[coordinate system]] as the reference image, this image is deformed until the mutual information between it and the reference image is maximized.
| |
| * Detection of [[phase synchronization]] in [[time series]] analysis
| |
| * In the [[infomax]] method for neural-net and other machine learning, including the infomax-based [[Independent component analysis]] algorithm
| |
| * Average mutual information in [[delay embedding theorem]] is used for determining the ''embedding delay'' parameter.
| |
| * Mutual information between [[genes]] in [[microarray|expression microarray]] data is used by the [[ARACNE]] algorithm for reconstruction of [[gene regulatory network|gene networks]].
| |
| * In [[statistical mechanics]], [[Loschmidt's paradox]] may be expressed in terms of mutual information.<ref name=everett56>[[Hugh Everett]] [http://www.pbs.org/wgbh/nova/manyworlds/pdf/dissertation.pdf Theory of the Universal Wavefunction], Thesis, Princeton University, (1956, 1973), pp 1–140 (page 30)</ref><ref name=everett57>[[Hugh Everett]], [http://www.univer.omsk.su/omsk/Sci/Everett/paper1957.html Relative State Formulation of Quantum Mechanics], ''Reviews of Modern Physics'' vol 29, (1957) pp 454–462.</ref> Loschmidt noted that it must be impossible to determine a physical law which lacks [[time reversal symmetry]] (e.g. the [[second law of thermodynamics]]) only from physical laws which have this symmetry. He pointed out that the [[H-theorem]] of [[Boltzmann]] made the assumption that the velocities of particles in a gas were permanently uncorrelated, which removed the time symmetry inherent in the H-theorem. It can be shown that if a system is described by a probability density in [[phase space]], then [[Liouville's theorem]] implies that the joint information (negative of the joint entropy) of the distribution remains constant in time. The joint information is equal to the mutual information plus the sum of all the marginal information (negative of the marginal entropies) for each particle coordinate. Boltzmann's assumption amounts to ignoring the mutual information in the calculation of entropy, which yields the thermodynamic entropy (divided by Boltzmann's constant).
| |
| | |
| * The mutual information is used to learn the structure of [[Bayesian network]]s/[[dynamic Bayesian network]]s, which explain the causal relationship between random variables, as exemplified by the GlobalMIT toolkit [http://code.google.com/p/globalmit/]: learning the globally optimal dynamic Bayesian network with the Mutual Information Test criterion.
| |
| * Popular cost function in [[Decision tree learning]].
| |
| | |
| == See also == | |
| * [[Pointwise mutual information]]
| |
| * [[Quantum mutual information]]
| |
| | |
| == Notes ==
| |
| <references/>
| |
| | |
| == References ==
| |
| * {{cite journal
| |
| | last = Cilibrasi
| |
| | first = R.
| |
| | coauthors = Paul Vitányi
| |
| | title = Clustering by compression
| |
| | journal = IEEE Transactions on Information Theory
| |
| | volume = 51
| |
| | issue = 4
| |
| | pages = 1523–1545
| |
| | year = 2005
| |
| | url = http://www.cwi.nl/~paulv/papers/cluster.pdf
| |
| | format = [[PDF]]
| |
| | doi = 10.1109/TIT.2005.844059
| |
| | ref = harv
| |
| }}
| |
| * Cronbach L. J. (1954). On the non-rational application of information measures in psychology, in H Quastler, ed., ''Information Theory in Psychology: Problems and Methods'', Free Press, Glencoe, Illinois, pp. 14–30.
| |
| * {{cite journal|first1=Kenneth Ward|last1=Church|first2=Patrick|last2=Hanks|title=Word association norms, mutual information, and lexicography|journal=Proceedings of the 27th Annual Meeting of the Association for Computational Linguistics|year=1989|url=http://delivery.acm.org/10.1145/90000/89095/p22-church.pdf?ip=87.193.196.98&id=89095&acc=OPEN&key=BF13D071DEA4D3F3B0AA4BA89B4BCA5B&CFID=238314452&CFTOKEN=56636212&__acm__=1375963734_fac6312f7247e2f4d7915da1a876acbe}}
| |
| * {{cite book|author=Guiasu, Silviu|year=1977|title=Information Theory with Applications|publisher=McGraw-Hill, New York|isbn=978-0070251090}}
| |
| * {{cite book
| |
| | last = Li
| |
| | first = Ming
| |
| | coauthors = Paul Vitányi
| |
| | title = An introduction to Kolmogorov complexity and its applications
| |
| | location = New York
| |
| | publisher = [[Springer-Verlag]]
| |
| |date=February 1997
| |
| | isbn = 0-387-94868-6 }}
| |
| * Lockhead G. R. (1970). Identification and the form of multidimensional discrimination space, ''Journal of Experimental Psychology'' '''85'''(1), 1–10.
| |
| * David J. C. MacKay. ''[http://www.inference.phy.cam.ac.uk/mackay/itila/book.html Information Theory, Inference, and Learning Algorithms]'' Cambridge: Cambridge University Press, 2003. ISBN 0-521-64298-1 (available free online)
| |
| * Haghighat, M. B. A., Aghagolzadeh, A., & Seyedarabi, H. (2011). A non-reference image fusion metric based on mutual information of image features. Computers & Electrical Engineering, 37(5), 744-756.
| |
| * [[Athanasios Papoulis]]. ''Probability, Random Variables, and Stochastic Processes'', second edition. New York: McGraw-Hill, 1984. ''(See Chapter 15.)''
| |
| * {{cite book|author=Witten, Ian H. & Frank, Eibe|title=2005|title=Data Mining: Practical Machine Learning Tools and Techniques|publisher=Morgan Kaufmann, Amsterdam|isbn=978-0-12-374856-0|url=http://www.cs.waikato.ac.nz/~ml/weka/book.html}}
| |
| * {{cite journal|author=Peng, H.C., Long, F., and Ding, C.|title=Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy|journal=IEEE Transactions on Pattern Analysis and Machine Intelligence|volume=27|issue=8|pages=1226–1238|year=2005|url=http://research.janelia.org/peng/proj/mRMR/index.htm}}
| |
| * {{cite journal|author=Andre S. Ribeiro, Stuart A. Kauffman, Jason Lloyd-Price, Bjorn Samuelsson, and Joshua Socolar|year=2008|title=Mutual Information in Random Boolean models of regulatory networks|journal=Physical Review E|volume=77|issue=1|arxiv=0707.3642}}
| |
| * {{cite journal
| |
| | last = Wells
| |
| | first = W.M. III
| |
| | coauthors = Viola, P., Atsumi, H., Nakajima, S., Kikinis, R.
| |
| | title = Multi-modal volume registration by maximization of mutual information
| |
| | journal = Medical Image Analysis
| |
| | volume = 1
| |
| | issue = 1
| |
| | pages = 35–51
| |
| | year = 1996
| |
| | pmid = 9873920
| |
| | doi = 10.1016/S1361-8415(01)80004-9
| |
| | url = http://www.ai.mit.edu/people/sw/papers/mia.pdf
| |
| | format = [[PDF]]
| |
| | ref = harv
| |
| }}
| |
| | |
| {{DEFAULTSORT:Mutual Information}}
| |
| [[Category:Information theory]]
| |
| [[Category:Entropy and information]]
| |