|
|
| (One intermediate revision by one other user not shown) |
| Line 1: |
Line 1: |
| In [[numerical analysis]], '''inverse iteration''' is an [[Iterative method|iterative]] [[eigenvalue algorithm]]. It allows one to find an approximate
| | Let me first begin with introducing by myself. My name is Santina. Doing 3d graphics could be the thing he loves most especially. Michigan is where my home is. In his professional life he is really a dentist. Go to my website to discover a out more: http://www.omeglegirls69.com/ |
| [[eigenvector]] when an approximation to a corresponding [[eigenvalue]] is already known.
| |
| The method is conceptually similar to the [[power method]] and is also known as the '''inverse power method'''.
| |
| It appears to have originally been developed to compute resonance frequencies in the field of structural mechanics.
| |
| <ref name=Pohlhausen>Ernst Pohlhausen, ''Berechnung der Eigenschwingungen statisch-bestimmter Fachwerke'', ZAMM - Zeitschrift für Angewandte
| |
| Mathematik und Mechanik 1, 28-42 (1921).</ref>
| |
| | |
| The inverse power iteration algorithm starts with number <math>\mu</math> which is an approximation for the [[eigenvalue]] corresponding to the searched [[eigenvector]],
| |
| and vector ''b''<sub>0</sub>, which is an approximation to the eigenvector or a random vector. The method is described by the iteration
| |
| :<math> b_{k+1} = \frac{(A - \mu I)^{-1}b_k}{C_k}, </math>
| |
| | |
| where ''C<sub>k</sub>'' are some constants usually chosen as <math>C_k= \|(A - \mu I)^{-1}b_k \|. </math> Since eigenvectors are defined up to multiplication by constant, the choice of ''C<sub>k</sub>'' can be arbitrary in theory; practical aspects of the choice of <math>C_k</math> are discussed below.
| |
| | |
| So, at every iteration, the vector ''b''<sub>''k''</sub> is multiplied by the inverse of the matrix <math>(A - \mu I)</math> and normalized.
| |
| It is exactly the same formula as in the [[power method]]
| |
| modula change of matrix ''A'', by <math>(A - \mu I)^{-1}. </math>
| |
| The better approximation <math>\mu</math> to the eigenvalue is chosen, the faster convergence one gets, however incorrect choice of <math>\mu</math> can lead to slow convergence or to the convergence of a different eigenvector. Usually in practice the method is used when good approximation for the eigenvalue is known, and hence one needs only few (quite often just one) iteration.
| |
| | |
| ==Theory and convergence ==
| |
| | |
| The basic idea of the [[power iteration]] is choosing an initial vector ''b'' (either an [[eigenvector]] approximation or a [[random]] vector) and iteratively calculating <math>Ab, A^{2}b, A^{3}b,...</math>. Except for a set of zero [[Measure (mathematics)|measure]], for any initial vector, the result will converge to an [[eigenvector]] corresponding to the dominant [[eigenvalue]].
| |
| | |
| The inverse iteration does the same for the matrix <math>(A - \mu I)^{-1}</math>, so it converges to eigenvector corresponding to the dominant eigenvalue of the matrix <math>(A - \mu I)^{-1}</math>.
| |
| Eigenvalues of this matrix are <math>(\lambda_1 - \mu)^{-1},...,(\lambda_n - \mu)^{-1}, </math> where <math> \lambda_i </math> are eigenvalues of ''A''.
| |
| The largest of these numbers correspond to the smallest of <math>(\lambda_1 - \mu),...,(\lambda_n - \mu). </math> It is obvious to see that eigenvectors of matrices ''A'' and <math>(A - \mu I)^{-1}</math> are the same. So:
| |
| | |
| '''Conclusion''': the method converges to the eigenvector of the matrix ''A'' corresponding to the closest eigenvalue to <math>\mu .</math>
| |
| | |
| In particular taking <math>\mu=0</math> we see that <math>(A)^{-k} b</math>
| |
| converges to the eigenvector corresponding the smallest in absolute value eigenvalue of ''A''.
| |
| | |
| === Speed of convergence ===
| |
| | |
| Let us analyze the [[rate of convergence]] of the method. | |
| | |
| The [[power method]] is known to [[Rate_of_convergence#Convergence_speed_for_iterative_methods |converge linearly]] to the limit, more precisely:
| |
| | |
| <math> \mathrm{Distance}( b^\mathrm{ideal}, b^{k}_\mathrm{Power~Method})=O \left( \left| \frac{\lambda_\mathrm{subdominant} }{\lambda_\mathrm{dominant} } \right|^k \right), </math>
| |
| | |
| hence for the inverse iteration method similar result sounds as:
| |
| | |
| <math> \mathrm{Distance}( b^\mathrm{ideal}, b^{k}_\mathrm{Inverse~iteration})=O \left( \left| \frac{\mu -\lambda_{\mathrm{closest~ to~ }\mu} }{\mu - \lambda_{\mathrm{second~ closest~ to~} \mu} } \right|^k \right). </math>
| |
| | |
| This is a key formula for understanding the method's convergence. It shows that if <math>\mu</math> is chosen close enough to some eigenvalue <math>\lambda </math>, for example <math> \mu- \lambda = \epsilon </math> each iteration will improve the accuracy <math> |\epsilon| /|\lambda +\epsilon - \lambda_{\mathrm{closest~ to~} \lambda} | </math> times. (We use that for small enough <math>\epsilon</math> "closest to <math>\mu</math>" and "closest to <math>\lambda </math>" is the same.) For small enough <math> |\epsilon|</math> it is approximately the same as <math> |\epsilon| /|\lambda - \lambda_{\mathrm{closest~ to~} \lambda}| </math>. Hence if one is able to find <math>\mu </math>, such the
| |
| <math>\epsilon </math> will be small enough, then very few iterations may be satisfactory.
| |
| | |
| === Complexity ===
| |
| | |
| The inverse iteration algorithm requires solving a [[System of linear equations|linear system]] or calculation of the inverse matrix.
| |
| For non-structured matrices (not sparse, not Toeplitz,...) this requires <math>O(n^{3})</math> operations.
| |
| | |
| == Implementation options ==
| |
| | |
| The method is defined by the formula:
| |
| | |
| :<math> b_{k+1} = \frac{(A - \mu I)^{-1}b_k}{C_k}, </math>
| |
| | |
| There are several details in its implementation.
| |
| | |
| * Calculate inverse matrix or solve system of linear equations.
| |
| | |
| We can rewrite the formula in the following way:
| |
| | |
| :<math> (A - \mu I) b_{k+1} = \frac{b_k}{C_k}, </math>
| |
| | |
| emphasizing that to find the next approximation <math> b_{k+1} </math>
| |
| we need to solve a system of linear equations.
| |
| There are two options: one may choose an algorithm that solves a linear
| |
| system, or to calculate an inverse matrix
| |
| <math>(A - \mu I)^{-1}</math> and then apply it to the vector.
| |
| Both options have complexity ''O(n<sup>3</sup>)'', the exact number depends on the chosen method. Typically, solutions of linear equation have slightly less complexity. The choice between the options depends on the number of iterations. If one solves the linear system the complexity will be ''k*O(n<sup>3</sup>)'', where ''k'' is number of iterations. Calculating the inverse matrix first and then applying it to the vectors ''b<sub>k</sub>'' is of complexity ''O(n<sup>3</sup>) + k* n<sup>2</sup>''. The second option is clearly preferable for large numbers of iterations. As inverse iterations are typically used when only a small number of iterations is needed one usually solves a linear system of equations.
| |
| | |
| * Tridiagonalization, [[Hessenberg form]].
| |
| | |
| If it is necessary to perform many iterations (or few iterations, but for many eigenvectors), then it might be wise to bring the matrix to the
| |
| upper [[Hessenberg form]] first (for symmetric matrix this will be [[tridiagonal matrix|tridiagonal form]]). Which costs <math>\begin{matrix}\frac{10}{3}\end{matrix} n^3 + O(n^2)</math> arithmetic operations using a technique based on [[Householder transformation|Householder reduction]]), with a finite sequence of orthogonal similarity transforms, somewhat like a two-sided QR decomposition.<ref name=Demmel>{{citation
| |
| | last = Demmel | first = James W. | authorlink = James Demmel
| |
| | mr = 1463942
| |
| | isbn = 0-89871-389-7
| |
| | location = Philadelphia, PA
| |
| | publisher = [[Society for Industrial and Applied Mathematics]]
| |
| | title = Applied Numerical Linear Algebra
| |
| | year = 1997}}.</ref><ref name=Trefethen>Lloyd N. Trefethen and David Bau, ''Numerical Linear Algebra'' (SIAM, 1997).</ref> (For QR decomposition, the Householder rotations are multiplied only on the left, but for the Hessenberg case they are multiplied on both left and right.) For [[symmetric matrix|symmetric matrices]] this procedure costs <math>\begin{matrix}\frac{4}{3}\end{matrix} n^3 + O(n^2)</math> arithmetic operations using a technique based on Householder reduction.<ref name=Demmel/><ref name=Trefethen/>
| |
| | |
| Solution of the system of linear equations for the [[tridiagonal matrix]]
| |
| costs ''O(n)'' operations, so the complexity grows like ''O(n<sup>3</sup>)+k*O(n)'', where ''k'' is an iteration number, which is better than for the direct inversion. However for small number of iterations such transformation may not be practical.
| |
| | |
| Also transformation to the [[Hessenberg form]] involves square roots and division operation, which are not hardware supported on some equipment like
| |
| [[digital signal processor]]s, [[FPGA]], [[ASIC]].
| |
| | |
| * Choice of the normalization constant ''C<sub>k</sub>'' and avoiding division.
| |
| | |
| On general purpose processors (e.g. produced by Intel) the execution time of addition, multiplication and division is approximately the same. But fast and/or low energy consuming hardware ([[digital signal processor]]s, [[FPGA]]
| |
| , [[ASIC]]) division is not supported by hardware, and so should be avoided.
| |
| For such hardware it is recommended to use ''C<sub>k</sub>=2<sup>n<sub>k</sub></sup>'', since division by powers of 2 is implemented by bit shift and supported on any hardware.
| |
| | |
| The same hardware usually supports only fixed point arithmetics: essentially works with integers. So the choice of the constant ''C<sub>k</sub>'' is especially important - taking too small value will lead to fast growth of the norm of ''b<sub>k</sub>'' and to the overflow; for too big ''C<sub>k</sub>'' vector ''b<sub>k</sub>'' will tend to zero.
| |
| The optimal value of ''C<sub>k</sub>'' is the eigenvalue of the corresponding eigenvector. So one should choose ''C<sub>k</sub>'' approximately the same.
| |
| | |
| == Usage ==
| |
| | |
| The main application of the method is the situation when an approximation to an eigenvalue is found and one needs to find the corresponding approximate eigenvector. In such situation the inverse iteration is the main and probably the only method to use.
| |
| So typically the method is used in combination with some other methods which allows to find approximate eigenvalues: the standard example is the [[bisection eigenvalue algorithm]], another example is the [[Rayleigh quotient iteration]] which is actually the same inverse iteration with the choice of the approximate eigenvalue as the [[Rayleigh quotient]] corresponding to the vector obtained on the previous step of the iteration.
| |
| | |
| There are some situations where the method can be used by itself, however they are quite marginal.
| |
| | |
| Dominant eigenvector.
| |
| The dominant eigenvalue can be easily estimated for any matrix.
| |
| For any [[Matrix_norm#Induced_norm|induced norm]] it is true that
| |
| <math>\left \| A \right \| \ge |\lambda| , </math> for any eigenvalue <math>\lambda</math>.
| |
| So taking the norm of the matrix as an approximate eigenvalue one can see that the method will converge to the dominant eigenvector.
| |
| | |
| Estimates based on statistics.
| |
| In some real-time applications one needs to find eigenvectors for matrices with a speed may be millions matrices per second. In such applications typically the statistics of matrices is known in advance and one can take as approximate eigenvalue the average eigenvalue for some large matrix sample,
| |
| or better one calculates the mean ratio of the eigenvalue to the trace or the norm of the matrix and eigenvalue is estimated as trace or norm multiplied on the average value the their ratio. Clearly such method can be used with much care and only in situations when the mistake in calculations is allowed.
| |
| Actually such idea can be combined with other methods to avoid too big errors.
| |
| | |
| == See also ==
| |
| *[[Power iteration]]
| |
| *[[Rayleigh quotient iteration]]
| |
| * [[List_of_numerical_analysis_topics#Eigenvalue_algorithms|List of eigenvalue algorithms]]
| |
| | |
| ==References==
| |
| {{reflist}}
| |
| | |
| ==External links==
| |
| *[http://www.physics.arizona.edu/~restrepo/475A/Notes/sourcea/node81.html Inverse Iteration to find eigenvectors], physics.arizona.edu
| |
| *[http://math.fullerton.edu/mathews/n2003/PowerMethodMod.html The Power Method for Eigenvectors], math.fullerton.edu
| |
| | |
| {{Numerical linear algebra}}
| |
| | |
| [[Category:Numerical linear algebra]]
| |