|
|
| (One intermediate revision by one other user not shown) |
| Line 1: |
Line 1: |
| {{Regression bar}}
| | Andera is what you can call her but she by no means really favored that title. Mississippi is where her house is but her spouse wants them to transfer. What I adore performing is football but I don't have the time lately. He works as a bookkeeper.<br><br>My site - [http://1.234.36.240/fxac/m001_2/7330 real psychic] |
| The method of '''iteratively reweighted least squares''' ('''IRLS''') is used to solve certain optimization problems. It solves [[objective function]]s of the form:
| |
| | |
| :<math>\underset{\boldsymbol\beta} {\operatorname{arg\,min}} \sum_{i=1}^n w_i (\boldsymbol\beta) \big| y_i - f_i (\boldsymbol\beta) \big|^2, </math>
| |
| | |
| by an [[iterative method]] in which each step involves solving a [[weighted least squares]] problem of the form:
| |
| | |
| :<math>\boldsymbol\beta^{(t+1)} = \underset{\boldsymbol\beta} {\operatorname{arg\,min}} \sum_{i=1}^n w_i (\boldsymbol\beta^{(t)}) \big| y_i - f_i (\boldsymbol\beta) \big|^2. </math>
| |
| | |
| IRLS is used to find the [[maximum likelihood]] estimates of a [[generalized linear model]], and in [[robust regression]] to find an [[M-estimator]], as a way of mitigating the influence of outliers in an otherwise normally-distributed data set. For example, by minimizing the least absolute error rather than the least square error.
| |
| | |
| Although not a linear regression problem, [[Weiszfeld's algorithm]] for approximating the [[geometric median]] can also be viewed as a special case of iteratively reweighted least squares, in which the objective function is the sum of distances of the estimator from the samples.
| |
| | |
| One of the advantages of IRLS over [[linear programming|linear]] and [[convex programming]] is that it can be used with [[Gauss–Newton]] and [[Levenberg–Marquardt]] numerical algorithms.
| |
| | |
| == Examples ==
| |
| | |
| === ''L''<sub>1</sub> minimization for sparse recovery ===
| |
| IRLS can be used for '''[[L1 norm|<math>\ell</math><sub>1</sub>]]''' minimization and smoothed '''[[Lp quasi-norm|<math>\ell</math><sub>p</sub>]]''' minimization, ''p'' < 1, in the [[compressed sensing]] problems. It has been proved that the algorithm has a linear rate of convergence for '''<math>\ell</math><sub>1</sub>''' norm and superlinear for '''<math>\ell</math><sub> ''t''</sub>''' with ''t'' < 1, under the [[restricted isometry property]], which is generally a sufficient condition for sparse solutions.<ref>{{Cite conference
| |
| | last1 = Chartrand | first1 = R.
| |
| | last2 = Yin | first2 = W.
| |
| | title = Iteratively reweighted algorithms for compressive sensing
| |
| | booktitle = IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2008
| |
| | pages = 3869–3872
| |
| | date = March 31 – April 4, 2008
| |
| | url = http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=4518498}}
| |
| </ref><ref>{{cite doi|10.1002/cpa.20303}}</ref> In most practical situations, the restricted isometry property is not satisfied.
| |
| | |
| === ''L<sup>p</sup>'' norm linear regression ===
| |
| To find the parameters '''''β''''' = (''β''<sub>1</sub>, …,''β''<sub>''k''</sub>)<sup>T</sup> which minimize the [[Lp space|''L<sup>p</sup>'' norm]] for the [[linear regression]] problem,
| |
| | |
| :<math>
| |
| \underset{\boldsymbol \beta}{ \operatorname{arg\,min} }
| |
| \big\| \mathbf y - X \boldsymbol \beta \|_p
| |
| =
| |
| \underset{\boldsymbol \beta}{ \operatorname{arg\,min} }
| |
| \sum_{i=1}^n \left| y_i - X_i \boldsymbol\beta \right|^p ,
| |
| </math>
| |
| | |
| the IRLS algorithm at step ''t''+1 involves solving the [[Linear least squares (mathematics)#Weighted linear least squares|weighted linear least squares]] problem:<ref>{{cite book
| |
| |chapter=6.8.1 Solutions that Minimize Other Norms of the Residuals
| |
| |title=Matrix algebra
| |
| |last=Gentle |first=James
| |
| |isbn=978-0-387-70872-0
| |
| |doi=10.1007/978-0-387-70873-7
| |
| |publisher=Springer |location=New York
| |
| |year=2007
| |
| }}</ref>
| |
| | |
| :<math>
| |
| \boldsymbol\beta^{(t+1)}
| |
| =
| |
| \underset{\boldsymbol\beta}{ \operatorname{arg\,min} }
| |
| \sum_{i=1}^n w_i^{(t)} \left| y_i - X_i \boldsymbol\beta \right|^2
| |
| =
| |
| (X^{\rm T} W^{(t)} X)^{-1} X^{\rm T} W^{(t)} \mathbf{y},
| |
| </math>
| |
| | |
| where ''W''<sup>(''t'')</sup> is the [[diagonal matrix]] of weights, usually with all elements set initially to: | |
| | |
| :<math>w_i^{(0)} = 1</math>
| |
| | |
| and updated after each iteration to:
| |
| | |
| :<math>w_i^{(t)} = \big|y_i - X_i \boldsymbol \beta ^{(t)} \big|^{p-2}.</math>
| |
| | |
| In the case ''p'' = 1, this corresponds to [[least absolute deviation]] regression (in this case, the problem would be better approached by use of [[linear programming]] methods,<ref name=Pfeil>William A. Pfeil,
| |
| ''[http://www.wpi.edu/Pubs/E-project/Available/E-project-050506-091720/unrestricted/IQP_Final_Report.pdf Statistical Teaching Aids]'', Bachelor of Science thesis, [[Worcester Polytechnic Institute]], 2006</ref> so the result would be exact) and the formula is:
| |
| | |
| :<math>w_i^{(t)} = \frac{1}{\big|y_i - X_i \boldsymbol \beta ^{(t)} \big|}.</math>
| |
| | |
| To avoid dividing by zero, [[Regularization (mathematics)|regularization]] must be done, so in practice the formula is:
| |
| | |
| :<math>w_i^{(t)} = \frac{1}{\text{max}(\delta, \big|y_i - X_i \boldsymbol \beta ^{(t)} \big|)}.</math>
| |
| | |
| where <math>\delta</math> is some small value, like 0.0001.<ref name=Pfeil />
| |
| | |
| == Notes ==
| |
| {{Reflist}}
| |
| | |
| == References ==
| |
| * [http://amath.colorado.edu/courses/7400/2010Spr/lecture23.pdf University of Colorado Applied Regression lecture slides]
| |
| * [http://sepwww.stanford.edu/public/docs/sep103/antoine2/paper_html/index.html Stanford Lecture Notes on the IRLS algorithm by Antoine Guitton]
| |
| * [http://www.mai.liu.se/~akbjo/LSPbook.html Numerical Methods for Least Squares Problems by Åke Björck] (Chapter 4: Generalized Least Squares Problems.)
| |
| * [http://graphics.stanford.edu/~jplewis/lscourse/SLIDES.pdf Practical Least-Squares for Computer Graphics. SIGGRAPH Course 11]
| |
| | |
| {{DEFAULTSORT:Iteratively Reweighted Least Squares}}
| |
| [[Category:Regression analysis]]
| |
| [[Category:Least squares]]
| |
Andera is what you can call her but she by no means really favored that title. Mississippi is where her house is but her spouse wants them to transfer. What I adore performing is football but I don't have the time lately. He works as a bookkeeper.
My site - real psychic