|
|
| Line 1: |
Line 1: |
| '''Augmented Lagrangian methods''' are a certain class of [[algorithm]]s for solving [[Constraint (mathematics)|constrained]] [[optimization (mathematics)|optimization]] problems. They have similarities to [[penalty method]]s in that they replace a constrained optimization problem by a series of unconstrained problems; the difference is that the augmented Lagrangian method adds an additional term to the unconstrained [[objective function|objective]]. This additional term is
| | Also, all these products only needed to take weekend breaks, holiday packages, there are any coupons pizza [http://Www.Thefreedictionary.com/coupons coupons] Here is more info about [http://lavvs.com/discounttirecouponssavingsinternetcoupons54467 Amazing Coupons] stop by our own web-page. . |
| designed to mimic a [[Lagrange multiplier]]. The augmented Lagrangian is not the same as the [[Lagrange multiplier|method of Lagrange multipliers]].
| |
| | |
| Viewed differently, the unconstrained objective is the [[Lagrange_multipliers#The_strong_Lagrangian_principle:_Lagrange_duality|Lagrangian]] of the constrained problem, with an additional penalty term (the '''augmentation''').
| |
| | |
| The method was originally known as the '''method of multipliers''', and was studied much in the 1970 and 1980s as a good alternative to penalty methods.
| |
| It was first discussed by [[Magnus Hestenes]] in 1969<ref>[[Magnus Hestenes|M.R. Hestenes]], "Multiplier and gradient methods", ''Journal of Optimization Theory and Applications'', 4, 1969, pp. 303–320</ref>
| |
| and by Powell in 1969<ref>M.J.D. Powell, "A method for nonlinear constraints in minimization problems", in ''Optimization'' ed. by R. Fletcher, Academic Press, New York, NY, 1969, pp. 283–298.</ref>
| |
| The method was studied by [[R. Tyrrell Rockafellar]] in relation to [[Fenchel duality]], particularly in relation to [[proximal-point method]]s, [[regularizion|Moreau–Yosida regularization]], and [[monotone operator|maximal monotone operator]]s: These methods were used in [[structural engineering|structural optimization]]<!-- French applied mathematicians, at least one in Texas, whose names escape senile K.W. -->. The method was also studied and implemented by [[Dimitri Bertsekas]], notably in his 1982 book,<ref>Dimitri P. Bertsekas, ''Constrained optimization and Lagrange multiplier methods'', Athena Scientific, 1996 (first published 1982)</ref> and with respect to [[Bregman divergence|entropic regularization]] (which accelerate the [[rate of convergence]] for his "exponential method of multipliers").
| |
| | |
| Since the 1970s, [[sequential quadratic programming]] (SQP) and [[interior point method]]s (IPM) have had increasing attention, in part because they more easily use [[sparse matrix]] [[subroutine]]s from [[numerical linear algebra|numerical]] [[numerical software|software libraries]], and in part because IPMs have proven complexity results via the theory of [[self-concordant function]]s. The augmented Lagrangian method was rejuvenated by the optimization systems [[Galahad library|LANCELOT]] and [[AMPL]], which allowed sparse matrix techniques to be used on seemingly dense but "partially separable" problems. The method is still useful for some problems.<ref>{{harvtxt|Nocedal|Wright|2006}}, chapter 17</ref> As of around 2007, there has been a resurgence of Augmented Lagrangian methods (and [[Augmented_Lagrangian_method#Alternating_direction_method_of_multipliers|ADMM]] in particular) in fields such as [[Total variation denoising|total-variation denoising]] and [[compressed sensing]]; for example, the [http://cascais.lx.it.pt/~mafonso/salsa.html SALSA] package was proposed in 2009.
| |
| | |
| A variant of the standard Augmented Lagrangian method that uses partial updates (similar to the [[Gauss-Seidel method]] for solving linear equations) is known as the '''[[Augmented_Lagrangian_method#Alternating_direction_method_of_multipliers|alternating direction method of multipliers]]''' or '''ADMM'''.
| |
| | |
| == General method ==
| |
| | |
| Let us say we are solving the following constrained problem:
| |
| :<math> \min f(\bold x) </math>
| |
| subject to
| |
| :<math> c_i(\bold x) = 0 ~\forall i \in I. </math>
| |
| | |
| This problem can be solved as a series of unconstrained minimization problems. For reference, we first list the [[penalty method]] approach:
| |
| :<math> \min \Phi_k (\bold x) = f (\bold x) + \mu_k ~ \sum_{i\in I} ~ c_i(\bold x)^2 </math>
| |
| The penalty method solves this problem, then at the next iteration it re-solves the problem
| |
| using a larger value of <math>\mu_k</math> (and using the old solution as the initial guess or "warm-start").
| |
| | |
| The augmented Lagrangian method uses the following unconstrained objective:
| |
| :<math> \min \Phi_k (\bold x) = f (\bold x) + \frac{\mu_k}{2} ~ \sum_{i\in I} ~ c_i(\bold x)^2 - \sum_{i\in I} ~ \lambda_i c_i(\bold x)</math>
| |
| and after each iteration, in addition to updating <math>\mu_k</math>, the variable <math>\lambda</math> is also updated according to the rule
| |
| :<math>\lambda_i \leftarrow \lambda_i - \mu_k c_i(\bold{x}_k) </math>
| |
| where <math>\bold{x}_k</math> is the solution to the unconstrained problem at the ''k''th step, i.e. <math>\bold{x}_k=\text{argmin} \Phi_k(\bold x) </math>
| |
| | |
| The variable <math>\lambda</math> is an estimate of the [[Lagrange multiplier]], and the accuracy of this estimate improves at every step. The major advantage of the method is that unlike the [[penalty method]], it is not necessary to take <math>\mu \rightarrow \infty</math> in order to solve the original constrained problem. Instead, because of the presence of the Lagrange multiplier term, <math>\mu</math> can stay much smaller.
| |
| | |
| The method can be extended to handle inequality constraints. For a discussion of practical improvements, see.<ref>{{harvtxt|Nocedal|Wright|2006}}, chapter 17</ref>
| |
| | |
| == Comparison with penalty methods ==
| |
| From,<ref>{{harvtxt|Nocedal|Wright|2006}}, chapter 17</ref> it is suggested that the augmented Lagrangian method is generally preferred to the quadratic penalty method
| |
| since there is little extra computational cost and the parameter <math>\mu</math> need not go to infinity, thus avoiding ill-conditioning.
| |
| | |
| == Alternating direction method of multipliers ==
| |
| The alternating direction method of multipliers (ADMM) is a variant of the augmented Lagrangian scheme that uses partial updates for the dual variables. This method is often applied to solve problems such as
| |
| | |
| <math> \min_x f(x) + g(x). </math>
| |
| | |
| This is equivalent to the constrained problem
| |
| | |
| <math> \min_{x,y} f(x) + g(y), \quad \text{subject to}\quad x = y. </math>
| |
| | |
| Though this change may seem trivial, the problem can now be attacked using methods of constrained optimization (in particular, the augmented Lagrangian method), and the objective function is separable in ''x'' and ''y''. The dual update requires solving a proximity function in ''x'' and ''y'' at the same time; the ADMM technique allows this problem to be solved approximately by first solving for ''x'' with ''y'' fixed, and then solving for ''y'' with ''x'' fixed. Rather than iterate until convergence (like the [[Jacobi method]]), the algorithm proceeds directly to updating the dual variable and then repeating the process. This is not equivalent to the exact minimization, but surprisingly, it can still be shown that this method converges to the right answer (under some assumptions). Because of this approximation, the algorithm is distinct from the pure augmented Lagrangian method.
| |
| | |
| The ADMM can be viewed as an application of the [[Douglas-Rachford splitting algorithm]], and the Douglas-Rachford algorithm is in turn an instance of the [[Proximal point algorithm]]; details can be found here.<ref>{{cite doi|10.1007/BF01581204|noedit}}</ref> There are several modern software packages that solve [[Basis pursuit]] and variants and use the ADMM; such packages include [http://yall1.blogs.rice.edu/ YALL1] (2009), [http://www.lx.it.pt/~mtf/SpaRSA/ SpaRSA] (2009) and [http://cascais.lx.it.pt/~mafonso/salsa.html SALSA] (2009).
| |
| | |
| == Software ==
| |
| Some well-known software packages that use the augmented Lagrangian method are <!-- [[MINOS (optimization software)|MINOS]], --> [[Galahad library|LANCELOT]]<!-- , --> and [[PENOPT|PENNON]].
| |
| The software [[MINOS (optimization software)|MINOS]] also uses an augmented Lagrangian method for some types of problems.
| |
| | |
| == See also ==
| |
| * [[Penalty method]]s
| |
| * [[Barrier method (mathematics)|Barrier method]]
| |
| * [[Barrier function]]
| |
| * [[Lagrange multiplier]]
| |
| | |
| ==References==
| |
| <references/>
| |
| | |
| ==Bibliography==
| |
| * {{Citation | last1=Nocedal | first1=Jorge | last2=Wright | first2=Stephen J. | title=Numerical Optimization | publisher=[[Springer-Verlag]] | location=Berlin, New York | edition=2nd | isbn=978-0-387-30303-1 | year=2006}}
| |
| | |
| {{optimization algorithms|convex}}
| |
| | |
| {{DEFAULTSORT:Augmented Lagrangian Method}}
| |
| [[Category:Optimization algorithms and methods]]
| |
Also, all these products only needed to take weekend breaks, holiday packages, there are any coupons pizza coupons Here is more info about Amazing Coupons stop by our own web-page. .