Faster Alternating Direction Method of Multipliers with a Worst-case O(1/n 2 ) Convergence Rate

Transcription

1 Faster Alteratig Directio Method of Multipliers with a Worst-case O1/ Covergece Rate WENYI TIAN XIAOMING YUAN July 17, 016 Abstract. The alteratig directio method of multipliers ADMM is beig widely used for various covex programmig models with separable structures arisig i specifically may scietific computig areas. The ADMM s worst-case O1/ covergece rate measured by the iteratio complexity has bee established i the literature whe its pealty parameter is a costat, where is the iteratio couter. Research o ADMM s worst-case O1/ covergece rate, however, is still i its ifacy. I this paper, we suggest applyig a rule proposed recetly by Chambolle ad Pock to iteratively update the pealty parameter ad show that ADMM with this adaptive pealty parameter has a worst-case O1/ covergece rate. Without strog covexity requiremet o the objective fuctio, our assumptios o the model are mild ad ca be satisfied by some represetative applicatios. We test the LASSO model ad umerically verify the sigificat acceleratio effectiveess of the faster ADMM with a worst-case O1/ covergece rate. Moreover, the faster ADMM is more user-favorable tha the ADMM with a costat pealty parameter i seses of that it ca pursue solutios with very high accuracy ad that it is ot sesitive to the iitial value of the pealty parameter. Keywords. Covex programmig, Alteratig directio method of multipliers, Covergece rate, Acceleratio, First order methods 1 Itroductio May applicatios ca be modeled as covex miimizatio problems with certai separable structures. For example, their objective fuctios may be separable ad represetable by a sum of two or more fuctios. A represetative case is where oe fuctio represets a datafidelity term ad the other is a regularizatio term; this case arises frequetly i areas such as iverse problems, image processig ad machie learig. For such a separable covex miimizatio model, the alteratig directio method of multipliers ADMM proposed origially i [18] see also [6, 16] turs out to be a bechmark solver ad it is beig widely used for may applicatios i a broad spectrum of areas. We refer the reader to [3, 14, 17] for some review papers o ADMM. The covergece of ADMM has bee well studied i earlier literature, e.g., [15, 16], ad recetly its worst-case O1/ covergece rate measured by the iteratio complexity has also bee established i [4, 5]. Here, is the iteratio couter ad we refer to Ceter for Applied Mathematics, Tiaji Uiversity, Tiaji 30007, Chia. twymath@gmail.com Departmet of Mathematics, Hog Kog Baptist Uiversity, Hog Kog, Chia. This author was partially supported by the Geeral Research Fud from Hog Kog Research Grats Coucil: HKBU xmyua@hkbu.edu.hk 1

2 [30, 31, 3] for some semial work of covergece rate aalysis i terms of the iteratio complexity. The mai goal of this paper is to ivestigate uder which sceario the ADMM has a faster worst-case O1/ covergece rate. Note that it still remais ope whether or ot the ADMM ca achieve a worst-case O1/ covergece rate uder the geeral settig where o special assumptios o the model are posed. This ca be partially uderstood by the result i [8] see Theorem 11 therei. To discuss the possibility of a worst-case O1/ covergece rate for ADMM, we cocetrate o the covex miimizatio model 1.1 mi x X fx + gax where X R d is closed ad covex, f : X, ] ad g : R m, ] are closed, proper, ad covex fuctios, ad A R m d is full colum rak. Throughout the solutio set of 1.1 is assumed to be oempty. The model 1.1 ca be writte as 1. mi fx + gy s.t. Ax y = 0 x X, y R m, where y R m is a auxiliary variable. The, the iterative scheme of ADMM for 1. reads as x +1 = argmi fx + σ Ax y + λ } x X σ 1.3 y +1 = argmi gy + σ Ax +1 y + λ } y R m σ λ +1 = λ + σax +1 y +1, with σ > 0 the pealty parameter ad λ R m the Lagrage multiplier. There are differet ways to uderstad the ADMM. For example, it ca be regarded as a splittig versio of the classical augmeted Lagragia method i [6, 33]; it was also explaied i [15] as a applicatio of the Douglas-Rachford splittig method DRSM, which was first proposed i [11] for liear heat equatios ad the geeralized i [7] to the oliear case, to the dual problem of 1.1; ad it was further aalyzed i [13] that the ADMM is a applicatio of the proximal poit algorithm PPA i [8, 9] from the maximal mootoe operator perspective. A importat variat of ADMM is the proximal versio x +1 = argmi fx + σ Ax y + λ + 1 } x X σ x x Q 1.4 y +1 = argmi gy + σ Ax +1 y + λ } y R m σ λ +1 = λ + σax +1 y +1, where Q R d d is a positive defiite matrix; see e.g. [1, 1]. I particular, whe Q = µi σa T A with µ > σa T A, it is easy to see that the x-subproblem i 1.4 reduces to x +1 = argmi fx + µ x x + 1 x X µ AT λ + σax y }.

3 Whe X = R d, the miimizatio problem i the equatio above amouts to computig the proximal operator of f: 1.5 prox γf x := argmi fy + 1 y γ y x}, with γ > 0. Note that the proximal operator 1.5 has a closed-form solutio for some iterestig cases such as fx = x 1. I this case, the proximal versio 1.4 reduces to the liearized versio of ADMM for 1.: x +1 = argmi fx + µ x x + 1 x X µ AT λ + σax y } 1.6 y +1 = argmi gy + σ Ax +1 y + λ } y R m σ λ +1 = λ + σax +1 y +1, We refer to, e.g., [39, 40, 4], for some efficiet applicatios of the liearized versio of ADMM. Note that we oly cosider the case where the x-subproblem is proximally regularized because it is useful eough for most of the ADMM s applicatios. Techically, there is o difficulty if both the subproblems are proximally regularized, see e.g., [1]. I [4], the problem 1.1 with X = R d was writte as the saddle-poit problem 1.7 mi max fx + Ax, λ g λ }, λ R m x R d where g λ := sup y y, λ gy} is the Fechel cojugate of gy see, e.g., [34]. The, the geeralized primal-dual algorithm was proposed to solve 1.7: λ +1 = prox σg λ + σa x 1.8 x +1 = prox τf x τa T λ +1 x +1 = x +1 + θx +1 x, where prox is defied i 1.5, θ [0, 1] is a combiatio parameter, τ > 0 ad σ > 0 are two costats satisfyig τσa < 1 whe θ = 1. For the scheme 1.8 with θ = 1, its worst-case O1/ covergece rate i the ergodic sese was established i [4]. The, the scheme 1.8 was exteded i [3] with the combiatio parameter θ [ 1, 1]; some correctio steps were combied with the primal-dual step ad the covergece was established uder the coditio τσ 1 + θ A T A < 1. 4 The covergece ad covergece rate for scheme 1.8 with θ [ 1, 1] were further established i [, 37] if f is strogly covex. As aalyzed i [4], the primal-dual algorithm 1.8 with θ = 1 is equivalet to the applicatio of the liearized also called precoditioed versio of ADMM with Q = σ 1 I τaa T ad τσa < 1. Moreover, i [4] see also [5], the authors suggested choosig the ivolved parameters θ, τ ad σ dyamically, istead of costats, i 1.8. That is, cosider the geeralized primal-dual algorithm with dyamical parameters: λ +1 = prox σ g λ + A x 1.9 x +1 = prox τf x τ A T λ +1 x +1 = x +1 + θ +1 x +1 x. 3

4 A worst-case O1/ covergece rate of 1.9 i the ergodic sese was established i [4] provided that f is further assumed to be strogly covex with the modulus γ ad these parameters satisfy the coditios 1.10 θ +1 = γτ, τ +1 = θ +1 τ, +1 = /θ +1. It turs out that these coditios are crucial for establishig the desired O1/ covergece rates i [4, 5]. Thus, compared with the primal-dual scheme 1.8 with costat parameters, the scheme 1.10 possesses a higher covergece rate i terms of the iteratio complexity despite that it additioally requires to determie three parameter sequeces. These existig results strogly ispire us to cosider uder which coditios the ADMM 1.3 with dyamically adjusted pealty parameters has a worst-case O1/ covergece rate. I the literature, there are some works related to how to derive a worst-case O1/ covergece rate for the ADMM; which either require stroger assumptios or are eligible oly for some variats of the origial ADMM scheme 1.3. I [9, 19], the followig more geeral problem was cosidered: 1.11 mi fx + gy s.t. Ax + By = b, where f : R d 1, ] ad g : R d, ] are closed, proper ad covex fuctios, A R m d1, B R m d ad b R m. It was show i [9] that if g is strogly covex ad the ADMM s pealty parameter σ satisfies the coditio σ < κυ g /B where υ g is the strog covexity modulus of g ad κ is the positive root of the equatio z 3 + z z 1, the the residual of the costraits i 1.11 is decreasig i order of o1/ i a oergodic sese while the measuremet of the error of the objective fuctio i the primal model 1.11 is still i order of o1/ 1. I [19], it was show that if both f ad g are strogly covex with g beig further assumed to be quadratic; ad if the pealty parameter σ is chose as σ 3 υ f υg < ρa T AρB T B, where υ f ad υ g are the strog covexity modulus of f ad g, respectively; ad ρ is the spectral radius of a matrix, the the acceleratio step proposed i [3] ca be combied 1 I our discussio, for simplicity, we do ot differetiate the O1/ ad o1/ also O1/ ad o1/ rates because of two reasos. First, they are of the same order i the worst-case ature; thus usually their differece is ot that sigificat. Secod, techically, for some basic operator splittig methods it is ot hard to improve a O1/ rate to o1/ or from O1/ to o1/, see, e.g., [7]. 4

5 with ADMM to yield a O1/ covergece rate. That is, for the scheme x +1 = argmi fx + σ Ax + Bŷ b ˆλ } x σ y +1 = argmi gy + σ Ax +1 + By b ˆλ } y σ 1.1 λ +1 = ˆλ σax +1 + By +1 b α +1 = α/ ŷ +1 = y +1 + α 1 y +1 y α +1 ˆλ +1 = λ +1 + α 1 α +1 λ +1 λ with α } iteratively updated from α 0 = 1, it has a worst-case O1/ covergece rate i a oergodic sese, where the accuracy of a iterate is measured by the error of the objective fuctio of the dual problem of I this paper, we will establish a worst-case O1/ covergece rate i the ergodic sese for both the origial ADMM scheme 1.3 ad the proximal versio 1.4, uder the coditio that the pealty parameter σ is iteratively adjusted by a specific rule similar as the oe i [4, 5]. The restrictio of the pealty parameter is mild ad ca be automatically determied with a give iitial value see.5. We also show that the proximity of the Lagrage multiplier λ } to the optimal value is reduced o a O1/ rate. Our assumptios o the model 1.1 are give at the begiig of Sectio. Note that we do ot assume ay strog covexity o the objective fuctio of the model 1.1 as existig work such as [7, 9, 19, 36]. Fially, we would metio that some covergece rate results i the asymptotical sese ca be established for ADMM if further assumptios are assumed. For example, the asymptotical liear covergece rate of ADMM was established i [, 0] for the special case of 1. where both fuctios are quadratic. But this type of aalysis is ot the focus of this paper. The rest of the paper is orgaized as follows. I Sectio, a faster ADMM with a worstcase O1/ covergece rate is proposed ad some remarks are give. We first prove the covergece for the faster ADMM i Sectio 3 ad the establish its worst-case O1/ covergece rate i Sectio 4. I Sectio 5, we elaborate o the coectio betwee the faster ADMM ad the primal-dual algorithm i [4]. I Sectio 6, we test the LASSO model ad report some prelimiary umerical results; some coclusios are also draw based o these umerical results. Faster ADMM with a Worst-case O1/ Covergece Rate As metioed, we cosider the origial ADMM 1.3 ad its proximal versio 1.4 simultaeously for 1.. So we relax the restrictio of Q i 1.4 ad oly require it to be positive semi-defiite i our discussios. To derive a worst-case O1/ covergece rate for the ADMM, our assumptios o the model 1.1 are summarized as follows. Assumptio: Both fx ad gx are proper, lower semicotiuous l.s.c. ad covex fuctios; gx is smooth ad g is Lipschitz cotiuous with costat 1 γ ; ad A is full colum rak. 5

6 We propose the faster ADMM with a worst-case O1/ covergece rate i Algorithm 1. First, we otice that with the above assumptio, it follows from [1, Theorem 18.15] that the followig iequality holds:.1 gµ gν + gν, µ ν + γ gµ gν, µ, ν R m. Algorithm 1: Faster ADMM with a worst-case O1/ covergece rate Specify a iteger κ > 0 as the frequecy of adjustig the pealty parameter ; a iitial value of σ 0 > 0; ad x 0, y 0, λ 0 X R m R m. Let 1 γ be the Lipschitz cotiuity costat of g. Choose a positive semi-defiite matrix Q R d d. For the + 1-th iteratio, perform the followig steps:. x +1 = argmi fx + Ax y + λ σ } + x X x x Q y +1 = argmi gy + Ax +1 y + λ } y R m λ +1 = λ + Ax +1 y +1 where the pealty parameter is updated every κ iteratios by the rule.3 = σ κ, where κ is the largest iteger o greater tha κ ad the sequece σ i} is give by.4 σ i+1 = σ i 1 + γ σi, with σ 0 = σ 0 > 0. Remark.1. Note that the sequece σ i } is specified with a give σ 0 ad the rule.3 adjusts the pealty parameter } after every κ iteratios by assigig each σ i to κ iteratios of the faster ADMM. cosecutively. Thus, the sequece } is also automatically determied with a give iitial value σ 0 ad a frequecy κ. For the extreme case where κ = 1, the we have σ 0 σ 1 σ κ = 1,, σ 0 σ 1 σ which meas the sequece } is iteratively updated by.5 +1 = 1 + γσ. This is precisely the formula for updatig the parameters of the accelerated primal-dual scheme i [4, 5]. If we choose κ > 1, e.g., κ = 10, the we have κ = 10, σ 0 σ 1 σ }} 9 σ 10 σ 11 σ 19 ; }} σ 0 σ 1 6

7 ad for the geeral κ, we have σ 0 σ κ 1 σ κ σ κ 1 σ sκ σ }}}} s+1κ 1, }} σ 0 σ 1 σ s For a iteger, it ca be decomposed as = sκ + j with 0 j κ 1. Thus, it follows, 0 j κ,.6 +1 =, 1 + γσ j = κ 1. Clearly, the sequece } is mootoically o-icreasig. Thus, it is easy to uderstad that if the sequece } is updated o a too high frequecy, i.e., κ is small, the the sequece } decreases too fast ad the step size for updatig the dual variable becomes too small. I this case, the efficiecy of the scheme. may be deteriorated. O the other had, if the sequece } is updated o a too low frequecy, i.e., κ is huge, as we shall show i Theorems 4.1 ad 4. see 4.18 ad 4.1, the coefficiet of the O1/ covergece rate to be established is too large ad it deteriorates the covergece also. So, i geeral we do ot recommed too extreme values of κ for the proposed faster ADMM.. As we shall umerically verify later, medium values such as κ = 5 or 10, usually ca result i very good umerical results eve though the optimal choice, we believe, still depeds o the specific applicatio of the abstract model 1. ad the data set uder cosideratio. 3 Covergece Recall our mai goal is to establish a worst-case O1/ covergece rate for Algorithm 1. First of all, i this sectio we prove the covergece of Algorithm 1. Let the Lagragia fuctio of 1. be defied as 3.1 Lx, y; λ := fx + gy + λ, Ax y with λ R m the Lagrage multiplier. Further, we defie Ω := X R m R m. The, solvig 1. is equivalet to fidig a saddle poit of Lx, y; λ. This is equivalet to solvig the variatioal iequality: fidig w = x, y, λ Ω such that 3. Θv Θv + F w, w w 0, w Ω, where 3.3 v = x, w = x y, F w = y λ AT λ λ, Θv = fx + gy. Ax + y To prove the covergece of the sequece w } geerated by Algorithm 1, we first give a lemma. Lemma 3.1. Let the sequece w = x, y, λ } be geerated by Algorithm 1. The we have 3.4 Θv Θv +1 + F w +1 + Φy, y +1 + M w +1 w, w w +1 γ gy gy +1, w Ω, 7

8 where 3.5 Φy, y +1 = σ A T y +1 y Q 0 0 y +1 y ad M = 0 I σ I Proof. First, the optimality coditio of the x-subproblem i. is 3.6 fx fx +1 + A T Ax +1 y + λ + σ Qx +1 x, x x +1 0, x X. Usig the updatig scheme for λ +1 i., we obtai 3.7 fx fx +1 + A T λ +1 + A T y +1 y + Qx +1 x, x x +1 0, x X. I additio, it follows from.1 ad the updatig scheme for λ +1 i. that 3.8 gy gy +1 λ +1, y y +1 γ gy gy +1, y R m. Together with 3.7, 3.8 ad the followig idetity 3.9 Ax +1 + y λ +1 λ = 0, we obtai the result 3.4. Theorem 3.1. Let w = x, y, λ be a saddle poit of 3.1 ad the sequece w = x, y, λ } be geerated by Algorithm 1. The we have 3.10 w +1 w M +1 w w M w w +1 M, where M is give i 3.5. Proof. From the y-subproblem i., it holds The, it follows from 3.4 with w = w that λ +1 = gy +1, λ = gy. M w +1 w, w w +1 Θv+1 Θv + F w +1, w +1 w Φy, y +1, w +1 w + γ λ λ +1. Takig w = w +1 i 3. ad addig F w +1, w +1 w to both sides, we have 3.1 Θv +1 Θv + F w +1, w +1 w 0. The optimality coditio of y-subproblem i. is 3.13 gy gy +1 λ +1, y y +1 0, y R m, ad it also satisfies 3.14 gy gy λ, y y 0, y R m. 8

9 Takig y = y i 3.13 ad y = y +1 i 3.14, ad summarizig them, we obtai 3.15 Φy, y +1, w +1 w = λ +1 λ, y +1 y 0. Therefore, it follows from 3.11, 3.1 ad 3.15 that 3.16 Usig the idetity M w +1 w, w w +1 γ λ λ +1. w +1 w M + M w +1 w, w w +1 = w w M w w +1 M, ad 3.16, we have 3.17 w +1 w M + γ λ λ +1 w w M w w +1 M. Moreover, accordig to.6, it holds γ σ 1 σ+1. Thus, it follows from 3.17, 3.18 ad the defiitio of M i 3.5 that the assertio 3.10 holds. The assertio 3.10 implies that the sequece w } geerated by Algorithm 1 is strictly cotractive with respect to the solutio set of 3.1, which essetially implies the covergece of the sequece w }. We prove a lemma ad the preset the covergece result. Recall the special case of the rule.3-.4 with κ = 1, i.e., the sequece } is updated by.5. The, as proved i [4], we have lim γ = 1, which meas O1/. Now, we geeralize this result to the geeral rule.3-.4 with a geeral frequecy κ. Lemma 3.. For } updated by the rule.3-.4 i Algorithm 1, we have Oκ/. γ Proof. For } updated by the rule.3-.4, we have lim s s σ s = 1. For a iteger, it ca be writte as = sκ + j, 0 j κ 1, where s = κ. Thus, we have γ lim = κ, because κ is a fixed iteger. This yields = σ κ Oκ/ ad completes the proof. Theorem 3.. Let w = x, y, λ } be the sequece geerated by Algorithm 1. The, the sequece w } coverges to a saddle poit w = x, y, λ of 3.1. The covergece of Algorithm 1 for model 1. is thus established i seses of 3.19 lim Ax y = 0, ad lim fx + gy } = fx + gy. Proof. Takig the summatio of 3.10 for from 0 to N, we have N w w +1 M w 0 w M 0, =0 9

10 which idicates 3.0 lim w w +1 M = 0. By the defiitio of M give i 3.5, we have 3.1 lim Qx x +1 = 0, lim y y +1 = 0 ad lim 1 λ λ +1 = 0. It follows from 3.10 that w w M is bouded. Recall the idetity 3.9, we kow that Ax Ax is bouded. Sice A is full colum rak ad Oκ/ show i Lemma 3., we coclude that the sequece w } has a cluster poit. Let us deote it by w. The, substitutig it ito 3.4 ad usig 3.1, we obtai Θv Θv + F w, w w 0, w Ω, which implies that w is a saddle poit of 3.1. Furthermore, by 3.9 ad 3.1, it immediately yields 3. lim Ax 1 y = lim λ λ 1 = 0. 1 Takig w = x, y, λ i 3., we get ad thus 3.3 lim if fx + gy fx + gy λ, Ax y, fx + gy } fx + gy. I additio, we set w = w i 3.4 ad simplify it as fx + gy fx +1 + gy +1 + λ, Ax +1 y +1 + y +1 y, Ax +1 y +1 + M w +1 w, w +1 w Thus, usig the boudedess of w w M, the results i 3.0, 3.1 ad 3., ad Oκ/ show i Lemma 3., we obtai 3.4 fx + gy lim sup fx + gy }. Therefore, 3.3 ad 3.4 imply the assertio 3.19 ad the proof is complete. 4 Worst-case O1/ Covergece Rate I this sectio, we establish a worst-case O1/ covergece rate for Algorithm 1. Our aalysis is based o the saddle-poit reformulatio of the model mi x X max λ R m Lx, λ := fx + Ax, λ g λ }. As aalyzed i [4], the followig partial primal-dual gap ca be used to measure the accuracy of a iterate geerated by Algorithm 1: 4. G B1 B x, λ = max λ B Lx, λ mi x B 1 Lx, λ, 10

11 where B 1 B is a ope subset of U := X R m cotaiig a solutio poit x, λ of the saddle-poit reformulatio 4.1. Accordig to [4], we kow that for ˆx, ˆλ i B 1 B, if G B1 B ˆx, ˆλ 0, the ˆx, ˆλ is also a solutio poit of 4.1. Hece, we ca defie x, λ B 1 B as a approximate solutio to 4.1 with a accuracy of ϵ if 4.3 G B1 B x, λ ϵ with ϵ > 0. We additioally eed the followig otatio for further aalysis: x σ A T A + Q u =, H λ = 1. 0 I Now we start to establish a worst-case O1/ covergece rate for Algorithm 1. First, based o the first-order optimality coditios of the subproblems i., we prove a lemma. Lemma 4.1. Let x, y, λ } be geerated by Algorithm 1. The, we have fx fx +1 + A T Ax +1 x + λ λ 1 + λ σ Qx +1 x, x x +1 0, x X ; 4.6 λ +1 = gy +1 ; 4.7 g λ g λ +1 Ax +1 1 λ +1 λ, λ λ +1 γ λ λ +1 0, λ R m. Proof. From the optimality coditios of the subproblems i., we have the first two assertios trivially. Furthermore, because of 4.6, we have 4.8 y +1 g λ +1. Sice g is Lipschitz cotiuous with costat 1 γ, it follows from [1, Theorem 18.15] that g is γ-strogly covex. The, together with 4.8, we obtai g λ g λ +1 γ 4.9 y +1, λ λ +1 λ λ +1 0, λ R m. Thus, usig the updatig scheme for λ i., we prove the assertio 4.7. The, we prove oe more lemma, based o which the worst-case O1/ covergece rate of Algorithm 1 ca be easily obtaied. Lemma 4.. Let u = x, λ } be geerated by Algorithm 1 ad U = X R m. The, we have 4.10 Lx+1, λ Lx, λ S +1 u S u, u U, θ +1 where ad θ := S u := 1 u u H + θ λ λ 1 + θ Ax x, λ λ 1. 11

12 Proof. First, it follows from 4.5 ad 4.7 that fx + Ax, λ +1 g λ fx +1 + Ax +1, λ g λ + A T 1 A + Qx +1 x, x x +1 + λ +1 λ, λ λ +1 Ax x +1, λ +1 λ Ax x+1, λ λ γ λ λ +1, u U. With 4.1, the defiitio of H i 4.4 ad the equality Ax x+1, λ λ we ca reformulate 4.1 as = Ax x, λ λ 1 Ax+1 x, λ λ 1, 4.14 Lx +1, λ Lx, λ +1 + H u +1 u, u u +1 Ax x +1, λ +1 λ Ax x, λ λ Ax+1 x, λ λ 1 + γ λ λ +1 Ax x +1, λ +1 λ Ax +1 x Further, usig the idetity σ Ax x, λ λ 1 1 σ 1 σ λ λ 1 + γ λ λ +1, u U. 1 H u +1 u, u u +1 = u u H u u +1 H u u +1 H, we have u u H + σ σ 1 σ λ λ 1 + Ax x, λ λ 1 1 Ax x +1 + x x +1 Q γ λ λ +1 + x +1 x Q + 1 λ +1 λ + Ax x +1, λ +1 λ + Lx +1, λ Lx, λ +1, u U. Recall.3,.4 ad.6. We have 4.16 θ +1 = +1, The, it easily yields from 4.15 that 1 + γ 1 θ S u 1 θ +1 S +1 u + Lx +1, λ Lx, λ +1, u U. The proof is complete. 1

13 Now, we establish a worst-case O1/ covergece rate i the ergodic sese for Algorithm 1. Theorem 4.1 O1/ covergece rate i the ergodic sese. Let u = x, λ } be the sequece geerated by Algorithm 1. Let σ 0 T = ad ũ = 1 σ 0 u i+1. σ i T σ i The, we have i=0 i= G B1 B x, λ c κ = O 1, where c > 0 is a costat. Proof. Multiplyig the iequality 4.10 i Lemma 4. by σ 0 σ i, summarizig it with i = 0, 1,,, ad usig the property of covex fuctios f ad g, we obtai 4.19 T L x, λ Lx, λ + σ 0 +1 S +1 u 1 u u 0 H 0, where λ 1 := λ 0 is set. It follows from Lemma 3. that Oκ/. With the defiitio of T, this immediately yields that T O /κ. Sice S +1 u is oegative, we have 4.0 L x, λ Lx, λ 1 T u u 0 H 0 c κ u u 0 H 0, u U. From the result of Theorem 3.1, we kow that the sequece u } is bouded, the its liear average ũ is also bouded. Therefore, for some ope bouded subset B 1 B of U cotaiig the sequece ũ }, the estimate 4.0 implies the assertio The proof is complete. The assertio 4.18 meas that ũ calculated by iteratios of Algorithm 1 is a approximate solutio of the saddle-poit reformulatio 4.1 with a accuracy of O1/. Therefore, a worst-case O1/ covergece rate i the ergodic sese is established for Algorithm 1. Moreover, we ca obtai a stroger covergece rate for the sequece of λ } i terms of the proximity to the optimal value λ i the followig theorem. This is a by-product of this paper. Theorem 4. Covergece rate of dual variable. Let u = x, λ be a solutio poit of the saddle-poit reformulatio 4.1, ad u = x, λ } be geerated by Algorithm 1. The, we have 4.1 λ +1 λ +1 σ 0 u u 0 H 0 = O κ. Proof. Takig u = u i 4.19, ad recallig the fact we have 4. L x, λ Lx, λ 0, σ 0 +1 S +1 u 1 u u 0 H 0. From 4.4 ad the defiitio of S i 4.11, we ca easily have λ λ +1 S +1 u. Therefore, the result 4.1 follows from the above two iequalities ad Lemma

14 5 Coectio with Primal-Dual Algorithms I this sectio, we elaborate o the coectio betwee Algorithm 1 with the primal-dual algorithm proposed i [4] for 1.1 with X = R d. Recall the relatioship betwee the primal-dual algorithm 1.9 ad the liearized versio of ADMM 1.6 for 1. with X = R d. Here, we cosider a symmetric versio of 1.9 with a exchage of the primal ad dual roles of the variables: x +1 = prox τf x τ A T λ 5.1 λ +1 = prox σ g λ + Ax +1 λ +1 = λ +1 + θ +1 λ +1 λ. First, usig the Moreau s idetity i [34]: prox σg λ + σprox g/σ λ/σ = λ, we ca reformulate the λ-subproblem i 5.1 as 5. λ +1 = λ + Ax +1 y +1, where 5.3 y +1 = argmi y x +1 = argmi x gy + Secod, the x-subproblem i 5.1 ca be rewritte as fx + θ where ad 5.5 = argmi x y Ax +1 λ }. 1Ax + Ax θ x x + C τ 1 I A T A fx + Ax y + 1 ϑ λ + 1 x x τ 1 I A T A ϑ := θ ϑ λ 1 y + λ C := τ A T λ + θ λ λ 1 1 λ + θ λ λ 1. Thus, the primal-dual algorithm 5.1 is equivalet to the followig scheme fx + Ax y + 1 ϑ λ x +1 = argmi x y +1 = argmi gy + y λ +1 = λ + Ax +1 y +1, + 1 x x τ 1 I A T A Ax +1 y + λ } }, + ϑ λ 1 which ca be viewed as a liearized versio of the ADMM with varyig pealty parameters x +1 = argmi fx + Ax y + 1 ϑ λ λ 1 } + ϑ x 5.6 y +1 = argmi gy + Ax +1 y + λ } y λ +1 = λ + Ax +1 y +1 } } 14

15 whose x-subproblem is proximally regularized by the term 1 x x τ 1 I A T A. Furthermore, as show i Lemma 4., ϑ = 0 if θ = / 1. Therefore, the scheme 5.6 ca be simplified as x +1 = argmi fx + Ax y + λ } x 5.7 y +1 = argmi gy + Ax +1 y + λ } y λ +1 = λ + Ax +1 y +1. This is precisely the applicatio of the stadard ADMM scheme 1.3 with varyig pealty parameters to 1.. Therefore, the coclusio is that if = θ 1 is satisfied, the the primal-dual algorithm 5.1 is the case of Algorithm 1 with Q = τ 1 I A T A. This coectio ca be regarded as a geeralizatio of the elaboratio i [35] o these two methods with costat parameters. Remark 5.1. Differet from the x-λ-λ scheme i 5.1, the accelerated scheme 1.9 discussed i [4] performs iteratios i order of λ-x-x. With a aalysis similar as above, we ca derive that the scheme 1.9 is equivalet to 5.8 where λ +1 = argmi λ z +1 = argmi f z + τ z x +1 = x τ Ax +1 + z +1, g λ + τ A T λ + z 1 δ x τ + 1 λ λ σ 1 I τ AA T A T λ +1 + z x } τ δ = τ θ τ 1 τ 1. If θ = τ /τ 1, the 5.8 ca be simplified as 5.9 λ +1 = argmi g λ + τ A T λ + z x 1 + λ τ λ λ σ 1 z +1 = argmi f z + τ A T λ +1 + z x } z τ x +1 = x τ Ax +1 + z +1, δ x 1 τ } I τ AA T } which is a applicatio of the proximal versio of the ADMM with varyig pealty parameters τ to the problem 5.10 mi f z + g λ λ R m,z R d s.t. A T λ + z = 0. 15

16 Note that the dual problem of 1. with X = R d ca be writte as 5.11 max λ R m f A T λ + g λ }. Thus, the problem 5.10 is equivalet to the dual problem 5.11 of 1. by itroducig variable z = A T λ ad regardig x i 5.9 as the Lagrage multiplier of the costrait i Numerical Results As metioed, the ADMM has foud may applicatios i a broad spectrum of areas. I this sectio, we take the LASSO model i [38] as a illustrative example to umerically verify the efficiecy of the proposed faster ADMM. The LASSO model is probably the simplest yet most represetative applicatio to which ADMM ca be applied. All the codes were writte by MATLAB R01a ad all experimets were performed o a desktop with Widows 8 system ad a ItelR CoreTM i5-4570s CPU processor.9ghz with a 8GB memory. 6.1 Experimet Setup The LASSO model i [38] is 6.1 mi αx x Dx c}, where x 1 := d i=1 x i, D R l d is a desig matrix usually with l d, l is the umber of data poits, d is the umber of features, c R l is the respose vector ad α > 0 is a regularizatio parameter. The LASSO model provides a sparse estimatio of x whe there are more features tha data poits. It ca also be explaied as a model for fidig a sparse solutio of the uder-determied system of liear equatios Dx = c. Obviously, model 6.1 ca be rewritte as 6. mi x,y αx Dy c s.t. x y = 0, which is a special case of model 1. with fx = αx 1, gy = 1 Dy c ad m = d, X = R d, A = I d d. For 6., it is easy to see that the assumptios posed i Sectio are satisfied. Particulary, the Lipschitz cotiuity costat of g is D T D ad γ = 1/D T D. Thus, applyig the proposed faster ADMM. with Q = 0 to 6., we obtai the scheme x +1 = argmi αx 1 + x y + λ } x y +1 = argmi y Dy c + x +1 y + λ } λ +1 = λ + x +1 y +1 where the pealty parameter } is updated by +1 = σ κ 16

17 with κ beig a give iteger ad σ s+1 = σ s 1 + σs /D T D, startig from a give σ 0 = σ 0. As metioed i may literatures, the x-subproblem i 6.3 has its closed-form solutio give by x +1 = S α/σ y λ, where S δ x is the soft-thresholdig operator [10] defied as S δ x i = 1 δ/ x i + x i, i = 1,,, d, ad the y-subproblem has its solutio give by y +1 = I + D T D 1 x +1 + λ + D T c. I our experimets, we specify the LASSO model as follows. We take l = 1500 ad d = 5000; the matrix D i 6.1 is geerated by the MATLAB fuctio rad with 1500 by 5000 etries; all colums are ormalized afterwards; the sparse vector x R 5000 is geerated by the MATLAB fuctio sprad with 100 ozero etries; the vector c is set as Dx + η with oise vector η N0, 0.001; the regularizatio parameter α is set as D T c /10, γ = 1/D T D. To implemet the scheme 6.3 ad avoid loss of efficiecy possibly caused by codig skills, we use the widely-used MATLAB ADMM package dowloaded at boyd/papers/admm/, with the oly slight revisio of usig a adaptive pealty parameter. For the iterate w +1 = x +1, y +1, λ +1 T geerated by 6.3, it is easy to see that it satisfies the variatioal iequality 6.4 Θv Θv +1 + F w +1, w w +1 + σ y +1 y 1 λ +1 λ for ay w R d R d R d, where x x+1, 0, λ λ +1 v = x, y T, Θv = fx + gy ad F w +1 = λ +1, λ +1, x +1 y +1 T. Thus, w +1 is a solutio of 6. if ad oly if y +1 = y ad λ +1 = λ, see similar aalysis i [3, 4, 41]. Hece, we ca use the stoppig criterio 1 } 6.5 max y +1 y, λ +1 λ dε, where ε > 0 is a tolerace. 6. Efficiecy Compariso We use the origial ADMM 1.3 with a costat pealty parameter as the bechmark to verify the O1/ covergece rate of Algorithm 1. We test differet costat pealty parameters for the origial ADMM 1.3 ad Algorithm 1 also starts from the same costat for each compariso. I Figure 1, we plot evaluatio of the objective fuctio value of the LASSO model 6.1 with respect to the iteratio umber for the origial ADMM

18 with a costat pealty parameter σ 0 deoted by ADMM ad Algorithm 1 with varyig pealty parameter startig from the same σ 0 deoted by FADMM. We report the results for σ 0 = 10, 0,, up to 000. We have tested a umber of other cases of σ 0 ad the compariso is similar except for some extreme cases where σ 0 is very small. For Algorithm 1, four choices of κ = 1, 5, 10, 0 are tested. For each case, we plot the evolutio of primal-dual residual i 6.5 with log-log scale axis i Figure 1. These curves show clearly that Algorithm 1 coverges absolutely faster tha the origial ADMM 1.3 with a give costat parameter. To discer the actual rate more clearly, we also set a bechmark rate of 100/ ad plot its decay. As we ca see, the speed of the decay of the primal-dual residuals is eve faster tha the bechmark rate of 100/ ; so the established O1/ covergece rate is just a worse-case estimate of the speed ad we ca easily witess faster speed empirically. Moreover, the values of the objective fuctio i each iteratio are also plotted i Figure 1, where the decay of the objective fuctio by Algorithm 1 is also much faster tha that of the origial ADMM, especially whe the iitial pealty parameter is large. I Table 1, we compare the iteratio umbers of these two ADMM schemes for some choices of σ 0 ad differet tolerace i the stoppig criterio 6.5. I this table, meas the stoppig criterio 6.5 is ot satisfied after 5000 iteratios. These data show that the origial ADMM 1.3 with a give costat pealty parameter ca easily fail, especially whe pursuig high-precisio solutios; while Algorithm 1 usually performs very well except for the extreme case where κ = 1. For most of the case, medium values of κ such as κ = 5 or 10 are good choices for our experimets. 6.3 Sesitivity I the literature, it is well kow that the efficiecy of the origial ADMM 1.3 with a costat pealty parameter heavily depeds o the value of this parameter ad it seems we still lack of ay geeral strategy to tue this costat. This ca also be see i Table 1. Ideed, it is the mai disadvatage of the ADMM 1.3 ad it usually requires users to tue this parameter to fid a specific value appropriate to a give problem. I this subsectio, we test the sesitivity to the iitial value of σ 0 of Algorithm 1. We oly report the results whe κ = 10 for succictess. I Figure, for differet cases of the tolerace ϵ i the stoppig criterio 6.5, we plot the evaluatios of the iteratio umbers with respect to differet choices of σ 0 whose values vary from 10 to 10 3 with a equal distace of 0.1, where the horizotal axis is i log scale. It is clearly demostrated that Algorithm 1 is much more robust to the value of the iitial pealty parameter especially whe σ 0 is larger equal tha 10.0, eve whe high-precisio solutios are pursued. This is a sigificat advatage for implemetig ADMM-type algorithms. 6.4 Coclusios Based o the umerical experimets, we fid that for most of the cases, Algorithm 1 with a give iitial value of the pealty parameter outperforms the origial ADMM 1.3 with the same costat pealty parameter; thus the theoretically faster O1/ covergece rate is umerically verified. Moreover, our prelimiary umerical results show that Algorithm 1 ca pursue solutios i very high precisios with few iteratios; this ca be hardly achieved by the origial ADMM 1.3 with a costat pealty parameter uless it is very well tued. Last, i our experimets, we also show that Algorithm 1 is much less sesitive to the iitial value of the pealty parameter. Ideed, for the LASSO model, Algorithm 1 is very robust with respect to the iitial value of σ 0. These features idicate that compared with the 18

19 primal dual residual ADMM FADMMκ = 1 FADMMκ = 5 FADMMκ = 10 FADMMκ = 0 100/ a Iitial σ 0 = 10.0 fx + gy ADMM FADMMκ = 1 FADMMκ = 5 FADMMκ = 10 FADMMκ = b Iitial σ 0 = 10.0 primal dual residual ADMM FADMMκ =1 FADMMκ =5 FADMMκ = 10 FADMMκ = 0 100/ c Iitial σ 0 = 50.0 fx + gy ADMM FADMMκ = 1 FADMMκ = 5 FADMMκ = 10 FADMMκ = d Iitial σ 0 = 50.0 primal dual residual ADMM FADMMκ =1 FADMMκ =5 FADMMκ = 10 FADMMκ = 0 100/ e Iitial σ 0 = fx + gy ADMM FADMMκ = 1 FADMMκ = 5 FADMMκ = 10 FADMMκ = f Iitial σ 0 = Figure 1: The decay of the primal-dual residual ad objective fuctio, for LASSO model by ADMM ad faster ADMM with κ = 1, 5, 10, 0 ad differet iitial pealty parameter σ 0, the tolerace ε = i the stoppig criterio 6.5. origial ADMM 1.3 with a costat pealty parameter, theoretically Algorithm 1 has a higher order of worst-case covergece rate ad umerically it performs more efficietly ad robustly. These user-favorable features make Algorithm 1 more attractive to a umber of applicatios. 19

20 primal dual residual ADMM FADMMκ =1 FADMMκ =5 FADMMκ = 10 FADMMκ = 0 100/ g Iitial σ 0 = 00.0 fx + gy ADMM FADMMκ = 1 FADMMκ = 5 FADMMκ = 10 FADMMκ = h Iitial σ 0 = primal dual residual ADMM FADMMκ =1 FADMMκ =5 FADMMκ = 10 FADMMκ = 0 100/ i Iitial σ 0 = fx + gy ADMM FADMMκ = 1 FADMMκ = 5 FADMMκ = 10 FADMMκ = j Iitial σ 0 = primal dual residual ADMM FADMMκ =1 FADMMκ =5 FADMMκ = 10 FADMMκ = 0 100/ k Iitial σ 0 = fx + gy ADMM FADMMκ = 1 FADMMκ = 5 FADMMκ = 10 FADMMκ = l Iitial σ 0 = Figure 1: co t The decay of the primal-dual residual ad objective fuctio, for LASSO model by ADMM ad faster ADMM with κ = 1, 5, 10, 0 ad differet iitial pealty parameter σ 0, the tolerace ε = i the stoppig criterio 6.5. Refereces [1] H. H. BAUSCHKE AND P. L. COMBETTES, Covex aalysis ad mootoe operator theory i Hilbert spaces, Spriger, New York, 011. [] D. BOLEY, Local liear covergece of the alteratig directio method of multipliers o quadratic or liear programs, SIAM J. Optim., 3 013, pp

21 ADMM FADMMκ = ADMM FADMMκ = 10 Iteratio Number Iteratio Number σ0 a ε = σ0 b ε = ADMM FADMMκ = ADMM FADMMκ = Iteratio Number Iteratio Number σ0 c ε = σ0 d ε = ADMM FADMMκ = ADMM FADMMκ = Iteratio Number Iteratio Number σ0 e ε = σ0 f ε = Figure : The iteratio umbers of ADMM ad faster ADMM with differet iitial parameter σ 0 ad κ = 10 for solvig LASSO model with differet tolerace i the stoppig criterio 6.5. [3] S. BOYD, N. PARIKH, E. CHU, B. PELEATO, AND J. ECKSTEIN, Distributed optimizatio ad statistical learig via the alteratig directio method of multipliers, Foud. Treds Mach. Learig, 3 010, pp [4] A. CHAMBOLLE AND T. POCK, A first-order primal-dual algorithm for covex problems with applicatios to imagig, J. Math. Imagig Visio, , pp

22 [5], O the ergodic covergece rates of a first-order primal-dual algorithm, Math. Program., 015. [6] T. F. CHAN AND R. GLOWINSKI, Fiite elemet approximatio ad iterative solutio of a class of mildly oliear elliptic equatios, Techical Report, Computer Sciece Departmet, Staford Uiversity, CA, [7] E. CORMAN AND X. M. YUAN, A geeralized proximal poit algorithm ad its covergece rate, SIAM J. Optim., 4 014, pp [8] D. DAVIS AND W. YIN, Covergece rate aalysis of several splittig schemes, arxiv: , 014. [9], Faster covergece rates of relaxed Peacema-Rachford ad ADMM uder regularity assumptios, arxiv: , 014. [10] D. L. DONOHO AND Y. TSAIG, Fast solutio of l 1 -orm miimizatio problems whe the solutio may be sparse, IEEE Tras. Iform. Theory, , pp [11] J. DOUGLAS, JR. AND H. H. RACHFORD, JR., O the umerical solutio of heat coductio problems i two ad three space variables, Tras. Amer. Math. Soc., , pp [1] J. ECKSTEIN, Some saddle-fuctio splittig methods for covex programmig, Optim. Methods Softw., , pp [13] J. ECKSTEIN AND D. P. BERTSEKAS, O the Douglas-Rachford splittig method ad the proximal poit algorithm for maximal mootoe operators, Math. Program., , pp [14] J. ECKSTEIN AND W. YAO, Uderstadig the covergece of the alteratig directio method of multipliers: Theoretical ad computatioal perspectives, Pac. J. Optim., , pp [15] D. GABAY, Applicatios of the method of multipliers to variatioal iequalities, i Augmeted Lagragia Methods: Applicatios to the Solutio of Boudary-Valued Problems, M. Forti ad R. Glowiski, eds., North-Hollad Publishig Co., Amsterdam, 1983, pp [16] D. GABAY AND B. MERCIER, A dual algorithm for the solutio of oliear variatioal problems via fiite elemet approximatio, Comput. Math. Appl., 1976, pp [17] R. GLOWINSKI, O alteratig directio methods of multipliers: A historical perspective, i Modelig, Simulatio ad Optimizatio for Sciece ad Techology, W. Fitzgibbo, Y. A. Kuzetsov, P. Neittaam?ki, ad O. Piroeau, eds., Spriger Netherlads, 014, pp [18] R. GLOWINSKI AND A. MARROCCO, Sur l approximatio par élémets fiis et la résolutio par péalisatio-dualité d ue classe de problèmes de Dirichlet o liéaires, R.A.I.R.O., R 1975, pp [19] T. GOLDSTEIN, B. O DONOGHUE, S. SETZER, AND R. BARANIUK, Fast alteratig directio optimizatio methods, SIAM J. Imagig Sci., 7 014, pp

23 [0] D. HAN AND X. M. YUAN, Local liear covergece of the alteratig directio method of multipliers for quadratic programs, SIAM J. Numer. Aal., , pp [1] B. S. HE, L.-Z. LIAO, D. HAN, AND H. YANG, A ew iexact alteratig directios method for mootoe variatioal iequalities, Math. Program., 9 00, pp [] B. S. HE, Y. F. YOU, AND X. M. YUAN, O the covergece of primal-dual hybrid gradiet algorithm, SIAM J. Imagig Sci., 7 014, pp [3] B. S. HE AND X. M. YUAN, Covergece aalysis of primal-dual algorithms for a saddle-poit problem: from cotractio perspective, SIAM J. Imagig Sci., 5 01, pp [4], O the O1/ covergece rate of the Douglas-Rachford alteratig directio method, SIAM J. Numer. Aal., 50 01, pp [5], O o-ergodic covergece rate of Douglas-Rachford alteratig directio method of multipliers, Numer. Math., , pp [6] M. R. HESTENES, Multiplier ad gradiet methods, J. Optim. Theory Appl., , pp [7] P.-L. LIONS AND B. MERCIER, Splittig algorithms for the sum of two oliear operators, SIAM J. Numer. Aal., , pp [8] B. MARTINET, Régularisatio diéquatios variatioelles par approximatios successives, Rev. Fraçaise Iformat. Recherche Opératioelle, , pp [9] J. MOREAU, Proximité et dualité das u espace hilbertie, Bull. Soc. Math. Frace, , pp [30] A. NEMIROVSKI, Prox-method with rate of covergece O1/t for variatioal iequalities with Lipschitz cotiuous mootoe operators ad smooth covex-cocave saddle poit problems, SIAM J. Optim., , pp [31] Y. NESTEROV, Gradiet methods for miimizig composite fuctios, Math. Program., , pp [3] Y. E. NESTEROV, A method for solvig the covex programmig problem with covergece rate O1/k, Dokl. Akad. Nauk SSSR, , pp I Russia. Traslated i Soviet Math. Dokl., , pp [33] M. J. D. POWELL, A method for oliear costraits i miimizatio problems, i Optimizatio, R. Fletcher, ed., Academic Press, New York, 1969, pp [34] R. T. ROCKAFELLAR, Covex aalysis, Priceto Uiversity Press, Priceto, NJ, [35] R. SHEFI, Rate of covergece aalysis for covex osmooth optimizatio algorithms, PhD Thesis, Tel Aviv Uiversity, Israel, 015. [36] M. TAO AND X. M. YUAN, O the optimal liear covergece rate of a geeralized proximal poit algorithm, arxiv: ,

24 [37] W. Y. TIAN AND X. M. YUAN, Covergece aalysis of primal-dual based methods for total variatio miimizatio with fiite elemet approximatio, submited, 014. [38] R. TIBSHIRANI, Regressio shrikage ad selectio via the lasso, J. Roy. Statist. Soc. Ser. B, , pp [39] X. F. WANG AND X. M. YUAN, The liearized alteratig directio method of multipliers for Datzig selector, SIAM J. Sci. Comput., 34 01, pp. A79 A811. [40] J. F. YANG AND X. M. YUAN, Liearized augmeted Lagragia ad alteratig directio methods for uclear orm miimizatio, Math. Comp., 8 01, pp [41] X. M. YUAN, Alteratig directio method for covariace selectio models, J. Sci. Comput., 51 01, pp [4] X. ZHANG, M. BURGER, AND S. OSHER, A uified primal-dual algorithm framework based o Bregma iteratio, J. Sci. Comput., , pp

25 Table 1: Iteratio umbers of ADMM ad faster ADMM for solvig the LASSO model with differet iitial parameter σ 0 ad differet tolerace ε i the stoppig criterio 6.5. meas the iteratio umber is beyod 5000 ε σ 0 = 1.0 ADMM FADMMκ = FADMMκ = FADMMκ = FADMMκ = σ 0 = 10.0 ADMM FADMMκ = FADMMκ = FADMMκ = FADMMκ = σ 0 = 0.0 ADMM FADMMκ = FADMMκ = FADMMκ = FADMMκ = σ 0 = 50.0 ADMM FADMMκ = FADMMκ = FADMMκ = FADMMκ = σ 0 = ADMM FADMMκ = FADMMκ = FADMMκ = FADMMκ = σ 0 = 00.0 ADMM FADMMκ = FADMMκ = FADMMκ = FADMMκ = σ 0 = ADMM 4471 FADMMκ = FADMMκ = FADMMκ = FADMMκ = σ 0 = ADMM FADMMκ = FADMMκ = FADMMκ = FADMMκ = σ 0 = ADMM FADMMκ = FADMMκ = FADMMκ = FADMMκ =