Worst-case evaluation complexity for unconstrained nonlinear optimization using high-order regularized models

Transcription

1 Worst-case evaluation comlexity for unconstrained nonlinear otimization using high-order regularized models E. G. Birgin, J. L. Gardenghi, J. M. Martínez, S. A. Santos and Ph. L. Toint 2 Aril 26 Abstract The worst-case evaluation comlexity for smooth (ossibly nonconvex) unconstrained otimization is considered. It is shown that, if one is willing to use derivatives of the objective function u to order (for ) and to assume Lischitz continuity of the -th derivative, then an ɛ-aroximate first-order critical oint can be comuted in at most O(ɛ (+)/ ) evaluations of the roblem s objective function and its derivatives. This generalizes and subsumes results known for = and = 2. Introduction Recent years have seen a surge of interest in the analysis of worst-case evaluation comlexity of otimization algorithms for nonconvex roblems (see, for instance, Vavasis [7], Nesterov and Polyak [6], Nesterov [4, 5], Gratton, Sartenaer and Toint [3], Cartis, Gould and Toint [3, 4, 5, 8], Bian, Chen and Ye [2], Bellavia, Cartis, Gould, Morini and Toint [], Graiglia, Yuan and Yuan [2], Vicente [8]). In articular the aer [6] was the first to show that a method using second derivatives can find an ɛ-aroximate first-order critical oint for an unconstrained roblem with Lischitz continuous Hessians in at most O(ɛ 3/2 ) evaluations of the objective function (and its derivatives), in contrast with methods using first-derivatives only, whose evaluation comlexity was known [4] to be O(ɛ 2 ) for roblems with Lischitz continuous gradients. The urose of the resent short aer is to show that, if one is willing to use derivatives u to order (for ) and to assume Lischitz continuity of the - th derivative, then an ɛ-aroximate first-order critical oint can be comuted in at most O(ɛ (+)/ ) evaluations of the objective function and its derivatives. This is achieved by the use of a regularization method very much in the sirit of the first- and second-order ARC methods described in [4, 5]. This work has been artially suorted by the Brazilian agencies FAPESP (grants 2/33-, 23/3447-6, 23/5475-7, 23/7375-, and 23/ ) and CNPq (grants 3432/2-7, 3957/24-, 3375/24-6, and 49326/23-7) and by the Belgian Fund for Scientific Research (FNRS). Deartment of Comuter Science, Institute of Mathematics and Statistics, University of São Paulo, Rua do Matão,, Cidade Universitária, 558-9, São Paulo, SP, Brazil. {egbirgin john}@ime.us.br Deartment of Alied Mathematics, Institute of Mathematics, Statistics, and Scientific Comuting, University of Caminas, Caminas, SP, Brazil. {martinez sandra}@ime.unicam.br Namur Center for Comlex Systems (naxys) and Deartment of Mathematics, University of Namur, 6, rue de Bruxelles, B-5 Namur, Belgium. hilie.toint@unamur.be

2 Birgin, Gardenghi, Martínez, Santos, Toint Comlexity with high-order models 2 2 A regularized -th order model and algorithm For, integer, consider the roblem min f(x), (2.) n x IR where we assume that f from IR n to IR is bounded below and -times continuously differentiable. We also assume that its -th derivative at x, the -th order tensor [ ] f xf(x) = x i... x i (x), i j {,...,n},j=,..., is Lischitz continuous, i.e. that there exists a constant L such that, for all x, y IR n, xf(x) xf(y) [] ( )! L x y. (2.2) In (2.2), [] is the tensor norm recursively induced by the Euclidean norm on the sace of -th order tensors, which is given by T [] def = max v = = v = T [v,..., v ], (2.3) where T [v,..., v j ] stands for the tensor of order q j resulting from the alication of the q-th order tensor T to the vectors v,..., v j. Let T (x, s) be the Taylor series of the function f(x + s) at x truncated at order T (x, s) def = f(x) + j= j! j xf(x)[s] j, (2.4) where the notation T [s] j stands for the tensor T alied j times to the vector s. Then Taylor s theorem, the identity ( ξ) dξ =, (2.5) the induced nature of [] and (2.2) imly that, for all x, s IR n, f(x + s) = T (x, s) + ( )! T (x, s) + ( )! T (x, s) + ( )! [ T (x, s) + ( ξ) ( )! ( ξ) xf(x + ξs)[s] dξ ( ξ) ( xf(x + ξs)[s] xf(x)[s] ) dξ ( ξ) xf(x + ξs)[s] xf(x)[s] dξ ] dξ max ξ [,] xf(x + ξs)[s] xf(x)[s] T (x, s) +! s max ξ [,] xf(x + ξs) xf(x) [] T (x, s) + L s +. (2.6)

3 Birgin, Gardenghi, Martínez, Santos, Toint Comlexity with high-order models 3 Following the more general argument develoed by Cartis, Gould and Toint [], consider now, for an arbitrary unit vector v, φ(α) = xf(x + αs)[v] and τ (α) = i= φ(i) ()α i /i!. Taylor s identity then gives that φ() τ () = ( 2)! Hence, since τ () = st (x, s)[v], ( xf(x + s) st (x, s))[v] = ( 2)! ( ξ) 2 [φ ( ) (ξ) φ ( ) ()] dξ. ( ξ) 2 [ xf(x + ξs) xf(x)][s] [v] dξ. Thus, using the symmetry of the derivative tensors, icking v to maximize the absolute value of the left-hand side and using (2.5), (2.3) and (2.2) successively, we obtain that xf(x + s) st (x, s) = ( 2)! ( 2)! [ ] s ( ξ) 2 ( xf(x + ξs) xf(x))[v] s dξ s [ ] [ ] ( ξ) 2 s dξ max ξ [,] ( xf(x + ξs) xf(x))[v] s ( )! max ξ [,] max w = = w ( xf(x + ξs) xf(x))[w,..., w ] s = = ( )! max ξ [,] xf(x + ξs) xf(x) [] s L s. s (2.7) In order to describe our algorithm, we also define the regularized Taylor series whose gradient is Note that m(x, s, σ) = T (x, s) + σ + s +, (2.8) sm(x, s, σ) = st (x, s) + σ s s s. (2.9) m(x,, σ) = T (x, ) = f(x). (2.) The minimization algorithm we consider is now detailed as Algorithm on the following age. Each iteration of this algorithm requires the aroximate minimization of m(x k, s, σ k ), but we may note that conditions (2.2) and (2.3) are relatively weak, in that they only require a decrease of the regularized -th order model and an aroximate first-order stationary oint: no global otimization of this ossibly nonconvex model is needed. Fortunately, this aroximate minimization does not involve additional comutations of f or of its derivatives at other oints than at x k, and therefore the exact method used and the resulting effort sent in Ste 2 have no imact on the evaluation comlexity. Also note that the numerator and

4 Birgin, Gardenghi, Martínez, Santos, Toint Comlexity with high-order models 4 Algorithm : AR Ste : Initialization. An initial oint x and an initial regularization arameter σ > are given, as well as an accuracy level ɛ. The constants θ, η, η 2, γ, γ 2, γ 3 and σ min are also given and satisfy θ >, σ min (, σ ], < η η 2 < and < γ < < γ 2 < γ 3. (2.) Comute f(x ) and set k =. Ste : Test for termination. Evaluate xf(x k ). If xf(x k ) ɛ, terminate with the aroximate solution x ɛ = x k. Otherwise comute derivatives of f from order 2 to at x k. Ste 2: Ste calculation. Comute the ste s k by aroximately minimizing the model m(x k, s, σ k ) with resect to s in the sense that the conditions m(x k, s k, σ k ) < m(x k,, σ k ) (2.2) and hold. sm(x k, s k, σ k ) θ s k (2.3) Ste 3: Accetance of the trial oint. Comute f(x k + s k ) and define ρ k = f(x k) f(x k + s k ) T (x k, ) T (x k, s k ). (2.4) If ρ k η, then define x k+ = x k + s k ; otherwise define x k+ = x k. Ste 4: Regularization arameter udate. Set [max(σ min, γ σ k ), σ k ] if ρ k η 2, σ k+ [σ k, γ 2 σ k ] if ρ k [η, η 2 ), [γ 2 σ k, γ 3 σ k ] if ρ k < η. (2.5) Increment k by one and go to Ste if ρ k η or to Ste 2 otherwise.

5 Birgin, Gardenghi, Martínez, Santos, Toint Comlexity with high-order models 5 denominator in (2.4) are strictly comarable, the latter being Taylor s aroximation of the former, without the regularization arameter laying any role. Iterations for which ρ k η (and hence x k+ = x k + s k ) are called successful and we def denote by S k = { j k ρ j η } the index set of all successful iterations between and k. We also denote by U k its comlement in {,..., k}, which corresonds to the index set of unsuccessful iterations between and k. Note that, before termination, each successful iteration requires the evaluation of f and its first derivatives, while only the evaluation of f is needed at unsuccessful ones. We first derive a very simle result on the model decrease obtained under condition (2.2). Lemma 2. The mechanism of Algorithm then guarantees that, for all k, T (x k, ) T (x k, s k ) σ k + s k +. (2.6) Proof. Observe that, because of (2.2) and (2.8), m(x k,, σ k ) m(x k, s k, σ k ) = T (x k, ) T (x k, s k ) σ k + s k + which imlies the desired bound. As a result, we obtain that (2.4) is well-defined for all k. We next deduce a simle uer bound on the regularization arameter σ k. Lemma 2.2 Suose that f is times continuously differentiable with Lischitz continuous -th derivative (i.e., that (2.2) holds). Then, for all k, [ def σ k σ max = max σ, γ ] 3L( + ). (2.7) ( η 2 ) Proof. Assume that σ k Using (2.6) and (2.6), we may then deduce that L( + ) ( η 2 ). (2.8) ρ k f(x k + s k ) T (x k, s k ) T (x k, ) T (x k, s k ) L( + ) σ k η 2 and thus that ρ k η 2. Then iteration k is very successful in that ρ k η 2 and σ k+ σ k. As a consequence, the mechanism of the algorithm ensures that (2.7) holds. Our next ste, very much in the line of the theory roosed in [5], is to show that the stelength cannot be arbitrarily small comared with the gradient of the objective function at the trial oint x k + s k.

6 Birgin, Gardenghi, Martínez, Santos, Toint Comlexity with high-order models 6 Lemma 2.3 Suose that f is times continuously differentiable with Lischitz continuous -th derivative (i.e., that (2.2) holds). Then, for all k, ( ) s k x f(x k + s k ). (2.9) L + θ + σ k Proof. Using the triangle inequality, (2.7), (2.9) and (2.3), we obtain that xf(x k + s k ) xf(x k + s k ) st (x k, s k ) + st (x k, s k ) + σ k s k s k s k +σ k s k L s k + sm(x k, s k, σ k ) + σ k s k [L + θ + σ k ] s k and (2.9) follows. We now bound the number of unsuccessful iterations as a function of the number of successful ones. Lemma 2.4 [5, Theorem 2.] The mechanism of Algorithm guarantees that, if for some σ max >, then k + S k σ k σ max, (2.2) ( + log γ ) + log log γ 2 log γ 2 ( σmax σ ). (2.2) Proof. The regularization arameter udate (2.5) gives that, for each k, γ σ j max[γ σ j, σ min ] σ j+, j S k, and γ 2 σ j σ j+, j U k. Thus we deduce inductively that We therefore obtain, using (2.2), that which then imlies that σ γ S k γ U k 2 σ k. S k log γ + U k log γ 2 log U k S k log γ log γ 2 + log γ 2 log ( σmax σ ( σmax σ ), ),

7 Birgin, Gardenghi, Martínez, Santos, Toint Comlexity with high-order models 7 since γ 2 >. The desired result (2.2) then follows from the equality k + = S k + U k and the inequality γ < given by (2.). Using all the above results, we are now in osition to state our main evaluation comlexity result. Theorem 2.5 Suose that f is times continuously differentiable with Lischitz continuous -th derivative (i.e., that (2.2) holds), and let f low be a lower bound on f. Then, given ɛ >, Algorithm needs at most f(x ) f low κ s ɛ + successful iterations (each involving one evaluation of f and its first derivatives) and at most ( f(x ) f low κ s + log γ ) + ( ) σmax log ɛ + log γ 2 log γ 2 σ iterations in total to roduce an iterate x ɛ such that xf(x ɛ ) ɛ, where σ max is given by (2.7) and where def κ s = + (L + θ + σ max ) +. η σ min Proof. At each successful iteration before termination, we have that f(x k ) f(x k + s k ) η (T (x k, ) T (x k, s k )) η σ min + s k + η σ min ( + )(L + θ + σ k ) + η σ min ( + )(L + θ + σ max ) + xf(x k + s k ) + ɛ +, where we used (2.4), (2.6), (2.5), (2.9), (2.7) and the fact that xf(x k +s k ) ɛ before termination. Thus we deduce that, for successful iterations and as long as termination does not occur, f(x ) f(x k+ ) =, j S k [f(x j ) f(x j + s j )] S k κ s ɛ + from which the desired bound on the number of successful iterations follows. Lemma 2.4 is then invoked to comute the uer bound on the total number of iterations. The comlexity bound of Theorem 2.5 can also be stated as the fact that, for a times continuously differentiable objective function with Lischitz continuous -th derivative, the global rate of convergence for the gradient s norm is O(k /(+) ).

8 Birgin, Gardenghi, Martínez, Santos, Toint Comlexity with high-order models 8 3 Final comments We have shown that, under suitable smoothness assumtions, an ɛ-aroximate stationary oint must be found by Algorithm in at most O(ɛ (+)/ ) iterations and function evaluations. This extension of results known for = and = 2 to arbitrary is made ossible by the introduction of two main innovations: weaker termination conditions on the model minimization subroblem (no global otimization is required at all) and a reformulation of the ratio of achieved versus redicted decreases where the model is limited to the Taylor aroximation. Of course, each iteration of the roosed algorithm requires the aroximate minimization of a tyically nonconvex regularized -th order model but this minimization does not involve additional comutation of the objective function of the original roblem or of its derivatives, and therefore its cost does not affect the evaluation comlexity of Algorithm. What numerical rocedure is best for this task is beyond the scoe of the resent note (for instance, one might think of alying an efficient first-order method on the model). Once this aer was submitted, the authors became aware of the interesting contribution by Dussault [] where the decrease measure (2.4) is also used to analyse a framework unifying the comlexity analysis of the cubic regularization algorithm and trust-region methods. It is of course interesting to consider if the extensions of the theories develoed for the firstand second-order cubic regularization methods for second-order otimality [8] or convexly constrained roblems [7] can be extended to higher-order regularization aroaches. We also note that Cartis et al. showed in [6] that a worst-case evaluation comlexity of order O(ɛ 3/2 ) is otimal for a large class of second-order methods alied on twice continuously differentiable roblems with Hölder continuous Hessians. The generalization of this otimality result for > 2 is also an oen question. Whether the aroach resented here has ractical imlications remains to be seen, since the aroximate model minimization could be costly even if comutation of f is avoided, and comuting derivatives for > 2 may often be out of reach. We conclude this aer by mentioning a simle extension which we anticiate could be useful in other contexts. We may, instead of minimizing f(x), slit the objective function into two arts and consider minimizing Φ(x) = h(x) + f(x) where h is bounded below and continuously differentiable. In this case, we then relace the model defined by (2.8) by m(x, s, σ) = h(x + s) + T (x, s) + σ s + /( + ) and, rovided we are ready to (aroximately) minimize this augmented model in Ste 2 of the algorithm, the above analysis remains unchanged. There are many ossible interesting choices for h(x): in the context of otimization with non-negative variables, a ossibility is, for examle, to choose h(x) = [max(x, )] 2 as a reformulation of the constraint x. What art of the objective function is easy enough to be included in the model m(x, s, σ) exlicitly and which art is better included using a Taylor series aroximation may deend on the roblem at hand, but it is interesting to note that the evaluation-comlexity bound resented in Theorem 2.5 is unaffected. Acknowledgements The authors are leased to thank Coralia Cartis and Nick Gould for valuable comments, in articular on the definition of the tensor Lischitz condition and associated material. Two anonymous referees also heled to imrove the final manuscrit.

9 Birgin, Gardenghi, Martínez, Santos, Toint Comlexity with high-order models 9 References [] S. Bellavia, C. Cartis, N. I. M. Gould, B. Morini, and Ph. L. Toint. Convergence of a regularized Euclidean residual algorithm for nonlinear least-squares. SIAM Journal on Numerical Analysis, 48(): 29, 2. [2] W. Bian, X. Chen, and Y. Ye. Comlexity analysis of interior oint algorithms for non-lischitz and nonconvex minimization. Mathematical Programming, Series A, 49:3 327, 25. [3] C. Cartis, N. I. M. Gould, and Ph. L. Toint. On the comlexity of steeest descent, Newton s and regularized Newton s methods for nonconvex unconstrained otimization. SIAM Journal on Otimization, 2(6): , 2. [4] C. Cartis, N. I. M. Gould, and Ph. L. Toint. Adative cubic overestimation methods for unconstrained otimization. Part I: motivation, convergence and numerical results. Mathematical Programming, Series A, 27(2): , 2. [5] C. Cartis, N. I. M. Gould, and Ph. L. Toint. Adative cubic overestimation methods for unconstrained otimization. Part II: worst-case function-evaluation comlexity. Mathematical Programming, Series A, 3(2):295 39, 2. [6] C. Cartis, N. I. M. Gould, and Ph. L. Toint. Otimal Newton-tye methods for nonconvex otimization. Technical Reort naxys-7-2, Namur Center for Comlex Systems (naxys), University of Namur, Namur, Belgium, 2. [7] C. Cartis, N. I. M. Gould, and Ph. L. Toint. An adative cubic regularization algorithm for nonconvex otimization with convex constraints and its function-evaluation comlexity. IMA Journal of Numerical Analysis, 32(4): , 22. [8] C. Cartis, N. I. M. Gould, and Ph. L. Toint. Comlexity bounds for second-order otimality in unconstrained otimization. Journal of Comlexity, 28:93 8, 22. [9] C. Cartis, N. I. M. Gould, and Ph. L. Toint. Evaluation comlexity of adative cubic regularization methods for convex unconstrained otimization. Otimization Methods and Software, 27(2):97 29, 22. [] C. Cartis, N. I. M. Gould, and Ph. L. Toint. Second-order otimality and (sometimes) beyond: characterization and evaluation comlexity in nonconvex otimization. Technical Reort (in rearation), Namur Center for Comlex Systems (naxys), University of Namur, Namur, Belgium, 26. [] J. P. Dussault. Simle unified convergence roofs for the trust-region and a new ARC variant. Technical reort, University of Sherbrooke, Sherbrooke, Canada, 25. [2] G. N. Graiglia, J. Yuan, and Y. Yuan. On the convergence and worst-case comlexity of trust-region and regularization methods for unconstrained otimization. Mathematical Programming, Series A, 52:49 52, 25. [3] S. Gratton, A. Sartenaer, and Ph. L. Toint. Recursive trust-region methods for multiscale nonlinear otimization. SIAM Journal on Otimization, 9():44 444, 28. [4] Yu. Nesterov. Introductory Lectures on Convex Otimization. Alied Otimization. Kluwer Academic Publishers, Dordrecht, The Netherlands, 24. [5] Yu. Nesterov. Modified Gauss-Newton scheme with worst-case guarantees for global erformance. Otimization Methods and Software, 22(3): , 27. [6] Yu. Nesterov and B. T. Polyak. Cubic regularization of Newton method and its global erformance. Mathematical Programming, Series A, 8():77 25, 26. [7] S. A. Vavasis. Black-box comlexity of local minimization. SIAM Journal on Otimization, 3():6 8, 993. [8] L. N. Vicente. Worst case comlexity of direct search. EURO Journal on Comutational Otimization, :43 53, 23.