Cumulative Step-size Adaptation on Linear Functions

Transcription

1 Cumulatve Step-sze Adaptaton on Lnear Functons Alexandre Chotard, Anne Auger, Nkolaus Hansen To cte ths verson: Alexandre Chotard, Anne Auger, Nkolaus Hansen. Cumulatve Step-sze Adaptaton on Lnear Functons. Parallel Problem Solvng From Nature, Sprnger, 0. <hal v> HAL Id: hal Submtted on 6 Jun 0 v, last revsed 9 Jun 0 v HAL s a mult-dscplnary open access archve for the depost and dssemnaton of scentfc research documents, whether they are publshed or not. The documents may come from teachng and research nsttutons n France or abroad, or from publc or prvate research centers. L archve ouverte plurdscplnare HAL, est destnée au dépôt et à la dffuson de documents scentfques de nveau recherche, publés ou non, émanant des établssements d ensegnement et de recherche franças ou étrangers, des laboratores publcs ou prvés.

2 Cumulatve Step-sze Adaptaton on Lnear Functons Alexandre Chotard, Anne Auger and Nkolaus Hansen TAO team, INRIA Saclay-Ile-de-France, LRI, Pars-Sud Unversty, France Abstract. The CSA-ES s an Evoluton Strategy wth Cumulatve Step sze Adaptaton, where the step sze s adapted measurng the length of a so-called cumulatve path. The cumulatve path s a combnaton of the prevous steps realzed by the algorthm, where the mportance of each step decreases wth tme. Ths artcle studes the CSA-ES on compostes of strctly ncreasng wth affne lnear functons through the nvestgaton of ts underlyng Markov chans. Rgorous results on the change and the varaton of the step sze are derved wth and wthout cumulaton. The step-sze dverges geometrcally fast n most cases. Furthermore, the nfluence of the cumulaton parameter s studed. Keywords: CSA, cumulatve path, evoluton path, evoluton strateges, step-sze adaptaton Introducton Evoluton strateges ESs are contnuous stochastc optmzaton algorthms searchng for the optmum of a real valued functon f : R n R. In the, λ-es, n each teraton, λ new chldren are generated from a sngle parent pont X R n by addng a random Gaussan vector to the parent, X R n X + σn 0, C. Here, σ R + s called step-sze and C s a covarance matrx. The best of the λ chldren, accordng to f, becomes the parent of the next teraton. To acheve reasonably fast convergence, step sze and covarance matrx have to be adapted throughout the teratons of the algorthm. In ths paper, C s the dentty and we nvestgate the so-called Cumulatve Step-sze Adaptaton CSA [,9]. In CSA, a cumulatve path s ntroduced, whch s a combnaton of all steps the algorthm has made, where the mportance of a step decreases exponentally wth tme. Arnold and Beyer studed the behavor of CSA on sphere, cgar and rdge functons [,,3,6] and on dynamcal optmzaton problems where the optmum moves randomly [4] or lnearly [5]. In ths paper, we study the behavour of the, λ-csa-es on compostes of strctly ncreasng wth affne lnear functons, e.g. f : x expx. Because the CSA- ES s nvarant under translaton, under change of an orthonormal bass rotaton and reflecton, and under strctly ncreasng transformatons of the f-value, we nvestgate, w.l.o.g., f : x x. Lnear functons model the stuaton when the current parent s far here nfntely far from the optmum of a smooth functon. To be far from the

3 optmum means that the dstance to the optmum s large, relatve to the step-sze σ. Ths stuaton s undesrable and threatens premature convergence. The stuaton should be handled well, by ncreasng step wdths, by any search algorthm. Solvng lnear functons s also useful to prove convergence ndependently of the ntal state on more general functon classes. In Secton we ntroduce the, λ-csa-es, and some of ts characterstcs on lnear functons. In Sectons 3 and 4 we study lnσ t wthout and wth cumulaton, respectvely. Secton 5 presents an analyss of the varance of the logarthm of the stepsze and n Secton 6 we summarze our results. Notatons In ths paper, we denote t the teraton or tme ndex, n the search space dmenson, N 0, a standard normal dstrbuton,.e. a normal dstrbuton wth mean zero and standard devaton. The multvarate normal dstrbuton wth mean vector zero and covarance matrx dentty wll be denoted N 0, I n, the th order statstc of λ standard normal dstrbutons N :λ, and Ψ :λ ts dstrbuton. If x = x,, x n R n s a vector, then [x] wll be ts value on the th dmenson, that s [x] = x. A random varable X dstrbuted accordng to a law L wll be denoted X L. The CSA-ES We denote wth X t the parent at the t th teraton. From the parent pont X t, λ chldren are generated: Y t, = X t +σ t ξ t, wth [[, λ]], and ξ t, N 0, I n, ξ t, [[,λ]]..d. Due to the, λ selecton scheme, from these chldren, the one mnmzng the functon f s selected: X t+ = argmn{fy, Y {Y t,,..., Y t,λ }}. Ths latter equaton mplctly defnes the random varable ξ t as X t+ = X t + σ t ξ t. In order to adapt the step-sze, the cumulatve path s defned as p t+ = cp t + c c ξ t wth 0 < c. The constant /c represents the lfe span of the nformaton contaned n p t, as after /c generatons p t s multpled by a factor that approaches /e 0.37 for c 0 from below ndeed c /c exp. The typcal value for c s between / n and /n. We wll consder that p 0 N 0, I n as t makes the algorthm easer to analyze. The normalzaton constant c c n front of ξ t n Eq. s chosen so that under random selecton and f p t s dstrbuted accordng to N 0, I n then also p t+ follows N 0, I n. Hence the length of the path can be compared to the expected length of N 0, I n representng the expected length under random selecton. The step-sze update rule ncreases the step-sze f the length of the path s larger than the length under random selecton and decreases t f the length s shorter than under random selecton: c σ t+ = σ t exp d σ p t+ E N 0, I n

4 where the dampng parameter d σ determnes how much the step-sze can change and s set to d σ =. A smplfcaton of the update consders the squared length of the path [4]: c pt+ σ t+ = σ t exp. 3 d σ n Ths rule s easer to analyse and we wll use t throughout the paper. Prelmnary results on lnear functons. Selecton on the lnear functon, fx = [x], s determned by [X t ] + σ t [ξ [ t ] [X t ] + σ t ξt, for all whch s equvalent to ] [ξ t ] [ ] ξ for all where by defnton [ ] t, ξ t, s dstrbuted accordng to N 0,. Therefore the frst coordnate of the selected step s dstrbuted accordng to N :λ and all others coordnates are dstrbuted accordng to N 0,,.e. selecton does not bas the dstrbuton along the coordnates,..., n. Overall we have the followng result. Lemma. On the lnear functon fx = x, the selected steps ξ t t N of the, λ- ES are..d. and dstrbuted accordng to the vector ξ := N :λ, N,..., N n where N N 0, for. Because the selected steps ξ t are..d. the path defned n Eq. s an autonomous Markov chan, that we wll denote P = p t t N. Note that f the dstrbuton of the selected step would depend on X t, σ t as t s generally the case on non-lnear functons, then the path alone would not be a Markov Chan, however X t, σ t, p t would be an autonomous Markov Chan. In order to study whether the, λ-csa-es dverges geometrcally, we nvestgate the log of the step-sze change, whose formula can be mmedately deduced from Eq. 3: σt+ ln = c pt+ 4 σ t d σ n By summng up ths equaton from 0 to t we obtan t ln σt = c t p k σ 0 d σ t n k=. 5 We are nterested to know whether t lnσ t/σ 0 converges to a constant. In case ths constant s postve ths wll prove that the, λ-csa-es dverges geometrcally. We recognze thanks to 5 that ths quantty s equal to the sum of t terms dvded by t that suggests the use of the law of large numbers to prove convergence of 5. We wll start by nvestgatng the case wthout cumulaton c = Secton 3 and then the case wth cumulaton Secton 4. 3 Dvergence rate of CSA-ES wthout cumulaton In ths secton we study the, λ-csa-es wthout cumulaton,.e. c =. In ths case, the path always equals to the selected step,.e. for all t, we have p t+ = ξ t. We have proven n Lemma that ξ t are..d. accordng to ξ. Ths allow us to use the standard law of large numbers to fnd the lmt of t lnσ t/σ 0 as well as compute the expected log-step-sze change. 3

5 Proposton. Let σ := d σn E N :λ. On lnear functons, the, λ-csa- ES wthout cumulaton satsfes almost surely lm t t ln σ t/σ 0 = σ, and for all t N, Elnσ t+ /σ t = σ. Proof. We have dentfed n Lemma that the frst coordnate of ξ t s dstrbuted accordng to N :λ and the other coordnates accordng to N 0,, hence E ξ t = E [ξ t ] + n = [ξ E t ] = E N:λ + n. Therefore E ξ t /n = E N:λ /n. By applyng ths to Eq. 4, we deduce that Elnσt+ /σ t = /d σ nen:λ. Furthermore, as EN :λ EλN 0, = λ <, we have E ξ t <. The sequence ξ t t N beng..d accordng to Lemma, and beng ntegrable as we just showed, we can apply the strong law of large numbers on Eq. 5. We obtan t ln σt = t ξ k σ 0 d σ t n k=0 a.s. E ξ = E N t d σ n d σ n :λ The proposton reveals that the sgn of E N :λ determnes whether the stepsze dverges to nfnty. In the followng, we show that E N :λ ncreases n λ for λ and that the, λ-es dverges for λ 3. For λ = and λ =, the step-sze follows a random walk on the log-scale. Lemma. Let N [[,λ]] be ndependent random varables, dstrbuted accordng to N 0,, and N :λ the th order statstc of N [[,λ]]. Then E N : = E N : =. In addton, for all λ, E N :λ+ > E N :λ. Proof. see [7] for the full proof The dea of the proof s to use the symmetry of the normal dstrbuton to show that for two random varables U Ψ :λ+ and V Ψ :λ, for every event E where U < V, there exsts another event E counterbalancng the effect of E,.e E u v P U, V duv = E v u P U, V duv. We then have E N:λ+ E N :λ. As there s a non-neglgble set of events E3, dstnct of E and E, where U > V, we have EN:λ+ > EN :λ. For λ =, N : N 0, so EN: =. For λ = we have EN: + N: = EN 0, =, and snce the normal dstrbuton s symmetrc EN: = EN:, hence EN: =. We can now lnk Proposton and Lemma nto the followng theorem: Theorem. On lnear functons, for λ 3, the step-sze of the, λ-csa-es wthout cumulaton c = dverges geometrcally almost surely and n expectaton at the rate /d σ nen:λ,.e. t ln σt a.s. E σt+ ln = E N t d σ n :λ. 6 σ 0 σ t 4

6 For λ = and λ =, wthout cumulaton, the logarthm of the step-sze does an addtve unbased random walk.e. ln σ t+ = ln σ t + W t where E[W t ] = 0. More precsely W t /d σ χ n/n for λ =, and W t /d σ N: +χ n /n for λ =, where χ k stands for the ch-squared dstrbuton wth k degree of freedom. Proof. For λ >, from Lemma we know that EN:λ > EN : =. Therefore EN:λ > 0, hence Eq. 6 s strctly postve, and wth Proposton we get that the step-sze dverges geometrcally almost surely at the rate /d σ EN:λ. Wth Eq. 4 we have lnσ t+ = lnσ t + W t, wth W t = /d σ ξ t /n. For λ = and λ =, accordng to Lemma, EW t = 0. Hence lnσ t does an addtve unbased random walk. Furthermore ξ = N:λ + χ n, so for λ =, snce N : = N 0,, ξ = χ n. In [7] we extend ths result on the step-sze to [X t ], whch dverges geometrcally almost surely at the same rate, gven Eexp x /n /d σ < wth x ξ. 4 Dvergence rate of CSA-ES wth cumulaton We are now nvestgatng the, λ-csa-es wth cumulaton,.e. 0 < c <. The path P s then a Markov chan and contrary to the case where c = we cannot apply a LLN for ndependent varables to Eq. 5 n order to prove the almost sure geometrc dvergence. However LLN for Markov chans exst as well, provded the Markov chan satsfes some stablty propertes: n partcular, f the Markov chan P s ϕ-rreducble, that s, there exsts a measure ϕ such that every Borel set A of R n wth ϕa > 0 has a postve probablty to be reached n a fnte number of steps by P startng from any p 0 R n. In addton, the chan P needs to be postve, that s the chan admts an nvarant probablty measure π,.e., for any borelan A, πa = R n P x, AπA wth P x, A beng the probablty to transton n one tme step from x nto A, and Harrs recurrent whch means for any borelan A such that ϕa > 0, the chan P vsts A an nfnte number of tmes wth probablty one. Under those condtons, P satsfes a LLN, more precsely: Lemma 3. [0, 7.0.] Suppose that P s a postve Harrs chan wth nvarant probablty measure π, and let g be a π-ntegrable functon such that π g = gx πdx < R n. Then /t t k= gp k a.s πg. t The path P satsfes the condtons of Lemma 3 and exhbts an nvarant measure [7]. We now obtan geometrc dvergence of the step-sze and get an explct estmate of the expresson of the dvergence rate. Theorem. The step-sze of the, λ-csa-es wth λ dverges geometrcally fast f c < or λ 3. Almost surely and n expectaton we have for 0 < c, t ln σt c E N :λ + c E N :λ. 7 σ 0 t d σ n }{{} >0 for λ 3 and for λ= and c< 5

7 Proof. For provng almost sure convergence of lnσ t /σ 0 /t we need to use the LLN for Markov chan. We refer to [7] for the proof that P satsfes the rght assumptons. We now focus on the convergence n expectaton. From Eq. 4 we have Elnσ t+ /σ t = c/d σ E p t+ /n, so E p t+ = E n [ ] = pt+ s the term we have to analyse. For, there s no selecton pressure for [ξ t ], so we are n these dmensons under random selecton. Hence, as [p 0 ] N 0,, [p ] N 0, also. By recurrence, we deduce that [p t ] N 0,. Therefore E n [ ] = pt+ = E [ ] p t+ + n. By recurrence we show that [ p t+ ] = ct+ [p 0 ] + t c c =0 [ c ξ t ]. When t goes to nfnty, the nfluence of [p 0] n ths equaton goes to 0 wth c t+, so we can remove t when takng the lmt: [pt+ lm E ] c t = lm E t c c [ ξ ] t t =0 We wll now develop the sum wth the square, such that we have ether a product [ ] [ ] ξ t ξ wth j, or [ t j ξ ] t j. Ths way, we can separate the varables by usng Lemma wth the ndependence of ξ over tme. To do so, we use the development formula n = a n = n n = j=+ a a j + n = a. We take the lmt of E [ ] p t+ and fnd that t s equal to lm c c t t t =0j=+ [ ] ξ t j + }{{} [ξ c +j ] E t =E[ξ t ] E[ξ t j] =E[N :λ ] 8 t [ξ c ] E t }{{} =E[N:λ ] 9 Now the expected value does not depend on or j, so what s left s to calculate t t =0 j=+ c+j and t =0 c. We have t t =0 j=+ c+j = t =0 c+ c t and when we separates ths sum n two, the rght hand sde c t goes to 0 for t. Therefore, the left hand sde converges to lm t =0 c + /c, whch s equal to lm t c/c t =0 c. And t =0 c s equal to c t+ / c, whch converges to /c c. So, by nsertng ths n Eq. 9 we get that E [pt+ ] c t c E N :λ + E N:λ, whch gves us the rght hand sde of Eq. 7. By summng Elnσ + /σ for = 0,..., t and dvdng by t we have the Cesaro mean /telnσ t /σ 0 that converges to the same value that Elnσ t+ /σ t converges to when t goes to nfnty. Therefore we have n expectaton Eq. 7. Accordng to Lemma, for λ =, EN : =, so the RHS of Eq. 7 s equal to c/d σ nen :. The expected value of N : s strctly negatve, so the prevous expresson s strctly postve. Furthermore, accordng to Lemma, EN:λ ncreases wth λ, as does EN :. Therefore we have geometrc dvergence for λ. From Eq. we see that the behavor of the step-sze and of X t t N are drectly related. Geometrc dvergence of the step-sze, as shown n Theorem, means that =0 6

8 also the movements n search space and the mprovements on affne lnear functons f ncrease geometrcally fast. Analyzng X t t N wth cumulaton would requre to study a double Markov chan, whch s left to possble future research. 5 Study of the varatons of ln σ t+ /σ t The proof of Theorem shows that the step sze ncrease converges to the rght hand sde of Eq. 7, for t. When the dmenson ncreases ths ncrement goes to zero, whch also suggests that t becomes more lkely that σ t+ s smaller than σ t. To analyze ths behavor, we study the varance of ln σ t+ /σ t as a functon of c and the dmenson. Theorem 3. The varance of ln σ t+ /σ t equals to Var ln σt+ Furthermore, E σ t = c [pt+ ] 4d σ n E t E N :λ [pt+ ] 4 E [pt+ ] + n + ce N :λ and wth a = c c. 0 [pt+ lm E ] 4 = a t a 4 k 4 + k 3 + k + k + k, where k 4 = E N:λ 4, k3 = 4 a+a+a a E N 3 3 :λ E N:λ, k = 6 a a E, N:λ k = a3 +a+3a a a 3 E N:λ E N:λ a and k = 4 6 a a a 3 E N :λ 4. Proof. σt+ c pt+ V ar ln = Var = c Var pt+ σ t d σ n 4d σn }{{} E p t+ 4 E p t+ The frst part of Var p t+, E p t+ 4, s equal to E n [ ] = pt+. We develop t along the dmensons such that we can use the ndependence of [p t+ ] wth [p t+ ] j for j, to get E n n [ ] [ ] = j=+ pt+ pt+ j + n [ ] 4 = pt+. For [ [pt+ p t+ ] s dstrbuted accordng to a standard normal dstrbuton, so E ] = 7

9 [pt+ ] 4 and E = 3. E p t+ 4 = = j=+ = = = E E = j=+ [pt+ ] [pt+ ] E + j + E j= n + n E = [pt+ ] 4 + n E [pt+ ] + E = [pt+ ] 4 n 3 + E = [pt+ ] + 3n + E [pt+ ] + n n + [pt+ ] 4 [pt+ ] 4 The other part left s E p t+, whch we develop along the dmensons to get E n [ ] = pt+ = E [ ] p + n t+, whch equals to E [ ] p t+ + n E [ ] p t+ +n. So by subtractng both parts we get E p t+ 4 E p t+ = E [ p t+ ] 4 E[ p t+ ] + n, whch we nsert nto Eq. to get Eq. 0. The development of E [ ] p t+ s the same than the one done n the proof of Theorem. We refer to [7] for the development of E [ ] 4 p t+, snce lmts of space n the paper prevents us to present t here. The varable [ p t+ s ndependent of the dmenson t s the sum of random varables wth dstrbuton N :λ, whch s ndependent of the dmenson. Therefore, from ] Eq. 0 we deduce that the varance of lnσ t+ /σ t behaves, for constant c, roughly lke /n and ts standard devaton lke / n. The standard devaton of ln σ t+ /σ t dvded by ts expected value computes to E [ ] 4 p E [ ] / t+ p t+ + n E [ ] p t+ and ncreases n n wth the dmenson, gven c s constant. Fgure shows the tme evoluton of lnσ t /σ 0 for 500 runs and c = left and c = / n rght. By comparng Fgure a and Fgure b we observe smaller varatons of lnσ t /σ 0 wth the smaller value of c. Fgure shows the relatve standard devaton of ln σ t+ /σ t.e. the standard devaton dvded by ts expected value. Lowerng c, as shown n the left, decreases the relatve standard devaton. To get a value below one, c must be smaller for larger dmenson. In agreement wth Theorem 3, n Fgure, rght, the relatve standard devaton ncreases lke n wth the dmenson for constant c three ncreasng curves. For the choce of c / + n /3, the relatve standard devaton appears to converge to 0.5 for n. Larger values lke c / + n /4 seem not approprate. 6 Summary We nvestgate throughout ths paper the, λ-csa-es on affne lnear functons composed wth strctly ncreasng transformatons. We fnd, n Theorem, the lmt dstrbuton for lnσ t /σ 0 /t and rgorously prove the desred behavour of σ wth λ : the 8

10 lnσ t /σ lnσ t /σ number of teratons a Wthout cumulaton c = number of teratons b Wth cumulaton c = / 0 Fg. : lnσ t /σ 0 aganst t. The dfferent curves represent the quantles of a set of samples, more precsely the 0 -quantle and the 0 -quantle for from to 4; and the medan. We have n = 0 and λ = 8. step-sze dverges geometrcally fast. In contrast, wthout cumulaton c = and wth λ =, a random walk on lnσ occurs, lke for the, -σsa-es [8] and also for the same symmetry reason. We derve an expresson for the varance of the step-sze ncrement. On lnear functons when c s kept constant and for n, the standard devaton s about n tmes larger than the step-sze ncrement. However wth c /n /3, the standard devaton remans below the actual ncrement, for n. Acknowledgments Ths work was partally supported by the ANR-00-COSI-00 grant SIMINOLE of the French Natonal Research Agency and the ANR COSINUS project ANR-08-COSI References. D. V. Arnold and H.-G. Beyer. Performance analyss of evolutonary optmzaton wth cumulatve step length adaptaton. IEEE Transactons on Automatc Control, 494:67 6, D. V. Arnold and H.-G. Beyer. On the behavour of evoluton strateges optmsng cgar functons. Evolutonary Computaton, 84:66 68, D.V. Arnold. Cumulatve step length adaptaton on rdge functons. In Parallel Problem Solvng from Nature PPSN IX, pages 0. Sprnger, D.V. Arnold and H.G. Beyer. Random dynamcs optmum trackng wth evoluton strateges. In Parallel Problem Solvng from Nature PPSN VII, pages 3. Sprnger, D.V. Arnold and H.G. Beyer. Optmum trackng wth evoluton strateges. Evolutonary Computaton, 43:9 308, D.V. Arnold and H.G. Beyer. Evoluton strateges wth cumulatve step length adaptaton on the nosy parabolc rdge. Natural Computng, 74: ,

11 Varlnσt + /σ t / lnσ t + /σ t c Varlnσt + /σ t / lnσ t + /σ t dmenson of the search space Fg. : Standard devaton of ln σ t+ /σ t relatvely to ts expectaton. Here λ = 8. The curves were plotted usng Eq. 0 and Eq.. On the left, curves for rght to left n =, 0, 00 and 000. On the rght, dfferent curves for top to bottom c =, 0.5, 0., / + n /4, / + n /3, / + n / and / + n. 7. A. Chotard, A. Auger, and N. Hansen. Cumulatve step-sze adaptaton on lnear functons: Techncal report chotard/ chotard0trcumulatve.pdf. 8. N. Hansen. An analyss of mutatve σ-self-adaptaton on lnear ftness functons. Evolutonary Computaton, 43:55 75, N. Hansen and A. Ostermeer. Adaptng arbtrary normal mutaton dstrbutons n evoluton strateges: The covarance matrx adaptaton. In Internatonal Conference on Evolutonary Computaton, pages 3 37, S. P. Meyn and R. L. Tweede. Markov chans and stochastc stablty. Cambrdge Unversty Press, second edton, A. Ostermeer, A. Gawelczyk, and N. Hansen. Step-sze adaptaton based on non-local use of selecton nformaton. In Proceedngs of Parallel Problem Solvng from Nature PPSN III, volume 866 of Lecture Notes n Computer Scence, pages Sprnger,