THE motivation for considering the selection of simplest

Transcription

1 Effcent Selecton of a Set of Good Enough Desgns wth Complexty Preference Shen Yan, Enlu Zhou, Member, IEEE, and Chun-Hung Chen, Senor Member, IEEE Abstract Many automaton or manufacturng systems are large, complex, and stochastc Snce closed-form analytcal solutons generally do not exst for such systems, smulaton s the only fathful way for performance evaluaton From the practcal engneerng perspectve, the desgns or soluton canddates wth low complexty called smple desgns have many advantages compared wth complex desgns, such as requrng less computng and memory resources, and easer to nterpret and to mplement Therefore, they are usually more desrable than complex desgns n the real world f they have good enough performance Recently, Ja [] dscussed the mportance of desgn smplcty and ntroduced an adaptve smulaton-based samplng algorthm to sequentally screen the desgns untl one smplest good enough desgn s found In ths paper, we consder a more generalzed problem and ntroduce two algorthms OCBA-mSG and OCBA-bSG to dentfy a subset of m smplest and good enough desgns among a total of K K > m desgns By controllng the smulaton allocaton ntellgently, our approach ntends to fnd those smplest good enough desgns usng a mnmum smulaton tme The numercal results show that both OCBA-mSG and OCBA-bSG outperform some other approaches on the test problems Note to Practtoners Ths paper was motvated by two problems from the real world: desgn of orderng polces n nventory control, and desgn of node actvaton rules n sensor networks In desgnng a good orderng polcy or a good sensor actvaton rule, smple desgns have varous advantages n practce, eg, easy to learn, to mplement, to operate, and to mantan More generally, practtoners want to fnd a desgn or a set of desgns whch not only have good performance but are also smple Due to the complexty of the systems, smulaton s a popular tool n ndustry to evaluate the performance of dfferent alternatve desgns before actual mplementaton Whle the advance of new technology has dramatcally ncreased computatonal power, effcency s stll a bg concern Our proposed approach ntellgently controls the smulaton of alternatve desgns, so that the smplest good enough desgns can be selected usng a mnmum computaton cost Our numercal experments show that the proposed approach s very effectve Index Terms optmal computng budget allocaton, rankng and selecton, smulaton-based optmzaton, complexty Shen Yan was a Master student n the Department of Industral & Enterprse Systems Engneerng, Unversty of Illnos at Urbana-Champagn, IL, 680 USA yanshen987@gmalcom Enlu Zhou s wth the faculty n the Department of Industral & Enterprse Systems Engneerng, Unversty of Illnos at Urbana-Champagn, IL, 680 USA enluzhou@llnosedu Chun-Hung Chen correspondng author s wth the faculty n Department of Electrcal Engneerng & Insttute of Industral Engneerng, Natonal Tawan Unversty, Tape, Tawan cchen9@gmuedu Ths work was supported by the Natonal Scence Foundaton under Grants ECCS and CMMI-3073, the Ar Force Offce of Scentfc Research under Grant FA , the Department of Energy under Grant DE-SC0003, Natonal Insttutes of Health under Grant RDK , and Natonal Scence Councl of Tawan under Grant NSC-00-8-E MY3 I INTRODUCTION THE motvaton for consderng the selecton of smplest good enough desgns comes from the real world, where smple desgns are preferred to the complex ones f the smple desgns are good enough to satsfy our requrements A typcal example s to fnd an orderng polcy n nventory control Consder the problem of orderng a certan amount of products at each perod to meet a stochastc demand whch follows a probablty dstrbuton In order to mnmze the expected cost ncludng holdng cost for excess nventory and shortage cost for unflled demand, we want to determne the optmal orderng polcy n each perod The optmal polcy can be found analytcally for some models to have the structure of a base-stock polcy or an s,s polcy [], [3], [4], [5] The basestock polcy s a threshold functon that maps the current stock nto the orderng amount Motvated by ths smple structure of the optmal polcy for some models, we can approxmate the optmal polces for other more complex nventory models by threshold polces In general, we can approxmate the optmal orderng polcy better wth a greater number of thresholds gven the rght values of these thresholds Then the problem s to decde the number and the values of the thresholds It s clear that wth more thresholds n the functon we have a more complex orderng polcy, makng t harder to determne the optmal values of these thresholds and to mplement n practce Conversely, wth a small number of thresholds, such as one base-stock polcy or two, we have a smple orderng polcy, whch s easer to compute and to mplement If the smple orderng polcy can acheve a requred cost crteron, t wll be more desrable than a complex orderng polcy, even though the complex polcy may yeld a lower cost Another example s the desgn of node actvaton rules n the wreless sensor networks WSNs, as descrbed n [6], [] Each node needs to collaborate wth ts neghbors n order to have enough power to montor an area of nterest Smlarly, gven that the requred probablty of correct detecton can be acheved, we prefer small communcaton radus of each node e, smple actvaton rule to large radus e, complex actvaton rule In ths paper, we use the word desgn to refer to the object under consderaton, such as the orderng polcy and the node actvaton rule n the prevous examples We consder the problem of selectng m m desgns that are smplest wth smallest complexty and good enough satsfyng a constrant on the performance measure Selecton of multple desgns s sometmes preferred because of at least two reasons Frst, a decson maker has to face dfferent objectves and constrants However, n many cases t s too complcated to nclude all

2 objectves and constrants n the smulaton model As a result, the decson maker may not lke the best desgn obtaned from the model By offerng a set of multple good desgns, the decson maker can choose the one he/she lkes by consderng more factors Second, the desgn space can be extremely large and the total smulaton cost s too expensve A common approach s to frst apply a smplfed model to screen out some good alternatves before the full-scale smulaton modelng analyss Offerng a set of multple good desgns s hghly useful for ths purpose The complexty of a desgn s represented by an nteger number, where smpler desgn has a smaller nteger number The complexty s a determnstc value known before smulaton However, the performance of a desgn s subject to system nose, and hence, t can only be estmated from smulaton, whch s often computatonally expensve For example, t takes a sgnfcant amount of computatonal effort to smulate the nventory system n order to evaluate the cost of a partcular orderng polcy Hence, our goal s to allocate a gven smulaton budget effcently to the desgns so as to maxmze the probablty of correctly selectng the m smplest good enough desgns out of a total of K desgns The above problem s closely related wth many known results n the lterature on rankng and selecton R&S Several recent R&S procedures are dscussed and compared by Branke et al [7] Some of the procedures can be further extended to more generalzed smulaton optmzaton problems eg, [8], [9], [0], [], [], [3] For subset selecton problem, Gupta [4] proposed the method of selectng a random sze subset contanng the best desgn wth a gven probablty of correct selecton Later, Santner [5] extended Gupta s method by mposng a maxmum sze m on the subset Koeng and Law [6] developed a two-stage procedure for selectng the top m desgns wth best performance, followng the results n Dudewcz and Dalal [7] Chen et al [8], [9], [0] developed the optmal computng budget allocaton OCBA procedure for the selecton of one best desgn, and later Chen et al [] extended OCBA to the selecton of the m best desgns However, all of ths work has focused on optmzng a sngle-objectve performance measure Problems of mult-objectve optmzaton and constrant optmzaton have also been studed Lee et al [], Teng et al [3], Chew et al [4], and Lee et al [5] extended the OCBA framework to effcently select desgns that optmze multple performance measures Branke and Mattfeld [6] proposed to search for solutons that are not only good but also flexble n dynamc schedulng Branke and Gamer [7] proposed an effcent samplng procedure n nteractve mult-crteron selecton In constrant optmzaton, Andradóttr et al [8] proposed a two-phase approach whch dentfes all the feasble systems frst and then selects the best from them Szechtman and Yücesan [9] used large devaton theory to deal wth feasblty determnaton Most recently, Pujowdanto et al [30] developed OCBA further for selectng one sngle best desgn under multple constrants of secondary performance measures The problem of consderng both complexty and performance evaluaton has only been consdered recently It appears to be a mult-objectve optmzaton or a constrant optmzaton problem, but t has ts unque problem structure that can be exploted to desgn a more effcent samplng procedure The most relevant problem to ours s probably the selecton of one smplest good desgn, for whch Ja [] proposed an Adaptve Samplng Algorthm ASA to mnmze the Type II error of the chosen desgn The relaton between complexty and performance n choosng polces has also been explored n the context of Markov decson processes [3], [3] In ths paper, we address ths problem of selectng multple smplest good enough desgns by proposng the algorthm OCBA-mSG, abbrevated for optmal computng budget allocaton for m smplest good enough desgns Based on OCBA-mSG, we develop another slghtly dfferent algorthm called OCBA-bSG for selectng the desgns wth the best performance from all the smplest good enough desgns, wth a slght ncrease of smulaton budget than OCBA-mSG Numercal results ndcate that both OCBA-mSG and OCBAbSG allocate the smulaton budget effcently to acheve a hgh probablty of correct selecton The rest of the paper s organzed as follows In Secton II, we defne the two problems of selectng m smplest good enough desgns and selectng the best m smplest good enough desgns, respectvely In Secton III, we state the man results proofs are ncluded n the Appendx and present the algorthms In Secton IV, we carry out the numercal experments on several test problems Fnally, we conclude our paper n Secton V II PROBLEM STATEMENT A Selectng m Smplest Good Enough Desgns Let θ denote a desgn, and Θ denote the set of all the K K > m desgns, e, Θ = {θ,θ,,θ K } To smplfy notatons, we wll also use the ntegers,,,k to denote the desgns n the followng when there s no ambguty The performance of the desgn θ k s measured by J k = E[Lθ k,ζ ], where ζ s a random vector that represents the uncertanty n the system, and Lθ k,ζ can only be evaluated through smulaton of the system The underlyng assumpton s that such smulaton s expensve A desgn s consdered better f ts performance measure J s smaller A good enough desgn s one that satsfes J k < J 0, where J 0 s a gven threshold on the performance In practce, J 0 can be set by the user or chosen based on a few plot runs whch return a rough estmate of the performance of the desgns Please note that the defnton of good enough here s the same as feasble, whch s dfferent from the defnton n the lterature on ordnal optmzaton cf [33] In the rest of the paper we wll use the words good enough and feasble nterchangeably Hence, the good enough set or the feasble set s defned as F = {k J k < J 0,k =,,,K}

3 3 The complexty of the desgn θ k s represented by the complexty Cθ k, whch s a determnstc value n the set {0,,,n},n < K, and s known before smulaton Note that Cθ s the result of the user mappng hs/her defnton of complexty to nteger numbers For example, the user could map the number of thresholds of the orderng polcy to an nteger; or the user could map a range of communcaton radus n the WSN problem to an nteger The complexty set C contans all the desgns wth complexty, defned as C = {k Cθ k =,k =,,,K} The set of m smplest and good enough desgns s defned as S m = {m,m,,m m Cθ m Cθ k, k F S m }, where F S m = {k F k S m } Notce that the set S m may not be unque, because t s possble that multple desgns n the set F have the same complexty For example, f there are m m < m desgns n F wth complexty 0 and m m > m m desgns n F wth complexty, then S m ncludes all the m desgns wth complexty 0 and any m m desgns of those m desgns wth complexty Fg gves a pctoral vew of the general case Suppose that all the desgns n the complexty sets C 0, C,, C t t n are not good enough or nfeasble and the frst feasble desgn appears n the set C t Moreover, suppose that the total number of feasble desgns n C t,c t+,,c t s less than m untl t reaches t + p 0 p n t Hence, n general we need to consder three types of subsets, whch we refer to as: nfeasble smplest subsets S d, = 0,,,t ; smplest good enough subsets S s, = 0,,, p; and nfeasble nonsmplest subsets S e, = 0,,, p In partcular, the smplest good enough subsets S s0,s s,,s sp satsfy p S s < m, =0 p =0 S s m, where denotes the cardnalty of the set Therefore, accordng to the defnton of the m smplest and good enough desgns, S m should nclude all the desgns n the subsets S s0,s s,,s sp and any m p =0 S s desgns n the subset S sp Snce there are already m desgns selected n the lower complexty sets C 0,,C t+p, there s no need to consder the hgher complexty sets In smulaton, we compute the sample mean J based on the samples on hand to estmate the performance J for each desgn, and then order the desgns to fnd the subsets {Ŝ s, = 0,,, ˆp}, {Ŝ d, = 0,,, ˆt } and {Ŝ e, = 0,,, ˆp} as estmates for S s, S d and S e respectvely accordng to the relatonshp shown n Fg Hence, the selected set Ŝ m should nclude all the desgns n Ŝ s0,ŝ s,,ŝ s ˆp and any m ˆp =0 Ŝ s desgns n Ŝ s ˆp To further classfy the subsets, we denote Ŝ s = ˆp =0 } { Ŝ s, Ŝ I = { =0Ŝe ˆp ˆt =0Ŝd } We defne the correct selecton CS m as the event that the desgns n Ŝ s are the true smplest good enough desgns, e, CS m = {J < J 0 & J j J 0, Ŝ s, j Ŝ I }, Fg : Relatonshp between subsets J j denotes the performance of a desgn whose complexty s and whose performance s the j th smallest n ts complexty set C where Ŝ s ncludes all the smplest good enough desgns, Ŝ I ncludes all the nfeasble ether smplest or non-smplest desgns The determnaton of Ŝ s and Ŝ I s based on the estmate sample mean of the performance of every desgn, and the accuracy of the estmate s determned by the number of smulatons carred out for that desgn Therefore, the decson varables that determne the probablty of correct selecton PCS m are the number of smulatons N,N,,N K allocated for the desgns θ,θ,,θ K, respectvely Ths wll be more clearly shown n the explct expresson 3 for PCS m later Gven a fxed total smulaton budget T, we want to fnd the optmal budget allocaton N,N,,N K to the K desgns n order to maxmze the probablty of correct selecton: max N,N,,N K PCS m st N + N + + N K = T B Selectng the Best m Smplest Good Enough Desgns Consder a smple example that there are two good enough desgns wth complexty 0, say A and B, so they are both smplest good enough desgns If we only need one smplest good enough desgn, then we can choose ether A or B However, f A and B have dfferent performance, say J A < J B, then we would prefer A, because t s better than B n performance and as smple as B Ths s what we refer to as the best smplest good enough desgn The formal defnton of the best m smplest good enough desgns s as follows: S b = {b,,b m F Cθ b < Cθ k OR J b < J k f Cθ b = Cθ k, k F S b }, where F S b = {k F k S b } The key dfference from S m s that S b ncludes all the feasble desgns n the subsets S s0,s s,,s sp and the best m p =0 S s desgns n the subset S sp That mples that the subset S sp should be further dvded nto two subsets: optmal subset S bl, and feasble nonoptmal set S a Fg gves a pctoral vew of the relatonshp

4 4 between the subsets Therefore, the optmal set S b satsfes { } Sbl S b = p =0 S s, S b = m In smulaton, we fnd estmates for these subsets based on the sample means of the desgns, and smlarly we denote Ŝ b = ˆp =0 } { Ŝ s, Ŝ I = { =0Ŝe ˆp ˆt =0Ŝd } Then the correct selecton CS b s defned as CS b = {J < J 0 & J j J k < J 0 & J s J 0, Ŝ b, j Ŝ bl, k Ŝ a, s Ŝ I } Our goal s to fnd the optmal budget allocaton N,N,,N K to the K desgns n order to maxmze the probablty of correct selecton gven a fxed total smulaton budget T : max N,N,,N K PCS b st N + N + + N K = T III MAIN RESULTS A Selectng m Smplest Good Enough Desgns We estmate PCS usng the same Bayesan model presented n [34] and [35] Assumng that the performance of each desgn, J, has a nonnformatve normal pror dstrbuton N0,ν wth ν extremely large, and a sample Jˆ for J s normally dstrbuted as NJ,σ, then the posteror dstrbuton of J has been shown n [34] to be J N J, σ, N where J = N N ˆ k= J k, and Jˆ, Jˆ,, Jˆ N d NJ,σ Thus, PCS m s as follows: PCS m = P{ J < J 0 & J j J 0, Ŝ s, j Ŝ I } = P{ J < J 0 } P{ J j J 0 }, 3 Ŝ s j Ŝ I where the second equaton s due to the ndependence between desgns The results are stated n the followng theorem, and the proof s contaned n the Appendx Theorem PCS m s asymptotcally as T maxmzed by the followng allocaton rule: N σ / J J 0 = N j σ j / J j J 0, 4 for all Ŝ s and j Ŝ I N k = 0 for all other k {,,,K} Remark From 4, we know that the smulaton budget for each desgn ncreases proportonally to ts correspondng sample varance If a desgn has a larger sample varance, more smulaton budget wll be allocated to t n order to obtan a more accurate estmate for the performance n the next teraton On the other hand, the smulaton budget for each desgn decreases proportonally to the dfference between ts sample mean and the good enough threshold J 0 The desgn whose sample mean of the performance s closer to J 0 wll be assgned more smulaton budget, snce t s more senstve to the feasblty test As there are already m desgns selected from Ŝ s ŜI n the lower complexty sets, there s no need to consder the hgher complexty sets, and hence, there s no more smulaton budget allocated to the desgns that are not n Ŝ s ŜI However, as more smulaton s carred out and the sample means are updated, Ŝ s and Ŝ I may become dfferent at the next teraton and contan some of the hgher-complexty sets that are not consdered n the prevous teraton B Selectng the Best m Smplest Good Enough Desgns It s hard to maxmze PCS b problem analytcally, and hence we maxmze a lower bound of PCS b, whch s called Approxmate Probablty of Correct Selecton APCS b [8] APCS b s defned as follows PCS b = P{ J < J 0 & J j J k < J 0 & J s J 0, Ŝ b, j Ŝ bl, k Ŝ a, s Ŝ I } P{ J < J 0 & J j µ & µ J k < J 0 & J s J 0, θ Ŝ b, j Ŝ bl, k Ŝ a, s Ŝ I } = P{ J < J 0 } P{ J j µ} Ŝ b j Ŝ bl k Ŝ a P{µ J s J 0 } 5 s Ŝ I APCS b, where the second equaton s due to the ndependence between the desgns It s easy to see that a larger APCS b yelds a better approxmaton for PCS b Followng a smlar procedure as n [], we determne the value of µ as stated n the followng Lemma Lemma Let θ [r] denote the desgn wth the largest sample mean n the subset Ŝ bl, and θ [r+] denote the desgn wth the smallest sample mean n the subset Ŝ a Then the µ value ntroduced n APCS b s gven by µ = ˆσ [r+] J [r] + ˆσ [r] J [r+], 6 ˆσ [r] + ˆσ [r+] where ˆσ = σ / N Therefore, nstead of solvng the maxmzaton problem, we consder the followng maxmzaton problem max N,N,,N K APCS b st N + N + + N K = T 7 The results are gven n the followng theorem, and the proof s contaned n the Appendx Theorem 3 APCS b s asymptotcally as T maxmzed by the followng allocaton rule: Case : If Ŝ a = e, the total number of feasble desgns s greater than m, then N σ / J J 0 = = N j σ j / J j µ = N x σx / J x µ = N s σ s / J s J 0 N y σ y / J y J 0, 8

5 5 for all Ŝ b, j Ŝ bl, s Ŝ I, x {k Ŝ a J k µ+j 0 }, y {k Ŝ a J k > µ+j 0 } N k = 0 for all other k {,,,K} Case : If Ŝ a = e, the total number of feasble desgns s less than or equal to m, then N σ / J J 0 = N s σs / J s J 0 9 for all Ŝ b Ŝbl and s Ŝ I N k = 0 for all other k {,,,K} Remark Theorem provdes some ntutve results We notce that at the two crtcal ponts µ and J 0 : µ s the threshold for the optmalty, J 0 s the threshold for the feasblty The desgns closer to these two ponts wll be assgned more smulaton budget For the subsets Ŝ b and Ŝ I, we are only nterested n determnng wether the desgns are good enough, and ndeed more smulaton budget s assgned to the desgns near J 0 Smlarly, for the subset Ŝ bl, we are only nterested n comparng the performance of the desgns, and more smulaton budget s assgned to the desgns around µ For the subset S a where both µ and J 0 are crtcal ponts, the last two terms n 8 mply that we should dvde the set nto two µ+j parts by the mdpont 0 : the desgns wth sample means n the range µ J x µ+j 0 wll be compared wth µ, and the ones closer to µ wll get more smulaton budget; the desgns fallng nto the range µ+j 0 < J y J 0 wll be compared wth J 0, and be assgned more smulaton budget f closer to J 0 Please see fg for a pctoral vew of the budget allocaton n the complexty set Cˆt+ ˆp, the hghest complexty set under consderaton Fg : Smulaton allocaton n the set Cˆt+ ˆp Remark 3 Comparng selectng S b wth S m, the dfference s n C t+p, where the subset S sp n selectng S m s dvded nto two subsets S bl and S a n selectng S b Theorem 3 mples that n addton to allocatng more smulaton budget to the desgns near J 0 n the sets C 0,C,C t+p, we also allocate smulaton budget to desgns near µ n the set C t+p As a result, the extra smulaton budget assgned to desgns near µ n selectng S b s approxmately /t + p + of the total smulaton budget n selectng S m If t = 0 and p = 0 e, there are more than m feasble desgns n the lowest complexty set C 0, then selectng S b needs approxmately double smulaton budget of that n selectng S m for the same accuracy of the sample means of the desgn performance On the other hand, f t + p s large, selectng S b requres lttle extra smulaton budget, and wll be preferred snce t yelds the best m desgns among all smplest and good enough desgns C OCBA-mSG and OCBA-bSG Based on the above results, we propose the Optmal Computng Budget Allocaton procedure for selectng m Smplest and Good enough desgns OCBA-mSG and that for selectng the Best m Smplest and Good enough desgns OCBAbSG Snce the two algorthms are smlar, we descrbe them together to save space and specfy the dfferent steps n the descrpton OCBA-mSG and OCBA-bSG Input: the total number of the desgns K, the number of desgns needed m, the total smulaton budget T, the smulaton budget ncrease at each teraton, the ntal smulaton budget for every desgn n 0, the good enough performance constrant J 0, and the upper bound of the total smulaton budget for one desgn NU Intalze: l = 0 Group the desgns accordng to ther complextes to obtan the complexty sets C 0,C,,C n Perform n 0 smulaton replcatons for all desgns to generate samples X k, k =,,,n 0, =,,,K Set N l = Kn 0 Loop: whle N l < T, do Update: For each desgn, compute the sample mean J = N l Nl k= X k, and the sample standard devaton σ = Nl k= X k J /N l Sort the desgns n each complexty set accordng to ther sample means n the ncreasng order Increase the computng budget N l+ = mn{n l +,T } Allocate: OCBA-mSG For each desgn θ, compute the smulaton budget N l+ accordng to 4 OCBA-bSG If the total number of feasble desgns s greater than m, compute µ accordng to 6, and compute the smulaton budget N l+ for each desgn θ accordng to 8 Otherwse, compute the smulaton budget N l+ for each desgn θ accordng to 9 3 Smulate: If N l+ NU or N l+ N l, we set Nl+ = N l, and do not smulate desgn θ at ths teraton Otherwse, perform N l+ N l smulatons for desgn θ to generate more samples X k, k = Nl +,N l +,,Nl+ 4 Update: l l + End of loop Output: output the feasble desgns startng from the lowest complexty set n the ncreasng order of ther sample means untl the total number of such desgns reaches m or all the desgns have been examned

6 6 Remark 4 In the above algorthms, we ntroduce an upper bound NU on the smulaton budget for one sngle desgn: f N NU, we stop allocatng new smulaton budget to that desgn That s because we obtan the smulaton budget allocaton rules under the asymptotc lmt T but the actual total smulaton budget T s fnte When T s nfnty, we can assgn a large amount of budget to one desgn at one teraton, and there wll always be enough budget left for other desgns f needed at future teratons Ths s not true when T s fnte Thus, we ntroduce NU and determne ts value n the followng way Snce the desgns near the crtcal ponts need more smulaton budget, we need to ensure each of such crtcal desgns wll be smulated at least once Hence, we approxmate the upper bound by countng the number of subsets related to the crtcal ponts after ntalzaton, where those subsets are Ŝ s, Ŝ d, Ŝ e n OCBA-mSG or Ŝ s, Ŝ bl, Ŝ a, Ŝ d, Ŝ e n OCBA-bSG cf Fg NU = T Kn 0 total number o f sets + n 0 Ths choce of upper bound works well n our numercal experments IV NUMERICAL EXPERIMENTS In ths secton, we demonstrate our methods OCBA-mSG and OCBA-bSG on some examples and also compare them wth two other methods - Equal Allocaton and Levn Search Equal Allocaton allocates the smulaton budget equally among all the desgns and do not use any nformaton such as the mean, the varance or the complexty of the desgn At teraton l, t allocates smulaton budget accordng to N l+ N l = /K, {,,,K} Levn Search method [36] allocates smulaton budget to the desgns sequentally n the order of the complexty It s useful when appled to fnd one smplest and good enough desgn [] frst smulates the desgns wth smallest complexty untl obtanng a certan accuracy for the estmates of the performance, based on whch the good enough desgns are selected If only less than m good enough desgns are found, t then contnues to smulate the desgns n the next complexty set untl eventually fndng m smplest good enough desgns eventually In our mplementaton, we smulate every desgn for n 0 tmes at ntalzaton, and order them accordng to ther sample means and complextes Snce t s hard to specfy a gven accuracy for the estmates n our examples, we evenly allocate the total remanng budget T Kn 0 /K to all the desgns beforehand, but smulate the desgns sequentally, e, start smulatng the frst smplest desgn for T Kn 0 /K tmes and then move on to the next one to repeat the same procedure Please note often exhbts some jump behavor n the PCS, because the PCS stays flat f the desgn currently under smulaton s not good enough and the PCS ncreases otherwse If the desrable set of desgns s found before utlzng all computng budget, wll termnate and the correspondng PCS curve wll level off n the fgures s the same as when utlzng all the T smulaton budget, but often acheves the fnal PCS earler In general, method performs better f the performance deterorates as the complexty ncreases In the numercal experments, we test three generc examples whch mmc dfferent scenaros n real world In Example, good desgns are also smple desgns In contrast, bad desgns are smple ones n Example In Example 3, we consder a problem wth a larger number of alternatve desgns We use PCS as the effcency measurement: for a gven total smulaton budget, the faster the PCS converges, the better the correspondng method s Here we estmate PCS usng Monte Carlo smulaton by computng the rato of the number of smulaton runs wth correct selectons to the total number of smulaton runs In addton, for convenence, we assume that desgn θ has complexty log, so the complexty s non-decreasng n Example Mean ncreases as complexty ncreases: There are 0 desgns n total, wth the th desgn havng Lθ,ζ dstrbuted accordng to the normal dstrbuton N,05 We want to fnd 5 smplest good enough desgns wth good enough constrant J 0 = 63 The ntal smulaton budget n 0 = 0, smulaton budget ncrement = 00, total smulaton budget T = 8000, and total number of smulaton runs = 0 4 The complexty sets are C 0 = {θ }, C = {θ,θ 3 }, C = {θ 4,θ 5,θ 6,θ 7 }, C 3 = {θ 8,,θ 5 } and C 4 = {θ 6,,θ 0 } The mean ncreases as the complexty ncreases, and the varance ncreases as the mean ncreases a OCBA-mSG: The correct selecton of the fve desrable desgns should nclude {θ,θ,θ 3 } and any two from {θ 4,θ 5,θ 6 } Fg 3 shows that OCBA-mSG converges faster than and performs well n ths example because of the small total number of desgns K and the small varance σ searches from the smplest sets {θ }, {θ,θ 3 },, and n ths example the correct selecton s {θ,θ,θ 3,θ 4,θ 5 }, so termnates n about 7 teratons PCS OCBA msg Total Smulaton Buget Fg 3: Example - selectng 5 smplest good enough desgns from 0 desgns wth dstrbuton N,05 and J 0 = 63 b OCBA-bSG: The correct selecton s {θ,θ,θ 3,θ 4,θ 5 } Fg 4 shows the smulaton result Example Mean decreases as complexty ncreases: There are 0 desgns, wth the th desgn havng Lθ,ζ

7 7 09 b OCBA-bSG: The correct selecton s {θ 4,θ 5,θ 8,θ 9,θ 0 } Fg 6 shows the smulaton result PCS PCS 04 OCBA bsg Total Smulaton Buget Fg 4: Example - selectng the best 5 smplest good enough desgns from 0 desgns wth dstrbuton N,05 and J 0 = 63 dstrbuted accordng to the normal dstrbuton N,05 We want to fnd 5 smplest good enough desgns wth good enough constrant J 0 = 73 The ntal smulaton budget n 0 = 0, smulaton budget ncrement = 00, total smulaton budget T = 8000, and total number of smulaton runs = 0 4 The complexty sets are the same as n Example The mean decreases as the complexty ncreases, and the varance ncreases as the mean decreases a OCBA-mSG: Correct selecton of the fve desrable desgns should nclude {θ 4,θ 5 } and any three from {θ 6,θ 7,θ 8,θ 9,θ 0 } Fg 5 shows the smulaton result All three methods converge slower than Example, but OCBA-mSG stll converges faster than and Desgns wth smaller means have larger varances, and the correct selecton s n the set {θ 4,θ 5,θ 6,θ 7, θ 8,θ 9,θ 0 } whch have relatvely large varances compared to other desgns, so OCBA-mSG converges slower than that n Example stll searches from the smplest sets whle the correct selecton s n the hgher complexty sets, so method also termnates later than Example 0 OCBA bsg Total Smulaton Buget Fg 6: Example - selectng the best 5 smplest good enough desgns from 0 desgns wth dstrbuton N,05 and J 0 = 73 3 Example 3 Md-scale problem: There are 65 desgns, wth the th desgn havng Lθ,ζ dstrbuted accordng to the normal dstrbuton N66,005 We want to fnd 5 smplest good enough desgns wth good enough constrant J 0 = 63 The ntal smulaton budget n 0 = 0, smulaton budget ncrement = 00, and total smulaton budget T = 8000 The complexty sets are C 0 = {θ }, C = {θ,θ 3 }, C = {θ 4,,θ 7 }, C 3 = {θ 8,,θ 5 }, C 4 = {θ 6,,θ 3 }, C 5 = {θ 3,,θ 63 } and C 6 = {θ 64,θ 65 } a OCBA-mSG: The correct selecton of the fve desrable desgns should nclude {θ 60,θ 6, θ 6,θ 63 } and any one from {θ 64,θ 65 } For ths md-scale problem, OCBA-mSG performs much better than and as shown n Fg 7 Detaled explanaton s smlar to that for OCBA-bSG n the followng PCS PCS OCBA msg OCBA msg Total Smulaton Buget Fg 5: Example - selectng 5 smplest good enough desgns from 0 desgns wth dstrbuton N,05 and J 0 = Total Smulaton Buget Fg 7: Example 3 - selectng 5 smplest good enough desgns from 65 desgns wth dstrbuton N66,005 and J 0 = 63 b OCBA-bSG: The correct selecton s {θ 60,θ 6,θ 6,θ 63,θ 65 } For ths md-scale problem, OCBAbSG performs much better than and as shown n

8 8 Fg 8 When the total desgn number K s large, converges slowly snce each desgn s assgned wth less smulaton budget at every teraton compared to Examples and For, the frst tme jumps n PCS s the tme that the total smulaton budget reaches 4900, whch s when t frst starts to smulate desgns n the set {θ 3,,θ 63 } wth means {34,,3} As we assgn the smulaton budget accordng to the order of the desgns n the same complexty set, here we smulate desgns n the order of θ 63,θ 6, Snce desgns θ 63,θ 6,θ 6 and θ 60 belong to the correct selecton set, jumps n PCS at ths pont The second jump n the PCS for happens n the end due to the smulaton budget allocaton to the desgn θ 65 PCS OCBA bsg Total Smulaton Buget Fg 8: Example 3 - selectng the best 5 smplest good enough desgns from 65 desgns wth dstrbuton N66,005 and J 0 = 63 V CONCLUSION In ths paper, we consdered the smulaton-based selecton of smplest good enough desgns, whch s motvated by reallfe applcatons We proved the optmal smulaton budget allocaton rules to asymptotcally maxmze the probablty of correct selecton or the approxmate probablty of correct selecton n the case of OCBA-bSG Based on the asymptotc results, we proposed the algorthm OCBA-mSG to effcently allocate the smulaton budget for selectng m smplest good enough desgns out of a total of K desgns, and also proposed a slghtly dfferent algorthm OCBA-bSG n order to fnd the best m smplest good enough desgns Numercal results show that both methods converge fast on all the test problems, whch ndcates OCBA-mSG and OCBA-bSG ndeed allocate smulaton budget effcently Whle our algorthms are motvated by the asymptotc results, an mportant future drecton s to analyze the fnte-tme performance of our algorthms VI APPENDIX A Appendx A: Proof of Theorem Snce J k N J k, σ k N, we have k P J k < J 0 = Φ J0 J k σ k / N k, where Φ s the error functon e, the cumulatve dstrbuton functon cdf of the standard normal dstrbuton By Lagrangan relaxaton of PCS m and Karush-Kuhn-Tucker KKT condton cf [37] for the maxmzaton problem, we get For S s, For S I, F = P{ J k < J 0 } P{ k S s k S I J k J 0 } λ K k= N T F = N P{ J k J 0 } k S s,k = k S I J0 J φ σ / J0 J λ = 0 0 N σ N F = N P{ J k < J 0 } k S s J0 J φ σ / N k S I,k = P{ J k J 0 } J0 J σ N λ = 0, where φ denotes the probablty densty functon pdf of the standard normal dstrbuton In order to fnd the relatonshp between N and N j, we need to consder + = 3 cases that, j belong to dfferent sets Case : S s, j S I Equatng 0 and, P{ J J0 J J0 J j J 0 }φ = P{ J < J 0 }φ σ / N J 0 J j σ j / N j σ N J j J 0 σ j Nj Takng logarthm on both sdes, we have logp { } J J 0 J J0 J j J 0 σ + log /N σ = logp { } J 0 J j J < J 0 σ + log j /N j J j J 0 σ j logn logn j Assumng N takes contnuous values, let N = α T Takng the asymptotc lmt of the above equaton as T, we have and we obtan lm T T {logp J j J 0 J 0 J α T + σ J0 J log logα T } σ = lm T T {logp J < J 0 J 0 J j α j T + σ j J0 J j log logα jt }, σ j α = J j J 0 α j J J 0 σ Case : S s, j S s, = j By equatng 0 and 0, smlarly as above we obtan σ j α = J j J 0 α j J J 0 σ σ j

9 9 Case 3: S I, j S I, = j By equatng and, smlarly as above we obtan α = J j J 0 α j J J 0 σ Combnng all three cases, we prove Theorem B Proof of Lemma Our dervaton of the value of µ follows the dea and method n Secton 33 n [] Specfcally, f we assume that all the desgns have equal varances, then we know σ j P J [r] µ P J µ, S bl, P J [r+] µ P J µ, S a To maxmze APCS b s equvalent to maxmzng the product of all the above terms The smallest terms P J [r] µ and P J [r+] µ have the most mpact on the value of the product Hence, to smplfy the problem, we consder the maxmzaton of the product of these two terms A good choce of µ can be determned by solvng the followng maxmzaton problem max N[r],N [r+] P J [r] µ P µ J [r+] st N [r] + N [r+] = T Followng the same approach n the proof of Theorem, we obtan the asymptotcally as T optmal soluton 6 C Proof for Theorem 3 Snce J N J, σ N, we have for S b, P J J0 J < J 0 = Φ σ / ; N for S bl, P J µ J µ = Φ σ / ; N for S a, Pµ J J0 J < J 0 = Φ σ / N µ J Φ σ / ; N for S I, P J J J 0 J 0 = Φ σ /, N where Φ s the error functon By Lagrangan relaxaton of APCS b and KKT condton, we have F = P{ k S b J k µ} P{µ J k < J 0 } k S bl k S a P{ J k J 0 } λ k S I K N k T k= Let φ denote the pdf of the standard normal dstrbuton We obtan the followng condtons For S b, 0 = F = λ + N k S b,k = P{µ J k J 0 } φ k S a k S I P{ J k < J 0 } J0 J σ / N P{ J k µ} k S bl J 0 J σ N For S bl, 0 = F = λ + N P{ k S b J k < J 0 } P{µ J k J 0 } φ k S a k S I For S a, k S bl,k = µ J σ / N P{ J k µ} µ J 3 σ N 0 = F = λ + N P{ J l µ} k S b k S bl P{µ J k < J 0 } J k J 0 } k S a,k = [ φ For S I, J0 J σ / N k S I P{ J 0 J φ σ N µ J σ / N µ ] J 4 σ N 0 = F = λ N P{ J k µ} k S b k S bl J k J 0 } P{µ k S a k S I,k = J J 0 φ σ / J J 0 5 N σ N In order to fnd the relatonshp between N and N j, we need to consder = 0 cases that θ and θ j belong to dfferent sets Case : θ S b, θ j S a Equatng and 4, we have P{ J < J 0 } [ φ J 0 J j J 0 J j σ j Nj φ µ J j µ J j σ j / N j σ j / N j σ j Nj = P{µ J J0 J j < J 0 }φ σ / J0 J N σ N Assumng N takes contnuous values, let N = α T Takng the asymptotc lmt of the above equaton as T, we have [ lm logp{ J < J 0 } + loga logσ j ] T T logα jt = lm T T {logp{µ J j < J 0 } J 0 J α T + σ J0 J log logα T }, 6 σ where J 0 J j A = φ σ j / J 0 J j φ α j T By L Hôptal s Rule, lm T T loga da/dt = lm T A = lm T exp J 0 J j σ j /α j J 0 J j σ j /α j J 0 J j T σ j /α j µ J j σ j / α j T µ J j µ J σ j /α j j ] µ J j µ J j T J σ j /α 0 J j µ J j j +

10 0 If J 0 J j J j µ, then exp 0 as T Hence, J 0 J j T σ j /α j lm T loga = J j µ α j T If J 0 J j < J j µ, then exp as T Hence, σ j J 0 J j T σ j /α j lm T loga = J 0 J j α j T σ j µ J j T σ j /α j µ J j T σ j /α j By applyng the above results of A to equaton 6, we get If J j J 0+µ, If J j > J 0+µ, α α j = µ J j J 0 J σ σ j α α j = J 0 J j J 0 J σ σ j Smlarly, we can obtan the relatonshp between N and N j for the other 9 cases If S a =, t reduces to the case of OCBA-mbG By combnng the results of all the 0 cases, we prove Theorem 3 ACKNOWLEDGMENT A prelmnary verson of the manuscrpt was presented at the 00 Wnter Smulaton Conference [38] REFERENCES [] Q S Ja, An adaptve samplng algorthm for smulaton-based optmzaton wth descrptve complexty preference, IEEE Transactons on Automaton Scence and Engneerng, vol 8, no 4, pp 70 73, 0 [] D P Bertsekas, Dynamc Programmng and Optmal Control Athena Scentfc, 005 [3] H P Gallher, P M Morse, and M Smond, Dynamcs of two classes of contnuous-revew nventory systems, Operatons Research, vol 7, no 3, pp , 959 [4] I Sahn, On the objectve functon behavor n s, S nventory models, Operatons Research, vol 30, no 4, pp , 99 [5] A Federgruen and Y S Zheng, An effcent algorthm for computng an optmal r, Q polcy n contnuous revew stochastc nventory system, Operatons Research, vol 40, no 4, pp , 99 [6] K Kar, A Krshnamurthy, and N Jagg, Dynamc node actvaton n networks of rechargeable sensors, IEEE/ACM Transactons on Networkng, vol 4, no, pp 5 6, 006 [7] J Branke, S E Chck, and C Schmdt, Selectng a selecton procedure, Management Scence, vol 53, no, pp 96 93, 007 [8] S H Jacobson and E Yucesan, Computatonal ssues for accessblty n dscrete event smulaton, ACM Transactons on Modelng and Computer Smulaton, vol 6, no, pp 53 75, 996 [9], Common ssues n dscrete-event smulaton and dscrete optmzaton, IEEE Transactons on Automatc Control, vol 47, no, pp , 00 [0] M C Fu, Optmzaton for smulaton: Theory vs practce, INFORMS Journal on Computng, vol 4, no 3, pp 9 5, 00 [] L P, Y Pan, and L Sh, Hybrd nested parttons and mathematcal programmng approach and ts applcatons, IEEE Transactons on Automaton Scence and Engneerng, vol 4, no 5, p , 008 [] W Chen, L P, and L Sh, An enhanced nested parttons algorthm usng soluton value predcton, IEEE Transactons on Automaton Scence and Engneerng, vol 8, no, pp 4 49, 0 [3] W P Wong, L H Lee, and W Jaruphongsa, Budget allocaton for effectve data collecton n predctng an accurate D effcency score, IEEE Transactons on Automatc Control, vol 56, no 6, pp 35 46, 0 [4] S S Gupta, On some multple decson selecton and rankng rules, Technometrcs, vol 7, no, pp 5 45, 965 [5] T J Santner, A restrcted subset selecton approch to rankng and selecton problems, The Annals of Statstcs, vol 3, no, pp , 975 [6] L W Koeng and A M Law, A procedure for selectng a subset of sze m contanng the l best of k ndependent normal populatons, Communcatons n Statstcs - Smulaton and Computaton, vol 4, pp , 985 [7] E J Dudewcz and S R Dalal, Allocaton of observaton n rankng and selecton wth unequal varances, Sankhya: The Indan Journal of Statstcs, vol 37, pp 8 78, 975 [8] C-H Chen, J Ln, E Yücesan, and S E Chck, Smulaton budget allocaton for further enhancng the effcency of ordnal optmzaton, Journal of Dscrete Event Dynamc Systems: Theory and Applcatons, vol 0, pp 5 70, 000 [9] C-H Chen, D He, and M Fu, Effcent dynamc smulaton allocaton n ordnal optmzaton, IEEE Transactons on Automatc Control, vol 5, no, pp , 006 [0] C-H Chen, E Yücesan, L Da, and H Chen, Effcent computaton of optmal budget allocaton for dscrete event smulaton experment, IIE Transactons, vol 4, no, pp 60 70, 00 [] C-H Chen, D H He, M Fu, and L H Lee, Effcent smulaton budget allocaton for selectng an optmal subset, INFORMS Journal on Computng, vol 0, no 4, pp , 008 [] L H Lee, E P Chew, S Y Teng, and D Goldsman, Optmal computng budget allocaton for mult-objectve smulaton models, n Proceedngs of 004 Wnter Smulaton Conference, 004, pp [3] S Y Teng, L H Lee, and E P Chew, Mult-objectve ordnal optmzaton for smulaton optmzaton problems, Automatca, vol 43, no, pp , 007 [4] E P Chew, L H Lee, S Y Teng, and C H Koh, Dfferentated servce nventory optmzaton usng nested parttons and MOCBA, Computers and Operatons Research, vol 36, no 5, pp , 009 [5] L H Lee, E P Chew, S Y Teng, and D Goldsman, Fndng the pareto set for mult-objectve smulaton models, IIE Transactons, vol 4, no 9, pp , 00 [6] J Branke and D Mattfeld, Antcpaton and flexblty n dynamc schedulng, Internatonal Journal of Producton Research, vol 43, no 5, pp , 005 [7] J Branke and J Gamer, Effcent samplng n nteractve mult-crtera selecton, n Proceedngs of the 007 INFORMS Smulaton Socety Research Workshop, 007 [8] S Andradóttr, D Goldsman, B W Schmeser, L W Schruben, and E Yücesan, Analyss methodology: Are we done? n Proceedngs of the 37th conference on Wnter smulaton, 005, pp [9] R Szechtman and E Yücesan, A new perspectve on feasblty determnaton, n Proceedngs of the 008 Wnter Smulaton Conference, 008, pp [30] N A Pujowdanto, L H Lee, C-H Chen, and C M Yap, Optmal computng budget allocaton for constraned optmzaton, n Proceedngs of 009 Wnter Smulaton Conference, 009, pp [3] Q-S Ja and Q-C Zhao, Strategy optmzaton for controlled markov process wth descrptve complexty constrant, Scence n Chna Seres F: Informaton Scences, vol 5, no, pp , 009 [3] Q-S Ja, On state aggregaton to approxmate complex value functons n large-scale Markov decson processes, IEEE Transactons on Automatc Control, vol 56, no, pp , 0 [33] Y-C Ho, Q-C Zhao, and Q-S Ja, Ordnal Optmzaton: Soft Optmzaton for Hard Problems Sprnger, 007 [34] C-H Chen, A lower bound for the correct subset-selecton probablty and ts applcaton to dscrete-event system smulatons, IEEE Transactons on Automatc Control, vol 4, no 8, pp 7 3, 996 [35] D He, S E Chck, and C H Chen, The opportunty cost and OCBA selecton procedures n ordnal optmzaton, IEEE Transactons on Systems, Man, and Cybernetcs Part C, vol 37, no 5, pp 95 96, 007 [36] L A Levn, Unversal sequental search problems, Problems of Informaton Transmsson, vol 9, no 3, pp 65 66, 973 [37] R C Walker, Introducton to Mathematcal Programmng Upper Saddle Rver, NJ: Prentce & Hall, 999 [38] S Yan, E Zhou, and C-H Chen, Effcent smulaton budget allocaton for selectng the best set of smplest good enough desgns, n Proceedngs of the 00 Wnter Smulaton Conference, 00, pp 5 59

11 PLACE PHOTO HERE Shen Yan receved her Bachelors of Engneerng degree wth hghest honors from Chnese Unversty of Hong Kong, Hong Kong n 009, and receved the Master of Scence degree n Industral Engneerng from the Unversty of Illnos at Urbana-Champagn n 0 Her research nterest s smulaton optmzaton PLACE PHOTO HERE Enlu Zhou receved the BS degree wth hghest honors n electrcal engneerng from Zhejang Unversty, Chna, n 004, and the PhD degree n electrcal engneerng from the Unversty of Maryland, College Park, n 009 Snce then she has been an Assstant Professor at the Industral & Enterprse Systems Engneerng Department at the Unversty of Illnos Urbana-Champagn Her research nterests nclude Markov decson processes, stochastc control, and smulaton optmzaton She s a recpent of the Best Theoretcal Paper award at the 009 Wnter Smulaton Conference and the 0 AFOSR Young Investgator award PLACE PHOTO HERE Chun-Hung Chen receved hs PhD degree n Engneerng Scences from Harvard Unversty n 994 He s a Professor of Systems Engneerng & Operatons Research at George Mason Unversty and s also afflated wth Natonal Tawan Unversty Dr Chen was an Assstant Professor of Systems Engneerng at the Unversty of Pennsylvana before jonng GMU Sponsored by NSF, NIH, DOE, NASA, MDA, and FAA, he has worked on the development of very effcent methodology for stochastc smulaton optmzaton and ts applcatons to ar transportaton system, semconductor manufacturng, healthcare, securty network, power grds, and mssle defense system Dr Chen receved the Natonal Thousand Talents Award from the central government of Chna n 0, the Best Automaton Paper Award from the 003 IEEE Internatonal Conference on Robotcs and Automaton, 994 Elahu I Jury Award from Harvard Unversty, and the 99 MasPar Parallel Computer Challenge Award Dr Chen has served as Co-Edtor of the Proceedngs of the 00 Wnter Smulaton Conference and Program Co-Char for 007 Informs Smulaton Socety Workshop He has served as a department edtor for IIE Transactons, assocate edtor of IEEE Transactons on Automatc Control, area edtor of Journal of Smulaton Modelng Practce and Theory, assocate edtor of Internatonal Journal of Smulaton and Process Modelng, and assocate edtor of IEEE Conference on Automaton Scence and Engneerng