Efficient Estimation of the Value of Information in Monte Carlo Models

Transcription

1 119 Effcent Estmaton of the Value of Informaton n Monte Carlo Models Tom Chave l,2 and Max Henron l,3 1 RockweJI Internatonal Scence Lab, 444 Hgh St., Palo Alto, CA Department of Engneerng-Economc Systems, Stanford Unversty 3 The Insttute for Decson Systems Research, Inc., 35 Cambrdge Ave., #38, Palo Alto, CA 9436 Abstract The expected value of nformaton (EVI) s the most powerful measure of senstvty to uncertanty n a decson model: t measures the potental of nformaton to mprove the decson, and hence measures the expected value of the outcome. Standard methods for computng EVI use dscrete varables and are computatonally ntractable for models that contan more than a few varables. Monte Carlo smulaton provdes the bass for more tractable evaluaton of large predctve models wth contnuous and dscrete varables, but so far computaton of EVI n a Monte Carlo settng also has appeared mpractcal. We ntroduce an approxmate approach based on preposteror analyss for estmatng EVI n Monte Carlo models. Our method uses a lnear approxmaton to the value functon and multple lnear regresson to estmate the lnear model from the samples. The approach s effcent and practcal for extremely large models. It allows easy estmaton of EVI for perfect or panal nformaton on ndvdual varables or on combnatons of varables. We llustrate ts mplementaton wthn Demos (a decson modelng system), and ts applcaton to a large model for crss transponaton plannng. 1. EVI: What's so, and What's New Any model s nevtably a smplfcaton of realty, and most of ts nput quanttes are nvarably uncertan. Senstvty analyss dentfes whch sources of uncertanty n a model affect ts outputs most sgnfcantly. In ths way, t helps a decson maker focus attenton on what assumptons really matter. It also helps a decson modeler to assgn prortes to hs efforts to mprove, refne, or extend hs model by dentfyng those varables for whch t wll be most valuable to fnd more complete data, to ntervew more knowledgeable experts, or to buld more elaborate submodels. The expected value of nformaton (EVI) on a varable x measures the expected ncrease n value y f we learn new nformaton about x and make a decson wth hgherexpected value n lght of that nformaton. It s the most powerful method of senstvty analyss because t analyes a varable's mportance n terms of the overall prescrpton for acton, and t expresses that mportance n the utlty or value unts of the problem. Other methods, such as rank-order correlaton, express mportance n terms of the correlaton between an uncertan varable and the output of the decson model. There are many cases where a varable can show hgh senstvty n ths way, yet stll have no effect on the selecton of an optmal decson. Determnstc perturbaton measures mportance n utlty or value unts, but t gnores nonlneartes and nteractons among varables, and also fals to measure a varable's mportance n terms of that varable's ablty to change the recommended decson.

2 12 Chave and Henron One calculates EVPI (Expected Value of Perfect Infor maton) n dscrete models by rollng back the decson tree. The computaton tself s straghtforward n the sense that, to compute EVI, one smply places at the front of the tree the chance varables to be observed. The EVPI s computed as the dfference between the expected value computed for ths scenaro and the expected value for the regular tree, wthout observatons. 2. Framework A decson model conssts of a set of n state varables x1,,x n, whch we wll denote by X. The decson maker has control of a decson varable D, whch can assume one of m possble values d1,..,dm. The value or utlty functon v(x,d) expresses the payoff to the decson maker when X obtans and decson d s chosen. Computng EVI wth contnuous varables s less ntutve, because we have no tdy way of reversng the uncertanty, unlke the dscrete case. Yet contnuous models are ncreasngly the norm for rsk and decson analyss, frst because dscretng nherently contnuous varables ntroduces unnecessary approxmaton, and second because Monte Carlo methods and ther varants (e.g., Latn hypercube) generate tractable, hghly effcent solutons to predctve models that contan thousands of varables. An especally useful feature of the Monte Carlo method s that, for a specfed error, the computatonal complexty ncreases lnearly n the number of uncertan varables [Morgan and Henron, 199]. Exact methods requre computaton tme that s exponental n the number of varables. There s thus a need to develop flexble, effcent methods for computng EVI on contnuous varables n a Monte Carlo settng. A ftexble method has (I) the ablty to compute EVI on sngle varables or on any combnaton of varables, and (2) the ablty to compute both perfect and partal values of nformaton. Perfect nformaton removes uncertanty entrely. Partal nformaton reduces uncertanty. We present a general framework for calculatng EVI based on preposteror analyss. Usng that framework, we develop a technque for computng EVI that depends on a lnear approxmaton to the value functon and on multple lnear regresson to estmate the constants for the lnear functon. We also dscuss a heurstc method for measurng the value of partal nformaton n terms of what we call the relatve nformaton multple (RIM). We have mplemented these methods n detachable computatonal modules usng Demos, a decson modelng system from Lumna, Inc., Palo Alto, CA. We demonstrate ther use on a large model to ad n mltary transportaton crss plannng. In a typcal decson model, the state varables are uncertan. We express pror knowledge about X n the form of a probablty dstrbuton, denoted {XI },where denotes a pror state of nformaton. The optmal Bayes' decson maxmng the expected value 1 s gven by l = Arg max ( l' ) v ( X ' d ) d I. The optmal decson gven perfect nformaton on state varable x, denoted d * _., s d x Arg max d We defne EVPI on x as.., (v(x,d)jx, ) EVPI (x) = (v (X, l x) I )- {v (X, d*) I ). In a smlar fashon, we defne the optmal expectedvalue decson gven the revelaton of evdence e, d* e as l = A rg max e (v( X,d)Je. ) d Then the EVI for evdence e s EVI(e) = (v(x,d.e)l )- ( v(x,l)l ) 2.1 Bnary decsons and Functon Let us consder a smplfed decson problem wth two decson alternatves: one of them s the optmal Bayes' decson d*; the other we denote tr". In vew of the uncertanty n the state varables, there must exst uncertanty n the outputs as well. Thus, for each 1. We use Howard's nferental notaton (see, for example, Howard, 197). {XIS) denotes the probablty densty of X condtonal on S; (XIS) denotes the expectaton of X condtonal on S.

3 Effcent Estmaton of the Value of Informaton n Monte Carlo Models 121 decson d;, there exsts a unque probablty dstrbuton on value { v(x,d;)l } (see Fgure 1). For notatonal convenence we let v(d) = (v(x, d) I ). In fact, we can use the ntuton behnd Fgure 2 to wrte an expresson for the general EVPI, whch s EVPI on all state varables. The absolute value of n the negatve shaded porton s the utlty that we could gan by chaosng cy nstead of d'; ts probablty s just ts correspondng value on the densty curve. Therefore, we have that EVPI = f ll {l } d. (EQ 1) We now defne = v(x,l) -v(x,tf). Functon s the pvotal element n our framework for computng EVI because t descrbes the dfference n value between the best and second-best decsons. In Fgure 2, we have graphed the probablty dstrbuton of. The shaded area represents the total probablty of makng a bad decson,.e., dong d' when cy would yeld hgher value. Explotng nformaton encoded n the shaded, negatve porton of the dstrbuton's curve wll provde the necessary clues to compute EVPI and EVI. FIGURE 1. Probablty strbutons on value for the two decsons d and cr-. Pro densty {v(x,tf)[ } {v(x,l)[s} 2.2 Preposteror Analyss Preposteror analyss helps us to calculate the effect on X of our seeng evdence e, gven a pror state of nformaton. At the heart of preposteror analyss s the specfcaton of a preposteror dstrbuton, whch s a a pror probablty dstrbuton on a posteror mean. Probablty theory provdes a prncpled bass for calculatng a preposteror dstrbuton, gven a pror and an adequate means of specfyng the effects of learnng new nformaton. How do we represent perfect nformaton on a contnuous random varable X? If X were known wth certanty, then ts varance would be equal to. Thus, we can thnk of evdence e as an nformaton-gatherng actvty that somehow reduces the varance of X to. Evdence e that provdes partal nformaton reduces the varance on the pror of X, wthout shrnkng that pror to. The followng lemma, taken from basc probablty theory, s known as the condtonal expectaton formula. Lemma 1: (XIs> = ((X] e)l ). FIGURE 2. Functon : the dfference n value between the best and second-best decsons. A further useful result s the followng lemma, whch gves the formula for condtonal varance. Lemma 2: Var( (X] e, )) = Var(Xls) -(V ar (XI e) I 1;). d * best Let J.l' and u' 2 denote the pror mean and varance of. They e computed from our pror uncertantes X and our value functon v. If we observe e, then we mght ask how e nfluences ; n partcular, we would lke to know howe affects J.l'. We wll denote the posteror mean of gven evdence by J.l".The dstrbuton { J.l" I } s a pror den sty on the posteror mean J.l" ; that s, t s a preposteror for the nner (XI e) on the rght densty. Substtutng J.l" hand sde of the equaton n Lemma 1 reveals that (EQ 2)

4 122 Chave and Henron Eq. 2shows that the mean of the preposteror dstrbuton { ll" I } s the same as the pror mean ll'. If cr" 2 denotes the posteror varance of after e, then applcaton of Lemma 2 shows that The type of ntegral gven n Eq. 4 s known as a lnear loss ntegral. In general, such an ntegral s mpossble to evaluate analytcally, so we must rely on statstcal tables or numercal approxmataton methods to evaluate t. Var {!l" 1 } == cr'2 - cr" 2. (EO 3) If the pror and posteror on are normal, then, as proved n [Raffa and Schlafer, 1961], the preposteror on s normal also. That s, the normal dstrbuton s conjugate to the normal samplng process. We thus requre to be dstrbuted normally. In Fgure 3, we show a pror on. a possble posteror on gven evdence e, and a preposteror densty { ll" I }. Note that the preposteror has the same mean as the pror, and that ts varance s the dfference between the pror varance and the posteror varance. FIGURE 3. Pror, posteror, and preposteror denstes. 3. Complexty and Non-Addtvty of EVI Inference n probablstc models wth dscrete varables s exponental n the number of varables, so we would expect the exact calculaton ofevi to be exponental n the number of varables also. For smplcty, we assume a sngle decson varable wth m alternatves. Let k; be the number of states for the th state varable x;. To evaluate a decson tree wth n state varables, we requre a number of value computatons at the leaves equal to m IJk;. = 1 Computng EVI on some subset of varables requres at least the same number of value computatons at the leaves, and thus we see that exact calculaton of EVI s exponental nn. Also note that EVI calculatons n dscrete models are possble for perfect nformaton only. If c; represents the value of nformaton on state varable X;, and C represents the value of nformaton on all the state varables smultaneously, then C'#L,c; The above relaton makes t dffcult to devse separable, or ncremental, procedures for computng EVI, because EVI wll often demonstrate nonlneartes for varyng combnatons of varables and for varyng cases of perfect and partal nformaton. The preposteror densty encodes a state of knowledge about n lght of what evdence e mght reveal. Its nterpretaton s the same as n Fgure 2. Because t s a probablty densty on value, we can ntegrate over ts negatve area to calculate the EVI of evdence e. Thus, we have EVI (e) J Ill" l {j!" l S} dj!" ' (EO 4) 4. Approxmaton of EVI We are ready to apply the precedng analyss to develop an effcent algorthm for estmatng EVI. We ntroduce a lnear approxmaton to the value functon, whch n tum allows us to derve an expresson for, the net dfference n value between two decson alternatves. Preposteror analyss on provdes a flexble mechansm for estmatng EVI.

5 Effcent Estmaton ot the Value of Informaton n Monte Carlo Models The Lnear Value Model We requre a key approxmatng assumpton: The value functon v(x,d;) can be approxmated by a frst-order (lnear) equaton for each decson d;. that s, we can wrte v; as a lnear functon of the x;. v (X, d) = L jx+ a 1 =l...n,j=l,2. j ror varance on xk. Snce e s perfect nformaton on x k, cr" =. Eq. 1 gves an approxmaton to the pror varance on, cr';. Gven e, we know that the kth term n the expresson n Eq. 1 must be equal to ero. We can thus wrte the posteror varance for gven e, cr" 2: (EQ 11) We assume, for now, that the x; are ndependent. (The assumpton s not necessary; we use t to smplfy our presentaton.) We denote the pror mean of x; by ; and denote ts pror varance by cr'2;. Our approxmatng assumpton allows us to perform smple but useful probablstc analyss. Frst, by lnearty of expectaton, we can wrte the mean (d ; ) as v (d ; ) = L,l\p'; + a r Second, the varance of { v(x,d;)} can be wrtten as Var{v(X,d;) } = ) cr ;. ; (EQ 5) (EQ 6) By our approxmatng assumpton, we can wrte a lnear approxmaton for v(x,d\ v(x,l) = and one for v(x,d+), L,13:X;+a*; (EO 7) (EQ 8) In vew of Eq. 3, the preposteror varance on s Var [J.l" ] (EQ 12) l 4.2 Monte Carlo methods: Estmaton of the Coeffcents In Monte Carlo smulaton, we generate a sample of n scenaros by samplng from the pror dstrubtons {XI }. A scenaro Xs s ann-tuple of state- varable assgnments to X. v(xs,d) s equal to the value or utlty generated by the sth scenaro for the th decson alternatve. We can estmate the expected value of each decson d1 as the average of the values v(xs,d) over the scenaro ndex s. The optmal Bayes' decson s the maxmum of those averages. (Naturally, hgher sample ses gve answers of greater precson.) We represent ths process for our bnary decson problem n Table 1: TABLE 1. Determnng the optmal decson n Monte Carlo decson ar alyss wth sample se= 1 and two decson alternatves Combnng Eqs. 5-8 wth the defnton of, we can wrte sth Scenaro Value wth d1 Value wth d2 expressons for the pror mean and varance of : XI v(x1,d2) v(l)-v(cf) J.l' = " (A - +) f..'. +(a* -a+). 1-'f I l (EQ 9) Xwo v(x1,d1) v(x1rod2) (EQ 1) Suppose that e expresses perfect nformaton on xk and no nformaton about the other x ;. Let cr" denote the poste- Average 1v(X d) s' I L s =I 1 too v (Xs, d1) 2: s=1 1

6 124 Chave and Henron l, arg max ( v (Xs, d) ) )=1,2 1 s"' I The only outstandng task s to estmate the constants for the lnear-approxmaton model. To ths end, we apply multple lnear regresson analyss to estmate the constants n Eqs. 7 and 8. Let be an ndex nto set of m decson alternatves, and letj be an ndex nto the n state varables. From [Shavelson, 1988], we can use multple lnear regresson to wrte constants for v;- value for the th decson alternatve n terms of the n state varablesas follows: source could tell me roughly twce as much as I know now, then the equvalent RIM s 2. A varable xk's contrbuton to the pror varance o-'2 s g1ven. m. Eq. 1 as (pk- k) rt* o ' k.for arim=rofevdence e on varable xk, the posteror varance o" 2 s gven by o-" = - (rt _A+) o ' r f'k -'k k' The preposteror varance s estmated as r- I 2 2 Var [)l" l S) = -r- ( - +) o-' k (EQ 14) (EQ 15) (EQ 13) The preposteror mean for partal nformaton stays the same, as n Eq Z Is Normal where Rj = correlaton(v,x). Ru= correlaton(x,x),s;= standard devaton(v;), and o ;=standard devaton(xj). We estmate these quanttes drectly from our Monte Carlo samples. Recall from Secton 2. 1 that the v; generate probablty dstrbutons n a Monte Carlo model. Thus, t makes sense to thnk of them as random varables wth correspondng sample correlatons and standard devatons. The a. are estmated as follows: 1 We wll assume that the x;, are normally dstrbuted. In lght of the followng proposton from probablty theory, our lnear-approxmatng assumpton requres to be normally dstrbuted also. Proposton: Let X; be a collecton of n normal random varables wth means gven by J..l and varances o 2 Defne the random varable Y as Y = a+'.x.. L,. I I ' aj = n (v)- L ;}'r ;,. I Then Y s normally dstrbuted also, wth mean gven by a+ r J..l' 4.3 Relatve Informaton Multple Suppose now that e expresses partal, rather than perfect, nformaton on xk. It s not mmedately obvous how to specfy partal nformaton on an uncertan varable. We suggest the followng method, based on our concept of a RIM. A RIM of evdence e on varable xk s defned to be the rato between the pror varance o ' and the posteror varance o " on xk after e has been seen. In ntutve terms, the RIM measures how much we could know relatve to what we know now. It s a multple on mssng but knowable nformaton. For example, f an nformaton and varance Observe that our approxmatng assumpton allows us to wrte the mean and varance of usng standard probablty formulae. There s nothng about our framework, however, that forces the actual dstrbuton of to belong to the same famly as do 's component dstrbutons. For exam-

7 Effcent Estmaton of the Value of Informaton n Monte Carlo Models 125 ple, f the x; are Posson, normal, and exponental, then s a hard-to-assess, mongrel dstrbuton. Assumng that the x are normal forces to be normal also. If the x; are nonnormal, then we must make an extra approxmatng assumpton that s normal also, although we must emphase that ths assumpton would not be analytcally true. A lmtng aspect of the technque presented here s that t measures EVI relatve to only two decsons. In [Chave, 1994], we show how to extend t to accommodate multple ( 3) decson alternatves. 4.5 Algorthm for EVI We now summare, n algorthmc form, our general technque for estmatng EVI n a Monte Carlo decson model: 1. Select the two decson alternatves generatn the hghest and second-hghest expected value, d and cr. 2. Defne varable as the dfference between v(x,d*) and v(x,d+). 3. Calculate regresson constants, +,a, and a+. I I 4. Usng Eqs. 2 and 9, calculate the mean (!l" l s> of the preposteror dstrbuton of : (!l" Js> = 11' = "c - +)!l' + (a -a+)..., J J j j 5. For perfect nformaton on X, defne 2 2 Var {!l" 1 S} = ( - +) a' ; 6. For partal nformaton on X wth RIM=k, defne Var{!l"JS} = k l( - +)2 a,2 7. For perfect nformaton on varables wth ndces n S, defne 8. For partal nformaton on varables wth ndces and a correspondng ordered set of RIM's k;, defne Var {!l" l S} = l: k ;- I 2 - k - ( - +) a'\. l l 9. For perfect nformaton on varables wth ndces n S mxed wth partal nformaton on varables wth ndces and a correspondng ordered set of RIM's k;. k; Var { " 1;} =I,-( - +) a' ;+I, ( - +) a' k ; je S 1.Defne the preposteror densty on, {!l" l S }, as Normal ( (!l" l s> = 11. Express EVI as EVI = ll' Var {!l" l S}) f ll"l {!l"js}d!l" 12. Perform the ntegraton n (11) numercally. 5. Applcaton We now descrbe an applcaton of our method to a large decson model developed at Rockwell's Palo Alto Scence Laboratory to support Course of Acton (COA) analyss for Noncombatant Evacuaton Operatons (NEO). Implemented n Demos, NEO-COA allows a user to nstantate a generc NEO plan wth specfc parameter values for locatons, forces, and destnatons of troops and cvlans. The model provdes nsghts nto the relatve strengths of alternatve plans by scorng them usng dfferent evaluaton metrcs, such as tme to complete the operaton. Because many of the elements of a real-world mltary plannng scenaro are not known wth certanty, several of the model's nputs are specfed as contnuous probablty dstrbutons. In the current verson of NEO-COA, there are three decson varables, or factors over whch a mltary planner exercses control: Securty forces: Securty forces vary n ther startng locatons, dates of avalablty, and capabltes n provdng securty. Safe havens: The places where cvlans gather to take shelter, safe havens dffer n terms of dstances from the assembly areas and port capactes. Transportaton assets: A confguraton of transportaton assets s a sequencng of transportaton capablty over a fxed perod of tme. "Three C-141 's avalable

8 126 Chave and Henron on day 2 and 5 C-14l's on days 3 through 1" s an example of a partcular transportaton confguraton. uncertantes took Demos 1 mnute, 53 seconds runnng on a Macntosh Ilfx computer. Because each of these decson varables currently possess three alternatves, there are a total of 27 avalable courses of acton. In addton, the NEO-COA model possesses over 1 dfferent nput varables; of those, currently nne are specfed as probablstc quanttes. Once the decson varables and nputs have been specfed, the model performs a dynamc smulaton of the flow of U.S. ctens (the non-combatants) from ther startng locatons wthn a country to a set of selected assembly areas, and then on to the safe havens. It also ncludes rsk factors assocated wth both U.S. ctens and U.S. mltary personnel as functons of tme. For example, rsk to U.S. ctens at the assembly areas can rse and fall over the course of an entre operaton n response to uncertan events, such as the arrval of securty forces. The functonal representatons of the rsk factors are then used to compute expected casualtes- cvlan and mltary - for varyng alternatves. A top level vew of the NEO-COA model as mplemented n Demos s shown n Fgure 4. There are three uncertantes for the NEO-COA model: Intal USCITS, probablty dstrbutons on the number of U.S. ctens n each of the three regons of the country (captal, north, and south) at the start of a crss plannng operaton; Country Regons Attrton Rsk, whch s the rsk posed to noncombatants over the course of an operaton; and Transfer Rate, whch s the speed at whch cvlans move from ther startng locatons to the assembly areas. Thus a total of nne contnuous probablty dstrbutons must be assgned; typcally, these are subjectve assessments provded by mltary planners usng the model. In Fgure 5, we show the results of applyng the EVI approxmaton technque to the NEO model for perfect nformaton. We see, for example, that the uncertanty about the number of Amercan ctens n the captal has EVI equal to about sx lves, and the uncertanty about the transfer rates n the captal has perfect nformaton value equal to more than seven lves. In all cases, the value of nformaton s hghest for uncertantes relatng to the captal regon, reflectng that the hghest number of ctens are concentrated there. The ntegral n Eq. 4 s evaluated numercally. Perfect nformaton calculatons on nne 6. Conclusons and Future Drectons We have descrbed a general analytc framework for estmatng EVI n a decson model usng preposteror analyss. It employs a lnear-approxmatng assumpton that allows us to wrte the value functon as a frst-order equaton n the nputs. We defne varable to be the dfference n value for the two decson alternatves. Multple lnear regresson on the nputs provdes the necessary constants for the lnear value equaton; we estmate the regresson constants from Monte Carlo sample nformaton. Applyng preposteror analyss to allows us to wrte an approxmaton to the value of perfect and partal nformaton for any combnaton of state varables. There are several areas n whch we plan to extend the work presented here. Frst, we would lke to develop a sster technque for approxmatng EVI on contnuous decson varables. Second, we would lke to examne how well our technque performs relatve to an exact, more costly approach. To ths end we wll apply our method to several large models, run t several tmes, and compare ts results to the correspondng exact answers. Thrd, we wlk apply statstcal proof technques to analye formally the algorthm's convergence and error characterstcs. 7. References Chave, Tom. (1994). "Recoverng Value oflnformaton from Parwse Peeks," Rockwell Palo Alto Scence Lab. Techncal Memorandum. Hennon. Max and Morgan, Granger. (199). Uncertanty: A Gulk ro Dealng wth Uncertanry n Quanttatve Rsk and Polcy Analyss. Cambrdge Unversty Press. Cambrdge. Howard. R. A. (197). "Proxmal Decson Analyss." Management Scence, 17, No. 9: Raffa, Howard and Schlafer, Raben. (1961 ). Appled Statstcal Decson Th ory, MIT Press, Boston. Shavelson, Rchard. (1988). Statstcal Reasonng for IM Behavoral Scences. 2nd ed. Allyn and Bacon, Inc, Boston.

9 Effcent Estmaton of the Value of Informaton n Monte Carlo Models 127 FIGURE 4. Top-level vew of the NEO-COA model. FIGURE 5. Approxmaton of perfect nformaton values. Md Value of Value of Perfect Informaton Key: Uncerta a nput vars Tl X Axs: Country Regons 7.5,..I C p"hl North CountrJ Regons South Ke11 Uncertan nput v rs - Number of UC ets - Attrton r te lll5l Tr nsfeor rat.>