Networks of Influence Diagrams: A Formalism for Representing Agents Beliefs and Decision-Making Processes

Transcription

1 Journal of Artfcal Intellgence Research 33 (2008) Submtted 11/07; publshed 9/08 Networks of Influence Dagrams: A Formalsm for Representng Agents Belefs and Decson-Makng Processes Ya akov Gal MIT Computer Scence and Artfcal Intellgence Laboratory Harvard School of Engneerng and Appled Scences Av Pfeffer Harvard School of Engneerng and Appled Scences gal@csal.mt.edu av@eecs.harvard.edu Abstract Ths paper presents Networks of Influence Dagrams (NID), a compact, natural and hghly expressve language for reasonng about agents belefs and decson-makng processes. NIDs are graphcal structures n whch agents mental models are represented as nodes n a network; a mental model for an agent may tself use descrptons of the mental models of other agents. NIDs are demonstrated by examples, showng how they can be used to descrbe conflctng and cyclc belef structures, and certan forms of bounded ratonalty. In an opponent modelng doman, NIDs were able to outperform other computatonal agents whose strateges were not known n advance. NIDs are equvalent n representaton to Bayesan games but they are more compact and structured than ths formalsm. In partcular, the equlbrum defnton for NIDs makes an explct dstncton between agents optmal strateges, and how they actually behave n realty. 1. Introducton In recent years, decson theory and game theory have had a profound mpact on the desgn of ntellgent systems. Decson theory provdes a mathematcal language for sngle-agent decson-makng under uncertanty, whereas game theory extends ths language to the multagent case. On a fundamental level, both approaches provde a defnton of what t means to buld an ntellgent agent, by equatng ntellgence wth utlty maxmzaton. Meanwhle, graphcal languages such as Bayesan networks (Pearl, 1988) have receved much attenton n AI because they allow for a compact and natural representaton of uncertanty n many domans that exhbt structure. These formalsms often lead to sgnfcant savngs n representaton and n nference tme (Dechter, 1999; Cowell, Laurtzen, & Spegelhater, 2005). Recently, a wde varety of representatons and algorthms have augmented graphcal languages to be able to represent and reason about agents decson-makng processes. For the sngle-agent case, nfluence dagrams (Howard & Matheson, 1984) are able to represent and to solve an agent s decson makng problem usng the prncples of decson theory. Ths representaton has been extended to the mult-agent case, n whch decson problems are solved wthn a game-theoretc framework (Koller & Mlch, 2001; Kearns, Lttman, & Sngh, 2001). The focus n AI so far has been on the classcal, normatve approach to decson and game theory. In the classcal approach, a game specfes the actons that are avalable to the agents, as well as ther utltes that are assocated wth each possble set of agents c 2008 AI Access Foundaton. All rghts reserved.

2 Gal & Pfeffer actons. The game s then analyzed to determne ratonal strateges for each of the agents. Fundamental to ths approach are the assumptons that the structure of the game, ncludng agents utltes and ther actons, s known to all of the agents, that agents belefs about the game are consstent wth each other and correct, that all agents reason about the game n the same way, and that all agents are ratonal n that they choose the strategy that maxmzes ther expected utlty gven ther belefs. As systems nvolvng multple, autonomous agents become ubqutous, they are ncreasngly deployed n open envronments comprsng human decson makers and computer agents that are desgned by or represent dfferent ndvduals or organzatons. Examples of such systems nclude on-lne auctons, and patent care-delvery systems (MacKe-Mason, Osepayshvl, Reeves, & Wellman, 2004; Arunachalam & Sadeh, 2005). These settngs are challengng because no assumptons can be made about the decson-makng strateges of partcpants n open envronments. Agents may be uncertan about the structure of the game or about the belefs of other agents about the structure of the game; they may use heurstcs to make decsons or they may devate from ther optmal strateges (Camerer, 2003; Gal & Pfeffer, 2003b; Rajarsh, Hanson, Kephart, & Tesauro, 2001). To succeed n such envronments, agents need to make a clear dstncton between ther own decson-makng models, the models others may be usng to make decsons, and the extent to whch agents devate from these models when they actually make ther decsons. Ths paper contrbutes a language, called Networks of Influence Dagrams (NID), that makes explct the dfferent mental models agents may use to make ther decsons. NIDs provde for a clear and compact representaton wth whch to reason about agents belefs and ther decson-makng processes. It allows multple possble mental models of delberaton for agents, wth uncertanty over whch models agents are usng. It s recursve, so that the mental model for an agent may tself contan models of the mental models of other agents, wth assocated uncertanty. In addton, NIDs allow agents belefs to form cyclc structures, of the form, I beleve that you beleve that I beleve,..., and ths cycle s explctly represented n the language. NIDs can also descrbe agents conflctng belefs about each other. For example, one can descrbe a scenaro n whch two agents dsagree about the belefs or behavor of a thrd agent. NIDs are a graphcal language whose buldng blocks are Mult Agent Influence Dagrams (MAID) (Koller & Mlch, 2001). Each mental model n a NID s represented by a MAID, and the models are connected n a (possbly cyclc) graph. Any NID can be converted to an equvalent MAID that wll represent the subjectve belefs of each agent n the game. We provde an equlbrum defnton for NIDs that combnes the normatve aspects of decson-makng (what agents should do) wth the descrptve aspects of decson-makng (what agents are expected to do). The equlbrum makes an explct dstncton between two types of strateges: Optmal strateges represent agents best course of acton gven ther belefs over others. Descrptve strateges represent how agents may devate from ther optmal strategy. In the classcal approach to game theory, the normatve aspect (what agents should do) and the descrptve aspect (what analysts or other agents expect them to do), have concded. Identfcaton of these two aspects makes sense when an agent can do no better than optmze ts decsons relatve to ts own model of the world. However, n open envronments, t s mportant to consder the possblty that an agent s devatng from ts ratonal strategy wth respect to ts model. 110

3 Networks of Influence Dagrams NIDs share a relatonshp wth the Bayesan game formalsm, commonly used to model uncertanty over agents payoffs n economcs (Harsany, 1967). In ths formalsm, there s a type for each possble payoff functon an agent may be usng. Although NIDs are representatonally equvalent to Bayesan games, we argue that they are a more compact, succnct and natural representaton. Any Bayesan game can be converted to a NID n lnear tme. Any NID can be converted to a Bayesan game, but the sze of the Bayesan game may be exponental n the sze of the NID. Ths paper s a revsed and expanded verson of prevous work (Gal & Pfeffer, 2003a, 2003b, 2004), and s organzed as follows: Secton 2 presents the syntax of the NID language, and shows how they buld on MAIDs n order to express the structure that holds between agents belefs. Secton 3 presents the semantcs of NIDs n terms of MAIDs, and provdes an equlbrum defnton for NIDs. Secton 4 provdes a seres of examples llustratng the representatonal benefts of NIDs. It shows how agents can construct belef herarches of each other s decson-makng n order to represent agents conflctng or ncorrect belef structures, cyclc belef structures and opponent modelng. It also shows how certan forms of bounded ratonalty can be modeled by makng a dstncton between agents models of delberaton and the way they behave n realty. Secton 5 demonstrates how NIDs can model I beleve that you beleve type reasonng n practce. It descrbes a NID that was able to outperform the top programs that were submtted to a competton for automatc rockpaper-scssors players, whose strategy was not known n advance. Secton 6 compares NIDs to several exstng formalsms for descrbng uncertanty over decson-makng processes. It provdes a lnear tme algorthm for convertng Bayesan games to NIDs. Fnally, Secton 7 concludes and presents future work. 2. NID Syntax The buldng blocks of NIDs are Bayesan networks (Pearl, 1988), and Mult Agent Influence Dagrams (Koller & Mlch, 2001). A Bayesan network s a drected acyclc graph n whch each node represents a random varable. An edge between two nodes X 1 and X 2 mples that X 1 has a drect nfluence on the value of X 2. Let Pa(X ) represent the set of parent nodes for X n the network. Each node X contans a condtonal probablty dstrbuton (CPD) over ts doman for any value of ts parents, denoted P (X Pa(X )). The topology of the network descrbes the condtonal ndependence relatonshps that hold n the doman every node n the network s condtonally ndependent of ts non-descendants gven ts parent nodes. A Bayesan network defnes a complete jont probablty dstrbuton over ts random varables that can be decomposed as the product of the condtonal probabltes of each node gven ts parent nodes. Formally, P (X 1,...,X n )= n P (X Pa(X )) We llustrate Bayesan networks through the followng example. =1 Example 2.1. Consder two baseball team managers Alce and Bob whose teams are playng the late nnngs of a game. Alce, whose team s httng, can attempt to advance a runner by nstructng hm to steal a base whle the next ptch s beng delvered. A successful 111

4 Gal & Pfeffer steal wll result n a beneft to the httng team and a loss to the ptchng team, or t may result n the runner beng thrown out, ncurrng a large cost to the httng team and a beneft to the ptchng team. Bob, whose team s ptchng, can nstruct hs team to throw a ptch out, thereby ncreasng the probablty that a stealng runner wll be thrown out. However, throwng a ptch out ncurs a cost to the ptchng team. The decsons whether to steal and ptch out are taken smultaneously by both team managers. Suppose that the game s not ted, that s ether Alce s or Bob s team s leadng n score, and that the dentty of the leadng team s known to Alce and Bob when they make ther decson. Suppose that Alce and Bob are usng pre-specfed strateges to make ther decsons descrbed as follows: when Alce s leadng, she nstructs a steal wth probablty 0.75, and Bob calls a ptch out wth probablty 0.90; when Alce s not leadng, she nstructs a steal wth probablty 0.65, and Bob calls a ptch out wth probablty There are sx random varables n ths doman: Steal and PtchOut represent the decsons for Alce and Bob; ThrownOut represents whether the runner was thrown out; Leader represents the dentty of the leadng team; Alce and Bob represent the utlty functons for Alce and Bob. Fgure 1 shows a Bayesan network for ths scenaro. Fgure 1: Bayesan network for Baseball Scenaro (Example 2.1) The CPD assocated wth each node n the network represents a probablty dstrbuton over ts doman for any value of ts parents. The CPDs for nodes Leader, Steal, PtchOut, andthrownout n ths Bayesan network are shown n Table 1. For example, the CPD for ThrownOut, shown n Table 1d, represents the condtonal probablty dstrbuton P (ThrownOut Steal, PtchOut). Accordng to the CPD, when Alce nstructs a runner to steal a base there s an 80% chance to get thrown out when Bob calls a ptch out and a 60% chance to get thrown out when Bob remans dle. The nodes Alce and Bob have determnstc CPDs, assgnng a utlty for each agent for any jont value of the parent nodes Leader, Steal, PtchOut and ThrownOut. The utlty for Alce s shown n Table 2. The utlty for Bob s symmetrc and assgns the negatve value assgned by Alce s utlty for the same value of the parent nodes. For example, when Alce s leadng, and she nstructs a runner to steal a base, Bob nstructs a ptch out, and the runner s thrown out, then Alce ncurs a utlty of 60, whle Bob ncurs a utlty of Note that when Alce does not nstruct to steal base, the runner cannot be thrown out, and the utlty for both agents s not defned for ths case. 112

5 Networks of Influence Dagrams Leader alce bob none (a) node Leader Steal Leader true false alce bob (b) node Steal PtchOut Leader true false alce bob (c) node PtchOut ThrownOut Steal PtchOut true false true true true false false true 0 1 false false 0 1 (d) node ThrownOut Table 1: Condtonal Probablty Tables (CPDs) for Bayesan network for Baseball Scenaro (Example 2.1) Leader Steal PtchOut ThrownOut Alce alce true true true 60 alce true true false 110 alce true false true 80 alce true false false 110 alce false true true alce false true false 10 alce false false true alce false false false 0 bob true true true 90 bob true true false 110 bob true false true 100 bob true false false 110 bob false true true bob false true false 20 bob false false true bob false false false 0 Table 2: Alce s utlty (Example 2.1) (Bob s utlty s symmetrc, and assgns negatve value to Alce s value). 113

6 Gal & Pfeffer 2.1 Mult-agent Influence Dagrams Whle Bayesan networks can be used to specfy that agents play specfc strateges, they do not capture the fact that agents are free to choose ther own strateges, and they cannot be analyzed to compute the optmal strateges for agents. Mult-agent Influence Dagrams (MAID), address these ssues by extendng Bayesan networks to strategc stuatons, where agents must choose the values of ther decsons to maxmze ther own utltes, contngent on the fact that other agents are choosng the values of ther decsons to maxmze ther own utltes. A MAID conssts of a drected graph wth three types of nodes: Chance nodes, drawn as ovals, represent choces of nature, as n Bayesan networks. Decson nodes, drawn as rectangles, represent choces made by agents. Utlty nodes, drawn as damonds, represent agents utlty functons. Each decson and utlty node n a MAID s assocated wth a partcular agent. There are two knds of edges n a MAID: Edges leadng to chance and utlty nodes represent probablstc dependence, n the same manner as edges n a Bayesan network. Edges leadng nto decson nodes represent nformaton that s avalable to the agents at the tme the decson s made. The doman of a decson node represents the choces that are avalable to the agent makng the decson. The parents of decson nodes are called nformatonal parents. There s a total orderng over each agent s decsons, such that earler decsons and ther nformatonal parents are always nformatonal parents of later decsons. Ths assumpton s known as perfect recall or no forgettng. TheCPD of a chance node specfes a probablty dstrbuton over ts doman for each value of the parent nodes, as n Bayesan networks. The CPD of a utlty node represents a determnstc functon that assgns a probablty of 1 to the utlty ncurred by the agent for any value of the parent nodes. In a MAID, a strategy for decson node D maps any value of the nformatonal parents, denoted as pa,toachoceford.letc be the doman of D. The choce for the decson canbeanyvaluenc. A pure strategy for D maps each value of the nformatonal parents to an acton c C.Amxed strategy for D maps each value of the nformatonal parents to a dstrbuton over C. Agent α s free to choose any mxed strategy for D when t makes that decson. A strategy profle for a set of decsons n a MAID conssts of strateges specfyng a complete plan of acton for all decsons n the set. The MAID for Example 2.1 s shown n Fgure 2. The decson nodes Steal and PtchOut represent Alce s and Bob s decsons, and the nodes Alce and Bob represent ther utltes. TheCPDsforthechancenodeLeader and ThrownOut are as descrbed n Tables 1a and 1d. A MAID defnton does not specfy strateges for ts decsons. These need to be computed or assgned by some process. Once a strategy exsts for a decson, the relevant decson node n the MAID can be converted to a chance node that follows the strategy. Ths chance node wll have the same doman and parent nodes as the doman and nformatonal parents for the decson node n the MAID. The CPD for the chance node wll equal the strategy for the decson. We then say that the chance node n the Bayesan network mplements the strategy n the MAID. A Bayesan network represents a complete strategy profle for the MAID f each strategy for a decson n the MAID s mplemented by a relevant chance node n the Bayesan network. We then say that the Bayesan network mplements that strategy profle. Let σ represent the strategy profle that mplements all 114

7 Networks of Influence Dagrams Fgure 2: MAID for Baseball Scenaro (Example 2.1) decsons n the MAID. The dstrbuton defned by ths Bayesan network s denoted by P σ. An agent s utlty functon s specfed as the aggregate of ts ndvdual utltes; t s the sum of all of the utltes ncurred by the agent n all of the utlty nodes that are assocated wth the agent. Defnton 2.2. Let E be a set of observed nodes n the MAID representng evdence that s avalable to α and let σ be a strategy profle for all decsons. Let U(α) be the set of all utlty nodes belongng to α. The expected utlty for α gven σ and E s defned as U σ (α E) = E σ [U E] = P σ (u E) u U U(α) U U(α) u Dom(U) Solvng a MAID requres computng an optmal strategy profle for all of the decsons, as specfed by the Nash equlbrum for the MAID, defned as follows. Defnton 2.3. A strategy profle σ for all decsons n the MAID s a Nash equlbrum f each strategy component σ for decson D belongng to agent α nthemaidsonethat maxmzes the utlty acheved by the agent, gven that the strategy for other decsons s σ. σ argmax U τ,σ (α) (1) τ ΔS These equlbrum strateges specfy what each agent should do at each decson gven the avalable nformaton at the decson. When the MAID contans several sequental decsons, the no-forgettng assumpton mples that these decsons can be taken sequentally by the agent, and that all prevous decsons are avalable as observatons when the agent reasons about ts future decsons. Any MAID has at least one Nash equlbrum. Exact and approxmate algorthms have been proposed for solvng MAIDs effcently, n a way that utlzes the structure of the network (Koller & Mlch, 2001; Vckrey & Koller, 2002; Koller, Meggdo, & von Stengel, 115

8 Gal & Pfeffer 1996; Blum, Shelton, & Koller, 2006). Exact algorthms for solvng MAIDs decompose the MAID graph nto subsets of nterrelated sub-games, and then proceed to fnd a set of equlbra n these sub-games that together consttute a global equlbrum for the entre game. In the case that there are multple Nash equlbra, these algorthms wll select one of them, arbtrarly. The MAID n Fgure 2 has a sngle Nash equlbrum, whch we can obtan by solvng the MAID: When Alce s leadng, she nstructs her runner to steal a base wth probablty 0.2, and reman dle wth probablty 0.8, whle Bob calls a ptch out wth probablty 0.3, and remans dle wth probablty 0.7. When Bob s leadng, Alce nstructs a steal wth probablty 0.8, and Bob calls a ptch out wth probablty 0.5. The Bayesan network that mplements the Nash equlbrum strategy profle for the MAID can be quered to predct the lkelhood of nterestng events. For example, we can query the network n Fgure 2 and fnd that the probablty that the stealer wll get thrown out, gven that agents strateges follow the Nash equlbrum strategy profle, s Any MAID can be converted to an extensve form game a decson tree n whch each vertex s assocated wth a partcular agent or wth nature. Splts n the tree represent an assgnment of values to chance and decson nodes n the MAID; leaves of the tree represent the end of the decson-makng process, and are labeled wth the utltes ncurred by the agents gven the decsons and chance node values that are nstantated along the edges n the path leadng to the leaf. Agents mperfect nformaton regardng the actons of others are represented by the set of vertces they cannot tell apart when they make a partcular decson. Ths set s referred to as an nformaton set. Let D be a decson n the MAID belongng to agent α. There s a one-to-one correspondence between values of the nformatonal parents of D n the MAID and the nformaton sets for α at the vertces representng ts move for decson D. 2.2 Networks of Influence Dagrams To motvate NIDs, consder the followng extenson to Example 2.1. Example 2.4. Suppose there are experts who wll nfluence whether or not a team should steal or ptch out. There s socal pressure on the managers to follow the advce of the experts, because f the managers decson turns out to be wrong they can assgn blame to the experts. The experts suggest that Alce should call a steal, and Bob should call a ptch out. Ths advce s common knowledge between the managers. Bob may be uncertan as to whether Alce wll n fact follow the experts and steal, or whether she wll gnore them and play a best-response wth respect to her belefs about Bob. To quantfy, Bob beleves that wth probablty 0.7, Alce wll follow the experts, whle wth probablty 0.3, Alce wll play best-response. Alce s belefs about Bob are symmetrc to Bob s belefs about Alce: Wth probablty 0.7 Alce beleves Bob wll follow the experts and call a ptch out, and wth probablty 0.3 Alce beleves that Bob wll play the best-response strategy wth respect to hs belefs about Alce. The probablty dstrbuton for other varables n ths example remansasshownntable1. NIDs buld on top of MAIDs to explctly represent ths structure. A Network of Influence Dagrams (NID) s a drected, possbly cyclc graph, n whch each node s a MAID. To avod confuson wth the nternal nodes of each MAID, we wll call the nodes of a NID blocks. Let D be a decson belongng to agent α n block K, andletβ be any agent. (In 116

9 Networks of Influence Dagrams partcular, β may be agent α tself.) We ntroduce a new type of node, denoted Mod[β,D] wth values that range over each block L n the NID. When Mod[β,D] takes value L, we say that agent β n block K s modelng agent α as usng block L to make decson D. Ths means that β beleves that α may be usng the strategy computed n block L to make decson D. For the duraton of ths paper, we wll refer to a node Mod[β,D] asa Mod node when agent β and decson D are clear from context. A Mod node s a chance node just lke any other; t may nfluence, or be nfluenced by other nodes of K. It s requred to be a parent of the decson D but t s not an nformatonal parent of the decson. Ths s because an agent s strategy for D does not specfy what to do for each value of the Mod node. Every decson D wll have a Mod[β,D] node for each agent that makes a decson n block K, ncludng agent α tself that owns the decson. If the CPD of Mod[β,D] assgns postve probablty to some block L, then we requre that D exsts n block L ether as a decson node or as a chance node. If D s a chance node n L, ths means that β beleves that agent α s playng lke an automaton n L, usng a fxed, possbly mxed strategy for D; fd s a decson node n L, ths means that β beleves α s analyzng block L to determne the course of acton for D. For presentaton purposes, we also add an edge K L to the NID, labeled {β,d}. (a) Top-level Block (b) block S (c) block P (d) Baseball NID Fgure 3: Baseball Scenaro (Example 2.1) We can represent Example 2.4 n the NID descrbed n Fgure 3. There are three blocks n ths NID. The Top-level block, shown n Fgure 3a, corresponds to an nteracton between Alce and Bob n whch they are free to choose whether to steal base or call a ptch out, respectvely. Ths block s dentcal to the MAID of Fgure 2, except that each decson node ncludes the Mod nodes for all of the agents. Block S, presented n Fgure 3b, corresponds to a stuaton where Alce follows the expert recommendaton and nstructs her player to steal. 117

10 Gal & Pfeffer Mod[Bob, Steal] Top-level S Mod[Alce, PtchOut] Top-level P Mod[Bob, PtchOut] Top-level 1 Mod[Alce, Steal] Top-level 1 (a) node Mod[Bob, Steal] (b) node Mod[Alce, PtchOut] (c) node Mod[Bob, PtchOut] (d) node Mod[Alce, Steal] Table 3: CPDs for Top-level block of NID for Baseball Scenaro (Example 2.1) In ths block, the Steal decson s replaced wth a chance node, whch assgns probablty 1totrue for any value of the nformatonal parent Leader. Smlarly, block P, presented n Fgure 3c, corresponds to a stuaton where Bob nstructs hs team to ptch out. In ths block, the PtchOut decson s replaced wth a chance node, whch assgns probablty 1 to true for any value of the nformatonal parent Leader. The root of the NID s the Top-level block, whch n ths example corresponds to realty. The Mod nodes n the Top-level block capture agents belefs over ther decson-makng processes. The node Mod[Bob, Steal] represents Bob s belef about whch block Alce s usng to make her decson Steal. Its CPD assgns probablty 0.3 to the Top-level block, and 0.7 to block S. Smlarly, the node Mod[Alce, PtchOut] represents Alce s belefs about whch block Bob s usng to make the decson PtchOut. Its CPD assgns probablty 0.3 to the Top-level block, and 0.7 to block P. These are shown n Table 3. An mportant aspect of NIDs s that they allow agents to express uncertanty about the block they themselves are usng to make ther own decsons. The node Mod[Alce, Steal] n the Top-level block represents Alce s belefs about whch block Alce herself s usng to make her decson Steal. In our example, the CPD of ths node assgns probablty 1 to the Top-level. Smlarly, the node Mod[Bob, PtchOut] represents Bob s belefs about whch block he s usng to make hs decson PtchOut, and assgns probablty 1 to the Top-level block. Thus, n ths example, both Bob and Alce are uncertan about whch block the other agent s usng to make a decson, but not about whch block they themselves are usng. However, we could also envson a stuaton n whch an agent s unsure about ts own decson-makng. We say that f Mod[β,D] atblockk equals some block L K, and β owns decson D, then agent β s modelng tself as usng block L to make decson D. In Secton 3.2 we wll show how ths allows to capture nterestng forms of bounded ratonal behavor. We do mpose the requrement that there exsts no cycle n whch each edge ncludes a label {α, D}. In other words, there s no cycle n whch the same agent s modelng tself at each edge. Such a cycle s called a self-loop. Ths s because the MAID representaton for a NID wth a self-loop wll nclude a cycle between the nodes representng the agent s belefs about tself at each block of the NID. In future examples, we wll use the followng conventon: If there exsts a Mod[α, D]node at block K (regardless of whether α owns the decson) and the CPD of Mod[α, D] assgns probablty 1 to block K, wewllomtthenodemod[α, D] from the block descrpton. In the Top-level block of Fgure 3a, ths means that both nodes Mod[Alce, Steal] and Mod[Bob, PtchOut], currently appearng as dashed ovals, wll be omtted. 118

11 Networks of Influence Dagrams 3. NID Semantcs In ths secton we provde semantcs for NIDs n terms of MAIDs. We frst show how a NID can be converted to a MAID. We then defne a NID equlbrum n terms of a Nash equlbrum of the constructed MAID. 3.1 Converson to MAIDs The followng process converts each block K n the NID to a MAID fragment O K,andthen connects them to form a MAID representaton of the NID. The key construct n ths process s the use of a chance node Dα K n the MAID to represent the belefs of agent α regardng the acton that s chosen for decson D at block K. ThevalueofD α depends on the block used by α to model decson D, as determned by the value of the Mod[α, D] node. 1. For each block K nthenid,wecreateamaido K. Any chance or utlty node N n block K that s a descendant of a decson node n K s replcated n O K,oncefor each agent α, and denoted N K α.ifn s not a descendant of a decson node n K, t s coped to O K and denoted N K. In ths case, we set N K α = N K for any agent α. 2. If P s a parent of N n K, thenp K α wll be made a parent of N K α n O K.TheCPD of N K α n O K wll be equal to the CPD of N n K. 3. For each decson D n K, we create a decson node BR[D] K n O K, representng the optmal acton for α for ths decson. If N s a chance or decson node whch s an nformatonal parent of D n K, andd belongs to agent α, thennα K wll be made an nformatonal parent of BR[D] K n O K. 4. We create a chance node D K α n O K for each agent α. We make Mod[α, D] K a parent of D K α. If decson D belongs to agent α, thenwemakebr[d] K a parent of D K α. If decson D belongs to agent β α, then we make D K β a parent of DK α. 5. We assemble all the MAID fragments O K nto a sngle MAID O as follows: We add an edge D L α D K β where L K f L s assgned postve probablty by Mod[β,D]K, and α owns decson D. Note that β may be any agent, ncludng α tself. 6. We set the CPD of Dα K to be a multplexer. If α owns D then the CPD of Dα K assgns probablty 1 to BR[D] K when Mod[α, D] K equals K, and assgns probablty 1 to Dα L when Mod[α, D] K equals L K. Ifβ α owns D then the CPD of Dα K assgns probablty 1 to Dβ K when Mod[α, D]K equals K, and assgns probablty 1 to Dβ L when Mod[α, D] K equals L K. To explan, Step 1 of ths process creates a MAID fragment O K for each NID block. All nodes that are ancestors of decson nodes representng events that occur pror to the decsons are coped to O K. However, events that occur after decsons are taken may depend on the actons for those decsons. Every agent n the NID may have ts own belefs about these actons and the events that follow them, regardless of whether that agent owns the decson. Therefore, all of the descendant nodes of decsons are duplcated for each agent n O K. Step 2 ensures that f any two nodes are connected n the orgnal block K, then 119

12 Gal & Pfeffer the nodes representng agents belefs n O K are also connected. Step 3 creates a decson node n O K for each decson node n block K belongng to agent α. The nformatonal parents for the decson n O K are those nodes that represent the belefs of α about ts nformatonal parents n K. Step 4 creates a separate chance node n O K for each agent α that represents ts belef about each of the decsons n K. Ifα owns the decson, ths node depends on the decson node belongng to α. Otherwse, ths node depends on the belefs of α regardng the acton of agent β that owns the decson. In the case that α models β as usng a dfferent block to make the decsons, Step 5 connects between the MAID fragments of each block. Step 6 determnes the CPDs for the nodes representng agents belefs about each other s decsons. The CPD ensures that the block that s used to model a decson s determned by the value of the Mod node. The MAID that s obtaned as a result of ths process s a complete descrpton of agents belefs over each other s decsons. We demonstrate ths process by convertng the NID of Example 2.4 to ts MAID representaton, shown n Fgure 4. Frst, MAID fragments for the three blocks Top-level, P, and S are created. The node Leader appearng n blocks Top-level, P, ands s not a descendant of any decson. Followng Step 1, t s created once n each of the MAID fragments, gvng the nodes Leader TL, Leader P and Leader S. Smlarly, the node Steal n block S and the node PtchOut n block P are created once n each MAID fragment, gvng the nodes Steal S and PtchOut P. Also n Step 1, the nodes Mod[Alce, Steal] TL,Mod[Bob, Steal] TL, Mod[Alce, PtchOut] TL and Mod[Bob, PtchOut] TL are added to the MAID fragment for the Top-level block. Step 3 adds the decson nodes BR TL [Steal] andbr TL [PtchOut] to the MAID fragment for the Top-level block. Step 4 adds the chance nodes PtchOut TL Bob, PtchOutTL Alce, StealTL Alce and Steal TL Bob to the MAID fragment for the Top-level block. These nodes represent agents belefs n ths block about ther own decsons or the decsons of other agents. For example, PtchOut TL Bob represents Bob s belefs about ts decson whether to ptch out, whle PtchOut TL Alce represents Alce s belefs about Bob s belefs about ths decson. Also followng Step 4, edges BR TL [PtchOut] PtchOut TL Bob and StealTL Alce StealTL Bob are added to the MAID fragment for the Top-level block. These represent Bob s belefs over ts own decson at the block. An edge Steal TL Alce StealTL Bob s added to the MAID fragment to represent Bob s belefs over Alce s decson at the Top-level block. There are also nodes representng Alce s belefs about her and Bob s decsons n ths block. In Step 5, edges Steal S Steal TL Bob and PtchOutP Ptchout TL Alce are added to the MAID fragment for the Top-level block. Ths s to allow Bob to reason about Alce s decson n block S, and for Alce to reason about Bob s decson n block P. Ths acton unfes the MAID fragments nto a sngle MAID. The parents of Steal TL Bob are Mod[Bob, Steal]TL, Steal S and Steal TL Alce. Its CPD s a multplexer node that determnes Bob s predcton about Alce s acton: If Mod[Bob, Steal] TL equals S, then Bob beleves Alce to be usng block S, nwhch her acton s to follow the experts and play strategy Steal S. If Mod[Bob, Steal] TL equals the Top-level block, then Bob beleves Alce to be usng the Top-level block, n whch Alce s acton s to respond to her belefs about Bob. The stuaton s smlar for Alce s decson Steal TL Alce and the node Mod[Alce, Steal]TL wth the followng excepton: When Mod[Alce, Steal] TL equals the Top-level block, then Alce s acton follows her decson node BR TL [Steal]. In the Appendx, we prove the followng theorem. 120

13 Networks of Influence Dagrams Theorem 3.1. Convertng a NID nto a MAID wll not ntroduce a cycle n the resultng MAID. Fgure 4: MAID representaton for the NID of Example 2.4 As ths converson process mples, NIDs and MAIDs are equvalent n ther expressve power. However, NIDs provde several advantages over MAIDs. A NID block structure makes explct agents dfferent belefs about decsons, chance varables and utltes n the world. It s a mental model of the way agents reason about decsons n the block. MAIDs do not dstngush between the real world and agents mental models of the world or of each other, whereas NIDs have a separate block for each mental model. Further, n the MAID, nodes smply represent chance, decson or utltes, and are not nherently nterpreted n terms of belefs. A Dα K node n a MAID representaton for a NID does not nherently represent agent α s belefs about how decson D s made n mental model K, andthe Mod K for agent α does not nherently represent whch mental model s used to make a decson. Indeed, there are no mental models defned n a MAID. In addton, there s no relatonshp n a MAID between descendants of decsons Nα K and Nβ K, so there s no sense n whch they represent the possbly dfferent belefs of agents α and β about N. 121

14 Gal & Pfeffer Together wth the NID constructon process descrbed above, a NID s a blueprnt for constructng a MAID that descrbes agents mental models. Wthout the NID, ths process becomes nherently dffcult. Furthermore, the constructed MAID may be large and unweldy compared to a NID block. Even for the smple NID of Example 2.4, the MAID of Fgure 4 s complcated and hard to understand. 3.2 Equlbrum Condtons In Secton 2.1, we defned pure and mxed strateges for decsons n MAIDs. In NIDs, we assocate the strateges for decsons wth the blocks n whch they appear. A pure strategy for a decson D n a NID block K s a mappng from the nformatonal parents of D to an acton n the doman of D. Smlarly, a mxed strategy for D s a mappng from the nformatonal parents of D to a dstrbuton over the doman of D. A strategy profle for a NID s a set of strateges for all decsons at all blocks n the NID. Tradtonally, an equlbrum for a game s defned n terms of best response strateges. A Nash equlbrum s a strategy profle n whch each agent s dong the best t possbly can, gven the strateges of the other agents. Classcal game theory predcts that all agents wll play a best response. NIDs, on the other hand, allow us to descrbe stuatons n whch an agent devates from ts best response by playng accordng to some other decson-makng process. We would therefore lke an equlbrum to specfy not only what the agents should do, but also to predct what they actually do, whch may be dfferent. A NID equlbrum ncludes two types of strateges. The frst, called a best response strategy, descrbes what the agents should do, gven ther belefs about the decson-makng processes of other agents. The second, called an actually played strategy, descrbes what agents wll actually do accordng to the model descrbed by the NID. These two strateges are mutually dependent. The best response strategy for a decson n a block takes nto account the agent s belefs about the actually played strateges of all the other decsons. The actually played strategy for a decson n a block s a mxture of the best response for the decson n the block, and the actually played strateges for the decson n other blocks. Defnton 3.2. Let N be a NID and let M be the MAID representaton for N. Letσ be an equlbrum for M. LetD be a node belongng to agent α n block K of N. Let the parents of D be Pa. By the constructon of the MAID representaton detaled n Secton 3.1, the parents of BR[D] K n M are Pa K α and the domans of Pa and Pa K α are the same. Let σ BR[D] K (pa) denote the mxed strategy assgned by σ for BR[D] K when Pa K α equals pa. The best response strategy for D n K, denoted θd K (pa), defnes a functon from values of Pa to dstrbutons over D that satsfy θ K D (pa) σ BR[D] K (pa) In other words, the best response strategy s the same as the MAID equlbrum when the correspondng parents take on the same values. Defnton 3.3. Let P σ denote the dstrbuton that s defned by the Bayesan network that mplements σ. Theactually played strategy for decson D n K that s owned by agent α, denoted φ K D (pa), specfes a functon from values of Pa to dstrbutons over D that satsfy φ K D(pa) P σ (Dα K pa) 122

15 Networks of Influence Dagrams Note here, that D K α s condtoned on the nformatonal parents of decson D rather than ts own parents. Ths node represents the belefs of α about decson K. Therefore, the actually played strategy for D n K represents α s belef about D n K, gven the nformatonal parents of D. Defnton 3.4. Let σ be a MAID equlbrum. The NID equlbrum correspondng to σ conssts of two strategy profles θ and φ, such that for every decson D n every block K, θd K s the best response strategy for D n K, andφk D s the actually played strategy for D n K. For example, consder the constructed MAID for our baseball example n Fgure 4. The best response strateges n the NID equlbrum specfy strateges for the nodes Steal and PtchOut n the Top-level block that belong to Alce and Bob respectvely. For an equlbrum σ of the MAID, the best response strategy for Steal n the Top-level block s the strategy specfed by σ for BR TL [Steal]. Smlarly, the best response strategy for Ptchout n the Top-level block s the strategy specfed by σ for BR TL [Ptchout]. The actually played strategy for Steal n the Top-level s equal to the condtonal probablty dstrbuton over Steal TL Alce gven the nformatonal parent LeaderTL. Smlarly, the actually played strategy for Ptchout s equal to the condtonal probablty dstrbuton over Ptchout TL Bob gven the nformatonal parent Leader TL. Solvng ths MAID yelds the followng unque equlbrum: In the NID Top-level block, the CPD for nodes Mod[Alce, Steal] and Mod[Bob, Ptchout] assgns probablty 1 to the Top-level block, so the actually played and best response strateges for Bob and Alce are equal and specfed as follows: If Alce s leadng, then Alce steals base wth probablty 0.56 and Bob ptches out wth probablty If Bob s leadng, then Alce never steals base and Bob never ptches out. It turns out that because the experts may nstruct Bob to call a ptch out, Alce s consderably less lkely to steal base, as compared to her equlbrum strategy for the MAID of Example 2.1, where none of the managers consdered the possblty that the other was beng advsed by experts. The case s smlar for Bob. A natural consequence of ths defnton s that the problem of computng NID equlbra reduces to that of computng MAID equlbra. Solvng the NID requres to convert t to ts MAID representaton and solvng the MAID usng exact or approxmate soluton algorthms. The sze of the MAID s bounded by the sze of a block tmes the number of blocks tmes the number of agents. The structure of the NID can then be exploted by a MAID soluton algorthm (Koller & Mlch, 2001; Vckrey & Koller, 2002; Koller et al., 1996; Blum et al., 2006). 4. Examples In ths secton, we provde a seres of examples demonstratng the benefts of NIDs for descrbng and representng uncertanty over decson-makng processes n a wde varety of domans. 4.1 Irratonal Agents Snce the challenge to the noton of perfect ratonalty as the foundaton of economc systems presented by Smon (1955), the theory of bounded ratonalty has grown n dfferent 123

16 Gal & Pfeffer drectons. From an economc pont of vew, bounded ratonalty dctates a complete devaton from the utlty maxmzng paradgm, n whch concepts such as optmzaton and objectve functons are replaced wth satsfcng and heurstcs (Ggerenzer & Selten, 2001). These concepts have recently been formalzed by Rubnsten (1998). From a tradtonal AI perspectve, an agent exhbts bounded ratonalty f ts program s a soluton to the constraned optmzaton problem brought about by lmtatons of archtecture or computatonal resources (Russell & Wefald, 1991). NIDs serve to complement these two prevalng perspectves by allowng to control the extent to whch agents are behavng rratonally wth respect to ther model. Irratonalty s captured n our framework by the dstncton between best response and actually played strateges. Ratonal agents always play a best response wth respect to ther models. For ratonal agents, there s no dstncton between the normatve behavor prescrbed for each agent n each NID block, and the descrptve predcton of how the agent actually would play when usng that block. In ths case, the best response and actually played strateges of the agents are equal. However, n open systems, or when people are nvolved, we may need to model agents whose behavor dffers from ther best response strategy. In other words, ther best response strateges and actually played strateges are dfferent. We can capture agent α behavng (partally) rratonally about ts decson D α n block K by settng the CPD of Mod[α, D α ] to assgn postve probablty to some block L K. There s a natural way to express ths dstncton n NIDs through the use of the Mod node. If D α s a decson assocated wth agent α, we can use Mod[α, D α ] to descrbe whch block α actually uses to make the decson D α.inblockk, fmod[α, D α ]sequaltok wth probablty 1, then t means that wthn K, α s makng the decson accordng to ts belefs n block K, meanng that α wll be ratonal; t wll play a best response to the strateges of other agents, gven ts belefs. If, however, Mod[α, D α ] assgns postve probablty to some block L other than K, t means that there s some probablty that α wll not play a best response to ts belefs n K, but rather play a strategy accordng to some other block L. In ths case, we say α self-models at block K. The ntroducton of actually played strateges nto the equlbrum defnton represents another advantage of NIDs over MAIDs, n that they explctly represent strateges for agents that may devate from ther optmal strateges. In some cases, makng a decson may lead an agent to behave rratonally by vewng the future n a consderably more postve lght than s objectvely lkely. For example, a person undergong treatment for a dsease may beleve that the treatment stands a better chance of success than scentfcally plausble. In the psychologcal lterature, ths effect s referred to as motvatonal bas or postve lluson (Bazerman, 2001). As the followng example shows, NIDs can represent agents motvatonal bases n a compellng way, by makng Mod nodes depend on the outcome of decson nodes. Example 4.1. Consder the case of a toothpaste company whose executves are faced wth two sequental decsons: whether to place an advertsement n a magazne for ther leadng brand, and whether to ncrease producton of the brand. Based on past analyss, the executves know that wthout advertsng, the probablty of hgh sales for the brand n the next quarter wll be 0.5. Placng the advertsement costs money, but the probablty of hgh sales wll rse to 0.7. Increasng producton of the brand wll contrbute to proft 124

17 Networks of Influence Dagrams f sales are hgh, but wll hurt proft f sales are low due to the hgh cost of storage space. Suppose now that the company executves wsh to consder the possblty of motvatonal bas, n whch placng the advertsement wll nflate ther belefs about sales to be hgh n the next quarter to probablty 0.9. Ths may lead the company to ncrease the producton of the brand when t s not warranted by the market and consequently, suffer losses. The company executves wsh to compute ther best possble strategy for ther two decsons gven the fact that they attrbute a motvatonal bas. A NID descrbng ths stuaton s shown n Fgure 5c. The Top-level block n Fgure 5a shows the stuaton from the pont of vew of realty. It ncludes two decsons, whether to advertse (Advertse) and whether to ncrease the supply of the brand (Increase). The node Sales represents the amount of sales for the brand after the decson of whether to advertse, and the node Proft represents the proft for the company, whch depends on the nodes Advertse, Increase and Sales. The CPD of Sales n the Top-level block assgns probablty 0.7 to hgh f Advertse s true and 0.5 to hgh f Advertse s false, as descrbed n Table 4a. The utlty values for node Proft are shown n Table 4.1. For example, when the company advertses the toothpaste, ncreases ts supply, and sales are hgh, t receves a reward of 70; when the company advertses the toothpaste, does not ncrease ts supply, and sales are low, t receves a reward of 40. Block Bas, descrbed n Fgure 5b, represents the company s based model. Here, the decson to advertse s replaced by an automaton chance node that assgns probablty 1 to Advertse = true. The CPD of Sales n block Bas assgns probablty 0.9 to hgh f Advertse s true and 0.5 to hgh f Advertse s false,as descrbed n Table 4b. In the Top-level block, we have the followng: 1. ThenodeMod[Company, Advertse] assgns probablty 1 to the Top-level block. 2. The decson node Advertse s a parent of the node Mod[Company, Increase]. 3. The node Mod[Company, Increase] assgns probablty 1 to block Bas when Advertse s true, and assgns probablty 0 to block Bas when Advertse s false. Intutvely, Step 1 captures the company s belefs that t s not based before t makes the decson to advertse. Step 2 allows the company s uncertanty about whether t s based to depend on the decson to advertse. Note that ths example shows when t s necessary for a decson node to depend on an agent s belefs about a past decson. Step 3 captures the company s belefs that t may use block Bas to make ts decson whether to ncrease supply, n whch t s over confdent about hgh sales. Solvng ths NID results n the followng unque equlbrum: In block Bas, thecompany s actually played and best response strategy s to ncrease supply, because ths s ts optmal acton when t advertses and sales are hgh. In block Top-level, we have the followng: If the company chooses not to advertse, t wll behave ratonally, and ts best response and actually played strategy wll be not to ncrease supply; f the company chooses to advertse, ts actually played strategy wll be to use block Bas n whch t ncreases supply, and ts best response strategy wll be not to ncrease supply. Now, the expected utlty for the company n the Top-level block s hgher when t chooses not to advertse. Therefore, ts best response strateges for both decsons are not to advertse nor to ncrease supply. Interestngly, f the company was never based, t can be shown usng backwards nducton 125

18 Gal & Pfeffer that ts optmal acton for the frst decson s to advertse. Thus, by reasonng about ts own possble rratonal behavor for the second decson, the company revsed ts strategy for the frst decson. (a) Block Top-level (b) Block Bas (c) NID Fgure 5: Motvatonal Bas Scenaro (Example 4.1) Sales Advertse low hgh true false (a) node Sales (Top-level Block) Sales Advertse low hgh true false (b) node Sales (Bas Block) Table 4: CPDs for Top-level block of Motvatonal Bas NID (Example 4.1) Example 4.2. Consder the followng extenson to Example 2.4. Suppose that there are now two successve ptches, and on each ptch the managers have an opton to steal or ptch out. If Bob ptches out on the frst ptch, hs utlty for ptchng out on the second ptch (regardless of Alce s acton) decreases by 20 unts because he has forfeted two ptches. Bob beleves that wth probablty 0.3, he wll succumb to socal pressure durng the second ptch and call a ptch out. Bob would lke to reason about ths possblty when makng the decson for the frst ptch. 126

19 Networks of Influence Dagrams Advertse Increase Sales Proft true true hgh 70 true true low 70 true false hgh 50 true false low 40 false true hgh 80 false true low 60 false false hgh 60 false false low 30 Table 5: Company s utlty (node Proft) fortop-level block of Motvatonal Bas NID (Example 4.1) In ths example, each manager s faced wth a sequental decson problem: whether to steal or ptch out n the frst and second ptch. The strategy for the second ptch s relevant to the strategy for the frst ptch for each agent. Now, each of the managers, f they were ratonal, could use backward nducton to compute optmal strateges for the frst ptch, by workng backwards from the second ptch. However, ths s only a vald procedure f the managers behave ratonally on the second ptch. In the example above, Bob knows that he wll be under strong pressure to ptch out on the second ptch and he wshes to take ths possblty nto account, whle makng hs decson for the frst ptch. Mod[Bob, PtchOut 2 ] Top-level 0.7 L 0.3 Table 6: CPD for Mod[Bob, PtchOut 2 ]nodentop-level block of Irratonal Agent Scenaro (Example 4.2) We can model ths stuaton n a NID as follows. The Top-level block of the NID s shown n Fgure 6a. Here, the decson nodes Steal 1 and PtchOut 1 represent the decsons for Alce and Bob n the frst ptch, and the nodes Steal 2 and Ptchout 2 represent the decsons for Alce and Bob n the second ptch. The nodes Leader, Steal 1, PtchOut 1 and ThrownOut 1 are all nformatonal parents of the decson nodes Steal 2 and PtchOut 2. For expostory convenence, we have not ncluded the edges leadng from node Leader to the utlty nodes n the block. Block L, shown n Fgure 6b, descrbes a model for the second ptch n whch Bob s succumbng to socal pressure and ptches out, regardless of who s leadng. Ths s represented by havng the block nclude a chance node PtchOut 2 whch equals true wth probablty 1 for each value of Leader. The node Mod[Bob, PtchOut 2 ] wll assgn probablty 0.3 to block L, and 0.7 probablty to the Top-level block, as shown n Table 4.1. The node Mod[Bob, PtchOut 2 ]snotdsplayednthetop-level block. By our conventon, ths mples that ts CPD assgns probablty 1 to the Top-level block, n whch Bob s reasonng about the possblty of behavng rratonally wth respect to the second ptch. In ths way, we have captured the fact that Bob may behave rratonally wth respect to the second ptch, and that he s reasonng about ths possblty when makng the decson for the frst ptch. 127