CS 188: Artificial Intelligence Lecture 7: Utility Theory Pieter Abbeel UC Berkeley Many slides adapted from Dan Klein 1 Maximum Expected Utility Why should we average utilities? Why not minimax? Principle of maximum expected utility: A rational agent should chose the action which maximizes its expected utility, given its knowledge Questions: Where do utilities come from? How do we know such utilities even exist? Why are we taking expectations of utilities (not, e.g. minimax)? What if our behavior can t be described by utilities? 2 1
Utilities Utilities are functions from outcomes (states of the world) to real numbers that describe an agent s preferences Where do utilities come from? In a game, may be simple (+1/-1) Utilities summarize the agent s goals Theorem: any rational preferences can be summarized as a utility function We hard-wire utilities and let behaviors emerge Why don t we let agents pick utilities? Why don t we prescribe behaviors? 3 Utilities: Uncertain Outcomes Getting ice cream Get Double Get Single Oops Whew 4 2
Preferences An agent must have preferences among: Prizes: A, B, etc. Lotteries: situations with uncertain prizes Notation: 5 Rational Preferences We want some constraints on preferences before we call them rational ( A B) ( B C) ( A C) For example: an agent with intransitive preferences can be induced to give away all of its money If B > C, then an agent with C would pay (say) 1 cent to get B If A > B, then an agent with B would pay (say) 1 cent to get A If C > A, then an agent with A would pay (say) 1 cent to get C 6 3
Rational Preferences Preferences of a rational agent must obey constraints. The axioms of rationality: Theorem: Rational preferences imply behavior describable as maximization of expected utility 7 MEU Principle Theorem: [Ramsey, 1931; von Neumann & Morgenstern, 1944] Given any preferences satisfying these constraints, there exists a real-valued function U such that: Maximum expected utility (MEU) principle: Choose the action that maximizes expected utility Note: an agent can be entirely rational (consistent with MEU) without ever representing or manipulating utilities and probabilities E.g., a lookup table for perfect tictactoe, reflex vacuum cleaner 8 4
Utility Scales Normalized utilities: u + = 1.0, u - = 0.0 Micromorts: one-millionth chance of death, useful for paying to reduce product risks, etc. QALYs: quality-adjusted life years, useful for medical decisions involving substantial risk Note: behavior is invariant under positive linear transformation With deterministic prizes only (no lottery choices), only ordinal utility can be determined, i.e., total order on prizes 9 Human Utilities Utilities map states to real numbers. Which numbers? Standard approach to assessment of human utilities: Compare a state A to a standard lottery L p between best possible prize u + with probability p worst possible catastrophe u - with probability 1-p Adjust lottery probability p until A ~ L p Resulting p is a utility in [0,1] 10 5
Money Money does not behave as a utility function, but we can talk about the utility of having money (or being in debt) Given a lottery L = [p, $X; (1-p), $Y] The expected monetary value EMV(L) is p*x + (1-p)*Y U(L) = p*u($x) + (1-p)*U($Y) Typically, U(L) < U( EMV(L) ): why? In this sense, people are risk-averse When deep in debt, we are risk-prone Utility curve: for what probability p am I indifferent between: Some sure outcome x A lottery [p,$m; (1-p),$0], M large 11 Example: Insurance Consider the lottery [0.5,$1000; 0.5,$0] What is its expected monetary value? ($500) What is its certainty equivalent? Monetary value acceptable in lieu of lottery $400 for most people Difference of $100 is the insurance premium There s an insurance industry because people will pay to reduce their risk If everyone were risk-neutral, no insurance needed! 12 6
Example: Human Rationality? Famous example of Allais (1953) A: [0.8,$4k; 0.2,$0] B: [1.0,$3k; 0.0,$0] C: [0.2,$4k; 0.8,$0] D: [0.25,$3k; 0.75,$0] Most people prefer B > A, C > D But if U($0) = 0, then B > A U($3k) > 0.8 U($4k) C > D 0.8 U($4k) > U($3k) 13 7