Simple Decision Making

Decision Making Under Uncertainty Ch 16 Simple Decision Making Many environments have multiple possible outcomes Some of these outcomes may be good; others may be bad Some may be very likely; others unlikely What s a poor agent s choice?? Non-Deterministic vs. Probabilistic Uncertainty Expected Utility? a b c {a,b,c} decision that is best for worst case Non-deterministic model ~ Adversarial search? a b c {a(pa),b(pb),c(pc)} decision that maximizes expected utility value Probabilistic model Random variable X with n values x 1,,x n and distribution (p 1,,p n ) E.g.: X is the state reached after doing an action A under uncertainty Function U of X E.g., U is the utility of a state The expected utility of A is EU[A] = i=1,,n p(x i A)U(x i ) One State/One Action Example A1 s0 U(S0) = 100 x 0.2 + 50 x 0.7 + 70 x 0.1 = 20 + 35 + 7 = 62 One State/Two Actions Example A1 s0 U1(S0) = 62 U2(S0) = 74 U(S0) = max{u1(s0),u2(s0)} = 74 A2 s1 s2 s3 0.2 0.7 0.1 100 50 70 s1 s2 s3 s4 0.2 0.7 0.2 0.1 0.8 100 50 70 80 1

Introducing Action Costs A1 s0 A2-5 -25 s1 s2 s3 s4 0.2 0.7 0.2 0.1 0.8 100 50 70 80 U1(S0) = 62 5 = 57 U2(S0) = 74 25 = 49 U(S0) = max{u1(s0),u2(s0)} = 57 Maximum Expected Utility (MEU) Principle A rational agent should choose the action that maximizes agent s expected utility This is the basis of the field of utility theory The MEU principle provides a normative criterion for rational choice of action Basic of Utility Theory In the language of utility theory, the complex scenarios are called lotteries to emphasize the idea that the different attainable outcomes are like different prizes and that the outcome is determined by chance Suppose a lottery L has two possible outcomes: 1. State A with probability p 2. State B with probability 1-p We write L = [p,a; 1-p,B]. Axioms of Utility Theory A>B means A is preferred over B A~B means no preference between A and B Orderability (A>B) (A<B) (A~B) Transitivity (A>B) (B>C) (A>C) Continuity A>B>C p [p,a; 1-p,C] ~ B Substitutability A~B [p,a; 1-p,C]~[p,B; 1-p,C] Monotonicity A>B (p q ([p,a; 1-p,B] >~ [q,a; 1-q,B])) Decomposability [p,a; 1-p, [q,b; 1-q, C]] ~ [p,a; (1-p)q, B; (1-p)(1-q), C] Utility functions Utility functions map states to real numbers. Utility theory has its roots in economics -> the utility of money Risk averse Risk seeking Certainty equivalent Risk neutral Utility scales and utility assessment Normalization Some Examples A or B? A: 80% chance of $4000 B: 100% chance of $3000 C or D? C: 20% chance of $4000 D: 25% chance of $3000 E or F? E: 1% chance of $1,000,000 F: 3% chance of $400,000 11 12 2

Some Examples A or B? A: 80% chance of $4000 B: 100% chance of $3000 C or D? C: 20% chance of $4000 D: 25% chance of $3000 E or F? E: 1% chance of $1,000,000 F: 2% chance of $550,000 Risk averse Choose B Risk neutral Choose C Risk seeking Choose E 13 Decision Networks A Visual Representation of Choices, Consequences, Probabilities, and Opportunities. Extend BNs to handle actions and utilities Also called influence diagrams Use BN inference methods to decide probabilities. Perform Value of Information calculations Decision Trees are populara Way of Breaking Down Complicated Situations Down to Easierto-Understand Scenarios. Decision Networks cont. Chance nodes: random variables, as in BNs Decision nodes: actions that an agent can take (as in decision trees) Utility/value nodes: the utility of the outcome state. Example of a Decision Tree Decision node Chance node Event 1 Event 2 Event 3 Umbrella Network Evaluating Decision Networks take/don t take umbrella P(have take) = 1.0 P(~have ~take)=1.0 have umbrella happiness U(have,rain) = -25 U(have,~rain) = 0 U(~have, rain) = -100 U(~have, ~rain) = 100 P(rain) = 0.4 weather forecast f w p(f w) sunny rain 0.3 rainy rain 0.7 sunny no rain 0.8 rainy no rain 0.2 Set the evidence variables for current state For each possible value of the decision node: Set decision node to that value Calculate the posterior probability of the parent nodes of the utility node, using BN inference Calculate the resulting utility for action Return the action with the highest utility 3

Decision Making: Umbrella Network Another Easy Example take/don t take umbrella P(have take) = 1.0 P(~have ~take)=1.0 Should I take my umbrella?? have umbrella happiness U(have,rain) = -25 U(have,~rain) = 0 U(~have, rain) = -100 U(~have, ~rain) = 100 P(rain) = 0.4 weather forecast f w p(f w) sunny rain 0.3 rainy rain 0.7 sunny no rain 0.8 rainy no rain 0.2 A Decision Tree with two choices. Go to Graduate School to get my master in CS. Go to work in the Real World Easy Example - Revisited What are some of the costs we should take into account when deciding whether or not to go to graduate school? Tuition and Fees Rent / Food / etc. Opportunity cost of salary Anticipated future earnings Simple Decision Tree Model Go to Graduate School to get my master in CS. Go to Work in the Real World Is this a realistic model? What is missing? 1.5 Years of tuition: $30,000, 1.5 years of Room/Board: $20,000; 1.5 years of Opportunity Cost of Salary = $100,000 Total = $150,000. PLUS Anticipated 5 year salary after Graduate School = $600,000. U(graduate school) = $600,000 - $175,000 = $425,000 First 1.5 year salary = $100,000 (from above), minus expenses of $20,000. Next five year salary = $330,000 U(no grad-school) = $410,000 Go to Graduate School The Yeaple Study (1994) Things he may have missed According to Ronald Yeaple, it is only profitable to go to one of the top 15 Business Schools otherwise you have a NEGATIVE NPV (net present value)! (Economist, Aug. 6, 1994) Benefits of Learning School Net Value ($) Harvard $148,378 Chicago $106,378 Stanford $97,462 MIT (Sloan) $85,736 Yale $83,775 Northwestern $53,526 Berkeley $54,101 Wharton $59,486 UCLA $55,088 Virginia $30,046 Cornell $30,974 Michigan $21,502 Dartmouth $22,509 Carnegie Mellon $18,679 Texas $17,459 Rochester - $307 Indiana - $3,315 North Carolina - $4,565 Duke - $17,631 NYU - $3,749 Future uncertainty (interest rates, future salary, etc) Cost of living differences Type of Job [utility function = f($, enjoyment)] Girlfriend / Boyfriend / Family concerns Others? Utility Function = f ($, enjoyment, family, location, type of job, prestige, gender, age, race) Many Human Factors Considerations 4

Example Joe s Garage Joe s garage is considering hiring another mechanic. The mechanic would cost them an additional $50,000 / year in salary and benefits. If there are a lot of accidents in Iowa City this year, they anticipate making an additional $100,000 in net revenue. If there are not a lot of accidents, they could lose $20,000 off of last year s total net revenues. Because of all the ice on the roads, Joe thinks that there will be a 70% chance of a lot of accidents and a 30% chance of fewer accidents. Assume if he doesn t expand he will have the same revenue as last year. Draw a decision tree for Joe and tell him what he should do. Joe s Decision - How to Decide? What would be his decision based on: Maximax? Maximin? Expected Return? Quick primer on Statistics and Probability Example - Answer Definitions: Expected Value of x: E(x) = xp ( x) ; as P(x) represents the probability of x. x (Note that P ( x) = 1 and that the xp( x) E( x) because P(x) represents x a probability density function) 2 2 Variance of x: X E[( x X) ] Standard Deviation = the sq. root of the variance Hire new mechanic Cost = $50,000 Don t hire new mechanic Cost = $0 70% chance of an increase in accidents Profit = $90,000 30% chance of a decrease in accidents Profit = - $20,000 Median = the center of the set of numbers ; or the point m such that P(x < m)< ½ and P(x > m)> ½. Estimated utility value of Hire Mechanic : EU =.7($90,000) +.3(- $20,000) - $50,000 = $7,000 Therefore Joe should hire the mechanic using Maximax or Expected Return; and not hire using Maximin. Problem: Jenny Lind Jenny Lind is a writer of romance novels. A movie company and a TV network both want exclusive rights to one of her more popular works. If she signs with the network, she will receive a single lump sum, but if she signs with the movie company, the amount she will receive depends on the market response to her movie. What should she do? Payouts and Probabilities Movie company Payouts Small box office: $200,000 Medium box office: $1,000,000 Large box office: $3,000,000 TV Network Payout Flat rate: Probabilities P() = 0.3 P() = 0.6 P() = 0.1 5

Jenny Lind - Payoff Table Using Expected Return Criteria Decisions Sign with Movie Company Sign with TV Network Prior Probabilities Small Box Office States of Nature Medium Box Office Large Box Office $200,000 $1,000,000 $3,000,000 0.3 0.6 0.1 EU movie = 0.3(200,000)+0.6(1,000,000)+0.1(3,000,000) = $960,000 EU tv = Therefore, using this criteria, Jenny should select the movie contract. EU = Expected Utility Something to Remember Jenny s decision is only going to be made one time, and she will earn either $200,000, $1,000,000 or $3,000,000 if she signs the movie contract, not the calculated expected return of $960,000!! Nevertheless, this amount is useful for decisionmaking, as it will maximize Jenny s expected returns in the long run if she continues to use this approach. Creating and Solving Decision Trees Three types of nodes Decision nodes - represented by squares ( ) Chance nodes - represented by circles (Ο) Terminal nodes - represented by triangles (optional) Solving the tree involves pruning all but the best decisions at decision nodes, and finding expected values of all possible states of nature at chance nodes Create the tree from left to right Solve the tree from right to left Jenny Lind Decision Tree Jenny Lind Decision Tree Sign with Movie Co. $200,000 $1,000,000 Sign with Movie Co. EU?.3.6 $200,000 $1,000,000 $3,000,000 EU?.1 $3,000,000 Sign with TV Network Sign with TV Network EU?.3.6.1 6

Jenny Lind Decision Tree - Solved EU= 960,000 Sign with Movie Co. EU= 960,000.3.6.1 EU=.3 900,000 Sign with TV Network.6.1 $200,000 $1,000,000 $3,000,000 Sam has the opportunity to buy a 1996 Spiffycar for $10,000, and he has a prospect who would be willing to pay $11,000 for the auto if it s in excellent mechanical shape. Sam determines that everything except for the transmission is in excellent shape. If the transmission is bad, it will cost $3000 to fix it. He has a friend who can run a test on the transmission. The test is not always accurate: 30% of the time it judges a good transmission to be bad and 10% of the time it judges a good transmission to be good. Sam knows that 20% of the 1996 Spiffycars have bad transmission. Draw a decision tree for Sam and tell him what he should do. T: Test judges that the transmission is bad. A: Transmission is good. P(T A) =.3 P(T not A) =.9 P(T) = P(T A) P(A) + P(T not A) P(not A) = (.3)(.8) + (.9)(.2) =.42 P(A T) = P(T A) P(A) / P(T) = (.3)(.8)/(.42) =.571429 Tran1: Test says the transmission is bad EU(Tran1) = (.571429)($11000) + (.428571)($8000) = $9722 Tran2: Test says the transmission is good EU(Tans2) = = $10897.42 Suppose Sam is in the same situation as the previous example, except that the test is not free. Rather, it costs $200. So Sam must decide whether to run the test, buy the car without running the car, or keep his $10,000. Draw a decision tree for Sam and tell him what he should do. 7

Mary s Factory EU(Tran1) = $9800 EU(Tran2) = $10,697 EU(Test) = $10320 EU(Tran3) = (.8)$11000 + (.2)$8000 = $10400 Mary is a manager of a gadget factory. Her factory has been quite successful the past three years. She is wondering whether or not it is a good idea to expand her factory this year. The cost to expand her factory is $1.5M. If she does nothing and the economy stays good and people continue to buy lots of gadgets she expects $3M in revenue; while only $1M if the economy is bad. If she expands the factory, she expects to receive $6M if economy is good and $2M if economy is bad. She also assumes that there is a 40% chance of a good economy and a 60% chance of a bad economy. Draw a Decision Tree showing these choices. Decision Tree Example Mary s Factory With Options Expand Factory Cost = $1.5 M Don t Expand Factory Cost = $0 40 % Chance of a Good Economy Profit = $6M 60% Chance Bad Economy Profit = $2M Good Economy (40%) Profit = $3M Bad Economy (60%) Profit = $1M A few days later she was told that if she expands, she can opt to either (a) expand the factory further, which costs 1.5M, but will yield an additional $2M in profit when economy is good but only $1M when economy is bad, (b) abandon the project and sell the equipment she originally bought for $1.3M, or (c) do nothing. EU Expand = (.4(6) +.6(2)) 1.5 = $2.1M EU No Expand =.4(3) +.6(1) = $1.8M $2.1 > 1.8, therefore Mary should expand the factory Draw a decision tree to show these three options for each possible outcome, and compute the NPV for the expansion. Decision Trees with Options Expected Return Value of the Options Good Market Bad Market Expand further yielding $8M (but costing $1.5) Stay at expanded levels yielding $6M Reduce to old levels yielding $3M (but saving $1.3 - sell equipment) Expand further yielding $3M (but costing $1.5) Stay at expanded levels yielding $2M Reduce to old levels yielding $1M (but saving $1.3 in equipment cost) Good Economy Expand further = 8M 1.5M = 6.5M Do nothing = 6M Abandon Project = 3M + 1.3M = 4.3M Bad Economy Expand further = 3M 1.5M = 1.5M Do nothing = 2M Abandon Project = 1M + 1.3M = 2.3M 8

Expected Return of the Project Decision Making: Umbrella Network If the forecast is rainy, should I take my umbrella?? So the EU of Expanding the factory further is: EU Expand = [.4(6.5) +.6(2.3)] - 1.5M = $2.48M Therefore the value of the option is 2.48 (new EU) 2.1 (old EU) = $380,000 You would pay up to this amount to exercise that option. take/don t take umbrella P(have take) = 1.0 P(~have ~take)=1.0 have umbrella happiness U(have,rain) = -25 U(have,~rain) = 0 U(~have, rain) = -100 U(~have, ~rain) = 100 P(rain) = 0.4 weather forecast f w p(f w) sunny rain 0.3 rainy rain 0.7 sunny no rain 0.8 rainy no rain 0.2 9