Nested logit. Michel Bierlaire

Transcription

1 Nested logit Michel Bierlaire Transport and Mobility Laboratory School of Architecture, Civil and Environmental Engineering Ecole Polytechnique Fédérale de Lausanne M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 1 / 54

2 Outline Outline 1 Red bus/blue bus paradox 2 Relaxing the independence assumption 3 The nested logit model 4 Airline itinerary example 5 Derivation 6 Summary M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 2 / 54

3 Red bus/blue bus paradox Simple choice model Mode choice Two alternatives: car and bus. There are red buses and blue buses. Car and bus travel times are equal: T. Only travel time is considered in the utility function. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 3 / 54

4 Red bus/blue bus paradox Red bus/blue bus paradox Model 1 U car = βt +ε car U bus = βt +ε bus Choice probability P(car {car, bus}) = P(bus {car, bus}) = e βt e βt +e βt = 1 2 M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 4 / 54

5 Red bus/blue bus paradox Red bus/blue bus paradox Model 2 U car = βt +ε car U blue bus = βt +ε blue bus U red bus = βt +ε red bus Choice probability e βt P(car {car, blue bus, red bus}) = e βt +e βt +e βt = 1 3 P(car {car, blue bus, red bus}) P(blue bus {car, blue bus, red bus}) = 1 3. P(red bus {car, blue bus, red bus}) M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 5 / 54

6 Red bus/blue bus paradox Red bus/blue bus paradox Conclusion If you paint the buses of a city red and blue, the mode share for public transportation increases from 50% to 66%. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 6 / 54

7 Red bus/blue bus paradox Explaining the paradox Model specification Only travel time appears in the utility function. Other attributes are captured by the error term. Some of them are shared by ε blue bus and ε red bus fare headway comfort convenience etc. Logit model Assumes that ε blue bus and ε red bus are independent. Inappropriate assumption in this case. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 7 / 54

8 Relaxing the independence assumption Outline 1 Red bus/blue bus paradox 2 Relaxing the independence assumption 3 The nested logit model 4 Airline itinerary example 5 Derivation 6 Summary M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 8 / 54

9 Relaxing the independence assumption Capturing the correlation Bus Car 7 7 Blue Red M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 9 / 54

10 Relaxing the independence assumption Capturing the correlation If bus is chosen then where V blue bus = V red bus = βt Choice probability U blue bus = V blue bus +ε blue bus U red bus = V red bus +ε red bus P(blue bus {blue bus, red bus}) = e βt e βt +e βt = 1 2 M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 10 / 54

11 Relaxing the independence assumption Capturing the correlation Bus Car 7 7 Blue Red M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 11 / 54

12 Relaxing the independence assumption Capturing the correlation What about the choice between bus and car? U car = βt +ε car U bus = V bus +ε bus with V bus = V bus (V blue bus,v red bus ) ε bus =? Idea Use a logit model at the higher level. Define V bus as the expected maximum utility of red bus and blue bus M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 12 / 54

13 Relaxing the independence assumption Expected maximum utility Definition For a set of alternative C, define U C max i C U i = max i C (V i +ε i ) and U C V C +ε C For logit E[max U i] = 1 i C µ ln e µv i + γ µ i C M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 13 / 54

14 Relaxing the independence assumption Expected maximum utility For logit U C = V C +ε C V C = 1 µ ln i C e µv i E[ε C ] = γ µ M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 14 / 54

15 Relaxing the independence assumption Expected maximum utility Back to the blue/red buses V bus = 1 µ b ln(e µ bv blue bus +e µ bv red bus ) = 1 µ b ln(e µ bβt +e µ bβt ) = βt + 1 µ b ln2 where µ b is the scale parameter for the logit model associated with the choice between red bus and blue bus M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 15 / 54

16 The nested logit model Outline 1 Red bus/blue bus paradox 2 Relaxing the independence assumption 3 The nested logit model 4 Airline itinerary example 5 Derivation 6 Summary M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 16 / 54

17 The nested logit model Nested Logit Model Probability model: car e µvcar P(car) = e µvcar +e µv bus = e µβt e µβt +e µβt+ µ µ b ln2 = µ µ b Extreme cases If µ = µ b, then P(car) = 1 3 (Model 2, logit with 3 alternatives) If µ b, then µ µ b 0, and P(car) 1 2 (Model 1, logit with 2 alternatives) M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 17 / 54

18 The nested logit model Nested Logit Model Probability model: bus e µv bus P(bus) = e µvcar +e µv bus = e µβt+ µ µ b ln2 e µβt +e µβt+ µ µ b ln2 = µ µ b Extreme cases If µ = µ b, then P(bus) = 2 3 (Model 2) If µ b, then µ µ b 0, then P(bus) 1 2 (Model 1) Utility of bus when µ b lim µ b βt + 1 µ b ln2 = βt M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 18 / 54

19 The nested logit model Nested Logit Model Choice probability P(car) P(bus) µ/µ b M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 19 / 54

20 The nested logit model Nested Logit Model 1 1 µ 2 /µ 2 b 0.8 Correlation Perfect correlation Model 1 Independent Model µ/µ b M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 20 / 54

21 The nested logit model Solving the paradox If µ µ b 0, we have P(car) = 1/2 P(bus) = 1/2 P(red bus bus) = 1/2 P(blue bus bus) = 1/2 P(red bus) = P(red bus bus)p(bus) = 1/4 P(blue bus) = P(blue bus bus)p(bus) = 1/4 M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 21 / 54

22 The nested logit model Nested logit model Comments A group of similar alternatives is called a nest Each alternative belongs to exactly one nest. The model is named Nested Logit The ratio µ/µ b must be estimated from the data 0 < µ/µ b 1 (between models 1 and 2) Going down the tree, µ s must increase, variance must decrease M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 22 / 54

23 Airline itinerary example Outline 1 Red bus/blue bus paradox 2 Relaxing the independence assumption 3 The nested logit model 4 Airline itinerary example 5 Derivation 6 Summary M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 23 / 54

24 Airline itinerary example Airline itinerary case study: Logit model Robust Parameter Coeff. Asympt. number Description estimate std. error t-stat p-value 1 One stop same airline dummy One stop multiple airlines dummy Round trip fare ($100) Elapsed time (0-2 hours) Elapsed time (2-8 hours) Elapsed time ( > 8 hours ) Leg room (inches), if male (non stop) Leg room (inches), if female (non stop) Leg room (inches), if male (one stop) Leg room (inches), if female (one stop) Being early (hours) Being late (hours) More than 2 air trips per year (one stop same airline) More than 2 air trips per year (one stop multiple airlines) Round trip fare / income ($100/$1000) Summary statistics Number of observations = 2544 L(0) = L(c) = L( β) = [L(0) L( β)] = ρ 2 = ρ 2 = M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 24 / 54

25 Airline itinerary example Logit model Non stop One stop same airline One stop multiple airlines M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 25 / 54

26 Airline itinerary example Nested logit model Non stop One stop Non stop One stop same airline One stop multiple airlines M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 26 / 54

27 Airline itinerary example Nested logit Marginal and conditional probabilities Pr(NS) = Pr(NS Non stop) Pr(Non stop {Non stop, One stop}), Pr(SAME) = Pr(SAME One stop) Pr(One stop {Non stop, One stop}), Pr(MULT) = Pr(MULT One stop) Pr(One stop {Non stop, One stop}). Note Pr(NS Non stop) = 1 M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 27 / 54

28 Airline itinerary example Nest one stop One stop One stop same airline One stop multiple airlines M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 28 / 54

29 Airline itinerary example Nest one stop Parameter number Description 1 One stop same airline dummy 2 One stop multiple airlines dummy normalized to 0 3 Round trip fare ($100) 4 Elapsed time (0-2 hours) no data to estimate 5 Elapsed time (2-8 hours) 6 Elapsed time ( > 8 hours ) 7 Leg room (inches), if male (non stop) alt. not in the model 8 Leg room (inches), if female (non stop) alt. not in the model 9 Leg room (inches), if male (one stop) 10 Leg room (inches), if female (one stop) 11 Being early (hours) 12 Being late (hours) 13 More than 2 air trips per year (one stop same airline) 14 More than 2 air trips per year (one stop multiple airlines) normalized to 0 15 Round trip fare / income ($100/$1000) M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 29 / 54

30 Airline itinerary example Nest one stop Binary choice SAME: One stop same airline MULT: One stop multiple airlines Specification of the utility functions Same as logit model. Up to normalization. MULT constant normalized to zero More than two air trips per year (MULT) normalized to 0 Elapsed time (0 2 hours) cannot be identified due to absence of data. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 30 / 54

31 Airline itinerary example Nest one stop Robust Parameter Coeff. Asympt. number Description estimate std. error t-stat p-value 1 One stop, same airline dummy Round trip fare ($100) Elapsed time (2 8 hours) Elapsed time (> 8 hours) Leg room (inches), if male (one stop) Leg room (inches), if female (one stop) Being early (hours) Being late (hours) More than two air trips per year (one stop, same airline) Round trip fare / income ($100/$1000) Summary statistics Number of observations = 846 L(0) = L(c) = L( ˆβ) = [L(0) L( ˆβ)] = ρ 2 = ρ 2 = M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 31 / 54

32 Airline itinerary example Upper level Non stop One stop M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 32 / 54

33 Airline itinerary example Upper level Parameter number Description 0 Non stop replaces parameter 2 1 One stop same airline dummy already estimated 2 One stop multiple airlines dummy not an alternative 3 Round trip fare ($100) already estimated 4 Elapsed time (0-2 hours) 5 Elapsed time (2-8 hours) already estimated 6 Elapsed time ( > 8 hours ) already estimated 7 Leg room (inches), if male (non stop) 8 Leg room (inches), if female (non stop) 9 Leg room (inches), if male (one stop) already estimated 10 Leg room (inches), if female (one stop) already estimated 11 Being early (hours) already estimated 12 Being late (hours) already estimated 12 More than 2 air trips per year (non stop) replaces parameter More than 2 air trips per year (one stop same airline) already estimated 14 More than 2 air trips per year (one stop multiple airlines) not an alternative 15 Round trip fare / income ($100/$1000) already estimated M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 33 / 54

34 Airline itinerary example Upper level Binary choice Non stop One stop Non stop: specification of the utility functions Same as logit. Up to normalization. Alternative specific constant More than 2 air trips per year Coefficients 0 and 12 are replacing 2 and 14 in logit. Coefficients already estimated at the lower level are not re-estimated. 5 coefficients must be estimated. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 34 / 54

35 Airline itinerary example Upper level One stop: specification Ṽ One stop = E[max(U SAME,U MULT )] = 1 µ One stop log(e µ One stopv SAME +e µ One stopv MULT ). Normalization µ One stop = 1 Ṽ One stop = log(e V SAME +e V MULT ). M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 35 / 54

36 Airline itinerary example Upper level Logit Pr(One stop {Non stop, One stop}) = Comment As µ One stop has been normalized, µ is identified. e µṽ One stop e µv NS +e µṽ One stop. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 36 / 54

37 Airline itinerary example Upper level Robust Parameter Coeff. Asympt. number Description estimate std. error t-stat p-value 0 Non stop dummy Elapsed time (0 2 hours) Leg room (inches), if male (non stop) Leg room (inches), if female (non stop) More than 2 air trips per year (non stop) µ Summary statistics Number of observations = 2544 L(0) = L(c) = L( ˆβ) = [L(0) L( ˆβ)] = ρ 2 = ρ 2 = t-test against 1 M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 37 / 54

38 Airline itinerary example Full model (sequential estimation) Robust Parameter Coeff. Asympt. number Description estimate std. error t-stat p-value 0 Non stop dummy One stop, same airline dummy Round trip fare ($100) Elapsed time (0 2 hours) Elapsed time (2 8 hours) Elapsed time (> 8 hours) Leg room (inches), if male (non stop) Leg room (inches), if female (non stop) Leg room (inches), if male (one stop) Leg room (inches), if female (one stop) Being early (hours) Being late (hours) More than 2 air trips per year (non stop) More than two air trips per year (one stop, same airline) Round trip fare / income ($100/$1000) µ Summary statistics Number of observations = 2544 L(0) = L(c) = L( ˆβ) = (= ) 2[L(0) L( ˆβ)] = ρ 2 = ρ 2 = M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 38 / 54

39 Airline itinerary example Logit vs. Nested logit Scale What is being estimated: µβ Examples Logit: µ = 1 (normalized) Nested logit: µ = Make sure to compare µβ across models. Parameter Logit Nested logit Scaled Round trip fare ($100) Elapsed time (0 2 hours) Leg room (inches), if female (non stop) M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 39 / 54

40 Airline itinerary example Logit vs. Nested logit Normalization of constants Parameter Nested logit Nested logit Nested logit number Description Logit (scaled) (scaled & shifted) 0 Non stop One stop same airline dummy One stop multiple airlines dummy > 2 trips/y (non stop) > 2 trips/y (one stop same airline) > 2 trips/y (one stop multiple airlines) M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 40 / 54

41 Airline itinerary example Logit vs Nested logit t-test Logit: µ = 1 Reject H 0 : µ = = Likelihood ratio test Logit: Nested logit: Likelihood ratio test: -2( )= Threshold: χ 2 1,0.95 = 3.84 Logit is rejected. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 41 / 54

42 Airline itinerary example Estimation Sequential estimation Estimate first the lower levels. Transfer the estimated utility function to estimate the upper level. Consistent estimator. Not efficient. Full information maximum likelihood All parameters estimated together. Consistent. Efficient. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 42 / 54

43 Airline itinerary example Full model (full information estimation) Robust Parameter Coeff. Asympt. number Description estimate std. error t-stat p-value 0 Non stop dummy One stop, same airline dummy Round trip fare ($100) Elapsed time (0 2 hours) Elapsed time (2 8 hours) Elapsed time (> 8 hours) Leg room (inches), if male (non stop) Leg room (inches), if female (non stop) Leg room (inches), if male (one stop) Leg room (inches), if female (one stop) Being early (hours) Being late (hours) More than two air trips per year (non stop) More than two air trips per year (one stop, same airline) Round trip fare / income ($100/$1000) µ Summary statistics Number of observations = 2544 L(0) = L(c) = L( ˆβ) = [L(0) L( ˆβ)] = ρ 2 = ρ 2 = M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 43 / 54

44 Airline itinerary example Normalize µ = 1, estimate µ m Robust Parameter Coeff. Asympt. number Description estimate std. error t-stat p-value 1 One stop same airline dummy One stop multiple airlines Round trip fare ($100) Elapsed time (0 2 hours) Elapsed time (2 8 hours) Elapsed time (> 8 hours) Leg room (inches), if male (non stop) Leg room (inches), if female (non stop) Leg room (inches), if male (one stop) Leg room (inches), if female (one stop) Being early (hours) Being late (hours) More than two air trips per year (one stop same airline) More than two air trips per year (one stop multiple airlines) Round trip fare / income ($100/$1000) µ m Summary statistics Number of observations = 2544 L(0) = L(c) = L( ˆβ) = [L(0) L( ˆβ)] = ρ 2 = ρ 2 = M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 44 / 54

45 Airline itinerary example Normalization of the nested logit model Best practice Normalize µ to 1. Estimate µ m for each nest. As 0 µ/µ m 1, then µ m 1. Large models Normalize all µ m to 1. Estimate µ. Note that it is not the most general specification. Imposing the same scale parameter for each nest is a strong assumption. Motivated only by models with a very high number of nests. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 45 / 54

46 Derivation Outline 1 Red bus/blue bus paradox 2 Relaxing the independence assumption 3 The nested logit model 4 Airline itinerary example 5 Derivation 6 Summary M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 46 / 54

47 Derivation Derivation from random utility Let C be the choice set. Let C 1,...,C M be a partition of C. The model is derived as M P(i C) = Pr(i m, C) Pr(m C). m=1 Each i belongs to exactly one nest m. P(i C) = Pr(i m) Pr(m C). Utility: error components U i = V i +ε i = V i +ε m +ε im. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 47 / 54

48 Derivation Derivation: Pr(i m) Pr(i m) = Pr(U i U j,j C m ) = Pr(V i +ε m +ε im V j +ε m +ε jm,j C m ) = Pr(V i +ε im V j +ε jm,j C m ) Assumption: ε im i.i.d. EV(0,µ m ) Pr(i m) = e µmv i j C m e µmv j. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 48 / 54

49 Derivation Derivation: Pr(m C) ( ) Pr(m C) = Pr maxu i maxu j, l m i C m j C l ( ) = Pr ε m +max(v i +ε im ) ε l +max(v j +ε jl ), l m, i C m j C l As ε im are i.i.d. EV(0,µ m ), max(v i +ε im ) EV(Ṽ m,µ m ), i C m where Ṽ m = 1 µ m ln i C m e µmvi. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 49 / 54

50 Derivation Derivation: Pr(m C) Denote to obtain max(v i +ε im ) = Ṽ m +ε m, i C m Pr(m C) = Pr(Ṽ m +ε m +ε m Ṽ l +ε l +ε l, l m). where Define to obtain ε m EV(0,µ m ). ε m = ε m +ε m, Pr(m C) = Pr(Ṽm + ε m Ṽl + ε l, l m). M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 50 / 54

51 Derivation Derivation: Pr(m C) Assumption: ε m i.i.d. EV(0,µ) Pr(m C) = Pr(Ṽ m + ε m Ṽ l + ε l, l m) = We obtain the nested logit model e µṽ m M p=1 eµṽp. P(i C) = = e µmv i e µṽ m j C m e µmv j M p=1 eµṽ p ( e µmv i exp µ µ m ln ) l C m e µmv l j C m e µmv j ( M p=1 exp µ µ p ln l C p e lp) µpv M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 51 / 54

52 Derivation Nested Logit Model µ If µ m = 1, for all m, NL becomes logit. Sequential estimation: Estimation of NL decomposed into two estimations of logit Estimator is consistent but not efficient Simultaneous estimation: Log-likelihood function is generally non concave No guarantee of global maximum Estimator asymptotically efficient Log likelihood for observation n is lnp(i n C n ) = lnp(i n C mn )+lnp(c mn C n ) where i n is the chosen alternative. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 52 / 54

53 Derivation Correlation Correlation matrix is block diagonal: 1 if i = j, Corr(U i,u j ) = 1 µ2 µ 2 if i j, i and j are in the same nest m, m 0 otherwise. Variance-covariance matrix is block diagonal: π 2 6µ 2 if i = j, Cov(U i,u j ) = π 2 6µ 2 π2 6µ 2 if i j, i and j are in the same nest m, m 0 otherwise. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 53 / 54

54 Summary Summary Independence assumption of logit may lead to erroneous forecasts Relaxing the assumption: nests Closed form model M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 54 / 54