Nested logit. Michel Bierlaire

Transcription

1 Nested logit Michel Bierlaire Transport and Mobility Laboratory School of Architecture, Civil and Environmental Engineering Ecole Polytechnique Fédérale de Lausanne M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 1 / 48

2 Outline Outline 1 Red bus/blue bus paradox 2 Relaxing the independence assumption 3 The nested logit model 4 Airline itinerary example 5 Derivation M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 2 / 48

3 Red bus/blue bus paradox Simple choice model Mode choice Two alternatives: car and bus. There are red buses and blue buses. Car and bus travel times are equal: T. Only travel time is considered in the utility function. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 3 / 48

4 Red bus/blue bus paradox Red bus/blue bus paradox Model 1 U car = βt +ε car U bus = βt +ε bus Choice probability P(car {car, bus}) = P(bus {car, bus}) = e βt e βt +e βt = 1 2 M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 4 / 48

5 Red bus/blue bus paradox Red bus/blue bus paradox Model 2 U car = βt +ε car U blue bus = βt +ε blue bus U red bus = βt +ε red bus Choice probability e βt P(car {car, blue bus, red bus}) = e βt +e βt +e βt = 1 3 P(car {car, blue bus, red bus}) P(blue bus {car, blue bus, red bus}) = 1 3. P(red bus {car, blue bus, red bus}) M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 5 / 48

6 Red bus/blue bus paradox Red bus/blue bus paradox Conclusion If you paint the buses of a city red and blue, the mode share for public transportation increases from 50% to 66%. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 6 / 48

7 Red bus/blue bus paradox Explaining the paradox Model specification Only travel time appears in the utility function. Other attributes are captured by the error term. Some of them are shared by ε blue bus and ε red bus fare headway comfort convenience etc. Logit model Assumes that ε blue bus and ε red bus are independent. Inappropriate assumption in this case. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 7 / 48

8 Outline Relaxing the independence assumption 1 Red bus/blue bus paradox 2 Relaxing the independence assumption 3 The nested logit model 4 Airline itinerary example 5 Derivation M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 8 / 48

9 Relaxing the independence assumption Capturing the correlation 7 Bus 7 Blue 7 7 Red 7 Car M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 9 / 48

10 Relaxing the independence assumption Capturing the correlation If bus is chosen then where V blue bus = V red bus = βt Choice probability U blue bus = V blue bus +ε blue bus U red bus = V red bus +ε red bus P(blue bus {blue bus, red bus}) = e βt e βt +e βt = 1 2 M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 10 / 48

11 Relaxing the independence assumption Capturing the correlation 7 Bus 7 Blue 7 7 Red 7 Car M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 11 / 48

12 Relaxing the independence assumption Capturing the correlation What about the choice between bus and car? U car = βt +ε car U bus = V bus +ε bus with V bus = V bus (V blue bus,v red bus ) ε bus =? Idea Use a logit model at the higher level. Define V bus as the expected maximum utility of red bus and blue bus M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 12 / 48

13 Relaxing the independence assumption Expected maximum utility Definition For a set of alternative C, define U C = max i C U i = max i C (V i +ε i ) and V C = E[U C ] For logit E[max U i] = 1 i C µ ln e µv i i C Actually, E[max i C U i ] = µ 1 ln i C eµv i + γ, but the constant term can be ignored. µ M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 13 / 48

14 Relaxing the independence assumption Expected maximum utility Back to the blue/red buses V bus = 1 µ b ln(e µ bv blue bus +e µ bv red bus ) = 1 µ b ln(e µ bβt +e µ bβt ) = βt + 1 µ b ln2 where µ b is the scale parameter for the logit model associated with the choice between red bus and blue bus M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 14 / 48

15 Outline The nested logit model 1 Red bus/blue bus paradox 2 Relaxing the independence assumption 3 The nested logit model 4 Airline itinerary example 5 Derivation M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 15 / 48

16 Nested Logit Model The nested logit model Probability model: car e µvcar P(car) = e µvcar +e µv bus = e µβt e µβt +e µβt+ µ µ b ln2 = µ µ b Extreme cases If µ = µ b, then P(car) = 1 3 (Model 2, logit with 3 alternatives) If µ b, then µ µ b 0, and P(car) 1 2 (Model 1, logit with 2 alternatives) M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 16 / 48

17 Nested Logit Model The nested logit model Probability model: bus e µv bus P(bus) = e µvcar +e µv bus = e µβt+ µ µ b ln2 e µβt +e µβt+ µ µ b ln2 = µ µ b Extreme cases If µ = µ b, then P(bus) = 2 3 (Model 2) µ If µ b 0, then P(bus) 1 2 (Model 1) M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 17 / 48

18 The nested logit model Nested Logit Model Choice probability P(car) P(bus) µ/µ b µ M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 18 / 48

19 Solving the paradox The nested logit model If µ µ b 0, we have P(car) = 1/2 P(bus) = 1/2 P(red bus bus) = 1/2 P(blue bus bus) = 1/2 P(red bus) = P(red bus bus)p(bus) = 1/4 P(blue bus) = P(blue bus bus)p(bus) = 1/4 M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 19 / 48

20 The nested logit model Nested logit model Comments A group of similar alternatives is called a nest Each alternative belongs to exactly one nest. The model is named Nested Logit The ratio µ/µ b must be estimated from the data 0 < µ/µ b 1 (between models 1 and 2) M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 20 / 48

21 Outline Airline itinerary example 1 Red bus/blue bus paradox 2 Relaxing the independence assumption 3 The nested logit model 4 Airline itinerary example 5 Derivation M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 21 / 48

22 Airline itinerary example Airline itinerary case study: Logit model Robust Parameter Coeff. Asympt. number Description estimate std. error t-stat p-value 1 One stop same airline dummy One stop multiple airlines dummy Round trip fare ($100) Elapsed time (0-2 hours) Elapsed time (2-8 hours) Elapsed time ( > 8 hours ) Leg room (inches), if male (non stop) Leg room (inches), if female (non stop) Leg room (inches), if male (one stop) Leg room (inches), if female (one stop) Being early (hours) Being late (hours) More than 2 air trips per year (one stop same airline) More than 2 air trips per year (one stop multiple airlines) Round trip fare / income ($100/$1000) Summary statistics Number of observations = 2544 L(0) = L(c) = L( β) = [L(0) L( β)] = ρ 2 = ρ 2 = M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 22 / 48

23 Logit model Airline itinerary example Non stop One stop same airline One stop multiple airlines M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 23 / 48

24 Nested logit model Airline itinerary example Non stop One stop Non stop One stop same airline One stop multiple airlines M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 24 / 48

25 Nested logit Airline itinerary example Marginal and conditional probabilities Pr(NS) = Pr(NS Non stop) Pr(Non stop {Non stop, One stop}), Pr(SAME) = Pr(SAME One stop) Pr(One stop {Non stop, One stop}), Pr(MULT) = Pr(MULT One stop) Pr(One stop {Non stop, One stop}). Note Pr(NS Non stop) = 1 M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 25 / 48

26 Nest one stop Airline itinerary example One stop One stop same airline One stop multiple airlines M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 26 / 48

27 Airline itinerary example Nest one stop Binary choice SAME: One stop same airline MULT: One stop multiple airlines Specification of the utility functions Same as logit model. Up to normalization. MULT constant normalized to zero More than two air trips per year (MULT) normalized to 0 Elapsed time (0 2 hours) cannot be identified due to absence of data. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 27 / 48

28 Nest one stop Airline itinerary example Robust Parameter Coeff. Asympt. number Description estimate std. error t-stat p-value 1 One stop, same airline dummy Round trip fare ($100) Elapsed time (2 8 hours) Elapsed time (> 8 hours) Leg room (inches), if male (one stop) Leg room (inches), if female (one stop) Being early (hours) Being late (hours) More than two air trips per year (one stop, same airline) Round trip fare / income ($100/$1000) Summary statistics Number of observations = 846 L(0) = L(c) = L( ˆβ) = [L(0) L( ˆβ)] = ρ 2 = ρ 2 = M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 28 / 48

29 Upper level Airline itinerary example Non stop One stop M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 29 / 48

30 Airline itinerary example Upper level Binary choice Non stop One stop Non stop: specification of the utility functions Same as logit. Up to normalization. Alternative specific constant More than 2 air trips per year Coefficients 0 and 12 are replacing 2 and 14 in logit. Coefficients already estimated at the lower level are not re-estimated. 5 coefficients must be estimated. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 30 / 48

31 Upper level Airline itinerary example One stop: specification Ṽ One stop = E[max(U SAME,U MULT )] = 1 µ One stop log(e µ One stopv SAME +e µ One stopv MULT ). Normalization µ One stop = 1 Ṽ One stop = log(e V SAME +e V MULT ). M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 31 / 48

32 Upper level Airline itinerary example Logit Pr(One stop {Non stop, One stop}) = Comment As µ One stop has been normalized, µ is identified. e µṽone stop e µv NS +e µṽ One stop. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 32 / 48

33 Upper level Airline itinerary example Robust Parameter Coeff. Asympt. number Description estimate std. error t-stat p-value 0 Non stop dummy Elapsed time (0 2 hours) Leg room (inches), if male (non stop) Leg room (inches), if female (non stop) More than 2 air trips per year (non stop) µ Summary statistics Number of observations = 2544 L(0) = L(c) = L( ˆβ) = [L(0) L( ˆβ)] = ρ 2 = ρ 2 = t-test against 1 M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 33 / 48

34 Airline itinerary example Full model (sequential estimation) Robust Parameter Coeff. Asympt. number Description estimate std. error t-stat p-value 0 Non stop dummy One stop, same airline dummy Round trip fare ($100) Elapsed time (0 2 hours) Elapsed time (2 8 hours) Elapsed time (> 8 hours) Leg room (inches), if male (non stop) Leg room (inches), if female (non stop) Leg room (inches), if male (one stop) Leg room (inches), if female (one stop) Being early (hours) Being late (hours) More than 2 air trips per year (non stop) More than two air trips per year (one stop, same airline) Round trip fare / income ($100/$1000) µ Summary statistics Number of observations = 2544 L(0) = L(c) = L( ˆβ) = (= ) 2[L(0) L( ˆβ)] = ρ 2 = ρ 2 = M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 34 / 48

35 Airline itinerary example Logit vs. Nested logit Scale What is being estimated: µβ Logit: µ = 1 (normalized) Nested logit: µ = 0.526µ m = Make sure to compare µβ across models. Normalization of constants Parameter Nested logit Nested logit Nested logit number Description Logit (scaled) (scaled & shifted) 0 Non stop One stop same airline dummy One stop multiple airlines dummy > 2 trips/y (non stop) > 2 trips/y (one stop same airline) > 2 trips/y (one stop multiple airlines) M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 35 / 48

36 Airline itinerary example Logit vs Nested logit Likelihood ratio test Logit: Nested logit: Likelihood ratio test: -2( )= Threshold: χ 2 1,0.95 = 3.84 Logit is rejected. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 36 / 48

37 Airline itinerary example Estimation Sequential estimation Estimate first the lower levels. Transfer the estimated utility function to estimate the upper level. Consistent estimator. Not efficient. Full information maximum likelihood All parameters estimated together. Consistent. Efficient. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 37 / 48

38 Airline itinerary example Full model (full information estimation) Robust Parameter Coeff. Asympt. number Description estimate std. error t-stat p-value 0 Non stop dummy One stop, same airline dummy Round trip fare ($100) Elapsed time (0 2 hours) Elapsed time (2 8 hours) Elapsed time (> 8 hours) Leg room (inches), if male (non stop) Leg room (inches), if female (non stop) Leg room (inches), if male (one stop) Leg room (inches), if female (one stop) Being early (hours) Being late (hours) More than two air trips per year (non stop) More than two air trips per year (one stop, same airline) Round trip fare / income ($100/$1000) µ Summary statistics Number of observations = 2544 L(0) = L(c) = L( ˆβ) = [L(0) L( ˆβ)] = ρ 2 = ρ 2 = M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 38 / 48

39 Airline itinerary example Normalize µ = 1, estimate µ m Robust Parameter Coeff. Asympt. number Description estimate std. error t-stat p-value 1 One stop same airline dummy One stop multiple airlines Round trip fare ($100) Elapsed time (0 2 hours) Elapsed time (2 8 hours) Elapsed time (> 8 hours) Leg room (inches), if male (non stop) Leg room (inches), if female (non stop) Leg room (inches), if male (one stop) Leg room (inches), if female (one stop) Being early (hours) Being late (hours) More than two air trips per year (one stop same airline) More than two air trips per year (one stop multiple airlines) Round trip fare / income ($100/$1000) µ m Summary statistics Number of observations = 2544 L(0) = L(c) = L( ˆβ) = [L(0) L( ˆβ)] = ρ 2 = ρ 2 = M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 39 / 48

40 Airline itinerary example Normalization of the nested logit model Best practice Normalize µ to 1. Estimate µ m for each nest. As 0 µ/µ m 1, then µ m 1. Large models Normalize all µ m to 1. Estimate µ. Note that it is not the most general specification. Imposing the same scale parameter for each nest is a strong assumption. Motivated only by models with a very high number of nests. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 40 / 48

41 Outline Derivation 1 Red bus/blue bus paradox 2 Relaxing the independence assumption 3 The nested logit model 4 Airline itinerary example 5 Derivation M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 41 / 48

42 Derivation Derivation from random utility Let C be the choice set. Let C 1,...,C M be a partition of C. The model is derived as M P(i C) = Pr(i m, C) Pr(m C). m=1 Each i belongs to exactly one nest m. P(i C) = Pr(i m) Pr(m C). Utility: error components U i = V i +ε i = V i +ε m +ε im. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 42 / 48

43 Derivation: Pr(i m) Derivation Pr(i m) = Pr(U i U j,j C m ) = Pr(V i +ε m +ε im V j +ε m +ε jm,j C m ) = Pr(V i +ε im V j +ε jm,j C m ) Assumption: ε im i.i.d. EV(0,µ m ) Pr(i m) = e µmv i j C m e µmv j. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 43 / 48

44 Derivation: Pr(m C) Derivation ( ) Pr(m C) = Pr maxu i maxu j, l m i C m j C l ( ) = Pr ε m +max(v i +ε im ) ε l +max(v j +ε jl ), l m, i C m j C l As ε im are i.i.d. EV(0,µ m ), max(v i +ε im ) EV(Ṽ m,µ m ), i C m where Ṽ m = 1 µ m ln i C m e µmvi. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 44 / 48

45 Derivation: Pr(m C) Derivation Denote to obtain max(v i +ε im ) = Ṽ m +ε m, i C m Pr(m C) = Pr(Ṽ m +ε m +ε m Ṽ l +ε l +ε l, l m). where Define to obtain ε m EV(0,µ m ). ε m = ε m +ε m, Pr(m C) = Pr(Ṽ m + ε m Ṽ l + ε l, l m). M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 45 / 48

46 Derivation: Pr(m C) Assumption: ε m i.i.d. EV(0,µ) Derivation Pr(m C) = Pr(Ṽm + ε m Ṽl + ε l, l m) = We obtain the nested logit model e µṽ m M p=1 eµṽ p. P(i C) = = e µmv i e µṽm j C m e µmv j M p=1 eµṽ p ( e µmv i exp µ µ m ln ) l C m e µmv l j C m e µmv j ( M p=1 exp µ µ p ln l C p e lp) µpv M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 46 / 48

47 Nested Logit Model Derivation µ If µ m = 1, for all m, NL becomes logit. Sequential estimation: Estimation of NL decomposed into two estimations of logit Estimator is consistent but not efficient Simultaneous estimation: Log-likelihood function is generally non concave No guarantee of global maximum Estimator asymptotically efficient Log likelihood for observation n is lnp(i n C n ) = lnp(i n C mn )+lnp(c mn C n ) where i n is the chosen alternative. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 47 / 48

48 Correlation Derivation Correlation matrix is block diagonal: 1 if i = j, Corr(U i,u j ) = 1 µ2 µ 2 if i j, i and j are in the same nest m, m 0 otherwise. Variance-covariance matrix is block diagonal: π 2 6µ 2 if i = j, Cov(U i,u j ) = π 2 6µ 2 π2 6µ 2 if i j, i and j are in the same nest m, m 0 otherwise. M. Bierlaire (TRANSP-OR ENAC EPFL) Nested logit 48 / 48