Models and optimal designs for conjoint choice experiments including a no-choice option

Transcription

1 Faculty of Economics and Applied Economics Models and optimal designs for conjoint choice experiments including a no-choice option Bart Vermeulen, Peter Goos and Martina Vandebroek DEPARTMENT OF DECISION SCIENCES AND INFORMATION MANAGEMENT (KBI) KBI 0704

2 Models and optimal designs for conjoint choice experiments including a no-choice option Bart Vermeulen Faculteit Economische en Toegepaste Economische Wetenschappen Katholieke Universiteit Leuven Peter Goos Faculteit Toegepaste Economische Wetenschappen Universiteit Antwerpen Martina Vandebroek Faculteit Economische en Toegepaste Economische Wetenschappen Universitair Centrum voor Statistiek Katholieke Universiteit Leuven Abstract In a classical conjoint choice experiment, respondents choose one profile from each choice set that has to be evaluated. However, in real life the respondent does not always make a choice: often he/she does not prefer any of the alternatives offered. Therefore, including a no-choice option in a choice set makes a conjoint choice experiment more realistic. In the literature three different models are used to analyze the results of a conjoint choice experiment with a no-choice option: the no-choice multinomial logit model, the extended nochoice multinomial logit model and the nested no-choice multinomial logit model. We develop optimal designs for each of these models using the D-optimality criterion and the modified Fedorov algorithm. We compare the optimal designs with a reference design that was constructed ignoring the no-choice option and we discuss the impact of the different designs and models on the precision of estimation and the predictive accuracy based on a simulation study. Keywords: Bayesian optimal design, choice based conjoint, conjoint analysis, D-optimality, multinomial logit model, nested logit Bart Vermeulen is funded by project G of the Fund for Scientific Research Flanders.

3 . Introduction The aim of a conjoint choice experiment is to model respondents choices as a function of the features of a product or service. In this type of experiment the respondent repeatedly chooses the alternative offering the maximum amount of utility from a number of choice sets each containing several alternatives. These experiments gain popularity for modelling market demand because of their ability to simulate market decisions realistically and because of the opportunity to estimate the impact of product or service features on market demand. In a classical conjoint choice experiment, the respondent is forced to choose one profile from each choice set. However, in real life the customer does not always make a choice: often he/she does not like any of the alternatives presented and does not buy any of the products or services offered. Therefore, including a no-choice option in a choice set makes the experiment more realistic. To conduct an efficient conjoint choice experiment with a small number of choice sets an optimal design has to be developed by choosing the appropriate alternatives and grouping them in choice sets in the best possible way. We examine whether an optimal no-choice design, i.e. a design constructed taking into account the presence of a no-choice option in the choice sets of the experiment, leads to better results in terms of the accuracy of the estimated model coefficients and the predicted probabilities compared to a reference design developed ignoring this no-choice option. In the next section we discuss the respondents motivation to choose this no-choice option and the advantage and disadvantage of including a no-choice option in a choice set. In Section 3, three models for analyzing the data from a conjoint choice experiment with a nochoice option are discussed: the no-choice multinomial logit model (NCMNL), the extended no-choice multinomial logit (ENCMNL) model and the nested no-choice multinomial logit model (NLMNL). In Section 4 we explain some basic notions of experimental design and introduce the D-optimality criterion, which we apply to the NCMNL, ENCMNL and NLMNL models respectively to develop optimal no-choice and reference design. The relative performances of the reference design and the optimal no-choice designs under different scenarios are compared in Section 5. In Section 6, we use a simulation study to measure the accuracy of the parameter estimates by the expected mean squared error of the parameter estimates and the prediction accuracy of the designs by the expected mean squared error of the predicted probabilities. Finally, in Section 7 we take a detailed look at the accuracy of the predictions based on a simulation study in which the data consists of a mixture of choices generated by the three models. 2

4 2. The no-choice option In this section we discuss several aspects of the no-choice option described in the literature. First of all we focus on the reasons why this option is attractive to respondents. Subsequently we discuss the advantage and disadvantage of incorporating this option in the design and model. In the literature that deals with the no-choice option in choice experiments, two reasons why a respondent would choose the no-choice option can be found. According to the rationale theory, which reduces decision making to the concept of utility, the consumer prefers the product that offers him the maximum amount of utility. None of the alternatives is considered as attractive when none of them offers the respondent sufficient utility. In that case, the benefits of continuing the respondent s search for better alternatives are larger than the costs. That is why the respondent chooses the no-choice option and looks for more useful alternatives. Psychological research provides another theory as to why a consumer chooses not to choose. The theory focuses strongly on avoiding intricate trade-offs and the related discomfort and fear of making the wrong choice. Baron and Ritov (994), for example, state that consumers prefer bearing the consequences of inaction rather than those of wrong action. This is the reason why a consumer prefers deferring his purchase over buying the wrong product or service when he/she feels uncomfortable choosing. In this situation the nochoice option is used as a way to avoid a choice conflict between two alternatives with nearly equal utilities. Johnson and Orme (996), however, found no evidence for such a behaviour and claim that respondents tend not to choose the no-choice option to avoid difficult decision making. In this article, we therefore assume that the respondents determine the utility for each option and choose the no-choice option if none of the alternatives offers sufficient utility. Consequently the meaning of the no-choice option given in this paper is None of the alternatives meet my requirements meaning the customer prefers to continue to look for better alternatives. The rationale theory enables us to use the multinomial model which is the focus of the following sections (Dhar (997) and Dhar and Simonson (2003)). The major advantage of including a no-choice option in a conjoint choice experiment is that a more realistic experiment is obtained. The experiment therefore leads to better estimates of the model parameters and to better predictions of market penetrations. As a matter of fact, forcing a respondent to make a choice in a conjoint choice experiment might lead to biased parameters when analyzing the choice data (Dhar (997) and Dhar and Simonson (2003)). That including a no-choice option in the experiment avoids the bias is a major advantage which should outweigh the disadvantage that, each time a respondent selects the no-choice 3

5 option, no information is collected concerning the relative attractiveness of the alternatives offered. 3. Multinomial logit models In this section, we discuss the multinomial logit model and the nested multinomial logit model. For each of these models, we first review the logit probability of choosing an alternative and the likelihood function of the corresponding model. Within the class of multinomial logit models, Haaijer, Kamakura and Wedel (200) describe two models for analyzing data from choice experiments that have a no-choice option: the NCMNL model and ENCMNL model. The use of these two models, described in Section 3., requires the independence of irrelevant alternatives assumption to be valid. The violation of this assumption necessitates the use of the nested logit model which is the subject of Section 3.2. In this paper, we refer to the nested logit model as the NLMNL model. 3. The NCMNL and ENCMNL model The most popular model to analyze choice data is the multinomial logit model. If a respondent n faces choice set k with J alternatives, then the utility of alternative j experienced by respondent n can be expressed as u = x β + ε. () ' nkj kj nkj The p-dimensional parameter vector β represents the importance of the attributes for the consumer in determining his/her utility. We assume that this vector is common for all respondents. The vector x kj has the same dimension as β and contains the levels of the attributes of the product or service represented by the j th alternative in choice set k. The error term ε nkj captures the influence of unobserved factors on the utility experienced by the respondent. All error terms are assumed to be independent and identically extreme value distributed. Under this assumption, the probability that respondent n chooses alternative j of choice set k is P nkj = J i= exp ' ( xkjβ ) ' ( x β ) exp ki. (2) If we assume that N respondents evaluate the same set of K choice sets, the log-likelihood function for the multinomial logit model becomes N K J ln( L( β )) = ln( ) n= k= j= y nkj P nkj. (3) The dummy variable y nkj equals one when respondent n prefers alternative j of choice set k and zero otherwise. The maximum likelihood estimate ˆ β for the parameter vector is obtained by maximizing the log-likelihood function. 4

6 Like Haaijer, Kamakura and Wedel (200), we distinguish two multinomial logit models for analyzing choice experiments with a no-choice option. The simplest model, the NCMNL model, represents the no-choice option by including an alternative having zero values for all attribute levels. Consequently the utility of the no-choice option in the NCMNL model is always zero. This method of coding creates an extra level for each attribute in the study which possibly leads to distorted parameter estimates. In the second model, the ENCMNL model, an extra no-choice dummy variable is used to represent the no-choice option. This offers the advantage of an enhanced model fit. The nochoice dummy variable acts as an additional two-level attribute and takes value zero for all real-choice options and value one for the no-choice option. The model parameter corresponding to the dummy variable is interpreted as the utility of choosing the no-choice option by the respondent. As the primary interest of researchers is not in the estimation of that model parameter, we develop optimal designs that focus on the precise estimation of the other model parameters, i.e. the part-worths of the original attributes. A problem with both the NCMNL and the ENCMNL models is that they require the strong assumption of the independence of irrelevant alternatives, commonly referred to as the IIAassumption, to be valid. Under this assumption, the relative odds of choosing alternative j over j depend only on the attributes of j and j no matter what the attributes of the other alternatives are. This implies that the unobserved parts of the utilities of the alternatives exhibit no correlation. While the IIA-assumption is a realistic one in some choice situations, it is not appropriate in others where it can lead to biased estimators and consequently to incorrect predictions (Train, 2003). Now, it turns out that the IIA-assumption is often violated when some of the alternatives in a choice set are more similar than others. This is exactly what happens in choice sets that have a no-choice option because there is a group of real options and one totally different option, the no-choice option. A nested logit model can therefore be used in such situations. 3.2 The nested no-choice multinomial logit model (NLMNL) In case the IIA-assumption is violated, the family of generalized extreme value models offers alternative ways to analyze the choice data. The most widely used model in this family is the nested multinomial logit model in which the probability of choosing alternative j of nest m in choice set k is modelled as P jkm = P(m)P(j m), (4) where ( ) P m = exp M i= ( λ V ) exp m m ( λv ) i i, (5) 5

7 and P ( j m) = J m i= exp ' ( xkjβ ) ' ( x β ) exp ki ' ( ) J m V m = ln exp x ki β. (7) i= (6) In these expressions, P(m) denotes the likelihood that nest m is chosen from a set of M nests, P(j m) is the conditional probability that alternative j is chosen out of the J m alternatives in nest m given that nest m was chosen, V m is the so-called inclusive value of nest m and λ m is the dissimilarity coefficient corresponding to nest m. The inclusive value V m is interpreted as the expected utility a respondent experiences from choosing nest m. The dissimilarity coefficients λ m usually lies between 0 and and measures to what extent the alternatives in nest m are different (for a detailed discussion, see Börsch-Supan (990)). Values for λ m close to zero therefore indicate that the alternatives in nest m are very similar and that the use of nesting is justified. A λ m value of one indicates that the alternatives in nest m are so different that grouping them in a nest is not useful. The parameter vector β can be estimated by the standard maximum likelihood method. The log-likelihood function for the nested logit model can be written as N K M J m ( L( )) = y ln( ln β P ) (8) n= k = m= j= Maximizing this likelihood function to find the maximum likelihood estimates is hard because of its potential non-concavity (Train, 2003). This necessitated us to use different starting values for the parameters and to repeat the optimisation procedure for each of them to obtain the maximum likelihood estimates. njkm jkm For accommodating the no-choice option in the nested logit model, we follow the work of Haaijer, Kamakura and Wedel (200), who suggested using one nest containing the nochoice option and another nest containing the real-choice options. The no-choice option is represented using zero values for all attributes. We refer to the resulting model as the NLMNL model. The rationale behind this model is a two-stage decision process. First, the respondent either selects the nest containing the real-choice options or the nest containing the no-choice option. In the former case, the second stage in the decision process consists of the respondent choosing one of the alternatives from the nest with real choices. 4. Constructing D-optimal designs for no-choice models In this section, we introduce the D-optimality criterion and discuss the problem of constructing an optimal design for non-linear models such as the NCMNL model, the ENCMNL model and 6

8 the NLMNL model considered here. Also, we explain how we constructed a reference design for evaluating the D-optimal designs for each of these models. 4. Constructing D-optimal designs for a conjoint choice experiment The aim of a conjoint choice experiment is to model how the respondents choices depend on the attributes of products and services. To achieve this goal in an efficient way, the experiment can be designed so that its information content is maximized somehow. The resulting experimental design is then said to be optimal. Finding an optimal design for a conjoint choice experiment involves selecting the alternatives to be presented to the respondents and arranging these alternatives in choice sets according to some optimality criterion. The literature on the optimal design of experiments distinguishes several criteria to select a designed experiment. The criteria that received attention in the marketing literature are the D-, A-, G- and V-optimality criteria (see, e.g., Huber and Zwerina (996), Sándor and Wedel (200) and Kessels, Goos and Vandebroek (2006a)). The most widely used of these is the D-optimality criterion which offers the advantage that the relative performance of different experimental designs with respect to it is independent of the coding strategy or scale used (Goos (2002)). In general, D-optimal designs maximize the determinant of the Fisher information matrix on the unknown parameters contained within the vector β and thereby minimize the volume of the confidence ellipsoid around β. As the Fisher information matrix is inversely proportional to the variance-covariance matrix of the parameter estimates, a D-optimal design is also said to minimize the generalized variance of the parameter estimates. The performance of a design in terms of the D-optimality criterion is expressed by the D-error p { det (, β) } det (, ) { } p D= I X = V X β, (9) where I(X,β) denotes the Fisher information matrix, V(X,β) is the variance-covariance matrix and the matrix X contains the attribute levels for all the alternatives in the experiment. The conjoint choice design having the smallest D-error is called the D-optimal design. Because of the nonlinearity of the NCMNL, ENCMNL and NLMNL models, the D-error not just depends on the matrix X but also on the unknown model parameters contained within the β vector. As a result of that, prior knowledge of the model parameters is required to develop an optimal design for the three models. However, this knowledge is not available when the experiment is planned. The three approaches adopted in the literature on the optimal design of conjoint choice experiments for circumventing this problem are reviewed by Kessels, Goos and Vandebroek (2006a). In one approach, zero prior values are assumed for all parameters in β to simplify the optimal design problem and to obtain theoretical results (see, e.g., Burgess 7

9 and Street, (2003, 2005), Schwabe et al. (2003) and Grasshoff et al. (2004)). The assumption of zero prior values is however strong as it comes down to assuming that the respondents are indifferent between all levels of all attributes. Huber and Zwerina (996) presented a second approach to handle the dependence on the unknown parameters. They suggested non-zero prior parameter values obtained from small-sample pretests or managers prior beliefs. The resulting optimal designs are referred to as locally optimal designs in the optimal design literature and lead to more precise parameter estimates and predictions than designs developed based on zero prior parameter values provided these prior values are sufficiently close to the true values. A third approach for dealing with the problem of the model s nonlinearity is the Bayesian optimal design approach introduced in the marketing literature by Sándor and Wedel (200). In this approach a prior distribution of possible parameter values is used instead of a single guess based on pretests or managers beliefs. When there is substantial uncertainty about the unknown parameters, the Bayesian optimal designs outperform the locally optimal design in terms of the D-error. We use the Bayesian approach assuming a prior distribution f(β) on the parameters. The Bayesian version of the D-error is denoted by D b -error and defined as the expected value of the D-error over this prior distribution: D b = Eβ I X βi = V X βi f β d { det (, )} p { det (, )} p ( ) β. (0) p R Because there is no analytical expression for this quantity, it is usually approximated by taking a large number of draws from the prior distribution and averaging the D-error over all draws. If we denote the draws by β, β 2,, β R with R the number of draws, the D b -error is calculated as R R p D { det (, )} p b = I X i { det V( X, )}. R β = R βi i= i= () The conjoint design that minimizes this D b -error is the Bayesian D-optimal design. To seek for a D-optimal design, we used the modified Fedorov algorithm introduced in Kessels, Goos and Vandebroek (2006a). This heuristic algorithm seeks optimal designs by exchanging profiles of a random start design by profiles from a candidate set. Several random starting designs were used to avoid local optima and the best design found over the different random starts is considered the D b -optimal design. 4.2 Optimal designs for the NCMNL and ENCMNL models When either the NCMNL or the ENCMNL model is used, the Fisher information matrix I(X,β) for a given β can be written as K (, β ) ' ( ) ' i i i i X i (2) i= I X = N X P p p 8

10 (Huber and Zwerina, 996), where X i is the part of X containing the attribute levels of the profiles in choice set i, p i is a vector in which the jth element represents the probability that the jth profile is chosen in choice set i and P i is a diagonal matrix with on the main diagonal the values of p i. We use expression () which approximates the D b -error to find an optimal no-choice design for the NCMNL model and for the ENCMNL model. Note that the number of rows in every X i equals the number of real-choice alternatives plus one for the no-choice option in every choice set. For the ENCMNL, every X i also has an extra column because of the dummy variable that is used for the no-choice option in that model. As a result of that, the dimension of the information matrix I(X,β) for the ENCMNL model is equal to p+ when p denotes the number of model parameter corresponding to the attributes under investigation. The designs we computed for the ENCMNL model were constructed such that they were D- optimal for the p parameters corresponding to the attributes under investigation. This is because the primary interest of researchers is in gathering information on the magnitudes of these parameters. In the literature on the optimal design of experiments, designs that are D- optimal for a subset of parameters are called a D s -optimal designs (see, e.g., Atkinson and Donev (992)). 4.3 Optimal design for the NLMNL model Goos, Vandebroek and Vermeulen (2007) show that the information matrix about the unknown parameters β and λ in the nested no-choice multinomial logit model is equal to I X ' DX X ' π, ' π X c ( X β, λ) = N, where the matrix D, the vector π and the constant c are all rather complicated functions of the probabilities λ, P jkm (j,m), P(m) and P(j m). As the interest of the researcher mainly is in estimating β, they prefer minimizing the generalized variance of the estimate of β. As a result, a D s -optimal design is needed here too. Finding it requires the maximization of det(x DX-c - X ππ X) instead of det(i(x,β,λ)). This is also done in a Bayesian fashion in this article. (3) 4.4 Reference design The estimation and prediction performances of the D-optimal designs for the NCMNL, ENCMNL and NLMNL models will be compared to a D-optimal design for a multinomial model that does not accommodate a no-choice option. In the sequel of this paper, the latter design will serve as a benchmark and will be referred to as the reference design. We have chosen the D-optimal design for the multinomial logit model as a reference design because, in terms of precision of estimation, it is the best alternative design option currently available in the literature on the design of conjoint choice experiments. 9

11 In the following sections we investigate how well the reference design described earlier performs relative to the optimal designs for the NCMNL, ENCMNL and NLMNL models when estimating these models and when using them to make predictions. The criteria we used to compare the four design options are the D b -error, the estimation precision and the accuracy of the predicted probabilities of choosing alternatives in different choice sets. 5. Design comparisons in terms of D b -error In this section, we compare the reference design and the optimal designs based on the D b - error. The results we report are for choice designs with 6 choice sets consisting of two realchoice alternatives and a no-choice option. Each real-choice option is described by two attributes having three levels and one attribute having two levels. Other number of choice sets, attributes and attribute levels produced similar results to the ones reported here. 5. Computational aspects In our calculations, we used effects-type coding for the attribute levels. For three-level attributes, the first, second and third level are then represented by the vectors [,0], [0,] and [-,-], respectively. For two-level attributes, the values and - are associated with the first and second level, respectively. Effects-type coding is illustrated in Table, where the coding of the no-choice option in each the three models considered in this paper is also shown. Notice that, in the ENCMNL, a dummy variable is used for the no-choice option. Attribute levels of the choice set Coding for NCMNL and NLMNL Coding for ENCMNL no - Level Level Level attr. attr. 2 attr. 3 Attr. attr. 2 attr. 3 attr. attr. 2 attr. 3 choice dummy No-choice option Table : Effects-type coding for the NCMNL, ENCMNL and NLMNL models. As explained in Section 4., calculating Bayesian D-optimal designs requires the specification of a prior distribution for the model parameters β and λ. The resulted presented in this paper were obtained assuming the prior parameters drawn from a p-variate normal distribution with mean [ ] and variance 0.5*I p, where I p is the p-dimensional identity matrix. This prior distribution is in agreement with the recommendations formulated in Kessels, Jones, Goos and Vandebroek (2006b). To calculate the D b -errors for the NCMNL and ENCMNL models and for computing the reference design, we used 000 draws from the normal prior distribution for each parameter of the model in order to approximate the Bayesian design criteria. For the ENCMNL model, a prior distribution was also needed for the 0

12 coefficient of the extra dummy variable. We used a normal distribution with mean -0.5 and variance 0.5 for that, and we utilized 000 draws from that distribution for calculating the D b - error for designs for that model. For the NLMNL model however, we used 250 draws only because its complicated information matrix slowed down the design construction algorithm, which was a modified Fedorov algorithm like the one in Kessels, Goos and Vandebroek (2006a). For the NLMNL, we also used 250 draws from the uniform distribution on the ]0;[ interval for the parameter λ. We include in the appendix the reference design and the optimal no-choice designs for the three models with 6 choice sets of three profiles. 5.2 Comparison of design options in terms of D b -error In this section, we compare the D b -errors for the reference design and the optimal no-choice designs for the NCMNL, ENCMNL and NLMNL models. For that purpose, we have calculated one D b -error for every combination of a design and a no-choice model. This does not only allow us to evaluate how much better it is to take into account the no-choice option when designing the choice experiments, but it also enables us to look at the robustness of the different designs when the model used to analyze the data is not the model used to construct the design. The 2 D b -errors are displayed in Table 2. The numbers in bold correspond to scenarios for which the model used for generating the design was the same as the one used for analyzing the data. Model used to Model used to design the choice experiment analyze data MNL NCMNL ENCMNL NLMNL NCMNL ENCMNL NLMNL Table 2: D b -errors of the D b -optimal designs constructed for the MNL model ignoring the nochoice option and the NCMNL, ENCMNL and NLMNL models which do take into account the no-choice option. It turns out that the reference design and the optimal designs for the NCMNL model and the ENCMNL model perform almost equally well for fitting a multinomial logit model with a nochoice option. These designs however perform poorly for fitting the NLMNL model compared to the optimal design for this model. The reverse is also true: the optimal design for the NLMNL model leads to poor D b -errors when used to fit the NCMNL and ENCMNL models.

13 6. Evaluating the performance of the optimal no-choice designs In this section we report the results of a simulation study to examine the accuracy of the estimated coefficients and the predicted probabilities for choosing an alternative. 6. Criteria for evaluation In order to examine the accuracy of the estimated coefficients we use the expected mean squared error where f ( ˆ ) ( ) ( ˆ = ) ( ˆ ) ( ˆ) EMSE ˆ β β β β β f β d ˆ β β, (3) k R β is the distribution of the estimated parameters and ˆ β and β are vectors containing the estimated and the true parameter values respectively. Evidently, small EMSE ˆ β values are preferred over large ones. In our computations, we approximated the EMSE ˆ β value by performing a large number of simulations for each set of parameters β and taking the average squared difference between ˆ β and β over all simulations. Denoting the number of simulations by R, the EMSE EMSE ˆ β values reported in this article are thus computed as R ˆ ( ) ( ˆ β = β ) ( ˆ β r β βr β). (4) R r= We used R = 000 in our simulations and computed EMSE ˆ β values for 60 draws of the parameters β coming from a p-variate normal distribution with mean [ ] and variance 0.5*I p. Values for the coefficient of the no-choice dummy variable in the ENCMNL and for λ in the NLMNL were again drawn from a normal distribution with mean -0.5 and variance 0.5 and from the uniform distribution on the interval ]0;[. In the next sections, the 60 EMSE ˆ β values for different scenarios are reported in side-by-side box-plots. To assess the accuracy of the predictions obtained from the estimated models, we also calculated the expected mean squared error of the predicted probabilities, EMSE ˆp. This measure compares the probabilities obtained from the estimated coefficients and the probabilities obtained with the real parameter values for all possible combinations of two realchoice options and the no-choice option. Formally, the EMSE ˆp value is defined as ( ) ( ˆ( ˆ) ( )) ( ˆ) ( ) ˆ( ˆ) ( ) EMSE ˆ pˆ β = p β p β p β p β f β dβ, (5) k R 2

14 where the vectors ( βˆ ) ˆp and ( β ) p contain the probabilities for all possible choice sets containing two real-choice options obtained using the estimated and the real parameter values, respectively. Like for the EMSE ˆ β, small values for the EMSE ˆp are desired and simulations were used to approximate it: R ' EMSE ˆ ( ) ( ˆ( ˆ ) ( )) ( ˆ( ˆ p β = p βp p β ) ( ) r p p βp p β r p ). (6) R r= Here too we used R = 000. Just like for the EMSE ˆ β, we computed EMSE ˆp values for 60 draws of the parameters under different scenarios and we report them using box-plots. In our simulation study, we started by assuming that the NCMNL was the true model. Using that model, data for 00 respondents were generated using the four different designs, namely the reference design and the optimal designs for the three no-choice models under investigations. Next, the data generated according to the reference design were analyzed using each of the NCMNL, ENCMNL and NLMNL models. Also, the data generated according to each optimal design were analyzed using the model for which the design was constructed. This led us to study six different scenarios for each of the data-generating models and explains why each of the Figures to 7 contains six box-plots. In each of these figures, the box-plots appear in pairs. The left pair of box-plots shows EMSE ˆ β or EMSE ˆp values obtained when analyzing the data using the NCMNL model. The middle pair of box-plot contains the results for situations in which the ENCMNL is used to analyze the data, and, finally, the right pair contains results for the NLMNL model. In each pair of box-plots, the left plot gives the results for the reference design and the right plot gives the results for the optimal design. 6.2 Choices generated by the NCMNL model When the choices are generated by the NCMNL model, all design and analysis options perform almost equally well in estimating the parameters. This can be seen in Figure. It turns out that, for none of the three models, there is a significant difference in the accuracy of the estimated parameters between the reference and the optimal design. In other words, taking into account the no-choice option when constructing a choice design does not lead to a substantial gain in terms of estimation accuracy. This complements the results of Street and Burgess (2004) who concluded that designs developed for a forced choice setting can be used to include a no-choice option at the cost of a slightly reduced estimation precision when zero prior parameters values are assumed. That the reference design and the optimal designs for the NCMNL and ENCMNL models perform similarly is logical because they are based on similar non-nested multinomial logit models. The optimal design for the NLMNL model is based on a different type of model, namely a nested one, and this explains the 3

15 slightly less accurate estimated parameters in the scenario where the data are generated by a non-nested multinomial model.. Figure : EMSE ˆ β values when choices are generated by the NCMNL model. The left, middle and right pairs of box-plots show the EMSE ˆ β values obtained when analyzing the data using the NCMNL, ENCMNL and NLMNL model, respectively. In each pair, the left and right plots give the results for the reference design and the optimal design, respectively. Figure 2: EMSE ˆp values when choices are generated by the NCMNL model. The left, middle and right pairs of box-plots show the EMSE ˆp values obtained when analyzing the data using the NCMNL, ENCMNL and NLMNL model, respectively. In each pair, the left and right plots give the results for the reference design and the optimal design, respectively. 4

16 By comparing the EMSE ˆp values in the box-plots in the left panel of Figure 2 with those in the other panels, it can be seen that the choice probabilities were estimated more precisely when the data were analyzed using the NCMNL model, no matter whether the reference design or the optimal design for that model is utilized. Also when the data are analyzed using the ENCMNL model, whether or not the optimal design is used makes no substantial difference. There is however a clear difference between the two design options when the NLMNL model is used for analyzing the data generated by the NCMNL model. That the difference is in favour of the reference design is because that design was optimized for a nonnested multinomial logit model, which is similar to the model used for generating the data. 6.3 Choices generated by the ENCMNL model When the choices are generated by the ENCMNL model, the conclusions are very similar to the situation in which the data were produced by means of the NCMNL model. This can be seen from Figures 3 and 4, in which the box-plots for the EMSE ˆ β and EMSE ˆp values are displayed. Again, there is no significant difference between the reference design and the optimal designs in terms of the accuracy of the estimated parameters and the predicted probabilities. However, as can be seen from the box-plots for the EMSE ˆ β values in Figure 3, analyzing the data using the ENCMNL is now substantially better than using any of the two other models. This result is true for both the reference design and the optimal design for the ENCMNL model. Notice also that the optimal design for the NLMNL performs better when the data are generated using the ENCMNL model than when using the NCMNL. This indicates that the ENCMNL model is conceptually more similar to the NLMNL model than the NCMNL model. The box-plots for the EMSE ˆp values in Figure 4 show that the predictions by means of the ENCMNL model are significantly better than those based on the other models. This result is true for the reference design as well as for the optimal design for that model. 6.4 Choices generated by NLMNL Finally, Figures 5 and 6 visualize the accuracy of the estimated parameters and predictions when the choices are generated by the NLMNL model. To generate data using that model, we drew a random value from a uniform distribution on the interval ]0;[ for the dissimilarity parameter λ for every draw of β. Even in the case where the data are generated using the NLMNL model, there is no substantial gain in terms of estimation accuracy and prediction accuracy from taking into account the no-choice option when designing the choice experiment. This can be seen by looking at the box-plots for the EMSE ˆ β values in Figure 5 and those for the EMSE ˆp values in Figure 6. What does matter is the model used to analyze 5

17 the data: the estimates and the predicted probabilities are far more precise when the NLMNL model is utilized for the analysis rather than the NCMNL or the ENCMNL model. Figure 3: EMSE ˆ β values when choices are generated by the ENCMNL model. The left, middle and right pairs of box-plots show the EMSE ˆ β values obtained when analyzing the data using the NCMNL, ENCMNL and NLMNL model, respectively. In each pair, the left and right plots give the results for the reference design and the optimal design, respectively. Figure 4: EMSE ˆp values when choices are generated by the ENCMNL model. The left, middle and right pairs of box-plots show the EMSE ˆp values obtained when analyzing the data using the NCMNL, ENCMNL and NLMNL model, respectively. In each pair, the left and right plots give the results for the reference design and the optimal design, respectively. The simulation results in Figures to 6 show that the differences between the three models for analyzing the data are larger than the differences between the design options. None of the 6

18 models appears to lead to accurate estimates and predicted probabilities when the model selected to analyze the data does not match the model used to generate the data. Figure 5: EMSE ˆ β values when choices are generated by the NLMNL model. The left, middle and right pairs of box-plots show the EMSE ˆ β values obtained when analyzing the data using the NCMNL, ENCMNL and NLMNL model, respectively. In each pair, the left and right plots give the results for the reference design and the optimal design, respectively. Figure 6: EMSE ˆp values when choices are generated by the NLMNL model. The left, middle and right pairs of box-plots show the EMSE ˆp values obtained when analyzing the data using the NCMNL, ENCMNL and NLMNL model, respectively. In each pair, the left and right plots give the results for the reference design and the optimal design, respectively. 7

19 7. Evaluating the performance of no-choice designs based on mixed responses In the previous section we assumed that the model generating the data is the same for all respondents. In this section, we relax this assumption and investigate the situation where respondents act according to different models. For that purpose, we generate choices using each of the three models investigated in this paper. For 00 of the respondents, we used the NCMNL model to generate the responses. For another 00 respondents, the ENCMNL model was used and, for the final 00, the NLMNL model was utilized. For all 300 respondents the same choice sets -given by one of the four designs- were used. The designs used were again the reference design and the optimal designs displayed in the appendix. The data generated using the reference design were analyzed using each of the models, whereas the data generated with one of the optimal designs were only analyzed using the model for which the designs was optimized. Box-plots of the EMSE ˆp values obtained in that fashion are displayed in Figure 7. The box-plots were constructed for 20 draws of the model parameters. The same distributions as before were used for these draws. Figure 7: EMSE ˆp values when one third of the choices are generated by the NCMNL model, the ENCMNL model and the NLMNL model each. The left, middle and right pairs of box-plots show the EMSE ˆp values obtained when analyzing the data using the NCMNL, ENCMNL and NLMNL model, respectively. In each pair, the left and right plots give the results for the reference design and the optimal design, respectively. 8

20 Figure 7 shows that there is no substantial improvement in prediction error from including the no-choice option in the design process. Furthermore the figure reveals that the predictions obtained from the NLMNL are substantially more precise than those produced by the other models. This difference is statistically significant. Consequently, the NLMNL model is most robust in predicting the choice probabilities when the respondents behaviour cannot be explained using a single model. 8. Conclusion In a classical conjoint choice experiment the respondent is forced to choose an alternative of the choice set even if none of the alternatives is valued highly. However, in reality the respondent always has the possibility to postpone his/her decision and to continue his/her search for better alternatives. Because more accurate predictions are expected from a more realistic choice experiment, it is believed that including a no-choice option in every choice set of a choice experiment leads to more accurate predictions. In this paper, three models were considered for analyzing data from choice experiments including a no-choice option. The no-choice multinomial logit model represents the attributes of the no-choice options by means of zero values. In the extended no-choice multinomial logit model, a no-choice dummy variable is added to the model. The nested no-choice logit model divides the alternatives in a no-choice nest and a real-choice nest clustering the real-choice options of the choice set. We developed optimal designs for these models and compared them with a reference design that was constructed ignoring the fact that the experiment involved a no-choice option in every choice set. The reference design was an optimal design for a multinomial model without assuming that a no-choice option was utilized during the experiment. The accuracy of the estimated parameters and the predicted probabilities of the optimal designs (that were constructed taking into account the presence of a no-choice option in each choice set) and the reference design (that was constructed ignoring the presence of the nochoice option) were examined by means of a simulation study. Remarkably, there was no substantial gain in terms of either estimation precision or prediction accuracy from using the optimal designs for the no-choice models compared to the reference design. An optimal design for the multinomial model, as computed in Huber and Zwerina (996), Sándor and Wedel (200) and Kessels, Goos and Vandebroek (2006a) can therefore be used also when a no-choice option is considered. Although we did not find differences between the different design options studied in this article, our simulation study revealed the importance of selecting the appropriate model for analyzing the data. Using the wrong model leads to less accurate estimates and predictions. In cases where there is uncertainty about which model to select, the nested multinomial logit model seems to produce the best results. 9

21 APPENDIX I. Reference design Choice set Profile Attr. Attr. 2 Attr. 3 Choice set Profile Attr. Attr. 2 Attr

22 II. Optimal design for the NCMNL model Choice set Profile Attr. Attr. 2 Attr. 3 Choice set Profile Attr. Attr. 2 Attr

23 III. Optimal design for the ENCMNL model Choice set Profile Attr. Attr. 2 Attr. 3 Choice set Profile Attr. Attr. 2 Attr

24 IV. Optimal design for the NLMNL model Choice set Profile Attr. Attr. 2 Attr. 3 Choice set Profile Attr. Attr. 2 Attr

25 References Atkinson, A.C. and Donev, A.N. (992), Optimum experimental designs, Oxford U.K., Clarendon Press Baron, J. and Ritov, I. (994), Reference points and omission bias, Organizational Behavior & Human Decision Processes, 59, Börsch-Supan, A. (990), On the compatibility of nested logit models with utility maximization, Journal of Econometrics, 46, Burgess, L. and Street, D.J. (2003), Optimal designs for 2 k choice experiments, Communications in Statistics: Theory and Methods, 32, Burgess, L. and Street, D.J. (2005), Optimal designs for choice experiments with asymmetric attributes, Journal of Statistical Planning and Inference, 34, Dhar, R. (997), Consumer preference for a no-choice option, Journal of Consumer Research, 24, Dhar, R. and Simonson, I. (2003), The effect of forced choice on choice, Journal of Marketing Research, 40, Goos, P. (2002), The Optimal Design of Blocked and Split-Plot Experiments, New York, Springer Goos, P., Vermeulen, B. and Vandebroek, M. (2007), D-optimal conjoint choice designs for a nested logit model when a no-choice option is used, Unpublished manuscript. Grasshoff, U., Grossmann, H., Holling, H. and Schwabe, R. (2004), Optimal designs for main-effects in linear paired comparison models, Journal of Statistical Planning and Inference, 26, Haaijer, R., Kamakura, W., and Wedel, M. (200), The no-choice alternative in conjoint choice experiments, International Journal of Market Research, 43,

26 Huber, J. and Zwerina, K. (996), The importance of utility balance in efficient choice designs, Journal of Marketing Research, 33, Johnson, R.M. and Orme, B.K. (996), How many questions should you ask in choice-based conjoint studies?, Sawtooth Software Technical Paper, available at Kamakura, W.A., Kim, B-D. and Lee, J. (996), Modeling preference and structural heterogeneity in consumer choice, Marketing Science, 5, Kessels, R., Goos, P. and Vandebroek, M. (2006a), A comparison of criteria to design efficient choice experiments, Journal of Marketing Research, 43, Kessels, R., Jones, B., Goos, P. and Vandebroek, M. (2006b), Recommendations on the use of Bayesian optimal designs for choice experiments, Research Report KBI_067, Department of Decision Sciences and Information Management, Katholieke Universiteit Leuven, 8 pp Sándor, Z. and Wedel, M. (200), Designing conjoint choice experiments using manager s prior beliefs, Journal of Marketing Research, 38, Schwabe, R., Grasshoff, U., Grossmann, H. and Holling, H. (2003), Optimal 2 K paired comparison designs for partial profiles, in PROBASTAT2002: Proceedings of the 4 th International Conference on Mathematical Statistics, Tatra Mountains Mathematical Publications, vol. 26, edited by Stulajter, F. and Wimmer, G., Street, D.J. and Burgess, L. (2004), Optimal stated preference choice experiments when all choice sets contain a specific option, Statistical Methodology, Train, K. (2003), Discrete Choice Methods with Simulations, Cambridge University Press Zwerina, K., Huber, J. and Kuhfeld, W.F. (996), A general method for constructing efficient choice designs, working paper, Fuqua School of Business, Duke University, NC 27708, Updated version available from SAS Institute ( 25