Adjusted Priors for Bayes Factors Involving Reparameterized Order Constraints

Transcription

1 Adjusted Priors for Bayes Factors Involving Reparameterized Order Constraints Supplementary Material Daniel W. Heck & Eric-Jan Wagenmakers April 29, 2016 Contents 1 The Product-Binomial Model Parameter Estimates Marginal Likelihoods Bayes Factors Marginal Likelihood of the Pair-Clustering Model 7 1

2 1 The Product-Binomial Model In the following, we compare the two product-binomial models from the main paper (balanced vs. unbalanced prior) with respect to parameter estimates, the marginal probability, and the Bayes factor. 1.1 Parameter Estimates To obtain parameter estimates for the models M θ u and M η u, we used JAGS (Plummer, 2003). An implementation of order constraints in JAGS that ensures a uniform distribution on the restricted parameter space was provided by Lee (2016) based on the function sort(). We confirmed that the resulting estimates were identical when using the Gibbs sampler for orderconstrained product-binomial models proposed by Myung, Karabatsos, and Iverson (2005) and when using the adjusted priors for the reparameterized model derived below. To compare differences in parameter estimates, we sampled observed frequencies for different numbers of order constraints (P = 2,..., 5) from a (discrete) uniform distribution, y i U(0, 1,..., n). Next, we fitted the balanced and unbalanced models M θ u and M η u and computed the differences in mean estimates for θ. Figure 1 shows the results of this simulation for sample sizes of n = 10 (first row) and n = 100 (second row). The mostly negative differences indicate that the unbalanced model M η u results in smaller estimates θ i for all i. Note that this follows directly from the larger weight of the prior distribution on small values as illustrated in the main paper. Figure 1 also shows that the differences in parameter estimates are more pronounced in the model with P = 5 compared to a model with only P = 2 order constraints. Again, this directly results from the larger discrepancy in the prior probability mass on θ illustrated in the main text. As expected, the data overwhelm the prior for large sample sizes and hence the discrepancy in parameter estimates for n = 100 almost vanishes. 2

3 P = 2 P = 3 P = 4 P = Bias of reparameterized model n = 10 n = Parameter θ i Figure 1: Distributions of mean posterior estimates for the difference in parameter estimates between the balanced prior model M θ u and the unbalanced prior model M η u, which both implement the order constraint θ 1... θ P. The simulation is based on uniformly sampled data points y i U(1, 2,..., n). 1.2 Marginal Likelihoods To compute the Bayes factor, we estimated the marginal likelihood of the data given the model by standard Monte Carlo integration (Robert & Casella, 2004). More precisely, for this simple model, we sampled parameter values from the prior distribution and averaged the likelihood given the sampled 3

4 parameters, p(y M θ k) 1 M M fk θ (y θ (m) ) using samples θ (m) πk(θ). θ (1) m=1 Note that for more complex MPT models, this sampling scheme is inefficient and should be replaced by importance sampling as for the empirical example below (Vandekerckhove, Matzke, & Wagenmakers, 2015). Here, we compare the marginal distributions of the balanced prior model M θ u and the unbalanced prior model M η u given only two independent conditions (P = 2) and the order constraint θ 1 θ 2. Note that, for P > 2, differences in the marginal distributions of the models M θ u and M η u will be even more pronounced due to the larger differences in the priors. However, since the direct comparison of the marginal likelihoods becomes increasingly complex for P > 2, we focus on the least critical case P = 2. If a relevant discrepancy occurs in this case, the issue of prior choice is even more important in the other cases. Figure 2 shows the marginal likelihoods of the models M θ u (left panel) and M η u (right panel) for all possibly observable data with a sample size of n = 10 based on 300,000 Monte Carlo samples. Note that the marginal likelihood depends on two observed relative frequencies, that is, p 1 = y 1 /n 1 varying on the x-axis and p 2 = y 2 /n 2 varying across separate lines. For reference, the dashed line shows all possible data with p 2 = 1, for which the order constraint θ 1 θ 2 holds descriptively across all possible values of p 1, thus resulting in relatively high marginal likelihoods. Figure 2 illustrates that the balanced prior model M θ u results in symmetric marginal likelihoods. In contrast, the unbalanced prior model M η u has larger marginal likelihoods if both relative frequencies p 1 and p 2 are small. Again, this mirrors the increased prior weight on small values of θ 1 and θ 2. 4

5 Balanced Prior Unbalanced Prior Marginal Probability Proportion p Observed proportion p 1 Figure 2: All possibly observable marginal likelihoods for the orderconstrained models with two parameters (θ 1 θ 2 ) and n 1 = n 2 = 10. As reference, data with a relative frequency of p 2 = 1.0 are shown by a dashed line. 1.3 Bayes Factors The discrepancy in marginal likelihoods discussed in the previous section directly affects any Bayes factor involving the order-constrained model. Specifically, the Bayes factor B ηθ between the unbalanced and the balanced prior model gives the multiplicative factor by which any Bayes factor involving the order constraint will differ. However, Bayes factors bear a more natural interpretation in terms of evidence and are more common in model comparisons. Therefore, in the following, we also show the discrepancy in the Bayes factor when using the balanced versus the unbalanced prior model. For this purpose, we computed the Bayes factor of the order-constrained model 5

6 against the null model M θ n, which restricts all parameters to be identical, θ 1 =... = θ P. Under a uniform prior, the marginal likelihood of the null model is given analytically by p(y M θ n) = = 1 0 [ P i=1 P ( ni y i=1 i ( ni where B(a,b) is the beta function. y i ) θ y i (1 θ) n i y i dθ (2) ) ] B( y i + 1, i i n i i y i + 1), (3) Figure 3 shows the log Bayes factors for all possibly observable data. Whereas the left panels show the Bayes factor for the balanced prior model M θ u versus the null model M θ n, the right panels show the Bayes factor for the unbalanced prior model M η u versus the null model M θ n. The sample sizes are n = 10 and n = 100 in the first and second row, respectively. To facilitate the interpretation, the horizontal line at zero indicates Bayes factors that are indifferent (log B ij = 0), and the gray area marks data for which the order constraint holds descriptively (i.e., for which p 1 p 2 ). It is clear from Figure 3 that the Bayes factors for the two models M θ u and M η u are markedly different. The qualitative differences are especially pronounced for n = 100 showing that model selection by Bayes factors depends on the prior even for large sample sizes. Most importantly, the symmetry of the marginal distribution of the balanced model M θ u transfers to the Bayes factor. Hence, the same amount of evidence in favor of the constraint θ 1 = θ 2 is obtained with observed proportions (p, p) and (1 p, 1 p) (these cases are shown by the boundary of the gray area, at which p 1 = p 2 ). In other words, it does not matter if we observe an equal amount of hits or an equal amount of misses. In contrast, this distinction has a large influence on the Bayes factor for the unbalanced prior model M η u. Here, the Bayes factor depends on the definition of a success. For instance, whereas the pair of prevalences (0,0) gives slight evidence in favor of the order constraint, the pair (1,1) provides 6

7 substantial evidence against it. Note that this dependence on how the data are coded is highly critical for empirical analyses because the coding scheme should not have an effect on statistical inferences. 2 Marginal Likelihood of the Pair-Clustering Model To compute the marginal likelihood for the four pair-clustering models, we used Monte Carlo integration with an importance-sampling distribution h that increases the probability of sampling parameters θ (m) with large likelihood. Specifically, we used a mixture of a uniform distribution and a betaapproximation of the posterior distribution (Vandekerckhove et al., 2015), h(θ i ) w Uniform(0, 1) + (1 w) Beta(a i, b i), (4) with a mixture weight of w =.05, where a i and b i are the beta-distribution parameters fitted to the posterior of each parameter. Given a set of sampled parameters θ (m) h(θ), the marginal likelihood is estimated by p(y M θ k) 1 M M m=1 fk θ(y θ(m) )πk θ(θ(m) ). (5) h(θ (m) ) To obtain standard errors for both the marginal likelihood and the logmarginal likelihood, we used 50 separate batches with 500,000 samples each. Note that the Bayes factor for the full model versus the balanced, orderconstrained model can also be computed directly using the encompassing prior approach (Klugkist, Kato, & Hoijtink, 2005). 1 Table 1 shows the marginal likelihood of the four models discussed in the main paper for Experiments 1, 3, and 4 of Riefer, Knapp, Batchelder, 1 The R code for both approaches the importance sampler and the encompassing prior is included in the supplementary material. 7

8 Bayes factor in favor of order vs. equality constraint Balanced Prior Unbalanced Prior 5 log B 0n n = 10 n = 100 Proportion p Observed proportion p 1 Figure 3: Possible values of the Bayes factor in favor of the order constraint θ 1 θ 2 compared to the null model θ 1 = θ 2 for different priors (columns) and different sample sizes (rows). The gray area shows observable data that descriptively satisfy the constraint p 1 p 2. As reference, data with p 2 = 1 that fulfill the constraint irrespective of the proportion p 1 are shown by a dashed line. 8

9 Bamber, and Manifold (2002). Whereas Experiment 1 experimentally manipulated the presentation rate of the words in the learning phase to validate the model, Experiment 4 compared healthy controls against alcoholics with brain damage. Similarly as discussed for Experiment 3 in the main text, the different priors on the reparameterized models have a pronounced impact on model selection in Experiment 1 and 4. This further underscores the importance of using adjusted priors on the auxiliary parameters that imply theoretically meaningful parameters on the original parameters of interest. Table 1: Estimated marginal likelihoods of the four competing models based on the data by Riefer, Knapp, Batchelder, Bamber, and Manifold (2002). Experiment Model M p(y M) SE p log p(y M) SE log p Full < (Presentation rate) Balanced Unbalanced Null < 0.01 Full (Schizophrenics) Balanced Unbalanced Null < 0.01 Full (Alcoholics) Balanced Unbalanced Null <

10 References Klugkist, I., Kato, B., & Hoijtink, H. (2005). Bayesian model selection using encompassing priors. Statistica Neerlandica, 59, Lee, M. D. (2016). Bayesian outcome-based strategy classification. Behavior Research Methods, 48, doi: /s Myung, J. I., Karabatsos, G., & Iverson, G. J. (2005). A Bayesian approach to testing decision making axioms. Journal of Mathematical Psychology, 49, doi: /j.jmp Plummer, M. (2003). JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. In Proceedings of the 3rd International Workshop on Distributed Statistical Computing (Vol. 124, p. 125). Vienna, Austria. Riefer, D. M., Knapp, B. R., Batchelder, W. H., Bamber, D., & Manifold, V. (2002). Cognitive psychometrics: Assessing storage and retrieval deficits in special populations with multinomial processing tree models. Psychological Assessment, 14, doi: / Robert, C. & Casella, G. (2004). Monte Carlo statistical methods. New York, NY: Springer Science & Business Media. Vandekerckhove, J. S., Matzke, D., & Wagenmakers, E. (2015). Model comparison and the principle of parsimony. In Oxford Handbook of Computational and Mathematical Psychology (pp ). New York, NY: Oxford University Press. 10