Day 10: Additional Scaling Issues

Transcription

1 Day 10: Additional Scaling Issues Kenneth Benoit Essex Summer School 2011 July 22, 2011

2 Problems to solve I: Conditional (non-)independence Words occur in order In occur words order. Occur order words in. No more training do you require. Already know you that which you need. (Yoda) Words occur in combinations carbon tax / income tax / inhertiance tax / capital gains tax / bank tax Sentences (and topics) occur in sequence (extreme serial correlation) Style may mean means we are likely to use synonyms very probable. In fact it s very distinctly possible, to be expected, odds-on, plausible, imaginable; expected, anticipated, predictable, predicted, foreseeable.) Rhetoric may lead to repetition. ( Yes we can! ) anaphora

3 Problems to solve II: Parametric (stochastic) model Poisson assumes Var(Y ij ) = E(Y ij ) = λ ij For many reasons, we are likely to encounter overdispersion or underdispersion overdispersion when informative words tend to cluster together underdispersion could (possibly) occur when words of high frequency are uninformative and have relatively low between-text variation (once length is considered) This should be a word-level parameter

4 Overdispersion in German manifesto data (from Slapin and Proksch 2008) Average Absolute Value of Residuals Log Word Frequency

5 How to account for uncertainty? Don t. (SVD-like methods, e.g. correspondence analysis) Analytical derivatives Parametric bootstrapping (Slapin and Proksch, Lewis and Poole) Non-parametric bootstrapping (and yes of course) Posterior sampling from MCMC

6 Steps forward Diagnose (and ultimately treat) the issue of whether a separate variance parameter is needed Diagnose (and treat) violations of conditional independence Explore non-parametric methods to estimate uncertainty

7 Diagnosis I: Estimations on simulated texts D10 D09 D08 D07 D06 D05 D04 D03 D02 D01 Poisson model, 1/!= D10 Negative binomial, 1/!=2.0

8 D02 DiagnosisD01 I: Estimations on simulated texts D10 D08 D07 D09 D05 D06 D03 D04 D01 D02 Negative binomial, 1/!= D10 Negative binomial, 1/!=0.8

9 D01 D02 Diagnosis I: Estimations on simulated texts D10 D09 D08 D06 D07 D05 D04 D03 D02 D01 Negative binomial, 1/!=

10 Diagnosis 2: Irish Budget debate of 2009 ff lenihan fg bruton odonnell green cuffe ryan gormley lab gilmore burton higgins quinn sf morgan ocaolain ff lenihan cowen fg bruton odonnell kenny green cuffe ryan gormley lab gilmore burton higgins quinn sf morgan ocaolain Wordscores LBG Position on Budget Normalized CA Position on Budget 2009 ff lenihan cowen fg bruton odonnell kenny green cuffe ryan gormley lab gilmore burton higgins quinn sf morgan ocaolain Classic Wordfish Position on Budget 2009

11 Germany, Diagnosis 3: German party manifestos (economic sections) (Slapin and Proksch 2008) (B) Economic Policy Party Position PDS Greens SPD CDU/CSU FDP Year

12 Diagnosis 4: What happens if we include irrelevant text? John Gormley s Two Hats ff lenihan fg bruton odonnell green cuffe ryan gormley lab gilmore burton higgins quinn sf morgan ocaolain ff lenihan cowen fg bruton odonnell kenny green cuffe ryan gormley lab gilmore burton higgins quinn sf morgan ocaolain Wordscores LBG Position on Budget Normalized CA Position on Budget 2009 Midwest 2010

13 Diagnosis 4: What happens if we include irrelevant text? John Gormley s Two Hats John Gormley: leader of the Green Party and Minister for the Environment, Heritage and Local Government As leader of the Green Party I want to take this opportunity to set out my party s position on budget [772 words later] Midwest 2010 I will now comment on some specific aspects of my Department s Estimate. I will concentrate on the principal sectors within the Department s very broad remit...

14 Diagnosis 4: Without irrelevant text Ministerial Text Removed ff lenihan fg bruton odonnell green cuffe ryan gormley lab gilmore burton higgins quinn sf morgan ocaolain ff lenihan cowen fg bruton odonnell kenny green cuffe ryan gormley lab gilmore burton higgins quinn sf morgan ocaolain Wordscores LBG Position on Budget Normalized CA Position on Budget 2009 Midwest 2010

15 The Way Forward Parametric Poisson model with variance parameter ( negative binomial with parameter for over- or under-dispersion at the word level, could use CML Block Bootstrap resampling schemes text unit blocks (sentences, paragraphs) fixed length blocks variable length blocks could be overlapping or adjacent More detailed investigation of feasible methods for characterizing fundamental uncertainty from non-parametric scaling models (CA and others based on SVD)

16 The Negative Binomial model Generalize the Poisson model to: f nb (y i λ i, σ 2 ) where : σ 2 is the variability (a new parameter v. Poisson) λi is the expected number of events for i λ is the average of individual λi s Here we have dropped Poisson assumption that λ i = λ i New assumption: Assume that λ i is a random variable following a gamma distribution (takes on only non-negative numbers) For the NB model, Var(Y i ) = λ i σ 2 for λ i > 0 and σ 2 > 0

17 The Negative Binomial model cont. For the NB model, Var(Y i ) = λ i σ 2 for λ i > 0 and σ 2 > 0 How to interpret σ 2 in the negative binomial when σ 2 = 1.0, negative binomial Poisson when σ 2 > 1, then it means there is overdispersion in Y i caused by correlated events, or heterogenous λ i when σ 2 < 1 it means something strange is going on When σ 2 1, then Poisson results will be inefficient and standard errors inconsistent Functional form: same as Poisson Variance of λ is now: E(y i ) = λ Var(y i ) = λ i σ 2 = e X i β σ 2

18 Problems to Solve III: Integrating non-parametric methods Non-parametric methods are algorithmic, involving no parameters in the procedure that are estimated Hence there is no uncertainty accounting given distributional theory Advantage: don t have to make assumptions Disadvantages: cannot leverage probability conclusions given distribtional assumptions and statistical theory results highly fit to the data not really assumption-free, if we are honest

19 Correspondence Analysis CA is like factor analysis for categorical data Following normalization of the marginals, it uses Singular Value Decomposition to reduce the dimensionality of the word-by-text matrix This allows projection of the positioning of the words as well as the texts into multi-dimensional space The number of dimensions as in factor analysis can be decided based on the eigenvalues from the SVD

20 Correspondence Analysis contd. There are also problems with bootstrapping: (Milan and Whittaker 2004) rotation of the principal components inversion of singular values reflection in an axis

21 How to account for uncertainty? Don t. (SVD-like methods, e.g. correspondence analysis) Analytical derivatives Parametric bootstrapping (Slapin and Proksch, Lewis and Poole) Non-parametric bootstrapping (and yes of course) Posterior sampling from MCMC

22 Methods of uncertainty accounting in text scaling MCMC Conditional ML SVD-based Algorithmic Uncertainty accounting (multinomial+)(poisson) (CA) (Wordscores) Posterior sampling Analytical??? Parametric bootstrap Non-parametric BS?

23 Data-driven versus parametric methods!"#"$!%&'()!"#$ %#&'()*+!"#, -&.&/$)*+0 1"")2).&- 3&.&/$).*+0 1"")2).&-!"##$%&"'($')$*+',-.%/% 2"/%%"'*%),-/'3 0"#(%)"#$% 1!1! %#&'()*+ *+!(,$!%&'()

24 Steps forward Diagnose (and ultimately treat) the issue of whether a separate variance parameter is needed Diagnose (and treat) violations of conditional independence Explore non-parametric methods to estimate uncertainty

25 Diagnosis I: Estimations on simulated texts D10 D09 D08 D07 D06 D05 D04 D03 D02 D01 Poisson model, 1/!= D10 Negative binomial, 1/!=2.0

26 D02 DiagnosisD01 I: Estimations on simulated texts D10 D08 D07 D09 D05 D06 D03 D04 D01 D02 Negative binomial, 1/!= D10 Negative binomial, 1/!=0.8

27 D01 D02 Diagnosis I: Estimations on simulated texts D10 D09 D08 D06 D07 D05 D04 D03 D02 D01 Negative binomial, 1/!=

28 Simulated text results Poisson model, 1/δ=0 Frequency Estimate of 1/δ Negative binomial, 1/δ=2.0 Frequency Estimate of 1/δ Negative binomial, 1/δ=0.8 Frequency Estimate of 1/δ

29 Diagnosis 2: Irish Budget debate of 2009 ff lenihan fg bruton odonnell green cuffe ryan gormley lab gilmore burton higgins quinn sf morgan ocaolain ff lenihan cowen fg bruton odonnell kenny green cuffe ryan gormley lab gilmore burton higgins quinn sf morgan ocaolain Wordscores LBG Position on Budget Normalized CA Position on Budget 2009 ff lenihan cowen fg bruton odonnell kenny green cuffe ryan gormley lab gilmore burton higgins quinn sf morgan ocaolain Classic Wordfish Position on Budget 2009

30 Budget debates: Analytical SEs Non-parametric bootstrap (blue) versus Analytical SEs (black) Cowen FF Lenihan FF Gormley Green Cuffe Green Ryan Green Morgan SF Ocaolain SF Odonnell FG Bruton FG Gilmore Lab Kenny FG Quinn Lab Higgins Lab Burton Lab

31 Budget debates: Bootstrapped SEs on CA CA with non-parametric bootstrap (blue) versus Analytical SEs (black) Cowen FF Lenihan FF Gormley Green Cuffe Green Ryan Green Morgan SF Ocaolain SF Odonnell FG Bruton FG Gilmore Lab Kenny FG Quinn Lab Higgins Lab Burton Lab

32 Germany, Diagnosis 3: German party manifestos (economic sections) (Slapin and Proksch 2008) (B) Economic Policy Party Position PDS Greens SPD CDU/CSU FDP Year

33 German manifestos: Poisson Scaled Analytical SEs Non-parametric bootstrap (blue) versus Analytical SEs (black) GREENS 1990 PDS 1990 PDS 1998 PDS 1994 GREENS 1994 GREENS 1998 PDS 2005 PDS 2002 GREENS 2002 GREENS 2005 SPD 1998 SPD 1990 SPD 1994 SPD 2005 SPD 2002 CDU 1990 CDU 1998 CDU 2002 CDU 2005 CDU 1994 FDP 2005 FDP 2002 FDP 1998 FDP 1990 FDP

34 German manifestos: Non-parametric bootstrap on CA CA with non-parametric bootstrap (blue) versus Analytical SEs (black) GREENS 1990 PDS 1990 PDS 1998 PDS 1994 GREENS 1994 GREENS 1998 PDS 2005 PDS 2002 GREENS 2002 GREENS 2005 SPD 1998 SPD 1990 SPD 1994 SPD 2005 SPD 2002 CDU 1990 CDU 1998 CDU 2002 CDU 2005 CDU 1994 FDP 2005 FDP 2002 FDP 1998 FDP 1990 FDP