A stochastic differential equation approach to the analysis of the 2017 and 2019 UK general election polls

Human dynamics and sociophysics build on statistical models that can shed light on and add to our understanding of social phenomena. We propose a generative model based on a stochastic differential equation that enables us to model the opinion polls leading up to the 2017 and 2019 UK general elections and to make predictions relating to the actual results of the elections. After a brief analysis of the time series of the poll results, we provide empirical evidence that the gamma distribution, which is often used in financial modelling, fits the marginal distribution of this time series. We demonstrate that the proposed poll-based forecasting model may improve upon predictions based solely on polls. The method uses the Euler–Maruyama method to simulate the time series, measuring the prediction error with the mean absolute error and the root mean square error, and as such could be used as part of a toolkit for forecasting elections.     

Forecasting government support in Irish general elections: Opinion polls and structural models

Election forecasting is a cottage industry among pollsters, the media, political scientists, and political anoraks. Here, we plow a fresh field in providing a systematic exploration of election forecasting in Ireland. We develop a structural forecast model for predicting incumbent government support in Irish general elections between 1977 and 2020 (the Iowa model). We contrast this structural model with forecasts from opinion polls, the dominant means of predicting Ireland’s elections to date. Our results show that with appropriate lead-in time, structural models perform similarly to opinion polls in predicting government support when the dependent variable is vote share. Most importantly, however, the Iowa model is superior to opinion polls in predicting government seat share, the ultimate decider of government fate in parliamentary systems, and especially significant in single transferable vote (STV) systems where vote and seat shares are not always in sync. Our results provide cumulative evidence of the potency of structural electoral forecast models globally, with the takeaway that the Iowa model estimating seat share outpaces other prediction approaches in anticipating government performance in Irish general elections.     

Accuracy of German federal election forecasts, 2013 & 2017

The present study reviews the accuracy of four methods (polls, prediction markets, expert judgment, and quantitative models) for forecasting the two German federal elections in 2013 and 2017. On average across both elections, polls and prediction markets were most accurate, while experts and quantitative models were least accurate. However, the accuracy of individual forecasts did not correlate across elections. That is, the methods that were most accurate in 2013 did not perform particularly well in 2017. A combined forecast, calculated by averaging forecasts within and across methods, was more accurate than three of the four component forecasts. The results conform to prior research on US presidential elections in showing that combining is effective in generating accurate forecasts and avoiding large errors.     

Additive cubic spline regression with Dirichlet process mixture errors

The goal of this article is to develop a flexible Bayesian analysis of regression models for continuous and categorical outcomes. In the models we study, covariate (or regression) effects are modeled additively by cubic splines, and the error distribution (that of the latent outcomes in the case of categorical data) is modeled as a Dirichlet process mixture. We employ a relatively unexplored but attractive basis in which the spline coefficients are the unknown function ordinates at the knots. We exploit this feature to develop a proper prior distribution on the coefficients that involves the first and second differences of the ordinates, quantities about which one may have prior knowledge. We also discuss the problem of comparing models with different numbers of knots or different error distributions through marginal likelihoods and Bayes factors which are computed within the framework of Chib (1995) as extended to DPM models by Basu and Chib (2003). The techniques are illustrated with simulated and real data.     

Playing the synthesizer with Canadian data: Adding polls to a structural forecasting model

Election forecasting has become a fixture of election campaigns in a number of democracies. Structural modeling, the major approach to forecasting election results, relies on ‘fundamental’ economic and political variables to predict the incumbent’s vote share usually a few months in advance. Some political scientists contend that adding vote intention polls to these models—i.e., synthesizing ‘fundamental’ variables and polling information—can lead to important accuracy gains. In this paper, we look at the efficiency of different model specifications in predicting the Canadian federal elections from 1953 to 2015. We find that vote intention polls only allow modest accuracy gains late in the campaign. With this backdrop in mind, we then use different model specifications to make ex ante forecasts of the 2019 federal election. Our findings have a number of important implications for the forecasting discipline in Canada as they address the benefits of combining polls and ‘fundamental’ variables to predict election results; the efficiency of varying lag structures; and the issue of translating votes into seats.     

Election forecasting: Too far out?

We consider two criteria for evaluating election forecasts: accuracy (precision) and lead (distance from the event), specifically the trade-off between the two in poll-based forecasts. We evaluate how much “lead” still allows prediction of the election outcome. How much further back can we go, supposing we tolerate a little more error? Our analysis offers estimates of the “optimal” lead time for election forecasts, based on a dataset of over 26,000 vote intention polls from 338 elections in 44 countries between 1942 and 2014. We find that optimization of a forecast is possible, and typically occurs two to three months before the election, but can be influenced by the arrangement of political institutions. To demonstrate how our optimization guidelines perform in practice, we consider recent elections in the UK, the US, and France.     

Long‐range dependence in Spanish political opinion poll series

This paper investigates the time series properties of partisanship for five political parties in Spain. It is found that pure fractional processes with a degree of integration, d, between 0.6 and 0.8 fit the time‐series behaviour of aggregate opinion polls for mainstream parties quite well, whereas values of d in the range of 0.3 to 0.6 are obtained for opinion polls related to smaller regional parties. Those results are in agreement with theories of political allegiance based on aggregation of heterogeneous voters with different degrees of commitment and pragmatism. Further, those models are found to be useful in forecasting the results of the last general elections in Spain. As a further contribution, new econometric techniques for estimation and testing of ARFIMA model are used to provide the previous evidence. Copyright © 2003 John Wiley & Sons, Ltd.     

Estimating floating voters: a comparison between the ecological inference and the survey methods

There are two main approaches to estimating the proportion of the electorate who are floating voters: the survey method and the ecological estimate method. Both the methods have their advantages and their problems. The main difficulties with the survey method are the coverage of the sample and the problems introduced by reliance on the quality of memory of the subjects. Ecological estimates have different problems, the principal of which is known as the ecological fallacy. The aim of this paper is to assess whether the survey and ecological estimates of voter swing between two elections are significantly different. For this purpose I will consider the 2006 and 2008 Italian Parliamentary elections. Given the short temporal gap between these two elections, both the methods should give reliable estimates, as the shorter the time between the two elections, the fewer the problems which will be encountered by subjects recalling the party they voted for in the previous one, and the fewer the changes which will have taken place in the composition of the population between the two elections. The ecological data I will employ comprise all the votes cast in both of the elections under consideration (2006 and 2008), at the polling station level. In Italy there are about 60,000 polling stations, and I will analyse the data from these using the Goodman Model. The survey data has been provided by Italian National Election Studies (ITANES), and consists of a large representative sample, obtained by interviews conducted by CATI.     

Local identifiability of the factor analysis and measurement error model parameter

Measurement error regression models are factor analysis models, the latent ‘correct’ regressors are the factors. There is however no common statistical method between the factor analysis and the regression model, because the covariance elements that are known identifying constituents of the former model are unknown in the latter. Instead, the idea that the data come from the same regression model, as with panel data, but can be grouped in two or more groups, each group having its own different regrressor generating process, is shown to supply credible restrictions. We generalize and compare relevant identifiability criteria and corresponding asymptotically efficient estimators that are recursive in the number of overidentifying restrictions.     

Correcting for survey effects in pre‐election polls

Pre‐election polls can suffer from survey effects, causing biases in forecasted election outcomes. We advocate a simple methodology to estimate the magnitude of survey effects, by collecting data both before and after the election. This method is illustrated by means of a field study with data concerning the 2009 European Parliament elections in the Netherlands. Our study provides empirical evidence of significant positive survey effects with respect to voter participation, especially for individuals with low intention to vote. For our data, the overall survey effect on party shares is small. This effect can be more substantial, for example, if political orientation and voting intention are correlated in the sample.     

Penalty and related estimation strategies in the spatial error model

Spatial autoregressive models are powerful tools in the analysis of data sets from diverse scientific areas of research such as econometrics, plant species richness, cancer mortality rates, image processing, analysis of the functional Magnetic Resonance Imaging (fMRI) data, and many more. An important class in the host of spatial autoregressive models is the class of spatial error models in which spatially lagged error terms are assumed. In this paper, we propose efficient shrinkage and penalty estimators for the regression coefficients of the spatial error model. We carry out asymptotic as well as simulation analyses to illustrate the gain in efficiency achieved by these new estimators. Furthermore, we apply the new methodology to housing prices data and provide a bootstrap approach to compute prediction errors of the new estimators.     

From 2000 Bush–Gore to 2006 Italian elections: voting at fifty-fifty and the contrarian effect

A sociophysical model for opinion dynamics is shown to embody a series of recent western hung national votes all set at the unexpected and very improbable edge of a fifty-fifty score. It started with the Bush–Gore 2000 American presidential election, followed by the 2002 Stoiber–Schröder, then the 2005 Schröder–Merkel German elections, and finally the 2006 Prodi-Berlusconi Italian elections. In each case, the country was facing drastic choices, the running competing parties were advocating very different programs and millions of voters were involved. Moreover, polls were given a substantial margin for the predicted winner. While all these events were perceived as accidental and isolated, our model suggests that indeed they are deterministic and obey to one single universal phenomena associated to the effect of contrarian behavior on the dynamics of opinion forming. The not hung Bush–Kerry 2004 presidential election is shown to belong to the same universal frame. To conclude, the existence of contrarians hints at the repetition of hung elections in the near future.     

A simple approach for the analysis of generalizea linear mixed models

A broad class of generalized linear mixed models, e.g. variance components models for binary data, percentages or count data, will be introduced by incorporating additional random effects into the linear predictor of a generalized linear model structure. Parameters are estimated by a combination of quasi-likelihood and iterated MINQUE (minimum norm quadratic unbiased estimation), the latter being numerically equivalent to REML (restricted, or residual, maximum likelihood). First, conditional upon the additional random effects, observations on a working variable and weights are derived by quasi-likelihood, using iteratively re-weighted least squares. Second, a linear mixed model is fitted to the working variable, employing the weights for the residual error terms, by iterated MINQUE. The latter may be regarded as a least squares procedure applied to squared and product terms of error contrasts derived from the working variable. No full distributional assumptions are needed for estimation. The model may be fitted with standardly available software for weighted regression and REML.     

Some further aspects of ecological regression analysis

Ecological regression analysis is concerned with the inference of individual level relationships from aggregate level data. A first method for this type of inference, which here is compared to Leo Goodman's (1953) method, was presented by the German statistician F. Bernstein (1932). His hypothetical example, voting behaviour depending on age, is reanalysed, using historical data from the Reichstag elections, with special emphasis on the NSDAP vote. Furthermore it is demonstrated, why age is a bad predictor is ecological regression analysis, and how unreliable predictors can be diagnosed by means of several simple coefficients.     

Semiparametric Estimator for Binary‐outcome Sample Selection: Prejudice Matters in Election

a semiparametric estimator for binary‐outcome sample‐selection models is proposed that imposes only single index assumptions on the selection and outcome equations without specifying the error term distribution. I adopt the idea in Lewbel (2000) using a ‘special regressor’ to transform the binary response Y so that the transformed Y becomes linear in the latent index, which then makes it possible to remove the selection correction term by differencing the transformed Y equation. There are various versions of the estimator, which perform differently trading off bias and variance. A simulation study is conducted, and then I apply the estimators to US presidential election data in 2008 and 2012 to assess the impact of racial prejudice on the elections, as a black candidate was involved for the first time ever in the US history.     

Modeling undecided voters to forecast elections: From bandwagon behavior and the spiral of silence perspective

Forecasting election results has been a highly attractive activity among political and social scientists. Different forecasting methods have been proposed, but those based on public opinion polls are the most common. However, there are challenges to using opinion polls, especially because they neglect undecided voters. Due to the significant number of undecided participants and their impact on voting outcomes, we analyze the potential behavior of undecided voters by considering opinion polls and sentiment based on voter expectation from the perspective of the bandwagon effect and the spiral of silence. We establish a hierarchical Bayesian forecasting model to predict voting results, and apply it to the 2016 United States presidential election and the 2016 Brexit referendum. The results of our model suggest that voting outcomes are more predictable when fully utilizing the impact of undecided voters. The results indicate that integrating aggregated polls into the hierarchical Bayesian framework is a strong predictor for forecasting outcomes, and they provide evidence for the influence of sentiment based on voter expectation in forecasting election results.     

Choosing union representation: the role of attitudes and emotions

In the United States, most unions are recognised by a majority vote of employees through union representation elections administered by the government. Most empirical studies of individual voting behaviour during union representation elections use a rational choice model. Recently, however, some have posited that voting is often influenced by emotions. We evaluate competing hypotheses about the determinants of union voting behaviour by using data collected from a 2010 representation election at Delta Air Lines, a US‐based company. In addition to the older rational choice framework, multiple regression results provide support for an emotional choice model. Positive feelings toward the employer are statistically significantly related to voting ‘no’ in a representation election, while positive feelings toward the union are related to a ‘yes’ vote. Effect sizes for the emotion variables were generally larger than those for the rational choice variables, suggesting that emotions may play a key role in representation election outcomes.     

Optimal weighted average power similar tests for the covariance structure in the linear regression model

This paper suggests a procedure for the construction of optimal weighted average power similar tests for the error covariance matrix of a Gaussian linear regression model when the alternative model belongs to the exponential family. The paper uses a saddlepoint approximation to construct simple test statistics for a large class of problems and overcomes the computational burden of evaluating the complicated integrals arising in the derivation of optimal weighted average power tests. Extensions to panel data models are considered. Applications are given to tests for error autocorrelation in the linear regression model and in a panel data framework.     

Combining forecasts: An application to elections

We summarize the literature on the effectiveness of combining forecasts by assessing the conditions under which combining is most valuable. Using data on the six US presidential elections from 1992 to 2012, we report the reductions in error obtained by averaging forecasts within and across four election forecasting methods: poll projections, expert judgment, quantitative models, and the Iowa Electronic Markets. Across the six elections, the resulting combined forecasts were more accurate than any individual component method, on average. The gains in accuracy from combining increased with the numbers of forecasts used, especially when these forecasts were based on different methods and different data, and in situations involving high levels of uncertainty. Such combining yielded error reductions of between 16% and 59%, compared to the average errors of the individual forecasts. This improvement is substantially greater than the 12% reduction in error that had been reported previously for combining forecasts.