Saturday, 17 June 2017

On #GE2017 Forecasting and Diagnosing the 'Polls Plus' Model

The 2017 General Election campaign, result and fallout are each undeniably remarkable.

How a governing party was able to go from a 20-point lead to a lead of just over 2 points (constituting a swing towards their opposition) over the campaign period defies all previous knowledge about the impact of campaigns, and the ability of parties to turn their fortunes around so dramatically in just six weeks.

Similarly, the dramatic improvement in the personal approval rating of Jeremy Corbyn is beyond anything we have seen before - previous wisdom dictated that first impressions were everything.

Also, for only the second time in modern polling history, polling undershot Labour Party voting intention by a significant margin - around 5% on average. Rather than there being an 8-point Conservative lead as suggested by the final polling average, the Conservatives were just 2.5% ahead on the day.

The resulting hung parliament, with the Conservatives short of an absolute majority by seven seats, was an outcome forecasted by only one - Ben Lauderdale for YouGov.

Many are suggesting that the collective failure of the political academic and wider data-driven community to predict the oncoming hung parliament (save for Lauderdale/YouGov) is indicative of some chronic level of problems. That there is something now structurally or fundamentally wrong with political science regarding election analysis and forecasting. In short: we simply don't get it any more.

I disagree.

Firstly, it is right that we acknowledge where we went wrong, where we can improve, and admit that currently we are still some way away from where we would like to be in terms of our reading of the current political climate and prediction of outcomes.

That said, some forecasts were simply not as far off as others, and while the data which we all rely on contains systematic error, some level of systematic error in our forecasting work is to be expected. Rather than abandon ship and proclaim forecasting as a doomed and dead enterprise, we should  continue to analyse and seek to improve our methods and give time for polling to re-calibrate and readjust. 

As a summary of this rather long piece:
  1. Rumours of the death of academic forecasting are much exaggerated: some polling/swing based models were much more accurate than others, and academic work continues to be very informative and useful. 
  2. Rather than academic methodology or knowledge being fundamentally flawed, the large underestimation of Labour Party voting intention in GE2017 polling was the leading cause in forecasting error - particularly for those closer to the mark. 
  3. The PME Politics uniform swing model was actually closer to the overall total Conservative seats than even the Lauderdale/YouGov model - which is not a good thing. 
  4. The PME Politics 'Polls Plus' model was thrown off by including a polling-error parameter set in the wrong direction (anticipating an undershot in Tory voting intention), and under-estimating the transfer of UKIP vote share to Labour. 
  5. Switching the polling-error parameter directly around in the 'Polls Plus' model produces a near perfect estimation of the Conservative seat total - 317.
  6. The PME Politics 'Polls Plus' model was the second most accurate GE2017 forecast, captured the upward drift of the Labour Party much quicker than most, and its regional modelling approach performed very well and is such a recommendation which I would urge future forecasters to use.

Differential Forecasting Error - Some Models Performed Much Better than Others

Firstly, the idea that all forecasters were equal in their miss, that we should all be thrown into the same scrap bucket, is simply not true. Firstly, and most obviously, the Lauderdale/YouGov forecast correctly projected the hung parliament, with a central forecast of 302 seats for the Conservatives.

The method employed in that model was use of multilevel regression analysis to predict outcomes constituency by constituency. This differs to polling average/uniform swing methods which the vast majority of forecasters - including myself - currently use. It also only included YouGov polls, which turned out to be among the most accurate this time around.

The much lauded (and rightly so) Lauderdale/YouGov forecasting model did predict a hung parliament, but was itself fairly off with its total Conservative seat projection (and thus their proximity to a Commons majority). Their final forecast of 302 seats for the Tories (not to be confused with an on the day poll which made a last minute change in methodology which boosted the Conservative Party lead to 7 points) was 16 seats away from the party's final figure.

16 away is by far the closest anyone managed to get, and combined with the correctly predicted result (hung parliament) makes Lauderdale/YouGov's forecast by far and away the GE2017 winner.

But were others really that much further off?

Separate forecasts produced by myself, Steven Fisher and Rosie Sharrocks, and Michael Thrasher were the next closest to the overall result, and projected a total Conservative Party seat count of 348, 349 and 350 (if you include the Speaker in the Conservative column, as the previous two did and as on-the-night results flows do) respectively. These forecasts thus overshot the projected Conservative seat total by 30, 31 and 32 seats. Twice the distance from the final total as Lauderdale/YouGov.

While half as good as the Lauderdale/YouGov model, these forecasts were not wildly wrong. The models used by each were much informed by ongoing, longstanding research into election forecasting and analysis, combining contemporary academic knowledge and research with polling estimates.

Other forecasts were indeed far wider of the mark. Some forecasted majorities for the Conservatives of around 100 seats (such as by Financial Times Election Analyst Matt Singh) and others close to 125 (such as by Ian Warren at Election Data). This is an overshot of around 60 and 70 respectively.

In short, there were degrees of error within our GE2017 forecasts, and I would argue that polling/swing based models have not yet had their day. That said, there is clearly an awful lot that we can learn from the kinds of models used successfully by both Lauderdale/YouGov and Chris Hanretty (EU Ref) before.

Polling Errors as Root Cause of Forecasting Error

What is also worth remembering is this: we are indeed in uncharted territory, and errors in the principal data which we all rely on to forecast with - polls - are still present. As the above introduction highlighted, this General Election broke many molds and conventions: the ability of opposition parties and their leader to recover from terrible polling ratings, campaigns (and particularly manifestos) making a difference, and polling significantly overestimating a Conservative Party lead.

This final point has much to do with why average polling/swing based estimates were as far off as they were (to varying degrees, as highlighted above).

Many of us, including myself, were correcting in our forecast models for an underestimate of the Conservative Party vote share, in line with what happened in 2015. In truth, while many of us were sceptical that the pollsters had fixed the issues raised by the Sturgis report into the 2015 polling miss, it appears that many of them actually substantially 'over-corrected', and were thus producing estimates of Labour Party voting intention that were far too low - almost 10 points too low in the case of BMG.

Simply put: we should expect average polling/swing based models to be inaccurate if polling continues to be inaccurate. Forecasts in 2015 were off largely because polling was not adequately capturing Conservative Party voting intention. Forecasts in 2017 were off largely because polling was not adequately reporting Labour Party voting intention.

It was a mistake for us to assume that the same error would continue from 2015 into 2017. Perhaps we could have taken a closer look at the changes made by pollsters since the Sturgis report, but I do not believe that we could have reasonably forecasted that the error would have flipped right around.

Diagnosing the PME Politics Forecasting Model

Turning to my own forecast, the worst fear of any modelling diagnostic was indeed realised in my case: the standard, flat uniform swing (UNS) model outperformed it (at least on the Conservative and Labour seat counts). That is to say, my attempts to make uniform swing more accurate actually made the forecast less accurate.

The final UNS model, which took the average swing to/from each party implied by the average of all polling and applies it to every 2015 constituency result in the country, projected a result of: Conservatives (332), Labour (237), Lib Dems (5), UKIP (0), SNP (54), Plaid (3) and Greens (1).

The UNS model was actually more accurate even than the Lauderdale/YouGov model; 332 is 14 seats away from the final Conservative total of 318, whereas the Lauderdale/YouGov model was 16 seats off. 

The final PME Politics 'Polls Plus' forecast by comparison projected: Conservatives (348), Labour (224), Lib Dems (10), UKIP (0), SNP (46), Plaid (3) and Greens (1).

So the UNS swing model predicted the Conservative and Labour Party seat totals much better, but the 'Polls Plus' model much more accurately predicted the SNP and Liberal Democrat seat totals.

Two quick fixes to the 'Polls Plus' model can provide much more accurate seat totals: removing the polling error parameter (which moved swing away from Labour and towards the Conservatives), and increasing the projected transfer of UKIP voters to Labour (initially set at 0.2, now set at 0.3).

These changes produce the following seat totals: Conservatives (330), Labour (239), Lib Dems (12), UKIP (0), SNP (47), Plaid (3) and Greens (1).

These two fixes could have been realistically anticipated and applied before the night, and would have greatly improved the forecast's accuracy (and indeed make it outperform the UNS model). 

A single and simple fix however, which could not have realistically have been foreseen, produces a near perfect result. Directly flipping the polling-error parameter around, so that it anticipates the impact of polls underestimating Labour (rather than the other way around as in 2015), moves the projected seat totals to the following: Conservatives (317), Labour (253), Lib Dems (11), UKIP (0), SNP (47), Plaid (3) and Greens (1).

To be clear, this is the exact same model but rather than average, pre-modeled swing being increased slightly in the Conservative direction, it is instead increased in the Labour direction, and by the same factor.

This is evidence that we should not abandon polling average or swing based models in favour of others: if polling averages become more accurate again in future, then these types of models will not miss in such a fashion as they have done this time around.

Where the 'Polls Plus' Model Performed Well

Despite calling the result incorrectly, and remaining 30 seats adrift of the final Conservative Party seat share, there were some elements of the model's performance which I am pleased with.

Firstly, the model responded quickly (and as it turned out accurately) to the narrowing of the Conservative Party lead over the campaign period. While it started out in similar territory as the aforementioned forecasts by Matt Singh and Ian Warren, the projected Conservative seat lead narrowed sharply as polling indicated a resurgent Labour Party was on its way to closing the gap. The final 'prediction tracker' graph below demonstrates the models' responsiveness to the campaign.

Secondly, the 'Polls Plus' model calculated and applied separate swing estimates for each of Britain's major regions: Scotland, Wales and London. These were informed by regional polling, mostly produced by YouGov. Without the separate calculations, the projected Conservative majority increases by around 10 seats. Thus, a regional approach to election modelling is certainly a recommendation I would make to future forecasting.

And in Conclusion...

There is no one approach to forecasting which has failed, or one which prevails - save of course for the Exit Poll managed by Professor John Curtice. Beyond this, there are no forecasting methods that we can be absolutely certain of. Beyond this, there are no election gurus.

As a political science community, we all have a lot to take away from this last General Election, and indeed still the one before, when it comes to our forecasting. Clearly, the success of the Lauderdale/YouGov multilevel regression method speaks for itself, but let's not write off average polling/swing based models quite yet - and certainly not while polling itself remains inaccurate.

As demonstrated above, a simple adjustment to counterbalance the underestimation of Labour voting intention in itself flipped the polling/swing based PME Politics 'Polls Plus' forecast from a Conservative majority to a near perfect projection of the result.

No comments:

Post a Comment