Friday, 22 December 2017

Political Studies 'Special Recognition' Award - Thank you.

This month I was honoured and thrilled to receive a Special Recognition award from the Political Studies Association alongside my exit poll colleagues.

Photo: Miranda Nunhofer... who also made a very good point about our (lack of) gender diversity!


Myself, Dr. Jonathan Mellon, and Professors John Curtice, Rob Ford, Steven Fisher, Jouni Kuha, Michael Thrasher, and Collin Rallings were all recognised by the Political Studies Association for our efforts in constructing and developing the exit poll during the day of the June 23rd 2017 election, and informing coverage of the election throughout the night and into the next day (it was a long old shift!).

Firstly, many many thanks to the Political Studies Association for the award!

It was my second time doing the exit poll (having joined the team in 2015), and the second time that we had released a shock prediction (Conservatives largest party, maybe majority in 2015, Conservatives largest party, no majority 2017) which - this year to a particular extent - proved to be pretty on the money. A spurious correlation, I'm certain!

But nonetheless, it has been brilliant to have each of our contributions to this amazing success recognised by the academic community. As John Curtice said in his acceptance speech, every single one of us brings something unique to the table and works absolutely flat out on the night and the week(s) proceeding it to bring the prediction and analysis together.

My job in the team is primarily to construct the database of contextual and political variables associated with each of our 650 constituencies before the election itself. On the night, I work hard running analysis and spotting trends that might make for interesting discussion and reporting on the election program. Also, in a new development this year, I had the fantastic job of trying to keep on top of and summarise the Twitter rumour mill and seeing how reports at counts and private meetings were matching up with our predictions!

It is an academic exercise at our end, but the entire operation would fall apart without the huge range of non-academic staff involved in and integral to every part the process - the exit poll really is one of those rare grounds where academia, the professional world (the BBC, ITV, Sky, Ipsos-Mori, Gfk), and the public (voters and viewers) truly, truly meet.

For all of the us, the planning and preparation for the exit poll (and for our work on the night covering the trends and patterns of results as they come in) takes months of work, hundreds (probably thousands) of emails and phone conversations, and a whole host of trial, retrial, error, and last minute 'save-the-day' fixes. Fortunately, we are blessed with an amazing squad of academics and a brilliant results team at the BBC who bring the whole thing together for the day and on the night. The actual polling is handled jointly by Ipsos-Mori and GfK, and is of the highest quality, organisation, and reliability. When those reports come into us during the day, we don't even think about second guessing them.

On the night of the results, we (the academics) are just one part of a big team of people in the BBC studio who are responsible for coordinating and managing the flow of results and predictions back and forth. Again, nothing is possible without those guys. Special shout outs go to Producer Tim Hammond, operational-organisers-in-Chief Rosie Sheed, Julia Walker (née Sherman), and Lizz Loxam, all the results inputters and BBC researchers, and Elect Systems Ltd gurus Richie Butler and Matt. There are so many more who help out the team and make the show what it is from every other conceivable angle - but the above people are the ones who I have contact with and make everything I do possible in particular. Thank you to everyone.

The exit poll and on-the-night election coverage operation is massive, and while it's amazing to be recognised for it, there's no pretending for a second that I (or any of us) would have got anywhere near this award without everyone mentioned above. Thank you.

Finally, I can't do a 'thank you' piece without acknowledging the role that my family, friends and non-exit poll colleagues have played, who have constantly and consistently been supporting and pushing me throughout my academic journey.

My father immigrated to this country in the 1980s with £10 in his back pocket, and the promise of a job as a security guard at a South London firm. My mother is the daughter of a travel agent (and later a Police Constable - the job my father also went into after I was born) and a typist. They (the whole family) worked tremendously hard to raise me and to give me the best chance in life - and taught me a valuable lesson in dedicating yourself to be the best version of you that you can be, and never to give up.

A huge thanks to the teachers at my schools - particularly William Farr Comprehensive School (as it was then) and 6th Form College, where the maths and history departments worked tirelessly with me keeping me on track, and spotting my interest and talent guided me into a life of studying politics and numbers.

A huge thanks to all my University Supervisors and Tutors who have also worked very hard keeping me on the academic straight and narrow (always had the glimmer in my eye of making it as the next Lionel Messi... nowadays it's more Steph Curry) - Dr Alistair McMillan, Professor Maria Grasso, Dr Maria Sobolewska, and Professor Rob Ford. The latter two were the ones who got me onto the exit poll team in the first place.

Lastly, my partner, Emma, my close friends, and colleagues too deserve special mentions for putting up with the lows and enjoying the highs with me. I would not have had my name up on that screen without them all.

So I want to dedicate this award (or perhaps more accurately - my 1/10 of it) to everyone who has put their faith and time into me.

But this is just the beginning - the road ahead to academic relevance and success is long and winding, and I fully intend to follow each and every one of its twists and turns. The next stop - PhD hand-in just over a month away!!!

Saturday, 17 June 2017

On #GE2017 Forecasting and Diagnosing the 'Polls Plus' Model

The 2017 General Election campaign, result and fallout are each undeniably remarkable.

How a governing party was able to go from a 20-point lead to a lead of just over 2 points (constituting a swing towards their opposition) over the campaign period defies all previous knowledge about the impact of campaigns, and the ability of parties to turn their fortunes around so dramatically in just six weeks.

Similarly, the dramatic improvement in the personal approval rating of Jeremy Corbyn is beyond anything we have seen before - previous wisdom dictated that first impressions were everything.

Also, for only the second time in modern polling history, polling undershot Labour Party voting intention by a significant margin - around 5% on average. Rather than there being an 8-point Conservative lead as suggested by the final polling average, the Conservatives were just 2.5% ahead on the day.

The resulting hung parliament, with the Conservatives short of an absolute majority by seven seats, was an outcome forecasted by only one - Ben Lauderdale for YouGov.

Many are suggesting that the collective failure of the political academic and wider data-driven community to predict the oncoming hung parliament (save for Lauderdale/YouGov) is indicative of some chronic level of problems. That there is something now structurally or fundamentally wrong with political science regarding election analysis and forecasting. In short: we simply don't get it any more.

I disagree.

Firstly, it is right that we acknowledge where we went wrong, where we can improve, and admit that currently we are still some way away from where we would like to be in terms of our reading of the current political climate and prediction of outcomes.

That said, some forecasts were simply not as far off as others, and while the data which we all rely on contains systematic error, some level of systematic error in our forecasting work is to be expected. Rather than abandon ship and proclaim forecasting as a doomed and dead enterprise, we should  continue to analyse and seek to improve our methods and give time for polling to re-calibrate and readjust. 

As a summary of this rather long piece:
  1. Rumours of the death of academic forecasting are much exaggerated: some polling/swing based models were much more accurate than others, and academic work continues to be very informative and useful. 
  2. Rather than academic methodology or knowledge being fundamentally flawed, the large underestimation of Labour Party voting intention in GE2017 polling was the leading cause in forecasting error - particularly for those closer to the mark. 
  3. The PME Politics uniform swing model was actually closer to the overall total Conservative seats than even the Lauderdale/YouGov model - which is not a good thing. 
  4. The PME Politics 'Polls Plus' model was thrown off by including a polling-error parameter set in the wrong direction (anticipating an undershot in Tory voting intention), and under-estimating the transfer of UKIP vote share to Labour. 
  5. Switching the polling-error parameter directly around in the 'Polls Plus' model produces a near perfect estimation of the Conservative seat total - 317.
  6. The PME Politics 'Polls Plus' model was the second most accurate GE2017 forecast, captured the upward drift of the Labour Party much quicker than most, and its regional modelling approach performed very well and is such a recommendation which I would urge future forecasters to use.

Differential Forecasting Error - Some Models Performed Much Better than Others

Firstly, the idea that all forecasters were equal in their miss, that we should all be thrown into the same scrap bucket, is simply not true. Firstly, and most obviously, the Lauderdale/YouGov forecast correctly projected the hung parliament, with a central forecast of 302 seats for the Conservatives.

The method employed in that model was use of multilevel regression analysis to predict outcomes constituency by constituency. This differs to polling average/uniform swing methods which the vast majority of forecasters - including myself - currently use. It also only included YouGov polls, which turned out to be among the most accurate this time around.

The much lauded (and rightly so) Lauderdale/YouGov forecasting model did predict a hung parliament, but was itself fairly off with its total Conservative seat projection (and thus their proximity to a Commons majority). Their final forecast of 302 seats for the Tories (not to be confused with an on the day poll which made a last minute change in methodology which boosted the Conservative Party lead to 7 points) was 16 seats away from the party's final figure.

16 away is by far the closest anyone managed to get, and combined with the correctly predicted result (hung parliament) makes Lauderdale/YouGov's forecast by far and away the GE2017 winner.

But were others really that much further off?

Separate forecasts produced by myself, Steven Fisher and Rosie Sharrocks, and Michael Thrasher were the next closest to the overall result, and projected a total Conservative Party seat count of 348, 349 and 350 (if you include the Speaker in the Conservative column, as the previous two did and as on-the-night results flows do) respectively. These forecasts thus overshot the projected Conservative seat total by 30, 31 and 32 seats. Twice the distance from the final total as Lauderdale/YouGov.

While half as good as the Lauderdale/YouGov model, these forecasts were not wildly wrong. The models used by each were much informed by ongoing, longstanding research into election forecasting and analysis, combining contemporary academic knowledge and research with polling estimates.

Other forecasts were indeed far wider of the mark. Some forecasted majorities for the Conservatives of around 100 seats (such as by Financial Times Election Analyst Matt Singh) and others close to 125 (such as by Ian Warren at Election Data). This is an overshot of around 60 and 70 respectively.

In short, there were degrees of error within our GE2017 forecasts, and I would argue that polling/swing based models have not yet had their day. That said, there is clearly an awful lot that we can learn from the kinds of models used successfully by both Lauderdale/YouGov and Chris Hanretty (EU Ref) before.

Polling Errors as Root Cause of Forecasting Error

What is also worth remembering is this: we are indeed in uncharted territory, and errors in the principal data which we all rely on to forecast with - polls - are still present. As the above introduction highlighted, this General Election broke many molds and conventions: the ability of opposition parties and their leader to recover from terrible polling ratings, campaigns (and particularly manifestos) making a difference, and polling significantly overestimating a Conservative Party lead.

This final point has much to do with why average polling/swing based estimates were as far off as they were (to varying degrees, as highlighted above).

Many of us, including myself, were correcting in our forecast models for an underestimate of the Conservative Party vote share, in line with what happened in 2015. In truth, while many of us were sceptical that the pollsters had fixed the issues raised by the Sturgis report into the 2015 polling miss, it appears that many of them actually substantially 'over-corrected', and were thus producing estimates of Labour Party voting intention that were far too low - almost 10 points too low in the case of BMG.

Simply put: we should expect average polling/swing based models to be inaccurate if polling continues to be inaccurate. Forecasts in 2015 were off largely because polling was not adequately capturing Conservative Party voting intention. Forecasts in 2017 were off largely because polling was not adequately reporting Labour Party voting intention.

It was a mistake for us to assume that the same error would continue from 2015 into 2017. Perhaps we could have taken a closer look at the changes made by pollsters since the Sturgis report, but I do not believe that we could have reasonably forecasted that the error would have flipped right around.

Diagnosing the PME Politics Forecasting Model

Turning to my own forecast, the worst fear of any modelling diagnostic was indeed realised in my case: the standard, flat uniform swing (UNS) model outperformed it (at least on the Conservative and Labour seat counts). That is to say, my attempts to make uniform swing more accurate actually made the forecast less accurate.

The final UNS model, which took the average swing to/from each party implied by the average of all polling and applies it to every 2015 constituency result in the country, projected a result of: Conservatives (332), Labour (237), Lib Dems (5), UKIP (0), SNP (54), Plaid (3) and Greens (1).

The UNS model was actually more accurate even than the Lauderdale/YouGov model; 332 is 14 seats away from the final Conservative total of 318, whereas the Lauderdale/YouGov model was 16 seats off. 

The final PME Politics 'Polls Plus' forecast by comparison projected: Conservatives (348), Labour (224), Lib Dems (10), UKIP (0), SNP (46), Plaid (3) and Greens (1).

So the UNS swing model predicted the Conservative and Labour Party seat totals much better, but the 'Polls Plus' model much more accurately predicted the SNP and Liberal Democrat seat totals.

Two quick fixes to the 'Polls Plus' model can provide much more accurate seat totals: removing the polling error parameter (which moved swing away from Labour and towards the Conservatives), and increasing the projected transfer of UKIP voters to Labour (initially set at 0.2, now set at 0.3).

These changes produce the following seat totals: Conservatives (330), Labour (239), Lib Dems (12), UKIP (0), SNP (47), Plaid (3) and Greens (1).

These two fixes could have been realistically anticipated and applied before the night, and would have greatly improved the forecast's accuracy (and indeed make it outperform the UNS model). 

A single and simple fix however, which could not have realistically have been foreseen, produces a near perfect result. Directly flipping the polling-error parameter around, so that it anticipates the impact of polls underestimating Labour (rather than the other way around as in 2015), moves the projected seat totals to the following: Conservatives (317), Labour (253), Lib Dems (11), UKIP (0), SNP (47), Plaid (3) and Greens (1).

To be clear, this is the exact same model but rather than average, pre-modeled swing being increased slightly in the Conservative direction, it is instead increased in the Labour direction, and by the same factor.

This is evidence that we should not abandon polling average or swing based models in favour of others: if polling averages become more accurate again in future, then these types of models will not miss in such a fashion as they have done this time around.

Where the 'Polls Plus' Model Performed Well

Despite calling the result incorrectly, and remaining 30 seats adrift of the final Conservative Party seat share, there were some elements of the model's performance which I am pleased with.

Firstly, the model responded quickly (and as it turned out accurately) to the narrowing of the Conservative Party lead over the campaign period. While it started out in similar territory as the aforementioned forecasts by Matt Singh and Ian Warren, the projected Conservative seat lead narrowed sharply as polling indicated a resurgent Labour Party was on its way to closing the gap. The final 'prediction tracker' graph below demonstrates the models' responsiveness to the campaign.


Secondly, the 'Polls Plus' model calculated and applied separate swing estimates for each of Britain's major regions: Scotland, Wales and London. These were informed by regional polling, mostly produced by YouGov. Without the separate calculations, the projected Conservative majority increases by around 10 seats. Thus, a regional approach to election modelling is certainly a recommendation I would make to future forecasting.

And in Conclusion...

There is no one approach to forecasting which has failed, or one which prevails - save of course for the Exit Poll managed by Professor John Curtice. Beyond this, there are no forecasting methods that we can be absolutely certain of. Beyond this, there are no election gurus.

As a political science community, we all have a lot to take away from this last General Election, and indeed still the one before, when it comes to our forecasting. Clearly, the success of the Lauderdale/YouGov multilevel regression method speaks for itself, but let's not write off average polling/swing based models quite yet - and certainly not while polling itself remains inaccurate.

As demonstrated above, a simple adjustment to counterbalance the underestimation of Labour voting intention in itself flipped the polling/swing based PME Politics 'Polls Plus' forecast from a Conservative majority to a near perfect projection of the result.

Thursday, 8 June 2017

#GE2017 Night - Oh What a Night!

As the dust settles on the election, it's time to write up my thoughts.

Quite frankly, who but a few could have predicted yesterday that the Conservatives would be losing their majority?

The exit poll did. And as part of the exit poll team, I have to say a huge congratulations to everyone on the team. From Psepho-in-Chief John Curtice, to fellow Curtice-minions Steve Fisher, Rob Ford, Jon Mellon, and Jouni Kuha. To our producer Tim and typist Tracey. To everyone involved in the operation at GfK and Ipsos -- and to all those who answered the exit poll!

We were delighted that everyone's immensely hard work paid off, and that we were able to successfully predict the outcome and guide the coverage on the night.

As for the result: only YouGov (Ben Lauderdale) had a hung parliament as their central forecast (before the exit poll). My own forecast suggested a Conservative seat total of 348 - a full 30 seats too high. It was however the closest model other than YouGov's that I know of (please do correct me if I am wrong there). I will run a full diagnostic on the model and write a report over the coming days.

In the meantime, full kudos to YouGov and Ben Lauderdale for their success in predicting the hung parliament.

To be where we are now given where we started when the snap election was called is quite incredible.

What happened since forecasts of a 100 seat majority for the Conservatives in May is:
1) the Conservatives put together a dismal campaign,
2) the Labour Party put together an impressive campaign and manifesto which confounded their critics, and
3) the majority of polls were highly erroneous by assuming and weighting to a far too low turnout of young people. This was, we all think, a real central part to the story: young voters coming out and winning Labour those key seats.

In truth, Labour were up mostly everywhere. It was a good performance (compared to the 2015 baseline). We can however draw out some interesting stories which begin to unravel a bit the complex and dynamic picture we now see before us on the British electoral map.

As well as some clear and well established stories about age and voting (swing to Labour was around 5% in seats with more than 10% of young people (aged 18-24), compared to around 2 in seats where there were less than 5% young people), there were also two other clear dynamics: regional performances and the Brexit vote.

On the first, the North East of England was the only region where the Conservatives out-performed Labour outside of Scotland. Here they made a serious impact in terms of winning over UKIP voters, and flipping seats (Middlesborough South and Cleveland and Southport, for example).

In Scotland, the Scottish Conservatives continued their strong advances under Ruth Davidson and have undoubtedly successfully established themselves as the second party there. Indeed, the Conservatives' ability to govern from this point are entirely thanks to their wins north of the border, 12 in all, without which they would be much too far adrift.

Elsewhere however, Labour were very much on top and achieved large swings from both the Conservatives and UKIP. Remarkably, the assumption that the vast majority of UKIP voters would go to the Conservatives simply didn't hold up outside Northern seats. The swings in the South and East in particular were mightily impressive (3.5% in the South East region for instance), knocking over seats which never should have been on the cards such as Ipswich and Canterbury.

The Conservatives also did well in high leave voting areas, particularly those in the North West and Midlands. In midlands seats where the Brexit vote was higher than 60%, the Conservatives rose on average by 10.5 points. This again did win them some contests, such as Walsall North and Mansfield.

Conversely, Labour were up on average 12 points in seats across the nation where the remain vote share was greater than 65%. They picked up wins across London, a high remain voting area, but also in Wales and the East where leave was in the majority.

For the Liberal Democrats, they will be pleased to have increased their overall seat numbers but their loses in England and Wales must be of great concern. They appeared to do well in England in high remain voting areas, against the Conservatives, and with well known candidates. Elsewhere, and particularly against Labour incumbents, they did not do well. In Scotland, the Lib Dems simply held up in many places while the SNP crashed all around them (which is the only real story there).

Young voters, the regions, and Brexit. What a night.

Final PME Politics #GE2017 Forecast - Conservative Overall Majority of 46 (348 Total Seats)

As of this morning, updated polling both nationally and regionally has been included into the model to produce the final PME Politics forecast.

The 'polls plus' model projects a result of:

Conservatives: 348
Labour: 224
Lib Dems: 10
UKIP: 0
Greens: 1
SNP: 46
Plaid: 3

This is based on an estimated swing of around +9 for the Conservatives on their 2015 results. The Butler swing at the aggregate level would therefore be -1 (a swing towards Labour).

Accounting for forecasting errors, the model is projecting a Conservative majority of somewhere between 60 and 30 seats, with the central projection being about 45.

After the results come in, I will run a full disgnostic on the model and write a piece discussing its accuracy, what worked well and what didn't, and what the next PME Politics forecast might look like.

Wednesday, 7 June 2017

#GE2017 - What Will Happen to Student Seats?

This is a quick note on something I expect to see come out in election results analysis tomorrow - the dispersal of student voters out from 'student seats' into the wider electorate.

A good deal of Universities are now finished for the summer, or are about to at the end of this week. This means that instead of voting in their term-time, 'student seats' (such as Sheffield Central, Manchester Central, and early declarers such as Newcastle Central), a whole bunch of student voters may be dispersing back into the student population.

This effect will be quite dramatic at least on the 'student seats' themselves. Take Sheffield Central for example - here, according to the 2011 Census, students made up 38% of the population (the highest in the country). But the population of students usually resident in the constituency is just 16% according to the same Census data. This suggests that there could well be under half the amount of students voting in Sheffield Central tomorrow than otherwise might in a May General Election.

This effect will be repeated across all student towns and cities. In Manchester Central the figures stand at 30% and 15% respectively, 31% and 14% in Liverpool Riverside, 28% and 12% in Cambridge, the list goes on and on.

This has two important consequences, both of which may well emerge as the election results come in over the course of Thursday night and Friday morning.

Firstly, we may see a substantial drop in turnout in student seats. Newcastle Central will declare early (probably 1st, I am told). Here, around 20% of the population are students. If even half of them are not voting in the seat (term time officially finishes on Friday), then we could see turnout decrease there dramatically. We should not however interpret this as indicative of a huge fall in turnout across the nation - it will only be student voters voting elsewhere.

Secondly, and perhaps less likely and definitely harder to detect, this dispersal of student voters could mean that instead of Labour 'racking up' votes in safe, student seats, these extra votes could well diffuse into the wider electorate. Though it is true that the highest concentration of this diffusing is likely to be into seats with younger, more educated populations (which naturally provide more students) which also would be more likely to be Labour strongholds.

That said, if a few hundred students are returning to vote in constituencies such as Derby North and Ealing Central and Acton, then if the result is tight as some polls are suggesting, they could well be the difference in these key battleground seats.

Much of this also depends on whether or not the students themselves will stick around to vote in student seats, or had the foresight to organise voting at home. It is certainly something to look out for!

The Other Side of the Exit Coin - Green Party Saving (Veggie) Bacon

Much has (quite rightly) been made of the potential impact that UKIP's exit in over 250 seats could have on Labour's ability to hang on in marginals up and and down the country. What is being discussed much less, and might be of crucial importance if the result does end up becoming as close as some pollsters and forecasters are projecting, is the impact that Green Party exits might generate.

The Greens are standing down in around 100 seats across the country (where they previously had contested in 2015). In 12 of such seats, the party posted a vote share of over 5%, from 5.1% in Cambridgeshire South East to 10% in York Central.

45 of the total seats in which the Green Party are exiting are tight marginal contests (where the swing required for them to change hands is less than 5%).

4 of the top 12 seats (where the Green vote share is larger than 5%) are marginals, including Brighton Kemptown and Lewes, and they are standing down in other crucial marginals where Labour are currently just about holding on - including the London seats of Brentford and Isleworth and Ealing Central and Acton.

Each of these seats typify a contest where Green exits could make all the difference:

In Brighton Kemptown, the Conservatives lead Labour by 1.5 points. The Green vote share in 2015 was 7%. If Labour are successfully able to mobilise even a third of those Green voters to turn out and put a cross by Lloyd Russell-Moyle's box tomorrow, then Labour would take the seat. Even in the face of a significant Tory swing, Green voters would still be enough to take Labour over the line and knock off a seat from the Conservative column.

In Lewes, the Liberal Democrats are campaigning hard to tack back the seat occupied by Norman Baker until the 2015 election. Then, the Conservatives took it by 2.1%, while the Greens achieved a vote share of 6.3%. So once again, if the Lib Dems are able to pick up those Green voters - whom the local party has specifically directed to vote for Kelly-Marie Blundell - then they may well be celebrating come Friday. The same is true of St Ives, another Con-Lib Dem marginal seat where the Greens achieved 5%+ in 2015 but are now standing down.

Finally, though in Brentford and Iselworth and Ealing Central and Acton the Greens racked up only around 3.5% of the popular vote, the Labour Party won these London battlegrounds in 2015 by the slimmest of margins - 1% and 0.5% respectively. As the Conservatives attempt to flip these seats - and others in the midlands such as Newcastle-Under-Lyme and Northern seats such as Halifax - they may well find that tactical Green voters provide an extra barrier which their candidates fail to breach.

Of course, in the face of a Conservative landslide, Green exits may well stop very little. But if the vote is indeed closer than the whitewash forecasted at the beginning of the campaign, tactical Green voters in marginal seats where their party has exited - and indeed where they have not - could make all the difference to the Conservative seat total.






Tuesday, 6 June 2017

Updated #GE2017 Forecast - Conservative Majority Increased to 46 (348 Seats)

Today's updates include adjustments to Scottish swings based on a new YouGov poll north of the border.

For the penultimate PME Politics forecast, a 3-day average of polls was taken and combined with the contextual model to produce the first increase in the predicted Conservative majority (compared with the last forecast) since early May.

The Uniform Swing (UNS) model calculated the following distribution of seats: Conservative (332), Labour (237), Lib Dems (5), UKIP (0), SNP (54), Plaid (3) and Greens (1). Again, this would constitute a more-than-modest increase of the 2015 Conservative majority of just 2 seats.

The implied swing from the last three days of polling has both Labour and the Conservatives up by around 6.5 points (from 2015). In practice, this would be a swing of around about 3% towards Labour.

The 'polls plus' model produced the following projected result:


This moves the Conservatives up by 3 seats from yesterday's forecast, reflecting a slight increase in their polling average, and more favourable Scottish swings (namely the SNP moving down a touch).

In fact, the SNP's current projected decline is now steep enough to suggest that Labour will take back Renfrewshire East from the nationalist party, according to the 'polls plus' model. 

The first expansion of the Tory lead in just under a month can be seen on the prediction tracking graph below.


The 'polls plus' model moves the projected marginals up one seat into the Labour defense list, with the Conservatives just predicted to take Wakefield by a threat, based on the current polling and contextual factors applied in the model.



Once again though - health warning - this function of the forecast is very much exploratory, and indicates the kind of seats that we might expect to currently be the battlegrounds, rather than constituting a firm prediction of the outcomes there.


Monday, 5 June 2017

Updated #GE2017 Forecast - Conservative Majority Cut Again to Just 40 (355 Seats)

Today's forecast contains an additional regional swing differential for London, which polling suggests is moving in a much different direction to that of the rest of the country. 

Today's #GE2017 Forecast is the last to use a 5-day rolling uniform swing, with future daily forecasts using a 3-day rolling average as the campaign and polling picks up speed moving towards the final day. There will also be a PME Politics Forecast on 'election-day-eve' which will use polls from June 7th only.


A simple uniform swing model using polls published in the last 5 days produces the following result: Conservatives (332), Labour (328), Lib Dems (4), UKIP (0), SNIP (54), Plaid (3), and Greens (1).

The averaging function of the most recent polls implies a Conservative swing of around +8.5, and a Labour swing of about +5%. This moves the result much closer together than that which the 6-point gap between the two parties produced in 2015.

An increase of just one seat and a majority of 14 for Theresa May would undoubtedly be seen as a very bad night.

The 'polls plus' model adds more seats to the Conservative's tally, but today predicts the smallest Conservative majority to date of just 40 seats.


Even a majority of 40 (compared to 14) would not be considered a good night for the Tories, given a) their position in the polls at the start of the campaign, and b) that this snap election was 'called' with the precise aim of giving the Prime Minister a large and commanding majority as she headed into the Brexit negotiations.

The projection tracking graph shows the predicted gap between Britain's largest political parties continuing to narrow, and also highlights the model's early pick-up of this trend (in the first week of May).



Finally, the predicted marginals continue to move back towards the 2015 result, with the model projecting that the fault line between Labour and the Conservatives when the dust settles on June 9th will be between Wakefield and Birmingham Northfield.

Croydon Central, a tight Tory-Lab marginal, is predicted to move into the Labour column - Labour are doing well in London according to recent polling. The 'progressive alliance' is also expected to tip Brighton Kemptown into Labour hands, with the Greens standing down and backing the Labour candidate. Although the reality with these forecasts is that the result in the seats below is currently too close to call.





Thursday, 1 June 2017

Updated #GE2017 Forecast - No Hung Parliament, but Conservative Majority of 60

Today's forecast includes an update to the Scottish swing mechanism to reflect a recent ICM poll which has the Conservatives on 25% north of the border, contrary to earlier polls placing them on around 30% across Scotland.

This week, YouGov released their forecast model, the central projection for which was a 'hung parliament'. Firstly, credit to YouGov for being bold. It always benefits the field when different approaches are tried, and when there is genuine diversity in methods and projections. Succeed or fail, this will only further our knowledge and accuracy in predicting elections.

While the hung parliament prediction obviously raised eyebrows, even a simple Uniform Swing (UNS) model using their most recent poll (31st May) which has the Conservatives just 3 points ahead (Con swing +3, Lab swing +9), would indeed lead to no overall majority result: Conservatives on 314, and Labour on 256.

The 5 day rolling average UNS model used as the basis of this forecast includes this YouGov poll and others dating back to May 26th. It produces the following result: Conservatives (341), Labour (227), Lib Dems (5), UKIP (0), SNP (55), Plaid (3), and Greens (1).

The 'polls plus' model, which combines a range of contextual and electoral information into the forecast, estimated a Conservative majority of 60, having gained 24 seats.


This result continues the trend of falling Conservative seat total projections in this forecast, which is visualised in the tracker graph below. The PME Politics forecast continues to sit in between the majority of academic forecasts (which still mostly project a majority of around 100), and the YouGov model.



The fall in the projected Conservative share in Scotland (according to a new ICM poll) brings them down to 5 Scottish seats, with Angus Roberston in Moray now looking safe.

The Con-Lab marginals line has shifted only slightly, with the model estimating that Labour will hang on in Bristol East, while the Conservatives take Scunthorpe.




Friday, 26 May 2017

Updated #GE2017 Forecast - Conservative majority of 64 Seats

**Note: Today's forecast contains three important updates. Firstly, an extra mechanism is introduced to try and counter for the traditional overstating of Labour Party and underestimation of Conservative Party support in British polling. Secondly, there are updates to Welsh and Scottish swings to take into account more recent polls. Finally, since we are moving closer to election day, the rolling average function has been tightened from 7 to 5 days.


This week, polling averages continued to move in Labour's direction over the week leading up to the Manchester terror attack, and the YouGov poll released last night had the two parties within 5% of each other for the first time in the race to date.

The story does now appear to be that Labour are set to improve on their 2015 vote share, achieved under former leader Ed Miliband.

The uniform swing model projected the following result from its now 5-day rolling average of current polling: Conservatives (344), Labour (223), Lib Dems (6), UKIP (0), SNP (55), Plaid (3), and Greens (1).

This would constitute an increased Conservative seat total of just 13 seats, and a majority of 38.

The 'polls plus' model, now including the 'Conservatives outperforming' mechanism as well as a host of other contextual factors (see methodology page for full information), produced the following prediction:




This result would give the Conservatives a majority of 64, with an improvement of 26 seats on their 2015 result. Though an increase, this would certainly not be the sort of outcome that Theresa May would have been gambling on when calling the early election back in April.

However, that Labour still find themselves so far behind in terms of seats, despite rallying in the polls and surpassing their 2015 benchmark will be frustrating for the party.

The prediction tracking graph below shows how this updated forecast represents a further narrowing of the projected result - and this is despite the inclusion of a new mechanism which certainly gives the Conservatives a boost compared to former predictions.



The predicted post-2017 marginals have thus swung back towards the 2015 line, with the current projection returning Bishop Auckland and Stoke-on-Trent North back into the Labour Party column. The line now sits between Bristol East and Harrow West, according to the 'polls plus' model.




As campaigns resume and further polling data is released, it will be fascinating to see what sort of impact the tragic events in Manchester on Monday have on the campaign. Will Labour's rise in the polls be halted? Can Theresa May escape the baggage acquired from a poorly received manifesto and associated screeching U-Turns?

Friday, 19 May 2017

Updated #GE2017 Forecast - Conservative Majority of 86

Today's forecast features a number of updates to the 'polls plus' model: namely a full account of where the Greens and UKIP are exiting in seats where they fought in 2015, and where incumbent MPs from 2015 are standing down. For full information on the effects that this information has on the model, see the methodology page of this website.

It has been a week since the last PME Politics forecast, when the projected Conservative majority dipped under 100 for the first time since the forecast was launched. Today, the seat projections move in further to give the Conservatives a majority of 86.

The continuing slide of the predicted Conservative majority is important - if this trend continues, we should soon abandon talk of a Conservative landslide (at least in terms of seats).

Over the past week, more polls than not have given Labour a positive swing compared to their 2015 vote share. In other words, in most of this week's polls voting intention for Jeremy Corbyn's Labour has surpassed Ed Miliband's 2015 result.

As such, the Uniform Swing (UNS) model, using the updated 7 day rolling average of Westminster voting intention polls, looks much better than previous for the Labour Party, and gives the following result: Conservatives 370, Labour 198, Lib Dems 5, UKIP 0, SNP 55, Plaid 3, Greens 1.

For the first time the UNS model has Labour close to the McCluskey 'good night' benchmark of 200 seats, reflecting the recent narrowing of the Conservative lead in the polls (though this narrowing is incredibly relative - even the closest polls still have Labour 15 points behind the Conservatives).

The 'polls plus' model does actually take Labour over the 200 mark for the first time in this forecast. The added contextual factors within the model suggest that just a few more Labour MPs will hang on in the face of the 'blue tide' than in the UNS scenario.

The full result from the 'polls plus' model is as follows:


The Lib Dems move perilously close to single figures in today's forecast, currently predicted to be only 2 seats better off than their disastrous 2015 result.

Despite Labour's improvement in the polls, the Conservatives continue to enjoy voting intention shares of close to 50%, thus maintaining a clear - if decreasing - lead in projected seats.




Finally, the 'polls plus' model predicts that the margin between Conservative and Labour seat wins now sits between Darlington and Coventry South. Interestingly, Copeland - taken by the Conservatives in that famous by-election earlier this year - is also close to the projected fault line between the country's two largest parties.

If Copeland were to fall the other way and once again return Trudy Harrison to Parliament, it would be no mean feat given past results in the seat at General Election time. Copeland's precarious position should serve as a reminder to jut how many traditionally solidly Labour seats still remain in jeopardy - polling rise or no. 




Friday, 12 May 2017

Updated #GE2017 Forecast - Conservative Majority of 96

Today's update of the #GE2017 Forecast came with an update to the redistribution of UKIP votes section of the 'polls plus' model (whereby votes are sent to other parties in seats where UKIP have withdrawn).

This section of the model now incorporates every constituency where UKIP are now standing aside (where they previously stood in 2015) in favour of the Conservatives, and the redistribution of their votes has been recalculated to reflect recent analysis conducted by Chris Hanretty of British Election Study data.

Interestingly, the model suggests that this updated list only moves one seat from the Labour to Conservative column (Chorley), such is the impact of their near 50% cut in the UKIP vote share at the national level. The model it seems was already reflecting Tory-UKIP gains by using the present (seemingly observable) swing of UKIP voters to the Conservatives in current polling

The uniform swing model calculated the following result:

Conservatives 376, Labour 190, Lib Dems 7, UKIP 0, SNP 55, Plaid 3, Greens 1.

This sizeable cut in the number of projected gains for the Conservatives reflects something of a revival recently in the Labour Party vote share across national polls. The 7-day rolling swing estimation (which both models use to predict seats) has Labour now just 0.3 points down from their 2015 result.

The story of this election seems to increasingly be solely about where those UKIP voters are going (over half of them appear to have already ditched the party, according to the polls), as opposed to any Labour to Conservative swing.

The 'polls plus' model moved the result to the following prediction:


The result represents Labour's highest seat projection of the forecast to date, reflective of a gradually declining swing away from the party in the polls since late April. It still projects however a healthy Conservative victory, giving them a majority of just short of 100 seats (96).

The forecast tracking graph shows the extent to which the gap between the parties has narrowed - not much in the grand scheme of the result, but certainly a significant shift in the projections.


The 'polls plus' model now projects the Conservative swing into Labour seats to stop at Southampton, Test. Ealing Central and Acton is projected to remain in Labour hands again for the first time since this feature of the forecast began. Chorley would also remain in Labour hands according to the model were it not for UKIP's withdrawal in the Lancashire seat, which is expected to push the Conservatives over the line there.


 


The news is certainly positive for Labour from a relative perspective, but they are still far from seriously denting the prospects of a significant Conservative majority in the House of Commons come June 8th.

Monday, 8 May 2017

Updated #GE2017 Forecast - Conservative Majority of 146

Today's forecast is the first update to PME Politics predictions since May 2nd. It includes the ICM poll released this afternoon, which gave the Conservatives a record (for ICM) lead of 22 points.

Much has happened since the last forecast, including last week's Local Election results (a resounding Conservative win) and Dianne Abbott's ill fated LBC interview.

Since then, the Conservatives have since apparently reestablished much of a polling lead which appeared to have been cut somewhat over the week previous by the Labour Party.

The rolling 7 day average of polls applied to the uniform swing model today produced the following result: Conservatives 403, Labour 165, Lib Dems 6, UKIP 0, SNP 54, Plaid 3, and Greens 1.

Applying the contextual and political variables to condition the swing according to various parameters in each seat using the 'Polls Plus' model resulted in a final projection of:


The projection has the Conservatives significantly up on the last forecast produced, with Labour someway down and both the SNP and Plaid also down a seat each. The Lib Dems are fairly stable, but seemed to have made some ground in recent polling which may see their forecasted seats rise, if it is sustained into later this week.



The current model forecast moves the Lab/Con marginal line further down the Labour path, putting the Tory Tanks right outside the outskirts of both Manchester (Worsley and Eccles South) and Sheffield (Penistone and Stocksbridge), having already blitzed through the previously safe Labour territories in the South of Bristol and Luton.




Just how might the Conservatives really deliver a result which sees them push towards 400 seats, only a few years after many questioned their ability to even build a majority again after the Thatcher and Major years?

The general pattern of last week's local election contest was of the Conservatives hoovering up UKIP votes in order to bypass the other parties and fly into 1st spot in over 500 council seats across the country. There is every reason to suspect that this may well happen again in next month's General Election, as former UKIP voters continue to abandon their previous party in support of Theresa May and her "Brexit means Brexit" agenda.

The number of Labour held seats in which the total Conservative plus UKIP vote share is greater than the Labour Party vote share is well over 150 (credit to Steve Fisher over at electionsetc.com for that one). If the Conservatives are once again successful in mopping up UKIP support, then the sky really may be the limit for the size of their landslide.

Friday, 5 May 2017

Thoughts from #LE2017 Part 4 - Conservatives Sweeping up the 'Left Behinds'

**Edit: in my dazed state at 9am on the morning after the elections, I incorrectly titled the manufacturing figure as 'manual labour'.

The Conservatives gradually winning over blue collar voters in England is nothing new - working class Conservatives have always existed. However, the rate at which working class voters are now seemingly traveling to the Conservatives it is certainly emerging as one of the stories of this local election contest - and will most likely be very prominent in the upcoming General Election in June.

As my colleague Professor Rob Ford pointed out, education continues to make its mark as a dividing line in contemporary British politics:




But as well as this, other indicators not just of social deprivation but social class are beginning to connect with Conservative Party swing. From the graph below we can see a steady upward relationship between Conservative Party swing (from 2013-2017) at the ward level in England and the percentage of workers in manufacturing (according to Census 2011 data). This is evidence of an ever-growing movement of working-class votes to Theresa May's Conservative Party. 



**Edit: in my dazed state at 9am on the morning after the elections, I incorrectly titled the manufacturing figure as 'manual labour'.

Additionally, deprivation also appears to play a role in increasing Conservative swing, with the % of people in each ward reporting to be in 'good health' (again according to Census 2011 data) relating negatively to Conservative Party vote share - in other words, when perceived health status is lower, so the swing to the Conservatives was higher. 



All of this points to one clear picture - a strong connection between social class and deprivation among voters, and Conservative Party swing. Put simply, the Conservative Party is hoovering up the 'left behind' voters, so many of whom flocked in their droves to UKIP just four years ago. 

However, this relationship doesn't hold north of Hardian's Wall. In fact, the opposite is true the other side of the border. For instance, in Scotland the Conservatives actually attracted greater swings in areas with lower rates of unemployment (Census 2011). 

This suggests two divergent trends in Conservative Party voter swings between England and Scotland; while in the former, the party is increasingly attracting blue collar workers in more deprived areas, it appears that the majority of its support in the latter is coming from middle-class areas. 




In each nation, the Conservatives have clearly and purposefully situated themselves on one side of the dominant debates - independence (from the United Kingdom and from the European Union), and it is very much paying off. 

The Conservatives are making inroads in areas no one ever would have reasonably thought possible, and their increasing ability to attract swings from very much working class, blue collar areas in England, whilst simultaneously appealing to a more middle-class, pro-Union base in Scotland is the key to that.

Thoughts from #LE2017 Part 3 - Lib Dem Protestors Return?

It appears that the Liberal Democrats are having some success in both the East and West of England in taking council seats from UKIP (who's torrid time continues - just the one seat won at the time of writing).

Thus far, they have taken seats from the anti-EU party in each of Cornwall, Norfolk, Buckinghamshire, Somerset, and Worcestershire local authorities.

Voters flipping from the UK's most anti-EU, Brexit supporting party to one of its most pro-EU, Remain supporting groupings and swinging seats will no doubt come as a surprise to many.

What we can suggest in response to this puzzle is that perhaps the Liberal Democrats, in taking seats from UKIP, are demonstrating a recapturing of their old 'protest vote' - an element of their support which was very much lost when they joined forced with the Conservatives in government in 2010.

Perhaps, with UKIP - undoubtedly the primary party of protest voting over the past four to five years - fading and voters looking for alternatives, the Liberal Democrats may be beginning to be rediscovered as a 'none of the above' option?

The evidence will be in future polling and survey data, but there does seem to be something to suggest that, in some circumstances at least, the Liberal Democrats are reattracting some of their old base. Protest, 'none of the above' voters were always an important part of that.


Thoughts from #LE2017 Part 2 - Uniform Swing

Another interesting thing to note is that we seem to be seeing much the same pattern from the South West of England right up through to the North East.

What was true of Somerset and Gloucestershire at the beginning of the counts - Conservatives taking seats from the Liberal Democrats despite both largely increasing their vote share, Labour support decreasing slightly, UKIP tanking - is currently holding as we move into the far North Eastern Council of Northumberland.

In short, there doesn't seem to be much regional variation in the patterns observed up until now. Across the country so far, there has been a swing to the Conservatives and Lib Dems from Labour and UKIP, with the main result being Conservatives taking control of seats (and councils).

Patterns in Wales have been slightly different in magnitude but directionally also quite similar (but with Plaid profiting from modest swings rather than the Lib Dems).

Something of a 'uniform swing' across the country - everywhere you look, it's broadly the same story.

As we move through the day, it will be interesting to see if any regions buck the trends. Scotland obviously will provide the most interest in this regard. What will happen with the SNP vote? Is their support also fading? If so, will it be the Conservatives that once again profit from those drifting voters?

Thursday, 4 May 2017

Thoughts from #LE2017 Part 1 - Lib Dem Fightback

As I work on analysing the 2017 Local Election and Metro-Mayor results with the BBC Team, I will be trying to pull out some interesting stories which might tell us something about the upcoming General Election on June 8th.

Number 1 - Lib Dem fightback being restrained by Conservatives

Early on, I spotted a trend of Liberal Democrat loses in Somerset - old South-West heartlands for the party where, if a Lib Dem Fightback is forthcoming, then it should be detectable there.

Commentary from the Guardian

The Liberal Democrats and the Conservatives did end up trading seats somewhat on the Council, but overall once Somerset had declared, the Liberal Democrats found themselves 6 seats down despite their vote share increasing by nearly 5% (on 2013).

What happened in Somerset is indicative of the struggles that the Lib Dems will face in between now and June 8th in kicking the engine into gear of their much-discussed comeback: they have increased their vote share, but the Conservatives have passed them at 100 miles per hour on the outside by mopping up UKIP votes (who, incidentally are absolutely crashing nationwide) and going even further ahead (and in some cases taking Lib Dem seats even when they increased their own vote share).

A similar thing happened in Gloucestershire County Council - the Liberal Democrats increased their vote share by 6.5%, but made 0 gains in terms of seats (conversely, the Conservatives took an extra 8 seats to gain overall control of the Council).

What can we read from this in terms of the upcoming General Election? To put it simply, the Liberal Democrats might increase their vote share in the South West by say 5%, but the Conservatives will go up 10%.

This does not translate into General Election seat wins. Lib Dems need a swing from the Conservatives to them, not for the Conservatives to go further ahead.

So something of a Lib Dem fightback could be on in terms of vote shares in the South West, but winning seats looks like a whole other ball game which appears to be out of reach so long as the Conservatives continue to hoover up UKIP voters.

This very much puts the brakes on the Lib Dem fightback in the South West.

Tuesday, 2 May 2017

Updated GE2017 Forecast: Conservative Majority of 120 Seats

NOTE: A mistake in the coding of the 'polls plus' model was noticed today - the Liberal Democrat comeback mechanism was not operating properly. This fix moved the party up by 3 seats in today's forecast, and likely would have moved them up by a similar amount in previous ones. 

Today's ICM poll gives the Conservatives another near 20-point lead over their Labour counterparts, on a 10% swing from their 2015 performance.

This poll moved the uniform swing model to the following: Conservatives 386, Labour 181, Lib Dems 6, UKIP 0, Greens 1, SNP 55 and Plaid 3.

The 'polls plus' model, which adds a variety of conditions to the swing in each seat if it meets certain criteria (explained in full on the methodology page of this website), gave this prediction:



This result would give Theresa May's Conservatives a majority of 120 seats, well over the 100 mark that many believe she is headed for. 

Events today surrounding Dianne Abbott's interview woes will not likely help Labour to close this gap, it will be interesting to see how Labour's polling does after the fact - will there be a backlash against (further) perceived incompetence? 

Elsewhere, the Lib Dem fightback is arguably one of the non-stories of the campaign so far, at least on national polling evidence. Reading the breakdowns (which we must all do with extreme caution), there does appear to be at least some weight in the argument that the 'fight back' will be a localised affair, and not necessarily detectable in terms of uniform swings or shares. 

The 'polls plus' model however does attempt to account for this (re: Lib Dem comeback mechanism above). But it's probably fair to say that a rise of 4 seats on 2015's performance would not be viewed with great satisfaction by the Lib Dem faithful.

The prediction tracking graph shows a steady and sizeable lead for the Conservatives throughout the period analysed. 




In terms of the (new) Con-Lab margins, the model suggests that the battlegrounds have now moved further into Labour territory, with the line now between Mansfield and Dudley North. Interestingly, the prediction suggests that even the 'progressive alliance' being formed in Ealing Central and Acton (controlled for in the poll plus model) will not be enough to stem the blue tide. 



Constituency
Predicted Winner
Predicted Margin
Stoke-on-Trent North
Labour
1.30
Wokington
Labour
0.96
Batley and Spen
Labour
0.78
Mansfield
Labour
0.04
Dudley North
Conservative
0.22
Wirral South
Conservative
0.23
Ealing Central and Acton
Conservative
0.53
Bury South
Conservative
0.80


If seats such as Mansfield, a Labour seat since Frank Bradley Varley took it from the Liberals in 1923 and never once represented by a Conservative MP, are at an alarming risk of falling to the Conservatives, there really are few Labour MPs who should be sitting comfortably.