Opinion Polling Accuracy

In September I did a series of factual posts on the opinion polls for the referendum. There is stuff we can learn from comparing the actual result to what the polls said beforehand. Especially as we have a General Election in a few months.

It is also worth pointing out that my day job is politically restricted and so I can only offer factual analysis and need to avoid political partisanship. That said I may say positive or negative things about party positions, that shouldn’t be construed as support for any party. Many political positions are not supported by relevant evidence but the public doesn’t often know or listen to evidence and politicians that want to be elected must listen to the public.

Many people, mainly pro-independence supporters, were surprised by the 55% no vote. The polls had narrowed in the final days and suggested a closer margin with perhaps a narrow Yes result. That said, the polls pretty much got the result right, as the graph below shows. It was only the magnitude of the margin that was wrong.

August and September 2014 opinion poll results on Scottish Independence

There are three factors that I think need to be taken into account when comparing the polling results to the election result in Scotland. One of these is peculiar to that referendum but the other two are general.

Cognitive bias in seeing polls as biased against them
Internet filter bubble insulating people from other opinions (not completely but limiting dissent from their views)
Fear of confrontation with pro-independence voters.

Cognitive bias

It was pretty common to hear pro-independence supporters claiming that the mainstream media was baised against them, and that the opinion polls weren’t showing an accurate picture. This is possibly a result of a framing effect in the polls commissioned on behalf of the Yes campaign. Several of them used the argument that their own polling showed a more positive outcome than other polls. In combination with confirmation bias (where you select data that confirms your hypothesis) this would explain their surprise.

Certainly within the data it is noticeable that some of the polls showing commissioning by the SNP are more pro-independence than the others. The level of variation isn’t huge, but the trend is clear, in each cluster of polls about the same the SNP commissioned set is the one with the highest level of support for independence. I cannot be sure why this is the case.

Frankly the political types should know better than to believe their own propaganda. That’s how you lose elections (and touch with the people).

Internet Filter Bubbles

So most internet services track us, and present information in a way that they think is most relevant to us. This is especially true of the likes of Facebook and Google (where you are logged in). Amazon recommendations are probably the most obvious example of it, but it is common practice. These work by looking at how you rate certain things, how other people rate those things and how well you compare.

What this does for people is that as they click ‘like’ on the posts that support their point of view, and comment to say ‘me too’, then that reinforces those posts from those people as the sort of thing to present. When compared with the ignored posts from the other side that get downgraded because they were ignored, then you tend to see more posts that support your preferred viewpoint. This then gives you the impression that most people agree with you, because most of the posts that you see agree with your opinion.

See this TED talk on internet filters for more. There is a need to go out of your way to get diverse opinions. Something we should all remember when it comes to working out who to vote for in May 2015.

Fear of Confrontation

This is the only one of the three that I think is peculiar to the independence referendum. Emotions were high for this, much higher than is usual for elections. All the more so for the Yes supporters who saw this as a once in a lifetime opportunity to break away from the Westminster Parliament that they felt had let down voters. I’ve personally heard from a few people that this was a real thing for them, that they daren’t express their real opinions for the status quo for fear of being hectored by Yes supporters.