26 October 2016
MSU Poll sampled “adult Montanans,” not registered or likely voters
That’s a big deal. It greatly diminishes the value of the just released poll for predicting the results of the 2016 general election on 8 November.
The MSU Poll, a tradition at Montana State University’s campus in Billings, is conducted by students, who make the telephone interviews, with faculty supervision. Here’s how the poll was conducted:
This report summarizes the results of a statewide random sample telephone survey of 590 adult Montanans. 70% of the survey respondents were contacted via cellphone and 30% were contacted via landlines. The sample was acquired from Marketing Systems Group. The poll was conducted from October 3-10, 2016.
By contrast, the Mason-Dixon poll, commissioned by Montana’s Lee Newspapers, sampled 1,003 likely voters during the 10–12 October window.
Although both polls sampled Montanans, they did not sample the same population of Montanans. Only 60 percent of the population sampled by MSU would have passed a likely voters screen set at the 71.4 percent turnout rate for the Presidential election in 2012. Mason-Dixon only polled likely voters. Mason-Dixon’s approach provides a much more accurate read on the election, but also requires making many more telephone calls. Mason-Dixon has the resources to make the extra calls. MSU’s pollsters probably do not.
Here’s how the population breakdown looks graphically:
Each of the shorter bars is a subset of the bars above it.
And here’s how the MSU and Mason-Dixon polls compare on the elections for Governor and Montana’s only seat in the U.S. House of Representatives:
The high number of undecided/no responses in the MSU poll may be the result of calling people not registered to vote.
The MSU Poll’s margin of error is one point wider than the Mason-Dixon’s MOE. That’s not a big deal, or even a little one. In a one-off poll such as this, how the sample was weighted — if it was weighted — to correct for the sampled population’s deviations from the true population demographic is likely to have a much more profound effect than the MOE.
How, then, should readers evaluate the MSU Poll? With wide error bars, and an understanding that it did not sample the same population as the Mason-Dixon poll.