In the heat of a historic presidential election, it’s easy to get swept away in the data and numbers from political polls that purport to predict the outcome this November. But with every news channel reporting on its own political poll findings, and with so many contradicting numbers, how can we make sense of the results?

There are many factors that can explain why two polls attempting to measure the same topic can reveal vastly different findings. The three major areas where a poll can produce bias, either intentionally or unintentionally, are the sampling procedures, the information-gathering method, and the wording of the questions.

The first general rule of thumb for sampling is that the larger the sample, the more likely the poll is to produce accurate results. But a large sample is not enough to ensure a representative sample, or a sample that closely matches the population that the poll is intended to study. In the case of political polls, a poor sampling method may over represent one political party, which will likely skew the results.   For example, a poll conducted by CNN among its own viewers is likely to yield different results than one by FOX News among its viewers, and neither of these polls is necessarily representative of the country. For more accurate results, it’s important to use a completely random sample of poll participants to help reduce sampling bias.

Another important sampling factor in political polling is the distinction between “voting-aged adults,” “registered voters,” and “likely voters.” While a national poll of “voting-aged adults” might yield accurate numbers for public opinion on an election, it is much less likely to predict election results than a poll of “likely voters.” There can even be bias or discrepancies between polls about the definition of a “likely voter” or ways of discerning someone’s intent to vote. Simply asking a respondent’s intent to vote is unlikely to be as accurate as asking when they last voted. If it was many years ago or they never voted in the past, it can be an indicator of actual intent. Interestingly, there can be anomalies, such as populist movements, as recent elections have shown. For example, record numbers of republican voters came out in the most recent primaries, including many voting-aged people who have never voted. In 2008, record numbers of new young voters came out to vote. These anomalies are not necessarily incorporated into poll numbers.

The second important source for political poll bias is the information-gathering method. Many political polls are conducted via telephone, which has advantages and disadvantages. While phone-conducted polls have much higher response rates than mail or internet methods, one major disadvantage is social desirability bias. If a respondent holds opinions that he or she worries will be unpopular with the interviewer, the respondent may be less forthright. People may be more honest about their political opinions through more anonymous methods like mail or internet. Additionally, whether a poll uses landlines, cell-phones, or a mix of both can affect the results, as landline users are more likely to be older while cell-phone-only users tend to skew younger.

The last major contributor to political-poll bias is the question phrasing. Every aspect of a question’s wording needs to be evaluated, from a single loaded word to the sequence of questions before or after. For example, a question phrased “Should the next president elect “more liberal”/“more conservative” supreme court justices?” is seeking agreement with a position versus a question that is more neutral and seeks on objective response. “Would you like the next president to elect supreme court justices without considering their own political leanings?” is less skewed and is less likely to elicit emotionally-fueled responses. A double-barreled question like “do you follow the election and plan to vote for candidate X” will inevitably cause ambiguity over whether the respondent is against candidate X or simply hasn’t been paying much attention to the election. Additionally, a very pointed question about a specific candidate such as probing a personality characteristic (e.g. “Do you believe X candidate is untrustworthy?”) may influence the overall opinion of that candidate. A more objective question would ask to rate the candidate on a range of personality traits, both positive and negative.

To understand the accuracy of a political poll, it is always important to consider all the factors that can influence the results, including the source of the sample, the survey medium, and the question construction. Potential pitfalls of poll and survey design can be avoided by using reputable third-party research firms and polling institutions who understand the potential biases and will strive to avoid them.

Contact Us to learn about how we ensure accurate quantitative research.