Every four years we get inundated with political polls as people suddenly find their two month attention-span for politics. There are literally dozens of polling agencies at almost every level tracking everything from local races to the Presidential Election. While in reality the state house and gubernatorial races have the most direct impact on us, we get fixated on the Presidential race probably because it affects us all no matter which state we call home. And the media fan that flame too.
As a continuing public service, I’m providing a philosophy for wading through the myriad of numbers. “Tempest-in-a Teapot” provided a link in an earlier comment here to
Real Clear Politics, a site I also use. It is tempting and easy to do no further analysis and accept their average numbers at face-value, but this is a mistake. The real value in Real Clear is not their suspect averages, but that they provide numbers from multiple polling companies and links to those companies’ web sites in one location. Real Clear usually posts polls within twenty-four hours of release. They also provide a few handy tools like the Electoral Vote map that you can change if you so choose to do your own analysis as well as points of view on the issues and candidates from both the right and the left. In that sense they are very “fair and balanced,” to steal a particular network’s motto.
When looking at poll numbers, one of the first questions that should cross your mind is, “who paid for the poll?” Not all polls are created equal and some exist not just to measure opinions and attitudes at a particular point in time (non-partisan polls do this), but to shape voter behavior come Election Day (partisan polls do this).
Real Clear does a good job labeling the polls that were paid for by either party (these are indicated by a (D) or (R) beside the pollster’s name). However, in addition to these obvious potentially biased polls, there are those conducted by independent but politically biased entities whose results can be suspect. It becomes important to do your research on the pollster.
Polls conducted by CNN/Time, SurveyUSA, MSNBC are consistently biased in favor of Democrats. FOX polls tend to favor Republicans. About the only time that any network’s involvement in a poll can be taken with a level of confidence in objectivity is when they team with a non-partisan pollster like Gallop, Rasmussen, Zogby, or Mason-Dixon. In those instances the independent polling agency designs the poll and oversees the collection of data by the network. There are others that are non-partisan, but these are the biggest ones with the best reputations. Quinnipiac provides independent polling data primarily in the Northeast and mid-Atlantic. Their polls tend to have large sample-sizes relative to the population and low margins of error.
Ideally, one would analyze sample size (but then you need to also understand the demographics and total population from which the sample was drawn), how respondents are selected, examine the pollster’s questions, their admitted causes of probable error, and their stated margin of error. For the average person this can be daunting and prohibitively time consuming. Therefore hone in on two things: first, the “who paid for it” question and second, the stated margin of error. Have a high level of confidence on a truly non-partisan poll with a stated margin of error below 3%. If the margin of error is above this level, then be suspect of the poll numbers. If the poll is partisan, be very suspect. If the difference between the two candidates' poll numbers is less than or equal to the margin of error, then consider the poll a "dead heat." Only when the difference is greater than the margin of error can you consider the poll leaning one way or the other.
Look at the poll’s history over several months. As stated earlier, an advantage of Real Clear is that they provide this information by state at a glance. If you see a particular poll with significant deviation from all the other agencies polling within the same timeframe, then be suspicious as to their method, or their purpose. If an obviously partisan poll gives the opponent an edge, then accept the number if it is within range of other polls conducted at the same time.
Out-of-range numbers are called “statistical outliers” and as a rule should not be included in averages. I am talking numbers that vary in the 50 – 100 percent range. For example, there are five polls taken in a given week. Four out of five produce results showing a difference between two candidates in a range of 5 – 8 percent favoring candidate “A”. The fifth poll shows a spread of 8% in the opposite direction (a 100% deviation), favoring candidate “B”. If the outlier is an obvious “partisan” poll, then without question discard the number. If the poll is non-partisan and its history over time shows it a consistent outlier, then include the number.
Look at when a particular poll was taken. If it is older than a couple weeks or before some major event that could be expected to influence numbers (Democratic and Republican conventions for example), discard the poll unless there are no other numbers, at which point you need to start weighing other factors or simply guess.
Using Real Clear’s averages can be misleading for the following reasons:
a. Real Clear treats all polls equally. There is no assumption of potential bias and obvious statistical outliers are included.
b. Real Clear averages often include significantly outdated polls that skew the resulting average.
c. Real Clear averages often include polls taken prior to significant events that one could reasonably expect to alter opinions.
d. Real Clear averages polls with differing margins of error. This compounds the first error.
In other words, they often wind up comparing apples to oranges to bananas. If one assumes that this fruit salad is therefore more accurate, one will often be mistaken.
Enter the final piece to analyzing poll numbers and that is in knowing the polity of a particular state in all these polls. What is the balance of rural vs urban, how have they voted in past elections and is there an observable trend? Is the population traditionally conservative or liberal (this apart from voting history)? Typically rural populations are conservative (there are exceptions) and urban populations are liberal (there are exceptions).
For example, Minnesota has not voted Republican since 1984, but the numbers of people voting for the Republican candidate have gotten progressively larger, even with an unpopular Republican on the ticket, the past two elections and the Republican margin of defeat has gotten smaller. These factors can be indicators of potential outcome and constitute a trend. Combined with the trend of how the numbers have been moving over the course of the election cycle, you may have a strong indicator of the direction a particular state is moving.
Always remember that polls are snapshot of opinion in a discreet window of time and they change, sometimes wildly within a few days depending upon events. In my next post I will compare the Real Clear Electoral Map, which is constructed using average poll data including the flaws I listed above, with my own map that includes the filtering techniques I’ve discussed in this post. Until then go to
Real Clear and try these methods on your own map to see how the map changes. Concentrate on the states they are calling "toss-ups" or "leaning" one way or the other.
Real Clear Politics Electoral Map 23 Sep 2008
