Method changes last year have substantially improved Gallup’s polling accuracy. That’s good news for early trend spotting
Gallup screwed up in 2010 and 2012. They called for a much wider Republican House victory in the former than what actually occurred, and they polled a Mitt Romney victory in the latter. That put them on a blacklist of sorts with some polling analysts. And while I am just one person, I want to say that I trust Gallup’s presidential approval numbers as much as any mainstream pollster at this point.
Why? The main question when looking at a poll is whether or not the numbers accurately reflect the current state of the country. Methodology is important, but without accuracy, methodology is rather useless. In Gallup’s case, solid technique has led to Gallup’s approval track being reliable since last October.
While Gallup was losing a lot of creditability in the presidential horse race, they were making adjustments to their presidential approval methodology. They added a higher percentage of cell phones, changed weights for the geographic distribution of Americans, and many other minor changes starting on 1 October 2012. These alterations made all the difference in the world (or, at least, the country).
From the beginning of August through the end of September 2012, Gallup pegged President Obama’s approval at only 47.1%. All other pollsters who also used live interviews and called all adults found an average presidential approval of 49.7%. Given undecideds, the difference between the two percentages is that of a president who is in major re-election trouble and one who is probably going to win a tight race.
From 1 October 2012 through the election in early November, Gallup all of a sudden was projecting Obama’s approval at 50.8%. We’re talking over 17,000 interviews in October and nearly 30,000 in the eight weeks prior, so that movement in Gallup’s Obama’s approval rating was outside any margin of error.
The change was not seen by other polling outfits. Although most pollsters switched to a “likely voter electorate”, the ones that continued to poll all adults discovered an average Obama approval of 49.7% – exactly the same as they had produced in the two months prior.
Thus, the movement seen in Gallup’s weekly numbers can only be ascribed to its change in methodology. A change that produced an approval rating indicating President Obama’s re-relection, unlike horserace numbers. The anti-Obama house effect in approval rating essentially disappeared overnight.
Since the election, Gallup has continued to keep the reliable work up. HuffPollster aggregates approval ratings from all pollsters and allows the ability to sort by population (adults, registered, and likely voters) and mode (live telephone, automated telephone, and internet). I have selected adults and live telephone, like Gallup, from all non-Gallup pollsters and compared this plot of local regression to Gallup’s trend* since October 2012.
What we see is what we’d want to see from a trusted pollster. Gallup’s numbers have tracked very well with the aggregate of the other pollsters. Both groups gave Obama post-re-election boosts. Both have shown Obama’s approval rating dropping since January. Both have Obama’s approval fall accelerating since the NSA story broke in early June.
The only difference you’ll notice is that Obama’s approval and disapproval ratings are slightly higher in the overall group. The disparity is only about a point for both. All it means is that other pollsters have designed questionnaires that end up pushing undecideds a little harder. The net approval of both pollster groups is the same.
So what’s this all mean? It means we have the ability to catch trends more quickly thanks to Gallup’s daily and weekly tracking data. Gallup also has very good crosstabs, which thanks to high sample sizes, have relatively small margins of error.
Of course, the major question going forward is whether or not Gallup can translate this success with all adults to registered and likely voters for elections. If they can’t, Gallup will continue to earn a bad rap. If they can, then Gallup’s reputation may be restored.
*Note: An average may produce slightly different results from the averaging technique spoken about earlier.