How Did The Polls Do in New Hampshire?

On Tuesday Morning, I wrote “Mitt Romney seems destined for a big New Hampshire 15-20% win, and Jon Huntsman looks to come up just short of a second place finish.” The final returns verify that Mitt Romney won by 16.4% and Jon Huntsman did finish in third place.

Did the rest of the results lineup how we expected them to?

Dick Bennett of the American Research Group has done a fine job calculating individual pollster accuracy. Most survey houses did a great job and their results fell within the expected confidence intervals.

Few of Bennett’s findings are surprising and generally fit with the conventional wisdom. He determines that the greatest pollster error occurred with Ron Paul’s projected vote percentage.

If you look at Bennett’s chart, you’ll note the smallest Paul errors were those that polled earlier in the period. Both NBC/Marist and Washington Times/JZ Analytics ended their polls either on the Wednesday or Thursday before the primary. The pollsters with the largest Paul error were all in the field through at least Sunday (January 8th).

How is it that the pollsters who polled later were the least accurate in terms of Paul? Were later polls missing something that earlier ones were not?

The answer is not as clear as we would like it to be, but Stefan Hankin has found that the Public Policy Polling and Suffolk did undercount both young and independent voters. Paul performed best among these groups.

This discovery seems to confirm the pre-primary thoughts of Jeff Winchell before the primary, although Romney’s percentage of the vote was not greatly affected by this factor.

Oddly, NBC/Marist actually had about the same percentage of Independents (38%) and young voters (10%) as later polls. Thus, the only conceivable conclusion that I can reach is that both early and late surveys were missing Paul voters, and that Paul did decline in the lead-up to the election. It’s quite possible then that Paul’s true percentage of the vote was ~26-28% early in the week.

I am not too worried about the polls undershooting Paul’s percentage going forward as the Iowa polls did a fine job gauging his support.

Bennett also finds that those pollsters late in the period were more likely to correctly gauge Huntsman’s share of the vote.

The timeline of the Huntsman error is consistent with the overall feeling that Huntsman was rising in the lead-up to the primary.

But I have to ask whether it is fair to “grade” pollsters who end their polls earlier on the same scale as those who ended their polls closer to the primary? Unlike what we know about general elections, voter opinion in primaries can change quickly. We saw that in Iowa with Rick Santorum and in New Hampshire with Jon Huntsman.

Finally, before I turn my attention to the great performance of the aggregators, I must point out that Bennett finds the least accurate overall pollster was the University of New Hampshire.

It was not that UNH polls were “bad”, but that they were the least great of the great. Indeed, UNH is a fine pollster, and I rely on their surveys for NH Congressional district polling.

I bring attention to UNH only because before the primary all I heard from pundits was how this was the “it” pollster in New Hampshire. The truth is that, on the whole, no pollster is that much better than any other one. Most pollsters (with the notable exception of Zogby Interactive) tend to be equally good, and Tuesday’s results confirm that belief.

As for the aggregate of the aggregates, and the aggregators who make up that aggregate, it was a fine evening.

The average absolute candidate error (1.6%) was down vs. the “difficult” to poll Iowa Caucuses (2.1%). The largest individual candidate error (3.9%) was also down from Iowa (5.2%). The aggregate of the aggregates as well as each aggregator correctly placed Mitt Romney’s first through Rick Perry’s sixth place performance.

This impressive predictive power is not surprising. While most of the public concentrated on the Democratic polling disaster of New Hampshire 2008, most failed to recognize that the polling on the Republican side was fantastic.

Among the individual aggregates, the smallest (statistically insignificant) error overall belonged to my old friends at HuffPollster.

Polls & Votes and Real Clear Politics tied for the smallest (again statistically insignificant) largest error. All but Polls & Votes with Rick Santorum had their largest one candidate error with Ron Paul.

Overall, the pollster’s graveyard did not receive a new body this past Tuesday. Along with Mitt Romney, polling improved upon its Iowa performance. The only question left to answer is whether the polls can maintain their momentum and make it three-for-three in South Carolina.