Sometimes, people respond in strange ways to survey questions.

For a recent project with Jim Stimson and Elizabeth Coggins, I spent a fair amount of time analyzing data from the Cooperative Congressional Election Study (CCES). Here’s a fun nugget from my exploration: a sizable proportion (21 percent) of respondents both support and oppose Obamacare. Simultaneously.

We can speculate wildly about why a fifth of respondents — in a sample that is disproportionately educated and interested in politics! — would give such a puzzling answer.

But in a bigger sense, surveys — as useful as they are — offer highly artificial settings where respondents will give answers. Not attitudes, nor opinions, nor preferences per se — just answers. We should keep that in mind before reading too much into public opinion reports.

The Conflict

Part of the CCES comprises a set of “roll call” votes. These present respondents with a policy position and require a simple yea/nay answer. Two of these questions ask about the Affordable Care Act: one asks the respondent to vote for or against Obamacare, the second asks respondents to vote for or against repealing Obamacare.

There is a logical connection between these two questions. In general, someone who wants to repeal the law would probably not vote for it; and those who want to keep the law around should vote for it to begin with.

Generally that works… but as the ‘jitter’ plot shows above, it doesn’t work that way for everyone. Each dot on the figure represents a single respondent. (I like imagining that I’m assigning people to stand in a corner of the room depending on their answers to questions. Maybe I have a power complex…) There are clearly a good number of respondents in two quadrants: those who either support Obamacare and want to keep it, and those who oppose Obamacare and want to repeal it. Makes sense.

But who are those respondents in the other two quadrants? Slightly more than 12 percent of them want to repeal Obamacare, despite saying that they would vote for the bill; and 9 percent would vote against the bill, but wouldn’t repeal it.

The latter group — the Vote Against / Don’t Repeal group — may be reasoning through the path dependency of Obamacare. Something like, “Well, I don’t like it, but it would endanger the health care system to repeal it now.” Or maybe they’re just ardent believers in the Democratic process: elected officials passed the bill, any who would I be to usurp them? I doubt either of these stories, but it’s not impossible.

The other group — the Vote For / Repeal It! group — is weirder, though. There’s really no logical connection between the two answers.

Surveys are weird…

Well, they are! Despite having used public opinion data in research for several years now, I took my first “real” political survey over the winter holidays. Gallup called and wanted to talk to me about global warming, and that sounded like fun.

It wasn’t. First, you get pretty tired of answering questions after the first twenty. Second, even as a well-educated, highly-informed and engaged observer of the political world, the survey made me feel dumb. There’s this unusual pressure in a survey to answer questions promptly, which is fine but sometimes you don’t have an easy answer right at the top of your mind. Besides, these issues are complicated! Global warming? Economics? Coal, nuclear, wind, oil? Health care mandates?

Stressed yet? Even informed and engaged respondents get a bit overwhelmed by the survey items, and by the need to provide clean answers to complicated questions. And sometimes the questions aren’t entirely clear. Are we asking if you would have voted for Obamacare back in 2010? Or would you vote for it today? Do some respondents miss the “repeal” part of the question? These are all possible points of confusion, introduced in a highly artificial environment, but for which it’s impossible to test without a specific instrument.

Here’s the uncomfortable truth about polls: we use them because they’re what we have. On many questions, they’re good for giving the general feeling in the public. “Will you vote for Mitt Romney, the Republican, or Barack Obama, the Democrat?” isn’t terribly difficult, and most respondents can give a decent answer.

But as the questions become more complicated, responses become less reliable. Accessing “true” attitudes on policy questions with a survey can sometimes be like removing a splinter from your finger with an axe:  In a sense it works, but it’s awfully messy.

And it gets messier when we try drawing relationships between multiple items, all of which have some weird characteristics, like non attitudes, weak attitudes, and non response. Aggregating to reduce the high dimensionality of multiple responses can help filter out some of the noise, but that’s a topic for another post.

Pundits and commentators roll out polls daily to elicit support for some position or another. Being an informed consumer of surveys means going beyond “What’s the Margin of Error?” (We are the Margin of Error, duh!)

It means realizing that a fair number of responses might carry little objective meaning. When pressed I’ll answer, but I honestly don’t know, don’t care or haven’t quite figured out my views yet. Treating these responses as some true-to-life measure of how the American people feel, or how they’ll act, can go pretty far afield.

–

Note: The CCES sample above is limited to the UNC module of 1,000 respondents. Expanding this to the full CCES sample of 55k+ doesn’t change anything, though, but does make the figure a bit messier.

Elections have consequences, and many of those consequences are — well, consequential. But every election brings endless speculation that this election was the election – the realignment, the death of the losing party, the upending of the current political era.

The midterm election in 2002 was “a disaster” that crushed Democrats and forced them, if they were smart, to tack toward the center. President Bush’s 2004 victory only solidified the Democrats’ fate. Until, of course, the 2006 and 2008 elections marked a resurgent Democratic left and the ultimate failure of the conservative project. Until 2010… well, you get the point.

Some of these speculations are more well-founded than others. In his last post for the New York Times’ Campaign Stops blog, Columbia professor Thomas Edsall asks if “Rush Limbaugh’s country [is] gone“. Pointing to some polling data and discussions with prominent Democratic pollsters, Edsall suggests that a new left-leaning electorate is emerging from the ashes of the political polarization and financial crisis of the late 2000s.

The argument is interesting, but we could probably reconsider some of the evidence he points to.

Mr. Edsall, for example, discusses a Pew Research survey showing that young voters, African Americans and Democrats with a favorable impression of socialism. This could mark the emergence of a potent leftism that could forever transform the American political landscape.

Source: Thomas Edsall, New York Times Campaign Stop blog.

The numbers say something else to me. I’m not sure that any more Americans would support actual socialist policies today than would have two or ten years ago. What likely changed is the affective charge of the term.

Take, for example, dissertation research by UNC’s K. Elizabeth Coggins on the emergence and relative decline of “liberalism” as a political identity. A paper (pdf) by Coggins, coauthored with Jim Stimson, explores how individuals attach meaning to such labels as “liberal” and “conservative”, and “how widely popular liberal policies like Social Security, Medicare, and workplace safety came unhinged from the ideological label which defines them. ”

If liberalism could come unhinged from its ideological content, it stands to reason that the same could happen to socialism. Over the past several years, many conservative commentators and Republican leaders have called President Obama’s policies “socialism”; and if the term might rally voters on the right, it may too help to redefine how many liberals think of “socialism”. If liberals begin associating Obamacare and higher taxes on top income earners as socialism, they may be more inclined toward the ideology.

The rest of Mr. Edsall’s case rests on striking differences between liberals and conservatives on an array of policy proposals. The gaps are stark, but they are not necessarily new. Self-identified liberals and conservatives have long held distinct views on an array of policy issues from education to welfare spending.

Liberal and Conservative Attitudes toward Increased Welfare Spending. (ANES Cumulative Data FIle)

Liberal and Conservative Attitudes toward Increased Education Spending. (ANES Cumulative Data FIle)

True, ideologically-leftist voters attach more consistently to the Democratic party, and conservatives self-identify more as Republicans, than in decades past. This upholds a partisan sorting hypothesis, but not to any particular narrative of either left- or right-of-center ideologies emerging as dominant.

Liberalism and Conservatism Over Time. (ANES Cumulative Data FIle)

In fact, there is little sign that the American electorate is moving either left or right. Macropartisanship is known to shift over time in response to economic conditions, the occupant of the White House and political shocks. Surely Republicans will want to rethink their strategy of appealing to minorities and to a lesser extent women; but it’s unlikely that Republicans will have to learn to live with an emerging leftism in the American electorate.

Last week, we examined the accuracy of several presidential forecasts. For those familiar with statistics and probability theory, the results proved unsurprising: the forecasts came reasonably close to the state-level outcomes, but the average forecast outperformed them all.

Put another way, the aggregate of aggregates performed better than the sum of its parts.

This year’s Senate races provide us another opportunity to test our theory. Today, I gathered the Senate forecasts from several prognosticators and compared them to the most recent Election Day returns. As before, I also computed the RMSE (root mean squared error) to capture how accurate each forecaster was on average.

We must note one modest complication: not all forecasters posited a point-estimate for every Senate race. Nate Silver put forward a prediction for every race; but Sam Wang of Princeton University only released 10 predictions for competitive races.

We accordingly compute two different RMSEs. The first, RMSE-Tossups, only computes the RMSE for those races for which each forecaster put forward a prediction. (There are nine races that fall into this category: Arizona, Connecticut, Massachusetts, Missouri, Montana, Nevada, North Dakota, Virginia and Wisconsin.)

The other calculation, RMSE-Total, shows each forecaster’s RMSE over all predictions. Wang, for example, is evaluated by his accuracy on the ten predictions he made; while Silver is evaluated on all 33 races.

The numbers in the above table give us a sense of how accurate each forecast was. The bigger the number, the larger the error. So what can we learn?

Alas! The average performs admirably yet again. It’s not perfect, of course; for some races, there are precious few forecasts to average over: Delaware, for instance, has only the 538 prediction.

To begin accounting for this, we weight the RMSE by the share of forecasts used to compute the average. If we limit our evaluation of the average to only those races with three or more available forecasts, the RMSE drops to 4.8.

What else emerges from the table? For one, the poll-only forecasts — especially the Wang, RCP and Pollster forecasts — perform better than Nate’s  mélange of state polls and economic fundamentals.

North Dakota, where Democrat Heidi Heitkamp bested Republican Rick Berg, provides a case in point. Pollster and RealClearPolitics both predicted a narrow win for Ms. Heitkamp. The 538 model considered the same polls upon which Pollster and RCP based their predictions; but the fundamentals in Mr. Silver’s model overwhelmed the polls. As a result, the 538 model predicted that Mr. Berg would win by more than five points. [See Footnote 1.]

In sum, however, all of the forecasts did reasonably well at calling the overall outcome. We can chalk this up to another victory for (most) pollsters and the quants who crunch the data.

1. Addendum: In the original post (text above is unchanged), I argued that Mr. Silver’s economic fundamentals pushed his ND forecast in the wrong direction. This undoubtedly contributed to his inaccuracy in North Dakota, but it wasn’t the main factor. As commenters pointed out, Silver’s model was selective in the polls it used to predict the outcome. As of the last run of the model, Silver’s polling average lined up fairly well with RCP (Berg +5) but not with Pollster (Heitcamp +0.3). Mea Culpa.

Thanks to data compiled by Kevin Collins at Princeton, we can examine the accuracy of some of the state-level forecasting models. Nate Silver’s 538 model performs marginally better than the pack. But the best predictor: an average of the forecasts.

To assess accuracy, we calculate the Root Mean Squared Error. To do so, we take the actual result in state i, , and a forecaster’s prediction in state i, , and calculate:

As you can see, higher values indicate that the forecaster made bigger errors. Put another way, the number shows us how badly each forecaster missed on average. 

Alex Jakulin, a statistician at Columbia, helpfully pointed out that a more useful metric may be the RMSE weighted by the importance of each state. We would expect misses to be larger in small states and should correct for that. Accordingly, we present the RMSE for each forecaster, and the RMSE weighted by the proportion of electoral votes controlled by each state.

All told, the forecasts did quite well. But look at what worked better: averaging over the forecasts. This makes good statistical sense: as Alex points out with a fun Netflix example, it makes more sense to keep as much information as possible. In a Bayesian framework, why pick just one “most probable” parameter estimate, instead of averaging over all possible parameter settings, with each weighted by its predictive ability?

During the Republican primaries, Harry Enten published a series of stories on this blog doing precisely that; and then, as now, the “aggregate of the aggregates” performed better than any individual prediction on its own.

On the whole, all the forecasting models did quite well. As the figure above shows, critics of these election forecasts ended up looking pretty foolish.

I only now wonder if the 2016 will see a profusion of aggregate aggregators; and if so, how much grief Jennifer Rubin will give them.

The polls, and the forecasters using them, performed pretty well. Harry Enten posted for the Guardian this morning about the overall success of pollsters in the 2012 cycle, and John Sides put up this great post/figure over at The Monkey Cage. (The figure was also picked up by Ezra Klein.)

I wanted to see the estimated error margins around the final pollster predictions. So here’s my take with 95 percent confidence bands:

The figure shows the difference in the predicted and actual Obama margin. Positive numbers mean that the pollster overstated Mr. Obama’s margin of victory; the negative implies the inverse.

Almost all of the polls came within a reasonable distance from the outcome. In fact, most contained the true outcome in their margins of error.

We’re not live-blogging the presidential race, but here is an interesting tidbit:

There is not much to update in the model, but we can update our expectations given current returns. Put another way: we can ditch simulations with results we know to be incorrect.

By updating our beliefs, Mr. Romney’s chances of winning drop from 32 percent to only 12 percent. As you can see in the figure, Mr. Romney’s distribution of electoral votes has become much more certain (as we would expect), but he’s lost most area under the curve to the right of 270.

The door on Mr. Romney’s probability of winning is closing, and quickly.

As we published yesterday, the Margin of Error forecast predicts Mr. Obama to secure reelection with 303 electoral votes to Mr. Romney’s 235. This translates roughly into a 68 percent chance for Mr. Obama to win, leaving a nonnegligible 32 percent chance of an upset victory by Mr. Romney.

One of the more interesting elements of the model is that it’s agnostic to state-level polling. Most of the highly-trafficked forecasting models (the gold standard is Nate Silver’s 538 model) use various methodologies for aggregating state- and national-level polling.

The MoE forecast, on the other hand, uses very few variables. The national popular vote is predicted using late-season approval data; the state-level votes are forecast using (a) previous election results; (b) August-November change in unemployment; (c) home state advantage; and (d) a regional dummy variable. No polls.

Despite the stark difference in methodology, the forecast comes in well in-line with most quantitative models. The most notable gap between our model and many of the poll-aggregation models is, in fact, that ours makes a far more conservative prediction of uncertainty.

So, what’s the deal with this fundamentals-based model, anyway? In my opinion, a fundamentals-based forecast brings some distinct advantages and disadvantages.  Continue reading →