Are ‘hidden’ Republicans still missing in the polls? Here’s what we’re doing to find out.
It won’t be long until we know the results of the 2022 midterm elections and how the polls fared.
No one can be certain of the outcomes on either front. Polls have systematically underestimated Republicans in recent cycles, and pollsters have blamed something called nonresponse bias — the possibility that supporters of Donald J. Trump will be less likely to respond to polls than demographically similar voters.
Non-response bias is a serious challenge for pollsters, who cannot hope to know anything about people who will not take their surveys. Without that information, interviewers have no way to even explain their existence, much less figure out a way to reach them.
So this fall, the New York Times tried something unusual: We paid people to take a survey.
In partnership with Ipsos, The Times paid respondents up to $25 for political research in Wisconsin, a battleground state where polls infamously got it wrong in 2016 and 2020.
Beginning in early September, Ipsos sent thousands of hard-to-miss priority mail and first-class envelopes to thousands of households across Wisconsin. The mailer contained a $5 bill and a letter promising an additional $20 if the respondent either completed the survey online or returned the attached questionnaire in the enclosed return envelope.
At the same time, we conducted a traditional Times/Siene telephone poll and an online probability-based panel poll at Ipsos/KnowledgePanel. This experimental design allows us to compare the results of two groups of respondents, which, in turn, might offer a portrait of the kind of people who are less likely to be represented in a typical survey—for example, a moderate who isn’t a political junkie, someone who really doesn’t enjoys talking to strangers and, yes, a Republican-leaning voter.
The data is still preliminary and it will likely be at least six months, if not longer, before we draw any definitive conclusions. But there is one immediate difference between the two groups, and that is the survey response rate: Nearly 30 percent of households have responded to the survey so far — a figure that is less than the 1.6 percent rate compared to the Times/Siena poll.
However, a first look at the main findings may be sobering for anyone hoping that a response rate of $25 and above will translate into the desired “hidden” Trump vote. While there were important differences between the high- and low-incentive polls — including some promising to improve on the Times/Siena polls and others going forward — there wasn’t necessarily clear evidence of a breakthrough into a vastly different group of respondents.
The state of affairs in the 2022 midterm elections
Election day is Tuesday, November 8.
While the mail poll showed Republican incumbent Ron Johnson and Democrat Mandela Barnes tied among registered voters — a number similar to other polls — raw, unadjusted respondents in mail and telephone polls favored Democrats by a substantial margin, including a margin of six points in the U.S. Senate race in a highly stimulating postal survey.
(It’s important to note that these poll results are “old” — both the mail poll and the Times/Siena poll were conducted over a period of several weeks, beginning in September. It would be a mistake to assume that these results are representative of voters today.)
Results by demographic group were also remarkably similar. White working-class voters—both before and after weighting—had almost identical party preferences in the two surveys. Registered voters in Wisconsin’s third district — the state’s Obama-Trump district — backed Mr. Barnes in both surveys.
The relatively small differences between the high-stimulus mail poll and other polls could be characterized as a good or bad thing for the polls — and that raises the stakes for today’s election.
On the one hand, the small to modest differences suggest that the Times/Siena poll and other low-incentive surveys remain competitive with the high-incentive survey with significantly higher response rates. On the other hand, it could be interpreted to mean that the “hidden Trump” vote remains out of reach – that $25 cannot reach a far more representative sample. (Of course, we won’t know which of these is true until the election is over.)
The Times/Ipsos Mail study offered relatively little evidence to support some of the most popular theories of nonresponse bias, such as social trust.
Surprisingly, compared to telephone surveys, a slightly higher proportion of mail survey respondents – 50 percent – say people can be trusted. Respondents who mailed were also equally likely to say they volunteered in their communities.
The pandemic has also been cited as a possible explanation for the 2020 polling error, on the assumption that liberals who stayed at home would likely be working the political polls. However, respondents to the telephone survey more often say that the pandemic has not changed how often they leave home. Mail respondents were only three percentage points more likely to say they had never worked from home.
And perhaps most surprising of all, mail survey respondents were just as likely to say they trust the mainstream media (interviewers do not introduce the Times/Siena poll as the New York Times survey) and even less likely than telephone survey respondents to say that the conservative media from trust. Opting out was just as common on the phone as it was in mail surveys.
In all these cases, it is possible that neither the phone nor the mail has penetrated to a group of people who do not usually respond. Maybe people who wouldn’t answer any polls will trust the mainstream media far less and simply stay out of reach.
However, there were some differences between the two studies. Mail survey did show more support for Republicans than the phone poll. The difference was usually quite small – only a point, sometimes two. Usually it wasn’t particularly significant, but it was there.
And the differences were greater in other questions, which offer a portrait of the kind of voters who do not respond to polls and who may be underrepresented in the political discourse.
Political moderation and disengagement
Respondents to the high-stimulus mail survey were far less politically engaged, more moderate, and less likely to vote than those who responded by telephone.
This shouldn’t come as much of a surprise: political polls are certainly more attractive to political junkies, who tend to vote and hold ideologically consistent views.
But the magnitude of the differences was striking: the proportion of mail respondents who said they were “moderate” was more than 12 points higher in the mail poll than in the Times/Siena poll, with both liberals and conservatives representing a smaller portion of the telephone sample.
Similarly, the proportion of people who said they were absolutely sure they would vote in November was higher in the telephone survey than in the mail survey. There are countless such examples in the data, from how often they poll to how closely they follow the news.
Another big difference between the two surveys is how much the respondents seemed to like interacting and interacting with strangers.
People who took part in a telephone survey, for example, were more likely to say they wanted a job that involved working with people, as opposed to working with their hands or on a computer.
And voters who took part in a mail-in survey were more likely to own or say they had considered getting a “no trespassing” or “no calling” sign. Respondents to a typical telephone survey were 10 percentage points more likely to say they had not considered putting up such a sign; Republicans had a commanding lead among those who did.
Perhaps related, the survey found a wide gap in preferences on immigration: 47 percent of respondents in the mail poll said undocumented immigrants should be deported back to their home country, compared with 38 percent in the Times/Siena poll.
It is important to note that a higher incentive and a higher response rate are not the only differences between the two surveys. The mail survey is self-administered; the telephone survey is conducted by a live interviewer. The presence of a live interviewer may have deterred some respondents from offering a potentially embarrassing answer such as support for deportation.
There were another set of differences that cannot easily be attributed to the deviation from interview to self-administered survey: those that suggest that the telephone survey reached a relatively mobile and unsettled group of people.
The mail survey had more people who were married, more people who were born in Wisconsin, and more people who had area codes in the state than the Times/Siena respondents.
It’s a pattern that’s consistent with already existing concerns about the types of voters who have cellphones available in voter registration files — the data set that underpins the Times/Siena poll. Almost by definition, voters who provided their cell phones when they registered to vote did so within the last decade (someone who registered in 1992 would almost certainly not have a cell phone to write down on the form). While the Times/Siena poll takes a number of steps to account for this problem, the results of the Ipsos poll suggest that it may not have been enough.
Whether differences in the number of out-of-state cell phone numbers would have helped sway the Times/Siena poll toward the final result is exactly the kind of question that will be addressed in the final analysis.
That analysis will begin shortly after the final results in Wisconsin, which we’ll be watching closely tonight.
Ruth Igielnik and Alicia Parlapiano contributed to the reporting.
#hidden #Republicans #missing #polls #Heres #find