April 12, 2013
I have only done a little bit of social science research, but it was enough to make me hate people. One study I helped with analyzed whether people from different countries had different answers on a certain psychological test. So we put up a website where people answered some questions about themselves (like “what country are you from?”) and then took the psychological test.
And so of course people screwed it up in every conceivable way. There were the merely dumb, like the guy who put “male” as his nationality and “American” as his gender. But there were also the actively malicious or at least annoying, like the people (yes, more than one) who wrote in “Martian”.
I think we all probably know someone like this, maybe a couple people like this.
I also think most of us don’t know someone who believes reptilian aliens in human form control all the major nations of Earth.
Public Policy Polling’s recent poll on conspiracy theories mostly showed up on my Facebook feed as “Four percent of Americans believe lizardmen are running the Earth”.
(of note, an additional 7% of Americans are “not sure” whether lizardmen are running the Earth or not.)
Imagine the situation. You’re at home, eating dinner. You get a call from someone who says “Hello, this is Public Policy Polling. Would you mind answering some questions for us?” You say “Sure”. An extremely dignified sounding voice says – and this is the exact wording of the question – “Do you believe that shape-shifting reptilian people control our world by taking on human form and gaining political power to manipulate our society, or not?” Then it urges you to press 1 if yes, press 2 if no, press 3 if not sure.
So first we get the people who think “Wait, was 1 the one for if I did believe in lizardmen, or if I didn’t? I’ll just press 1 and move on to the next question.”
Then we get the people who are like “I never heard it before, but if this nice pollster thinks it’s true, I might as well go along with them.”
Then we get the people who are all “F#&k you, polling company, I don’t want people calling me when I’m at dinner. You screw with me, I tell you what I’m going to do. I’m going to tell you I believe lizard people are running the planet.”
And then we get the people who put “Martian” as their nationality in psychology experiments. Because some men just want to watch the world burn.
Do these three groups total 4% of the US population? Seems plausible.
I really wish polls like these would include a control question, something utterly implausible even by lizard-people standards, something like “Do you believe Barack Obama is a hippopotamus?” Whatever percent of people answer yes to the hippo question get subtracted out from the other questions.
Alas, not all weird poll answers can be explained that easily. On the same poll, 13% of Americans claimed to believe Barack Obama was the Anti-Christ. Subtracting our Lizardman’s Constant of 4%, that leaves 9% of Americans who apparently gave this answer with something approaching sincerity.
(a friend on Facebook pointed out that 5% of Obama voters claimed to believe that Obama was the Anti-Christ, which seems to be another piece of evidence in favor of a Lizardman’s Constant of 4-5%. On the other hand, I do enjoy picturing someone standing in a voting booth, thinking to themselves “Well, on the one hand, Obama is the Anti-Christ. On the other, do I really want four years of Romney?”)
Some pollsters are starting to consider these sorts of things symptomatic of what they term symbolic belief, which seems to be kind of what the Less Wrong sequences call Professing and Cheering or Belief As Attire. Basically, people are being emotivists rather than realists about belief. “Obama is the Anti-Christ” is another way of just saying “Boo Obama!”, rather than expressing some sort of proposition about the world.
And the same is true of “Obama is a Muslim” or “Obama was not born in America”.
But sometimes it’s not some abstruse subtle bias. Sometimes it’s not a good-natured joke. Sometimes people might just be actively working to corrupt your data.
Another link I’ve seen on my Facebook wall a few times is this one: Are Climate Change Sceptics More Likely To Be Conspiracy Theorists? It’s based on a paper by Stephen Lewandowsky et al called NASA Faked The Moon Landing, Therefore Climate Science Is A Hoax – An Analysis Of The Motivated Rejection Of Science.
The paper’s thesis was that climate change skeptics are motivated by conspiracy ideation – a belief that there are large groups of sinister people out to deceive them. This seems sort of reasonable on the face of it – being a climate change skeptic requires going against the belief of the entire scientific establishment. My guess is that there probably is a significant link here waiting to be discovered.
Unfortunately, it’s… possible Stephan Lewandowsky wasn’t the best person to investigate this? Aside from being a professor of cognitive science, he also runs Shaping Tomorrow’s World, a group that promotes “re-examining some of the assumptions we make about our technological, social and economic systems” and which seems to be largely about promoting global warming activism. While I think it’s admirable that he is involved in that, it raises conflict of interest questions. And the way his paper is written – starting with the over-the-top title – doesn’t do him any favors.
(if the conflict of interest angle doesn’t make immediate and obvious sense to you, imagine how sketchy it would be if a professional global warming denier was involved in researching the motivations of global warming supporters)
But enough of my personal opinions. What’s the paper look like?
The methodology goes like this: they send requests to several popular climate blogs, both believer and skeptic, asking them to link their readers to an online survey. The survey asks people their beliefs on global warming and on lots of conspiracy theories and fringe beliefs.
On first glance, the results are extremely damning. People who rejected climate science were wildly more likely to reject pretty much every other form of science as well, including the “theory” that HIV causes AIDS and the “theory” that cigarettes cause cancer. They were more willing to believe aliens landed at Roswell, that 9-11 was an inside job, and, yes, that NASA faked the moon landing. The conclusion: climate skeptics are just really stupid people.
But a bunch of global warming skeptics started re-analyzing the data and coming up with their own interpretations. They found that many large pro-global-warming blogs posted the link to the survey, but very few anti-global-warming blogs did. This then devolved into literally the worst flame war I have ever seen on the Internet, centering around accusations about whether the study authors deliberately excluded large anti-global warming blogs, or whether the authors asked the writers of anti-global-warming blogs and these writers just ignored the request (my impression is that most people now agree it was the latter). In either case, it ended up with most people taking the survey being from the pro-global-warming blogs, and only a few skeptics.
More interestingly, they found that pretty much all of the link between global warming skepticism and stupidity was a couple of people (there were so few skeptics, and so few conspiracy believers, that these couple of people made up a pretty big proportion of them, and way more than enough to get a “significant” difference with the global warming believers). Further, most of these couple of people had given the maximally skeptical answer to every single question about global warming, and the maximally credulous answer to every single question about conspiracies.
The danger here now seems obvious. Global warming believer blogs publish a link to this study, saying gleefully that it’s going to prove that global warming skeptics are idiots who also think NASA faked the moon landing and the world is run by lizardmen or whatever. Some global warming believers decide to help this process along by pretending to be super-strong global warming skeptics and filling in the stupidest answers they can to every question. The few real global warming skeptics who take the survey aren’t enough signal to completely drown out this noise. Therefore, they do the statistics and triumphantly announce that global warming skepticism is linked to stupid beliefs.
The global warming skeptic blogosphere has in my opinion done more than enough work to present a very very strong case that this is what happened (somebody else do an independent look at the controversy and double-check this for me?) And Professor Lewandowsky’s answer was…
…to publish a second paper, saying his results had been confirmed because climate skeptics were so obsessed with conspiracy theories that they had accused his data proving they were obsessed with conspiracies of being part of a conspiracy. The name of the paper? Recursive Fury. I have to hand it to him, this is possibly the most chutzpah I have ever seen a single human being display. (the paper is now partially offline as the journal investigates it for ethical something something)
The lesson from all three of the cases in this post seems clear. When we’re talking about very unpopular beliefs, polls can only give a weak signal. Any possible source of noise – jokesters, cognitive biases, or deliberate misbehavior – can easily overwhelm the signal. Therefore, polls that rely on detecting very weak signals should be taken with a grain of salt.