bad faith uses of scientific 'rigor'

I have conflicted feelings about productivity books, but even as I increasingly reject the emphasis on productivity, I do find that there are some gems in these books that are helpful to me as I try to keep my life organized across all of its dimensions. While rereading one of these books over the summer, I came across the following quote (which appears to be a misquotation of Oliver Wendell Holmes, Jr.):

I wouldn’t give a fig for the simplicity on this side of complexity; I would give my right arm for the simplicity on the far side of complexity.

That’s not bad advice in terms of personal organization, but I’ve also thought about it a lot over the last few months in terms of my attitude toward science. There’s a skepticism that exists “on this side” of science (i.e., that never makes it to really understanding what science is) that is tremendously dangerous, as evidenced by far too much of what is going on in the world around us. Yet, there’s a skepticism of the scientific process “on the far side” of science (i.e., that understands science well enough to see its flaws and issues) that I think is critically important. If appreciation for science is tragically lacking in the general populace, I think skepticism of science from its “far side” is tragically lacking among scientists themselves. I think that Catherine D’Ignazio and Lauren Klein’s Data Feminism is one of the best texts I’ve read on these critiques, but I know I have a lot more reading to do in this area.

Part of this critical view of science is recognizing the ways that the assumptions and procedures that guide scientific inquiry are not, in fact, objective instruments of scientific discovery but rather traditional, human heuristics that can fall prey to all of the subjectivities, biases, and bad faith actions that empirical investigations are supposed to help us overcome. There are plenty of examples of this phenomenon (consider, for example, how many of the pioneers of modern statistics also embraced scientific racism), but I’d like to share a story that happened to me a while ago.

In the summer of 2020, Amy Chapman and I presented some of our preliminary work on the #DezNat hashtag from Mormon Twitter at that year’s online International Conference on Social Media & Society. Some time later, the participants in this hashtag found the YouTube video where we’d presented our work and let us know on Twitter what they thought of it. We were critical of the movement (given the anti-feminist and far-right influences on their social media activity), and so we were unsurprised when they were critical of us. Some of the criticism was juvenile, some of it was mildly offensive (we got off fairly light, all things considered), but the two critiques that interested me the most were attempts to critique our work in scientific terms.

If memory serves (I have screenshots, but I can’t remember where), there were two different accounts that tried to discredit our research based on supposed failings of scientific “rigor.” One account told us that his dissertation supervisor would have been appalled by our sample size, and the other called into question our credentials given that he had two degrees in psychology but had never heard of “educational psychology,” the field that shows up on my CV. These are, of course, both laughable criticisms. We weren’t setting out to do generalizable research, and didn’t even do any statistical analysis, so sample size isn’t something we were really concerned about. What’s more, our entirely-human qualitative analysis of 1,400 tweets is an impressive sample size for the kind of work that we were doing: I’d never previously spent that much time coding social media data, and I’m not really in a rush to do that much work any time soon. What’s more, educational psychology is a well-established field within psychology, I got my PhD from a program that is really, really strong in educational psychology—and besides, the “ed psych” part of my degree has never really been my specialty, so it’s weird to pick on that in particular.

It’s hard for me to imagine that either of these criticisms was made in good faith. The critics weren’t going to like our research no matter what, and these were just handy excuses. Yet, it stands out to me that both criticisms are based on heuristics that hold some weight. Sample size is important when evaluating research. A researcher’s credentials are important for determining if they’re a legitimate source. Yet, so long as these remain shallow heuristics for rigor instead of tools employed in a broader, deeper process of evaluating scholarly arguments, they are vulnerable to being misused to shut down research that people don’t like.

Indeed, the entire reason I’ve been thinking about these stories recently is because of a conversation I had with other members of the editorial team at the Journal of the Mormon Social Science Association. Mormon Studies is a field that includes people with a very wide range of different relationships to the “Smith-Rigdon” family of religions, including its harshest critics and its most devoted apologists. Yet, because it’s an academic field, most people understand that it’s scientific rigor that matters. While I wasn’t privy to all the details of the case in question, the concern that we were discussing involved the extent to which the critiques made of a particular article under review represented legitimate scientific critiques or an attempt to shut down conclusions that someone didn’t like using scientific rigor as an excuse.

Both of these examples involve Mormon Studies, but I hope it’s clear that this is a danger that exists across all scientific disciplines. Science is a process of argumentation, and scientific heuristics we employ within our arguments—they are not unimpeachable, objective standards of truth.

• macro
• Work
• Journal of the Mormon Social Science Association
• research
• publishing
• DezNat
• productivity
• Catherine D'Ignazio
• Lauren Klein
• data feminism