The College Fix

Issues with past phrasing of questions concerning open-mindedness

A new study suggests that intellectual diversity is important in studying psychology, especially when dealing with issues such as religiosity and political attitudes.

It’s an exercise in self-examination from University of Toronto Prof. Keith Stanovich, who is joined by York University Prof. Maggie Toplak. The study is based on Stanovich’s previous work with another co-author on “studies of actively open-minded thinking,” as the study’s subtitle says.

Stanovich told The College Fix in an email interview he wanted to get out this study quickly to remedy a possible error in the research literature he helped create.

“The need for intellectual diversity in psychological science,” published in the journal Cognition earlier this month, deals with the concept of actively open-minded thinking.

AOT is measured by questionnaires designed to test certain aspects of an individual’s reasoning: “the willingness to consider alternative opinions, the sensitivity to evidence contradictory to current beliefs, the willingness to postpone closure, and reflective thought,” according to the study’s abstract.

The study re-examines studies from the past two decades that found religious-minded individuals were less open-minded than other respondents.

Stanovich and Toplak were “startled” from observing high negative correlations between AOT and religious belief, which meant that the more open-minded people are, the less likely they are to hold religious beliefs.

However, the earlier studies used the ambiguous word “belief,” which signaled to religious respondents that the questions were referring to religious beliefs, rather than a generic belief, such as Coke being superior to Pepsi. The lack of religiosity among the research team may have blinded them to this possibility, they admitted.

The Cognition study was highlighted by New York University social psychologist Jonathan Haidt, who called it “a nice (and humble) example of psychologists discovering that their lack of religious diversity in the lab (all secular) led to an error in scale construction.”

https://twitter.com/JonHaidt/status/1106891588846977024

Those with “strongly held religious convictions might well see the word ‘belief’ as referring to their spiritual beliefs—a class of beliefs that are not going to be easily altered by evidence,” the study states. This could explain why religious respondents would respond negatively to questions concerning changing beliefs after finding new evidence or abandoning beliefs entirely.

A person would most likely feel more strongly about a belief in God than the merits of Coke versus Pepsi – hence the high negative correlation for religious respondents to questions concerning changing one’s beliefs.

“This non-equivalence never occurred to us at the time we were making up the belief identification items, perhaps because of our own secular biases,” Stanovich and Toplak speculate.

“That is the reason I wanted to get my Cognition paper out quickly so that this potential problem in the literature (the problem of inflating negative correlations with religiosity) could be quickly brought to people’s attention,” Stanovich told The Fix.

He’s not sure that the biases had done damage yet, but Stanovich wanted to ensure that the potential problems in the phrasing of questions would be addressed.

The study demonstrates a self-awareness from the authors that is surprising in academic research: They publicly admit that their biases interfered with decades of conclusions.

If the earlier studies had indicated that religious-minded individuals were more open-minded, the authors state they would have applied more scrutiny and “are sure that we would have pondered much longer and harder about why there was a correlation in the first place.”

The new study concludes that issues of bias and open-minded thinking are not unique to one set of beliefs or another. It also clarifies that the issue is not relegated to religious and secular respondents, but to beliefs of all kinds.

“Intolerance and lack of warmth correlations with conservatism, low intelligence, or openness virtually disappear once a more diverse range of social groups are included in the rating task,” the duo wrote:

It seems that liberals too can show relative intolerance—they just express it toward different groups (businesspeople, Christian fundamentalists, the wealthy, the military) that do not share their worldview or values. Our studies here are similar in that we found that correlations with religiosity and ideology were attenuated once AOT stimuli put protected values at risk more equally across the religiosity/ideological spectrum.

Accounting for potential bias against religious respondents can potentially be avoided with increased religious representation and intellectual diversity in psychological studies, they concluded. Otherwise, questions could continue to be phrased in ways that are misleading to religious individuals and create situations where they are unfairly maligned for being close-minded.

“The problem of religious people being less represented in lab research is simply a subset of the problem of the ideological monoculture in the social sciences,” Stanovich told The Fix. “This ideological monoculture is of course highly correlated with the monoculture of anti-religiosity.”

MORE: New book combats popular myths about science

IMAGE: Billion Photos/Shutterstock

Like The College Fix on Facebook / Follow us on Twitter