How do you best measure self-reported attitudes on socially sensitive topics? For instance, how much faith would you have in responses to a question asking whether someone wanted to have Black neighbors? Some people might think that such an item would create “socially desirable responding”, meaning that people would report a socially acceptable response even if they felt otherwise.
One alternative to asking people such direct questions is to use more subtle, indirect questions. These indirect questions deal with racial issues, but people may feel more freedom to express negative racial attitudes if they can be justified by referring to other factors. For example, someone who wants to know about general racial attitudes might ask a question about affirmative action. Someone holding anti-Black attitudes may feel free to express disagreement with affirmative action because such an opinion can be justified by arguing that no race should be given preferential treatment. However, while these indirect items may be lower on socially desirable responding, they may also be imperfect measures of racial attitudes because they introduce a lot of potentially irrelevant opinions into responses (like actual beliefs about educational policy).
So researchers interested in racial attitudes are stuck with a tradeoff. Do they use very direct items that may suffer from socially desirable responding? Or do they use indirect items that are lower on socially desirable responding but may introduce irrelevant information into responses? In a recent paper, I tested whether more direct or indirect self-report items are better measures of racial attitudes.
But first, let's discuss how can you test which questions best tap into racial attitudes. One way is to compare how well responses to one measure correspond with responses to another, related measure. For instance, height and weight are distinct constructs, but they are correlated; that is, knowing someone’s height will help a lot if you had to guess their weight. I’ll also get stronger correlations with people’s weight if I use a very precise measure of height, like a ruler with markings down to the centimeter, than if I use an imprecise measure of height, like a ruler that only includes four-inch increments. Using less precise measures results in more error, and this error only reduces the correlation between any two measures. As a result, better, more precise measures will minimize error and maximize the correlation with a related outcome.
I adopted a similar approach by comparing how well different self-reported items of racial attitudes correlated with an implicit measure of racial attitudes (the Implicit Association Test, or IAT). A lot of previous work suggests that explicit, self-reported attitudes and implicit attitudes are distinct, but related, so the best measure of self-reported racial attitudes should have the strongest correlation with the IAT.
During two years of data collection on the Project Implicit Demonstration Site, more than 800,000 participants completed an IAT and a subset of over 400 self-report questions. Each question was also rated by other researchers on whether it was a more direct or indirect measure of racial attitudes, and whether responses could produce socially desirable responding.
Results showed that more direct items, like comfort with having Black neighbors, were thought to produce more socially desirable responding. However, the more direct items also produced the strongest correlations with the IAT, suggesting they were better measures of racial attitudes. The answer from these data seems clear: the best way to measure self-reported racial attitudes is to use very direct items, even if such items have the chance to produce more socially desirable responding.
This is not to say that there was no impact of socially desirable responding in the data. In a previous paper, domains with higher potential for socially desirable responding, like race, had much lower correlations between explicit and implicit attitudes than domains with lower potential for socially desirable responding, like preferences for shorts vs. pants. This suggests that some participants likely altered their responses when asked about more socially sensitive issues. But combining those earlier results with this recent work still indicates that when looking within any one attitude, like race, it’s better to use more direct items.
Finally, it’s worth noting that all participants completed the measures online, so it’s possible that results would look different in answers had to be reported in person. But the current data suggest that the best way to understand someone’s self-reported racial attitudes is to simply ask about them.
The article, “The best way to measure explicit racial attitudes is to ask about them”, is in press at Social Psychological and Personality Science. All measures, data, codebooks and analysis scripts are available here: https://osf.io/e9shx/.