Should the risk of social desirability bias in survey studies be assessed at the level of each pro-environmental behaviour?

Reducing social desirability bias risk in survey studies (YouTube, 2m 16s)

Introduction

Social desirability bias and self-reported pro-environmental behaviour

Environmental psychology research typically relies on self-report questionnaires, asking people to report on their past pro-environmental behaviours or future behavioural intentions. The validity of self-reported pro-environmental behaviours can be negatively affected by social desirability bias – an individual’s tendency to over-report behaviours that are encouraged by the society to construct a more favourable self-image (Fisher, 1993; Lusk & Norwood, 2009; Powell, 2003; Paulhus, 1984; Kaiser et al., 1999). Socially desirable responding can increase measurement errors (Cote & Buckley, 1988) and negatively impact the validity of the results (Zerbe & Paulhus, 1987).

Social desirability bias has been identified as a key reason for survey respondents substantially over reporting pro-environmental behaviours (Kaiser et al., 1999; Bruni & Schultz, 2010; Cerri et al., 2019). One way of managing social desirability bias is to control for it by using a validated general trait level social desirability scale such as the MarlowCrowne scale (Crowne & Marlowe, 1960; Vesely & Klockner, 2020; Paulhus, 1991). This is the currently dominant approach used when aiming to increase the validity of self-reported past pro-environmental behaviours (e.g., Bratt et al., 2015; Panno et al., 2016; Moon et al., 2016) and future intentions (Lapinski et al., 2017; Haws et al., 2014).

Despite the dominance of this approach, empirical evidence is mounting that the overall correlation is low between the person-specific tendency to answer survey questions in socially desirable ways (trait social desirability) and the self-reported pro-environmental behaviours (Milfont, 2009; O’Brien et al., 2018; Vilar et al., 2020). A meta-analysis of 29 such studies report pooled correlations ranging from 0.06 to 0.11 (Vesely & Klöckner, 2020), suggesting a weak association only, especially in view of higher association with other selfreported behaviours, including domestic violence (-.18; Sugarman & Hotaling, 1997) and alcohol abuse (0.29; Welte & Russell, 1993). A possible explanation of Vesely and Klöckner’s (2020) findings might be that their analysis does not account for differences across pro-environmental behaviours. The authors explicitly note the substantial heterogeneity across the studies included in their meta-analysis (Vesely & Klöckner, 2020). Responding to the call by Vesely and Klöckner (2020) to tailor social desirability scales to a specific context, we propose a behaviour-specific approach to capture social desirability.

Behaviour-specific social desirability bias

Prior studies from other contexts suggest that investigating the value of behaviour-specific indicators of social desirability holds substantial promise. For example, in the context of selfefficacy – the belief in one’s ability to perform and have control over one’s behaviours across different contexts (Bandura, 1997) – behaviour-specific self-efficacy beliefs consistently emerge as stronger predictors of self-reported emotion regulation (e.g., self-injury; Hasking & Rose, 2016), including for the behaviour of risky drinking (Oei et al., 2007) and recycling (Tabernero & Hernandex, 2011). It is reasonable to assume, therefore, that beliefs about the social desirability of a specific pro-environmental behaviour might be more predictive of selfreported behaviours than a person-specific indicator. For example, even if a person scores highly on trait social desirability, they are unlikely to misreport a specific behaviour unless they perceive that behaviour as socially desirable or undesirable (Randall & Fernandes, 1991). Likewise, a person who scores low on trait social desirability might still overreport their engagement in specific behaviours that they believe are highly favourable in the environmental context.

Whether a survey respondent perceives a specific behaviour as socially desirable or not is, of course, not trivial to capture. Asking how socially desirable they perceive the behaviour to be is highly abstract and likely to elicit a very general response. Perceived social evaluation of engaging in a behaviour (i.e., how positively or negatively society views a behaviour) and how embarrassing it would be to admit or not admit engaging in a behaviour may represent more specific, tangible aspects that indicate the extent to which a behaviour is perceived as socially desirable (Fisher, 1993). People want to avoid embarrassment or disapproval of their response (Nyborg et al., 2016; Fisher, 1993), even in an anonymous survey. People rather lie to the researcher than admit to performing that behaviour (negative behaviours) or not performing that behaviour (positive behaviours; Fisher, 1993). If someone believes it is embarrassing to admit to a specific behaviour (e.g., littering) they will respond in a socially desirable manner to avoid feeling embarrassed. Avoiding embarrassment might be a key driver of over-reporting pro-environmental behaviours. Theoretically, a person is unlikely to over report a behaviour they believe is favoured by society and if they believe that it would not be embarrassing to admit engagement in that behaviour. In this study, we focus specifically on those two factors (social evaluation and embarrassment in admitting the behaviour) in attempt to capture behaviour-specific social desirability bias.

Aim of the current study

This study challenges the currently dominant approach of relying on measures of personspecific inclination to respond in socially desirable ways to questions about proenvironmental behaviours (trait social desirability) and offers and alternative. We propose that behaviour-specific social desirability is driven by social evaluation in whether the behaviour is perceived positively or negatively by society, and perceived embarrassment in admitting the behaviour. We test this assumption empirically and insights from our investigation are of immediate value to researchers who use self-report surveys to measure pro-environmental behaviours. The approach we propose, at the very least, offers to survey researchers an additional approach of detecting the risk of data being biased by socially desirable responding. Optimally, it offers a superior alternative to the currently dominant paradigm of person-specific (trait) social desirability.

Method

Participants

We collected data via a self-administrated survey using Prolific Scholar. All respondents were informed that the anonymity and confidentiality of their responses is protected under the National Health and Medical Research Council’s ethics guidelines. All studies in this article are approved by the university’s human research ethics committee (blind for review). In the participant information sheet, we listed all researchers with their contact details to increase respondent trust (Holbrook et al., 2003). We recruited 400 respondents who live in Australia and have travelled in the past five years as previous research recommends at least 100 participants per condition for an effect size of d = .40 and a power of .80 (Brysbaert, 2019).

Behaviours

We investigated 11 behaviours that differ in their likelihood of being affected by social desirability. We used a combination of home and vacation behaviours because the context of the behaviour could influence the effect of social desirability bias (Kormos & Gifford, 2014). Five behaviours are pro-environmental in nature and visible to others: not wasting food at a buffet on vacation; separating waste on vacation; turning off air-conditioning when leaving the hotel room; not littering on vacation; following the designated trail in a national park. We expect these behaviours to be most socially desirable. Two behaviours are pro-environmental in nature and occur in a more private setting: taking a short shower on vacation; recycling correctly at home. These pro-environmental behaviours have frequently been discussed in the literature as being difficult to accurately measure due to socially desirable responding (Miller et al., 2015; Lee & Jeong, 2018; Juvan et al., 2018; Ganglmair-Wooliscroft & Wooliscroft, 2017). Four behaviours are neutral and should not be affected by social desirability bias: visiting as many attractions as you can on vacation; eating meals at cafes and restaurants on vacation; doing the planning and booking before a vacation; and shopping online in the everyday context. Respondents indicated how often they perform the behaviour. For example, for waste separation on vacation, we asked respondents to indicate, in relation to the statement “When I am on vacation, I separate my waste”, how often this is the case using a slider scale ranging from never (0) to always (100). Responses were coded such that higher scores indicate higher reported frequency in displaying the behaviours.

Experimental conditions

After reporting the frequency of engaging in each behaviour, survey respondents were randomly assigned to one of two conditions. One group of respondents (person-specific condition, n=173) completed a short version of the Marlowe-Crowne social desirability scale (Crowne & Marlowe, 1960; Reynolds, 1982), which measures the extent to which a person is affected by social desirability bias. The original Marlowe-Crowne scale correlates highly
with other social desirability scales (Paulhus, 1984) and the short form reduced the risk of respondent fatigue without sacrificing validity (Reynolds, 1982). The measure contains 12 true-false items, such as “I sometimes feel resentful when I don’t get my way”. People with higher scores are assumed to have a higher propensity to respond in socially desirable ways (Zerbe & Paulhus, 1987). We calculated Pearson’s correlations to assess the association between survey respondents’ social disability scores and their self-reported behaviours. We expect small correlations for the pro-environmental behaviours and non-significant correlations for behaviours that are not likely influenced by social desirability bias.

The second group of respondents (behaviour-specific condition, n=183) rated, for each behaviour, societal beliefs (positive or negative perceptions of the behaviour in their society), and embarrassment when admitting to displaying the behaviours. Societal beliefs were recorded on a slider scale ranging from 0 (very negative) to 100 (very positive), for example, “Australian society views turning off the air-conditioner when leaving the hotel room as…”. Perceived embarrassment was recorded on a slider scale ranging from 0 (not embarrassed at all) to 100 (very embarrassed), for example, “If others knew that I take more food from the buffet than I can eat on vacation, I would feel…” The exact questions are provided in the Supplementary Information.

We correlated the summated score across the two measurements with their self-report behaviours. For socially desirable behaviours that are publicly visible, we expect a stronger correlation of behaviour-specific social desirability with the self-report pro-environmental behaviours (compared to the person-specific social desirability). For private-sphere socially desirable behaviours that are less likely to be affected by social desirability bias, we expect behaviour-specific social desirability to have a weaker correlation with self-reported behaviour than person-specific social desirability.

Results

Figure 1 illustrates the results, with Table 1 in the Appendix containing the exact correlation coefficients. Among the pro-environmental behaviours that are visible to others, only one (leaving no plate waste at buffet, r=.24, p<.05) was significantly correlated with person-specific social desirability (small-moderate correlation). This result suggests that the general propensity of a person to give a socially desirable responses is only of limited value in assessing the risk that that person reports biased self-reported pro-environmental behaviours. In contrast, behaviour-specific social desirability is moderately correlated with four pro-environmental behaviours: not leaving plate waste at buffets (r=.30, p<.001); turning off the air conditioner when leaving a hotel room (r=.33, p<.001); not littering on vacation (r=.35, p<.001); and following designated trails in national parks (r=.38, p<.001). Separating waste on vacation is the exception; only a weak correlation (r=.18, p<.05) is detected.

For pro-environmental behaviours that are not visible to others, neither person-specific nor behaviour-specific social desirability is significantly correlated with stated frequencies. For neutral behaviours – expected not to be affected by social desirability bias – the correlation between self-reported behaviours and person-specific social desirability bias is insignificant. A weak-moderate significant correlation exists between behaviour-specific social desirability bias and the self-reported behaviour of visiting as many attractions as possible on vacation (r=.28, p<.01).

Discussion

Accounting for social desirability bias is critically important to ensuring the validity of conclusions drawn from survey studies investigating pro-environmental behaviours. Environmental psychologists typically rely on person-specific (trait) indicators of social desirability to assess the risk of social desirability bias affecting self-reported proenvironmental behaviours. There is mounting evidence, however, that such person-specific indicators are not highly correlated with the self-reported frequency of socially desirable proenvironmental behaviours (Vesely & Klockner, 2020; Vilar et al., 2020). This study investigates the potential of a behaviour-specific approach to assessing the risk of responses being affected by social desirability.

Our results indicate that the person-specific (trait) social desirability bias is particularly unreliable in assessing the risk of social desirability bias from pro-environmental behaviours that visible to others – with only one significant correlation out of five behaviours.

Behaviour-specific social desirability bias, however, was significantly correlated with all five behaviours. For pro-environmental behaviours that are not visible to others, there was no difference between the two approaches. For behaviours that should not be affected by the behaviour-specific social desirability bias, the behaviour-specific social desirability bias has demonstrated a significant correlation with one behaviour (visiting as many attractions as possible on vacation). A plausible explanation is that visiting as many attractions as possible on vacation could be considered as a socially desirable behaviour, with previous literature demonstrating that people tend to over emphasise the role of experiential activities in creating travel happiness in comparison with the material products (Gilovich, 2015; Zhang et al., 2014). Overall, the person-specific (trait) social desirability approach was outperformed by the behaviour-specific social desirability approach in accessing socially desirable responding in self-report pro-environmental behaviours.

Our study offers initial empirical support for interpretations offered in previous studies that person-specific social desirability may be too general, thus not being informative in terms of the risk of people over-reporting specific pro-environmental behaviours (Vesely & Klockner, 2020; Ewert & Galloway, 2009). Our initial empirical evidence demonstrates that self-reported pro-environmental behaviours are more affected by behaviour-specific social desirability than by a person’s inherent inclination to give socially desirable responses in surveys. This insight has immediate methodological implications for researchers using selfreport methods to measure pro-environmental behaviours; they may wish to replace the person-specific measures with behaviour-specific measures or, at least, capture behaviourspecific risk of socially desirable responding in addition to the general person-specific inclination to do so.

Our study has its limitations: We investigated a limited numbers of pro-environmental behaviours and two behaviour-specific social desirability constructs. Future research could replicate this study with different sets of behaviours and test if measures other than societal evaluation and embarrassment improve correlations further. We did not account for cultural differences across respondents; we used Australian residents only in our study. Yet it is well understood that cultural background determines perceptions of social desirability (Kim & Kim, 2016; Keillor et al., 2001). Accounting for cultural differences among respondents, therefore, represents a key future avenue of investigation. Finally, we do not have an available objective measure of an actual pro-environmental behaviour. While it would be difficult to replicate our study using objective measures (given the large number of different behaviours which cannot be easily observed), future research could replicate this study with individual behaviours using objective behavioural measures to capture the gap between real behaviour and self-reported behaviour and investigate if this gap is explained by behaviour specific social desirability.

References

Bandura, A. (1997). Self-efficacy: The exercise of control. W.H. Freeman.

Bratt, C., Stern, P. C., Matthies, E., & Nenseth, V. (2015). Home, car use, and vacation: The structure of environmentally significant individual behavior. Environment and Behavior, 47(4), 436-473.

Brick, C., & Lai, C. K. (2018). Explicit (but not implicit) environmentalist identity predicts pro-environmental behavior and policy preferences. Journal of Environmental Psychology, 58, 8-17.

Brick, C., Sherman, D. K., & Kim, H. S. (2017). “Green to be seen” and “brown to keep down”: Visibility moderates the effect of identity on pro-environmental behavior. Journal of Environmental Psychology, 51, 226-238.

Bruni, C. M., & Schultz, P. W. (2010). Implicit beliefs about self and nature: Evidence from an IAT game. Journal of Environmental Psychology, 30(1), 95-102.

Brysbaert, M. (2019). How many participants do we have to include in properly powered experiments? A tutorial of power analysis with reference tables. Journal of Cognition, 2(1), 16.

Cerri, J., Thøgersen, J., & Testa, F. (2019). Social desirability and sustainable food research: A systematic literature review. Food Quality and Preference, 71, 136-140.

Cote, J. A., & Buckley, M. R. (1988). Measurement error and theory testing in consumer research: An illustration of the importance of construct validation. Journal of Consumer Research, 14(4), 579-582.

Crowne, D. P., & Marlowe, D. (1960). A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology, 24(4), 349-354.

Ewert, A., & Galloway, G. (2009). Socially desirable responding in an environmental context: Development of a domain specific scale. Environmental Education Research, 15(1), 55-70.

Fisher, R. J. (1993). Social desirability bias and the validity of indirect questioning. Journal of Consumer Research, 20(2), 303-315.

Ganglmair-Wooliscroft, A., & Wooliscroft, B. (2017). Ethical behaviour on holiday and at home: Combining behaviour in two contexts. Journal of Sustainable Tourism, 25(4), 589-604.

Gilovich, T., Kumar, A., & Jampol, L. (2015). A wonderful life: Experiential consumption and the pursuit of happiness. Journal of Consumer Psychology, 25(1), 152-165.

Hasking, P., & Rose, A. (2016). A preliminary application of social cognitive theory to Nonsuicidal self-injury. Journal of Youth and Adolescence, 45(8), 1560-1574.

Haws, K. L., Winterich, K. P., & Naylor, R. W. (2014). Seeing the world through GREENtinted glasses: Green consumption values and responses to environmentally friendly products. Journal of Consumer Psychology, 24(3), 336-354.

Holbrook, A. L., Green, M. C., & Krosnick, J. A. (2003). Telephone versus face-to-face interviewing of national probability samples with long questionnaires: Comparisons of respondent satisficing and social desirability response bias. Public Opinion Quarterly, 67(1), 79-125.

Huang, C. (2013). Relation between self-esteem and socially desirable responding and the role of socially desirable responding in the relation between self-esteem and performance. European Journal of Psychology of Education, 28(3), 663-683.

Juvan, E., Grün, B., & Dolnicar, S. (2018). Biting off more than they can chew: Food waste at hotel breakfast buffets. Journal of Travel Research, 57(2), 232-242.

Kaiser, F. G., Wölfing, S., & Fuhrer, U. (1999). Environmental attitude and ecological behaviour. Journal of Environmental Psychology, 19(1), 1-19.

Keillor, B. D., Owens, D., & Pettijohn, C. (2001). A cross—cultural/cross national study of influencing factors and socially desirable response biases. International Journal of Market Research, 43(1), 1-19.

Kim, S. H., & Kim, S. (2016). National culture and social desirability bias in measuring public service motivation. Administration & Society, 48(4), 444-476.

Kormos, C., & Gifford, R. (2014). The validity of self-report measures of proenvironmental behavior: A meta-analytic review. Journal of Environmental Psychology, 40, 359371.

Lapinski, M. K., Zhuang, J., Koh, H., & Shi, J. (2017). Descriptive norms and involvement in health and environmental behaviors. Communication Research, 44(3), 367-387.

Lee, W., & Jeong, C. (2018). Effects of pro-environmental destination image and leisure sports mania on motivation and pro-environmental behavior of visitors to Korea’s national parks. Journal of Destination Marketing & Management, 10, 25-35.

Lusk, J. L., & Norwood, F. B. (2009). Bridging the gap between laboratory experiments and naturally occurring markets: An inferred valuation method. Journal of Environmental Economics and Management, 58(2), 236-250.

Milfont, T. L. (2009). The effects of social desirability on self-reported environmental attitudes and ecological behaviour. The Environmentalist, 29(3), 263-269.

Miller, D., Merrilees, B., & Coghlan, A. (2015). Sustainable urban tourism: understanding and developing visitor pro-environmental behaviours. Journal of Sustainable Tourism, 23(1), 26-46.

Moon, S., Bergey, P. K., Bove, L. L., & Robinson, S. (2016). Message framing and individual traits in adopting innovative, sustainable products (ISPs): Evidence from biofuel adoption. Journal of Business Research, 69(9), 3553-3560.

Nyborg, K., Anderies, J. M., Dannenberg, A., Lindahl, T., Schill, C., Schlüter, M., & De Zeeuw, A. (2016). Social norms as solutions. Science, 354(6308), 42-43.

O’Brien, L. V., Meis, J., Anderson, R. C., Rizio, S. M., Ambrose, M., Bruce, G., & Kashima, Y. (2018). Low Carbon Readiness Index: A short measure to predict private low carbon behaviour. Journal of Environmental Psychology, 57, 34-44.

Oei, T. P., Hasking, P., & Phillips, L. (2007). A comparison of general self-efficacy and drinking refusal self-efficacy in predicting drinking behavior. The American Journal of Drug and Alcohol Abuse, 33(6), 833-841.

Panno, A., Carrus, G., Maricchiolo, F., & Mannetti, L. (2015). Cognitive reappraisal and proenvironmental behavior: The role of global climate change perception. European Journal of Social Psychology, 45(7), 858-867.

Paulhus, D. L. (1984). Two-component models of socially desirable responding. Journal of Personality and Social Psychology, 46(3), 598-609.

Paulhus, D. L. (1991) Measurement and control of response bias. In J.P. Robinson, P.R. Shaver & L.S. Wrightsman (Eds.), Measurement of personality and social psychological attitudes (pp. 17-59). Academic Press.

Powell, R. J. (2013). Social desirability bias in polling on same-sex marriage ballot measures. American Politics Research, 41(6), 1052-1070.

Randall, D. M., & Fernandes, M. F. (1991). The social desirability response bias in ethics research. Journal of Business Ethics, 10, 173-190.

Reynolds, W. M. (1982). Development of reliable and valid short forms of the Marlowe‐Crowne Social Desirability Scale. Journal of Clinical Psychology, 38(1), 119-125.

Seebauer, S., Fleiß, J., & Schweighart, M. (2017). A household is not a person: Consistency of pro-environmental behavior in adult couples and the accuracy of proxyreports. Environment and Behavior, 49(6), 603-637.

Sugarman, D. B., & Hotaling, G. T. (1997). Intimate violence and social desirability: A metaanalytic review. Journal of Interpersonal Violence, 12(2), 275-290.

Tabernero, C., & Hernández, B. (2011). Self-efficacy and intrinsic motivation guiding environmental behavior. Environment and Behavior, 43(5), 658-675.

Vesely, S., & Klöckner, C. A. (2018). Global social norms and environmental behavior. Environment and Behavior, 50(3), 247-272.

Vesely, S., & Klöckner, C. A. (2020). Social desirability in environmental psychology research: Three meta-analyses. Frontiers in Psychology, 11, 1395.

Vilar, R., Milfont, T. L., & Sibley, C. G. (2020). The role of social desirability responding in the longitudinal relations between intention and behaviour. Journal of Environmental Psychology, 70, 101457.

Welte, J. W., & Russell, M. (1993). Influence of socially desirable responding in a study of stress and substance abuse. Alcoholism: Clinical and Experimental Research, 17(4), 758-761.

Zerbe, W. J., & Paulhus, D. L. (1987). Socially desirable responding in organizational behavior: A reconception. Academy of Management Review, 12(2), 250-264.

Zhang, J. W., Howell, R. T., Caprariello, P. A., & Guevarra, D. A. (2014). Damned if they do, damned if they don’t: Material buyers are not happier from material or experiential consumption. Journal of Research in Personality, 50, 71-83.

Appendix

Correlations between pro-environmental behaviour intentions and both person and behaviour-specific social desirability.

Value from social desirability scale was extracted from group one