Measuring Growth Mindset; A Validation of a Three-item Scale (PDF)

MEASURING GROWTH MINDSET 1 Measuring Growth Mindset: A Validation of a Three-item Scale and a Single-item Scale in Youth and Adults Beatrice Rammstedt, David J. Grüning, Clemens M. Lechner GESIS – Leibniz Institute for the Social Sciences, Mannheim, Germany Note: This is the second (revised) version of this preprint. The revised manuscript is currently under review and has not yet been accepted for publication. Author notes. Editorial correspondence concerning this manuscript should be addressed to Beatrice Rammstedt, GESIS – Leibniz Institute for the Social Sciences, PO Box 12 21 55, 68072 Mannheim, Germany. E-mail: [email protected]  MEASURING GROWTH MINDSET 2 Abstract A growth mindset is the belief that personal characteristics, specifically intellectual ability, are malleable and can be developed by investing time and effort. In recent decades, numerous studies have investigated the associations between growth mindset and academic achievement, and large intervention programs have been established to train adolescents to develop a stronger growth mindset. However, methodological research on the adequacy of the measures used to assess growth mindset is scarce. In our study, we conducted one of the first comprehensive assessments of the psychometric properties of Dweck’s widely used three-item Growth Mindset Scale in two samples – adolescents (age 14–19 years) and adults (age 20–64 years) – and empirically demonstrate the comparability (i.e., measurement invariance) of the scale across these age groups. Furthermore, we identified and validated a single-item measure to assess growth mindset in research settings with severe time constraints. Our results show that both the short (three-item) and ultra-short (single-item) scales have acceptable psychometric properties in terms of of reliability, comparability, and validity. However, our analyses did not yield support for some of the central tenets of mindset theory, calling for future research on the criterion validity of growth mindset. Keywords: growth mindset, single-item scale, German validation, adolescents and adults, Big Five  MEASURING GROWTH MINDSET 3 Measuring Growth Mindset: Validation of a Three-Item and a Single-Item Scale in Adolescents and Adults A growth mindset is the belief that personal characteristics, specifically intellectual ability, are malleable and can be cultivated. In her theory on growth mindset, Carol Dweck (1999, 2006) distinguished two mindsets: a growth mindset and a fixed mindset. Whereas people with a growth mindset believe that their ability and intelligence can be developed over time, those with a fixed mindset believe that they were born with a certain invariant amount of ability that cannot be increased through effort and experience over time. According to Dweck’s theory, these mindsets differentially affect achievement motivation: Students with a fixed mindset tend to avoid challenges and negative feedback and to give up easily, whereas students with a growth mindset embrace challenges, persist in the face of setbacks, and learn from criticism (Dweck, 2016). The growth mindset and its implications for academic achievement have received an enormous amount of attention from policymakers, educators, and the media over the past two decades (e.g., Eisenberg, 2005; Paul, 2013; Smith, 2014). The White House (Obama Administration) even convened a special meeting in May 2013 entitled “Excellence in Education: The Importance of Academic Mindsets,” and Boaler (2013) hailed the findings of mindset research as the basis of “the mindset revolution that is reshaping education.” Subsequently, funding for mindset research was sought as a “national education priority” (Rattan et al., 2015, p. 723), resulting in a vast body of—mostly applied— research testing the basic assumptions of Dweck’s theory. In addition, given the postulated association between a growth mindset and academic achievement, extensive interventions were funded to increase academic performance and reduce rates of school dropout among adolescents (Sisk et al., 2018; Yeager et al., 2019).  MEASURING GROWTH MINDSET 4 Several meta-analyses have summarized the findings of the numerous studies on the antecedents and consequences of having a growth versus a fixed mindset. Their results indicate that a growth mindset is indeed positively associated with various self-regulatory processes and negatively associated with psychological distress (Burnette et al., 2013; Burnette, et al., 2020). This supports Dweck’s assumption that a growth mindset is related to achievement motivation. Regarding her second central assumption—namely, that a growth mindset is predictive of academic achievement—the evidence is less clear. Several comprehensive national/regional and international large-scale studies, for example, a study of mindset among students in California’s CORE school districts (Claro & Loeb, 2019) and the OECD’s Programme for International Student Assessment (PISA; OECD, 2019), support the hypothesized link between growth mindset and academic achievement. In PISA, for example, students who reported a fixed mindset scored on average about one fourth of a standard deviation (23– 32 scale points) lower in all skills assessed.1 However, in other studies (both small- and large-scale), the association between growth mindset and academic achievement was found to be zero, or even negative, thus suggesting a detrimental effect of a growth mindset on achievement (e.g., Bahník & Vranka, 2017). A recent meta-analysis by Sisk and colleagues (2018) summarized the findings of the existing 273 studies on growth mindset and found only a very weak positive association overall between growth mindset and academic achievement. The effectiveness of growth mindset interventions is also debated. Based on their meta-analysis of 43 such interventions, Sisk et al. (2018) concluded that overall effects of mindset interventions on academic achievement were weak (but see Yeager & Dweck, 2020, who criticized this conclusion, arguing that the effect size is acceptably large). However, some results of the studies reviewed in the 1 Notably, this effect was with differences between 12 to 17 points, that is, a standardized effect size between .04 to .06, comparatively small in Germany.  MEASURING GROWTH MINDSET 5 meta-analysis indicated that students with low socioeconomic status or students who are academically at risk might benefit from such interventions. Given the vast amount of attention that growth mindset has received in both academic and applied circles, and the vast body of research on the construct, one would expect that this line of research would be based on a set of well-validated scales for the assessment of growth mindset. This, however, is not the case. There are few, primarily two, established scales assessing growth mindset, namely a more comprehensive, six- to eight-item scale2 (Dweck, 1999) and its short-scale version, comprising three-items, both developed as part of Dweck’s (Dweck et al., 1995) broader Implicit Theories Scale. Surprisingly little research has been conducted on the psychometric properties of the scales (see Mikiff et al., 2017), especially regarding the short, three-item scale. Regarding the eight- item scale, the few studies conducted to date have yielded mixed results regarding the quality of the existing growth mindset scales. Levy and Dweck (1999) as well as Erdley and Dweck (1993) report only moderate internal consistencies (.62 and .71, respectively) and stabilities across a short one-week interval (.70 and .64, respectively). In addition, two studies proved that the eight items did not fit to a unidimensional model (Midkiff et al., 2017; Troche & Kunz, 2020). For the three-item scale Dweck herself conducted a series of validation studies proving its high internal consistency (.94 to .98) and retest-stability (.80 across a two-week interval) as well as its unidemensionality (Dweck et al., 1995). In these studies she could also show that growth mindset measured by the three-item scale is largely independent from other constructs such as intelligence or optimism. The current trend of assessing growth mindset not only in individual diagnostic settings but increasingly also in large-scale surveys with extreme time constraints raises the need for a highly 2 This eight-item scale is sometimes referred to as the Implicit Theories of Intelligence Scale (IT IS; see, e.g., Troche & Kunz, 2020) or the Growth Mindset Scale (Midkiff et al., 2017)  MEASURING GROWTH MINDSET 6 efficient, parsimonious, and valid assessment of this construct. Because of the lack of validation studies to guide item selection, current large-scale studies have adopted different subsets of the existing growth mindset items in an ad hoc manner, thus yielding different solutions for the different surveys. The latter may gravely hamper the comparability and replicability of the respective research findings. Whereas the CORE survey of 4th–7th grade students in California selected four items from Dweck’s eight-item scale (Claro & Loeb, 2019), PISA used a single-item measure by selecting one item from Dweck’s three-item scale (OECD, 2021), and a nationwide survey of high school students in Chile used two items from Dweck’s six-item scale (Claro et al., 2016). To the best of our knowledge, no studies have psychometrically validated a short (e.g., three-item) or ultra-short (single-item) measure of growth mindset for use in settings where questionnaire space is severely restricted. Unsurprisingly, given the scope of growth mindset, the vast majority of studies on the construct focus on primary and secondary school settings, and thus investigate children and adolescents. Also, the above-mentioned original growth mindset measures were developed and validated for these populations. However, the relevance of a growth mindset for adults is often stressed (see Han & Stieha, 2020, for a review of recent applications in human resource development). The few studies investigating growth mindset in adult samples (usually college students) have simply adopted the usual growth mindset items verbatim without methodologically testing their appropriateness for this age group (see, e.g., Midkiff et al., 2017; Thompson et al., 2013). In sum, research on growth mindset does not currently rest on a solid psychometric footing. In addition to this general need for further scrutiny of the psychometric quality of commonly used growth- mindset measures, there is also a need for short and ultra-short measures of growth mindset for use in survey research—and a need for scales that can be validly applied to adult samples. The Present Study  MEASURING GROWTH MINDSET 7 The aim of the present study is twofold: First, given the lack of empirical validation studies, we aim to validate Dweck’s widely used three-item Growth Mindset Scale and examine its psychometric properties in terms of descriptive statistics, reliability, factorial validity, and its nomological network. To this end, we have developed a new German-language version of the instrument. We will conduct a validation in parallel for the typical target group of adolescents as well as for adults who are no longer in education, for which no validated instruments assessing growth mindset exist so far. We therefore focus our analyses on testing the applicability (including measurement invariance) of the scale in adolescent and adult populations. Therefore, we use an adolescent sample and an adult sample, both of which are heterogeneous with regard to the target population. Our validation strategy will also include the assessment of the nomological network for the growth mindset measure. However, given the lack of such psychometric validation studies on growth mindset, it is difficult to infer clear-cut hypotheses regarding the criterion validity of the scale. First, for adult populations—and especially for adults who are no longer in education—empirical findings are extremely scarce, and it is somewhat unclear what meaningful criterion variables might be. For adolescents, very few associations between growth mindset and external criteria are substantial and replicable. Although Dweck’s mindset theory postulates a link between growth mindset and academic achievement, this link was not replicated in several recent studies (e.g. Bahnik & Vranka, 2017; see also meta-analyses by Sisk et al., 2018). Moreover, socioeconomic differences in growth mindset are debated (see Destin et al., 2019; King & Trinidad, 2021). Similarly, a stronger fixed mindset was postulated for girls/women (see Dweck, 1999), which was also not replicated empirically (e.g. Spinath & Stiensmeier-Pelster, 2001). With regard to age and personal characteristics such as optimism, self- esteem, and intelligence, theoretical assumptions and empirical findings are in agreement that these are largely unrelated to growth mindset (e.g., Dweck et. al, 1995; Dweck, 1999). Similarly, basic  MEASURING GROWTH MINDSET 8 dimensions of personality in terms of the Big Five - investigated only in one study so far (Zamarro et al., 2016)- seem to be mostly independent from growth mindset. Only for aspects of self-regulation there is support for a positive—albeit small— association to growth mindset (see meta-analysis by Burnette et al., 2020). Thus, based on previous studies and on theoretical assumptions, we expect the growth mindset measure to be mostly independent from other personal as well as socio-demographic variables. Also, with regard to academic achievement, the existing data is not clear enough to formulate directed hypotheses. Only with regard to self regulation, we can assume a small positive association. The second aim of our study is to identify and validate an ultra-short, single-item measure of growth mindset to be used in large-scale settings (either in adolescent or adult populations) in which even the three-item scale may be too lengthy. By doing so, we hope to support large-scale studies in using a highly time-effective but validated alternative to the three-item Growth Mindset Scale. In sum, with the present paper, we attempt to increase the comparability of future studies by providing extensive information on the psychometric properties of two very short, viable options for assessing growth mindset. Method Samples We used data from a large multi-thematic, four-wave survey in which several measurement instruments were validated. The survey comprised a total of four assessment waves and two samples residing in Germany, one targeting adolescents aged 14 to 19 years (Sample A), the other targeting adults aged 20 to 64 years (Sample B). The adolescent sample had a quota for gender, whereas the adult sample had quotas for age, gender, and educational attainment according to the German  MEASURING GROWTH MINDSET 9 Microzensus 2011. The survey was conducted by the online survey provider respondi AG. Respondents received a small monetary remuneration for participation. For our analyses of growth mindset, we used data from those adolescents and adults who participated in the second survey wave (fielded between February 2 and February 21, 2021; n = 365 adults; n = 362 adolescents), in which we first assessed growth mindset. A retest of growth mindset followed approximately four months later in the fourth and final survey wave (fielded between May 21 and June 6, 2021; n = 263 adults; n = 171 adolescents). Some additional measures that we used to assess the nomological network of growth mindset were also taken from the first (January 21 to February 1, 2021; n = 365 adults and n = 362 adolscents) and third (March 8 to March 30, 2021; n = 300 adults and n = 256 adolescents) survey waves. The median spacing between the two assessments was 109 days. For an independent assessment of the psychometric properties of our proposed single item measure, we (a) analyzed the German data of PISA 2018 (OECD, 2020; here Sample C), in which the same item (but with a 4-point response scale) was assessed based on N = 4,235 15-year olds and assessed the single item in a separate online sample (Sample D) of N = 794 adults (fielded between February 17 and February 26, 2022). Sample D had the same quotas for gender, age, and education and was collected via the same provider as the initial adult sample. Table 1 shows the sociodemographic profiles of the samples A, B, and D. The raw data of these samples is available from the OSF project website at https://osf.io/etx7j/?view_only=9b20f680eafa40cab883f76628b4cff1 (anonymized link). Instruments Growth Mindset  MEASURING GROWTH MINDSET 10 We measured growth mindset using a newly developed German-language adaptation of Dweck’s three-item measure (Dweck et al., 1995), which is the most widely used growth mindset short scale in applied research to date (e.g., Blackwell et al., 2007; Romero et al., 2014; Yeager et al., 2019). All three items are formulated as fixed mindset statements. They are answered on a 6-point rating scale ranging from 1 (fully agree) to 6 (fully disagree). We translated the three items using the TRAPD approach (Harkness et al., 2010). Two experts in personality assessment—both of whom are native speakers of German with a very good command of English—translated the items from English to German. Two independent experts in cross-cultural research and personality assessment then reviewed these translations and suggested improvements as necessary. At a reconciliation meeting in which all the aforementioned experts participated, we resolved any remaining disagreements through discussion, and agreed on the final translation. The original English-language source version and the final translations of the instruction, items, and response-scale labels are provided in the Appendix. Note that for all questions in the questionnaire (including growth mindset and the following criterion variables), there was a questionnaire split such that adolescents were addressed by the informal you (German “Du”), whereas adults were addressed with the formal you (German “Sie”). Correlational Variables Based on previous research and theoretical assumptions we selected a set of correlates to validate the three-item Growth Mindset Scale as well as the single-item measure derived from it and explore the nomological network of the construct. These were mainly assessed in our Samples A and B, some of them also in the Sample D. First, to explore whether there are differences in growth mindset across different subsegements of the population, we investigated associations between growth mindset and sociodemographic characteristics. The latter included sex, age, education (clustered according to  MEASURING GROWTH MINDSET 11 the highest degree into low, intermediate and high. Low refers to a lower secondary level providing a basic general education; intermediate also refers to lower secondary level that provides a more extensive general education and an opportunity to continue on to upper secondary level; high refers to upper secondary level that leads to a higher education entrance qualification), parental education (only in Sample A), employment status (only in Samples B and D; coded as unemployed (1) vs. (self- )employed (2)), and income (only in Samples B and D; assessed by 17 categories ranging from <300€ to ≥10.000€ /month). Second, to establish the nomological network of growth mindset in relation to other established individual-difference constructs, we investigated associations between growth mindset and other key personality constructs, namely, the Big Five (measured using the BFI-2-S; Soto & John, 2017; German adaptation Rammstedt et al., 2018), self-regulation skills, and goal regulation (all only in Samples A and B). The measures of the latter two constructs were taken from the German-language version of the Behavioral, Emotional, and Social Skills Inventory (BESSI-G; Lechner et al., 2021). Third, to test the (concurrent) criterion validity, we tested associations between growth mindset and indicators of achievement and ability, namely, self-reported final overall grade and grades in various subjects (math, German, English, history, and biology) in Sample A; and crystallized intelligence (gc) as well as fluid intelligence (gf) in Samples A and B. We assessed crystallized intelligence (gc) with the short version of the Berliner Test zur Erfassung fluider und kristalliner Intelligenz (BEFKI GC-K; Schipolowski et al., 2013). BEFKI contains 12 items that cover basic knowledge from humanities, natural and social sciences. Reliability of the 12-item BEFKI sum score in our sample was α = .68.  MEASURING GROWTH MINDSET 12 To assess fluid intelligence (gf), we used 12 items from the International Cognitive Assessment Resource (ICAR; Condon & Revelle, 2014) assessing Verbal Reasoning (VR), Letter and Number Series (LN), and Matrix Reasoning (MR). Reliability of the 12-item sum score of ICAR in our sample was α = .73. A full documentation of the criteria variables used can be found on the OSF project website at https://osf.io/etx7j/?view_only=9b20f680eafa40cab883f76628b4cff1 (anonymized link). Data Analysis We analyzed the quality of our German-language adaptation of the three-item Growth Mindset Scale in terms of its descriptives, reliability, its nomological network, and factorial validity. To investigate the factorial (structural) validity of the three items, we estimated a unidimensional confirmatory factor analysis (CFA) model via the R package lavaan using a robust maximum likelihood estimator (MLR) in Samples A and B. Because a single-factor model with three indicators is just-identified (df = 0) and would not allow for model fit assessment via fit indices, we tested an essentially tau-equivalent model (i.e., a model in which all items have identical factor loadings). The essentially tau-equivalent model is also more parsimonious. We assessed model fit via Hu and Bentler’s (1999) commonly used heuristics for fit indices (i.e., , CFI ≥ .950, RMSEA ≤ .060, and SRMR ≤ .080). To test the applicability and comparability of the three-item Growth Mindset Scale for adolescent and adult populations, we investigated exact measurement invariance using multiple-group CFA. We tested three levels of invariance3: metric invariance (same factor loadings), scalar invariance (same factor loadings and item intercepts), and strict invariance (additionally same residual variances). To evaluate 3 Note that testing configural measurement invariance is neither possible nor necessary when using an essential tau- equivalent factor model.  MEASURING GROWTH MINDSET 13 invariance across samples, we compared the fit of these models using the differences in goodness of fit (ΔGOF), Δχ2, and the sample-size adjusted Bayesian Information Criterion (aBIC) (see Chen, 2007; Putnick & Bornstein, 2016; Rutkowski & Svetina, 2014). Regarding ΔGOF, we followed the simulation-based guidelines proposed by Chen (2007), which stipulate that differences of ΔCFI ≥ .010, ΔRMSEA ≥ .015, ΔSRMR ≥ .030 when moving from a configural to a metric invariance model suggest loading non-invariance, whereas differences of ΔCFI ≥ .010, ΔRMSEA ≥ .015, ΔSRMR ≥ .010 suggest intercept non-invariance when comparing scalar to metric invariance. Regarding aBIC, lower values indicate a better balance between model fit and complexity (or parsimony). Results As outlined above, our study aimed, first, to validate the German-language adaptation of Dweck’s three-item Growth Mindset Scale and to investigate its applicability to adolescent and adult samples. Second, we aimed to identify and validate an ultra-short, single-item measure for the assessment of growth mindset in research settings with severe time constraints. Validation of the Three-Item Growth Mindset Scale Based on adolescent and adult data from Samples A and B we investigated the quality of our German-language adaptation of the three-item Growth Mindset Scale in terms of its descriptives, reliability, and factorial validity. We also investigated its associations with a set of sociodemographic variables, personality traits, and indicators of achievement and abilities. To investigate the applicability of the scale to adolescent and adult populations, we conducted all analyses separately for adolescents and adults, and we compared the results and formally tested the measurement invariance of the scale across the two samples. Descriptive Statistics and Reliability  MEASURING GROWTH MINDSET 14 The upper part of Table 2 shows the means, standard deviations, and skewness of the three-item scale, as well as its reliability coefficients in terms of Cronbach’s alpha and test–retest stability over a roughly 4-month (i.e., 109 days in the median) period. Detailed results on the item level are provided on the OSF project website (https://osf.io/etx7j/?view_only=9b20f680eafa40cab883f76628b4cff1). Means, standard deviations, and skewness were highly comparable across the adolescent and adult samples (Samples A and B). Indicators of reliability were also similar for both samples. Internal consistency coefficients were high in both populations, with α = .83 in the adolescent sample and α = .90 in the adult sample. Test–retest stability, rtt, over a 4-month period was somewhat lower than the internal consistencies, with rtt = .67 in the adolescent sample and rtt = .45 in the adult sample. Note that rtt reflects not only unreliability (i.e., classical measurement error) but also true changes and state fluctuations in a construct, rendering it a conservative estimate of scale reliability. Factorial Validity As shown in Table 3, an essentially tau-equivalent single-factor CFA model showed good fit in both the adolescent (Sample A) and the adult sample (Sample B; e.g., CFI ≥ .985, RMSEA ≤ .094). All standardized factor loadings were high (λ > .75). The German-language version of the Item 3 had comparatively the lowest standardized loading on the common factor in both samples (.75 in Sample A and .83 in Sample B), whereas the German-language version of the Item 2 loaded highest on average (.81 in Sample A and .92 in Sample B). Measurement Invariance As can be seen from the right-hand side of Table 3, results suggest that the three-item Growth Mindset Scale is largely measurement invariant across adolescents and adults, even on the strict level. Thus, researchers can make valid comparisons with the mean of the three-item Growth Mindset Scale’s manifest scores and perform meaningful regression analyses across the two age groups.  MEASURING GROWTH MINDSET 15 Associations with External Criteria To investigate the nomological network of the three-item Growth Mindset Scale, we correlated the measure with different socio-demographic, psychological and achievement indicators which have been investigated in earlier validation studies. Based on previous findings we assume – as outlined in the introduction – mostly zero associations. Only with regard to self- and goal regulation meta-analytic findings suggest small positive correlations. The left-hand side of Table 4 shows—separately for adolescents and adults—the correlations between the three-item Growth Mindset Scale and sociodemographic variables (age, sex, own and parental education, employment status, income), central personality characteristics (the Big Five, self- regulation, goal regulation) and aspects of achievement and ability (school grades, crystallized intelligence, and fluid intelligence). The overall picture indicates two things: First and in line with our assumption, growth mindset showed mostly zero-associations with the investigated sociodemographic variables and personality characteristics (with an average association of |.07| in the adolescent sample and |.06| in the adult sample). Second and more importantly for the current research question, results were highly comparable across adolescents and adults (column vector correlation after applying the Fisher-z transformation to the individual correlations: r = .73). Looking at the associations with socio-demographic variables, we found support for the previous finding that, on average, males and females do not differ in terms of growth mindset. There were no strong age differences, either. Further, we found no effects of own or parental education, indicating that socioeconomic background and educational attainment are unrelated to growth mindset.