Maintaining Success Rates; Does Statway® sustain its impact (PDF)

CARNEGIE MATH PATHWAYS TECHNICAL REPORT Maintaining Success Rates: Does Statway® sustain its impact as it scales to new classrooms and institutions? March 2017 Melrose Huang and Hiroyuki Yamada Carnegie Foundation for the Advancement of Teaching Stanford, CA  CARNEGIE STATWAY SUCCESS 2 Abstract Statway is an accelerated developmental mathematics intervention program for college students who are not yet prepared to succeed in a college-level math course. The Carnegie Foundation for the Advancement of Teaching created the initiative to increase student success rates in developmental mathematics and subsequent credit-bearing, college-level mathematics. Statway differs from traditional developmental mathematics sequences in that it allows students to complete both their developmental and college-level mathematics requirements in a single pathway over the course of one academic year. A previous study (Yamada & Bryk, 2016) found that students—across all sex and race/ethnic groups—who enrolled in Statway, had higher rates of completion for both developmental and college-level math compared to carefully matched students in traditional developmental math programs. Statway provides a multifaceted approach to addressing challenges that impede student success through psychosocial and academic supports, as well as lessons based on examples relevant to students’ lives. The course is structured so the same cohort of students is enrolled in a consecutive two- term sequence with the same faculty instructor, students work collaboratively, and instructors learn and apply new and more interactive teaching methods. Faculty also receive professional development and support through a networked improvement community (NIC) that connects them with other practitioners, administrators, and researchers from diverse institutions across the nation. We assessed the effectiveness of the program in its third and fourth years by utilizing a multilevel modeling approach with propensity score matching to compare Statway student outcomes with their counterparts in traditional developmental mathematics programs. We found that in comparison to traditional developmental mathematics students, Statway students had a significantly higher likelihood of successfully attaining college-level mathematics credit— even when comparison group students were allowed twice the amount of time to achieve the same outcome. These results are consistent across all sex and race/ethnic subgroups, as well as in almost all participating classrooms and institutions. Additionally, in replicating earlier findings from the first two years of the program, the results suggest that Statway continues to advance equitable outcomes even as it scales to greater numbers of students in an increasing number of institutions. This report also suggests potential areas for future investigation and research.  CARNEGIE STATWAY SUCCESS 3 This report provides an important sequel to the Carnegie Foundation’s previous assessment of the effectiveness of the first two years of Statway®, a mathematics program designed to expedite the process by which students may fulfill their developmental requirements and earn college mathematics credit. Employing a multilevel modeling technique with propensity score matching, the previous peer-reviewed study found that Statway considerably improved college mathematics success rates among students from the 2011 and 2012 cohorts (Yamada & Bryk, 2016). Since then, Statway has continued to scale within current institutions and spread to new ones, nearly doubling its enrollment and expanding its geographic reach to three new states, and expanding in total four-fold since it began five years ago (Huang, Hoang, Yesilyurt, & Thorn, 2016). Scaling innovations in education is notoriously difficult (Cannata & Nguyen, 2015). Thus, the current study, which focuses on the third and fourth years of Statway’s implementation, seeks to further substantiate Statway as an evidence-based initiative by replicating results from the previous analysis. To better inform our efforts to address challenges in scaling Statway, the study also examines variation in performance across instructors and institutions. The Developmental Mathematics Problem Level of attainment in higher education predicts income and is a primary means of attaining upward social mobility (Deming & Dynarski, 2009). As of 2014, 12.3 million individuals were enrolled in community colleges in order to develop relevant knowledge and skills to participate meaningfully in the workforce (American Association of Community Colleges, 2016). For some, successfully completing community college serves as the final step toward earning an associate degree or professional certification. For others, community college represents a stepping stone for transitioning to a four-year institution and working toward a higher degree. Unfortunately, without first achieving college-level mathematics credit, many of these students can never realize their dreams. Approximately 60% of community college students are assigned to take one or more developmental mathematics courses. However, three years after enrollment, 80% of these students do not successfully complete the course, thereby precluding them from going on to earn college-level mathematics credit (Bailey, Jeong, & Cho, 2010). Minority students are disproportionately required to take more developmental mathematics courses, but are also less likely to achieve college-level mathematics credit than white students are (Bailey et al., 2010). Developmental mathematics sequences thus act as gatekeepers in granting or denying students, particularly those from traditionally disadvantaged groups, the means of continuing to progress toward their academic and professional goals. Consequently, developmental mathematics reform is one of the most salient education and equity issues of our time. Prior research suggests that a number of factors in how developmental mathematics courses are traditionally designed and implemented may adversely affect student outcomes. At present, developmental mathematics programs are generally offered as a sequence of foundational courses that increase in difficulty, starting with basic arithmetic and culminating in intermediate algebra (Stigler, Givvin, & Thompson, 2010). Prior to even attempting a college- level mathematics course, students must first successfully make it through this sometimes lengthy sequence (Hodara, 2013). Depending on their placement test scores, students may be  CARNEGIE STATWAY SUCCESS 4 required to take three or more of these prerequisite courses; most students, however, are placed at least two levels below college math, meaning a minimum of a year in non-credit- bearing math (Cullinane & Treisman, 2010). Among those who do attempt a developmental mathematics course, even those who successfully complete one or multiple courses, may find themselves struggling to persevere through the entirety of the sequence (Bailey et al., 2010). Structural barriers to student success and enrollment in developmental mathematics courses may be further exacerbated by a myriad of other factors. For instance, researchers have found little use of evidence-based classroom activities and practices that may encourage student success by facilitating deeper learning (Mesa, 2011; National Research Council, 2002). In addition, much of the material presented in conventional developmental mathematics courses lacks accessibility and relevance by failing to demonstrate real world applications of mathematics or incorporate student interests (Carnevale & Desrochers, 2003; Edwards, Sandoval, & McNamara, 2015). Lamentably, students may believe that successfully learning mathematics is a skill some people are born with, while others are inherently incapable of becoming successful mathematics students. Such beliefs may derive from previous negative experiences with mathematics, which may also induce excessive anxiety among students when they are confronted with challenging concepts (Blackwell, Trzesniewski, & Dweck, 2007; Haynes, Perry, Stupinsky, & Daniels, 2009). Students who identify with traditionally marginalized groups may be especially vulnerable; in particular, African-Americans and females may internalize stereotypes of themselves as poor mathematics students (Cohen, Garcia, Purdie-Vaughns, Apfel, & Brzustoski, 2009). Additionally, students with limited language and literacy skills, such as students from immigrant backgrounds, may find it difficult to effectively communicate mathematical reasoning or understand mathematics problems (Gomez, Rodela, Lozano, & Mancevice, 2013). Statway’s Theory of Improvement Statway was initiated by the Carnegie Foundation for the Advancement of Teaching as an innovative means to improve upon the generally dismal success rate of students in developmental mathematics. Statway offers students an accelerated pathway by which they can meet their developmental mathematics requirements and achieve college-level mathematics credit in statistics within a single academic year. The theory behind the program’s design is underpinned by six key drivers (see Figure 2): 1) an accelerated pathway through college-level mathematics, 2) curriculum and instruction principles that emphasize productive struggle, explicit connections, and deliberate practice, 3) explicit integration of socioemotional supports in the form of productive persistence, 4) language and literacy supports, 5) professional development for faculty, and 6) support of a networked improvement community comprised of researchers, practitioners, college administrators, and designers (Bryk, Gomez, Grunow, & LeMahieu, 2015). Designing an accelerated pathway through college-level mathematics. Statway offers an accelerated and simplified means by which students can meet their developmental and college- level mathematics requirements. As shown in Figure 1, under traditional developmental mathematics programs, students placed into elementary algebra within the sequence would  CARNEGIE STATWAY SUCCESS 5 require one and a half years, at minimum, to achieve college-level mathematics credit. Others, who may repeatedly fail developmental mathematics courses and/or face scheduling conflicts or do not have full course loads due to family and work duties, may require even more time to achieve the same outcome (Bailey et al., 2010; Cullinane & Treisman, 2010). Accordingly, Statway is offered as an intensive two-course sequence that is intended to be taken consecutively by the same cohort of students and with the same faculty member. Its curriculum integrates elementary and intermediate algebra with college-level statistics, and generally meets liberal arts or general education requirements for introductory college-level mathematics at both participating community colleges and four-year universities. In particular, its content is designed to prepare students working toward non-STEM majors (Cullinane & Treisman, 2010; Hodara, 2013). Instructional emphasis on productive struggle, explicit connections, and deliberate practice. Statway’s instructional system is rooted in three evidence-based practices that promote student learning: productive struggle, explicit connections, and deliberate practice. Through productive struggle, students work through rich mathematics assignments that challenge them to make sense of mathematical concepts on their own, while still offering enough scaffolding such that students can feasibly complete the task (Schmidt & Bjork, 1992; Hiebert & Grouws, 2007). The idea of explicit connections suggests that understanding mathematical concepts and procedures in relation to one another can foster deeper learning, whereas traditional mathematics instruction generally places too much emphasis on developing procedural knowledge as discrete process, while offering little to bolster students’ conceptual knowledge (Boaler, 1998; Hiebert & Grouws, 2007). Similarly, rather than emphasizing repetitive tasks and procedural knowledge through rote memorization, deliberate practice entails assigning tasks specifically designed to bridge gaps in student knowledge, while enriching student understanding of core concepts (Ericsson, 2008; Ericsson, Krampe, & Tescher-Römer, 1993; Pashler, Rohrer, Cepeda, & Carpenter, 2007). Integration of socioemotional supports via Productive Persistence. Statway hosts an intervention package made up of established change ideas that foster a sense of productive persistence among students. These activities strive to empower students’ belief in themselves as mathematics learners (Dweck, 2006), mitigate their mathematics-induced anxiety (Jamieson, Mendes, Blackstock, & Schmaeder, 2010), and heighten their sense of belonging in the classroom (Walton & Cohen, 2011). Other activities focus on helping students develop good learning strategies (Dweck, Walton, & Cohen, 2011; Yeager & Walton, 2011). The productive persistence change package includes activities such as formalizing team members’ roles during group work, committing to contributing to a healthy learning environment and personal engagement via a contract, and completing a reading about the malleability of the brain and its potential to acquire new knowledge and skills. Inclusion of language and literacy supports. Language and literacy supports are embedded in Statway’s curriculum and related materials to assist student acquisition of key vocabulary, identification of core concepts in question prompts and written materials, as well as  CARNEGIE STATWAY SUCCESS 6 development of critical thinking and reasoning skills. These structures work in conjunction to promote student understanding and learning, while eliminating language as a barrier to that learning (Gomez et al., 2013; Gomez, Gomez, Rodela, Horton, Cunningham, & Ambrocio, 2015). Addressing faculty professional development needs. Instructors accustomed to teaching traditional mathematics courses may find it difficult to adapt to Statway’s holistic and multifaceted approach. Accordingly, onboarding and continuous professional development opportunities are offered to assist faculty members in transitioning from a teacher-centered perspective to a student-centered perspective that is less dependent upon lecture-heavy instruction, focusing instead on group work and discussion of mathematics problems (Edwards et al. 2015; Grubb, 1999; Grubb & Grabiner, 2013; Mesa, 2011). Avenues for professional development include attending in-person trainings, participating in regular online sync meetings, and mentoring by faculty members with more experience in successfully implementing Statway. Support of a networked improvement community. As discussed earlier in this report, Statway is structured as a networked improvement community. Within the NIC, instructors, administrators, and researchers work together on a measurable and specific aim. That work is supported by research of the underlying problem and the system through which it exists, as well as a working theory on how to address the problem (Bryk, Gomez, & Grunow, 2011; Dolle, Gomez, Russell, & Bryk, 2013). Members of the NIC do this by leveraging the key tenets of improvement science, including iteratively developing, testing, and refining ideas for change. Learning is communicated throughout and accelerated by the network, which works to incorporate validated change ideas into the extant system. In the case of Statway, the NIC provides an opportunity for postsecondary practitioners, improvement coaches, and educational researchers to collaborate both within and across institutions. As a learning community, they implement common measures, procedures, and concepts to achieve quantifiable improvement in Statway at both the outcome and process levels (Bryk et al., 2015). In light of the previous study’s promising results and the evolving nature of the Statway initiative, the aim of the present study was to determine whether Statway’s initial positive results were sustainable beyond the first two years of its establishment. In particular, our focus centered on whether Statway continued to be effective in the face of scaling and operating across an increasingly diverse range of contexts. To investigate these areas of interest, we used a hierarchical linear modeling (HLM) approach (Raudenbush & Bryk, 2002) with propensity score matching (Rosenbaum & Rubin, 1983) to increase both the validity of causal inference and the quality of the matched sample by accounting for potential selection bias in Statway group membership (e.g., students with certain characteristics may have been more likely to participate in Statway, which may have artificially inflated its success rates). The HLM analyses also address the nested structure of our sample (i.e., students enrolled in classrooms within colleges), which may be relatively homogeneous compared to a random sample, and thus manifest problems when using conventional linear modeling techniques (Osborne, 2000). We then compared college mathematics performance between Statway and the non-Statway  CARNEGIE STATWAY SUCCESS 7 matched comparison groups, again applying an HLM approach. Although matching was conducted separately for Year 3 and Year 4, data from the two cohorts were combined prior to examining college mathematics credit achievement. Beyond assessing the average effect of Statway, we also explored variation in performance to ensure that Statway benefits a wide spectrum of students, instructors, and institutions within the NIC. This is extremely important from an improvement science perspective as it seeks to ensure effective implementation for all participants and across diverse contexts, in contrast to typical attributional analyses that seek only to examine mean effects across populations. Additionally, we analyzed student performance across a variety of sex and race/ethnic categories to determine Statway’s capacity to achieve equitable outcomes for all. These analytic results will be examined to identify systemic shortcomings and prioritize strategic action within the NIC to further improve student success and reliably achieve quality at scale (Bryk et al., 2015). Method Participants Students were drawn from the Year 3 and Year 4 Statway cohorts that began the sequence in the fall of 2013 and 2014, respectively. The Year 3 cohort consisted of 2,283 students enrolled in 158 sections taught by 70 faculty members (Sowers & Yamada, 2015). These students attended 19 community colleges and three four-year universities across seven states: California, Connecticut, Florida, Minnesota, New York, Texas, and Washington. (For more information, please see Appendix). The Year 4 cohort comprised 2,862 students enrolled in 166 sections taught by 81 faculty members across 22 community colleges and four four-year universities in 10 states, including the aforementioned states with the addition of Hawaii, Massachusetts, and Wyoming (Huang et al., 2016). Across these two cohorts (N = 5,145), more than 60% of the students were female and nearly 70% were minorities. Data Collection and Study Design The institutional researchers of each participating college supplied data on student background characteristics, prior course enrollment, and performance. The Year 3 analytic sample consisted of 1,422 students enrolled across 18 community colleges and two four-year universities. The Year 4 analytic sample comprised 1,284 students across 12 community colleges and one four-year university.1 Figure 3 illustrates the standard design used in this study. It followed the previous study design (Yamada & Bryk, 2016) to take into account the amount of time students who enrolled in traditional developmental mathematics sequences typically require to attain college-level mathematics credit (see Figure 1). Specifically, we selected students who enrolled in developmental mathematics courses one year prior to Statway students. This method provided 1 Due to a lack of or insufficient institutional data, two colleges were excluded from the Year 3 analyses. In Year 4, three participating colleges only offered the first course within Statway’s two-course sequence; accordingly, they were excluded from the analytic sample. An additional ten colleges were excluded from the Year 4 analyses due to either a lack of or insufficient institutional data. The data used in our analysis were submitted prior to November 2016.  CARNEGIE STATWAY SUCCESS 8 comparison group students with twice the amount of time to achieve college-level mathematics credit as their Statway counterparts, and allowed us to measure the final outcome for each group at the same time point. Thus, we compared students who began traditional developmental mathematics in Fall 2012 with students who began Statway in Fall 2013, and compared students who began traditional developmental mathematics in Fall 2013 with students who began Statway in Fall 2014. College-level mathematics achievement was then measured for each cohort in Spring 2014 and Spring 2015, respectively. We followed this approach for students in all participating institutions with the exception of those who attended two community colleges, one of which only implemented an accelerated one-term form of Statway, and another which implemented both the standard one-year and accelerated one- term forms of Statway. That is, Statway students in these institutions were to complete the full Statway sequence and acquire college-level mathematics credit in a single academic term. Thus, for the purpose of these analyses, students in these institutions’ comparison groups were provided a full academic year to achieve the same outcomes. We utilized a two-level hierarchical linear model (HLM) to calculate propensity scores for each student. Propensity score matching is a statistical technique applied to observational data to reduce possible selection bias—where certain kinds of students may have been more likely to enroll in Statway, leading to more positive outcomes than there otherwise would have been—and, accordingly, increase the validity of causal inference (Rosenbaum & Rubin, 1983). There are two main steps involved in this procedure: (a) obtain a propensity score per student, which represents the likelihood of a student enrolling in Statway, and (b) identify students whose propensity scores are similar to each other as matches. Typically, a logistic regression approach is used with a set of factors or covariates hypothesized to influence student enrollment in Statway. We selected a total of 32 student-level covariates including student background characteristics, as well as prior course taking and success patterns during the two years prior to the Statway term, to generate propensity scores from HLM (Hong & Raudenbush, 2005, 2006; Raudenbush & Bryk, 2002; Yamada & Bryk, 2016). These covariates were selected based on prior research findings and advice from institutional researchers in the participating colleges. Specifically, student background characteristics including sex and race/ethnicity were included because they may influence student performance in developmental mathematics (Bailey et al., 2010). Prior course records were also taken into account because they may serve as more accurate proxies of students’ professional and educational goals than declared academic majors (Jenkins & Cho, 2012). Table 1 displays a comprehensive list of covariates as well as their descriptive statistics. Some student background data were not available in institutional records, and were therefore treated as “unknown” when matching students. Prior to matching in both Years 3 and 4, Statway students were more likely to be female and/or Hispanic than non- Statway students. Additionally, Statway students were generally more likely to have previously attempted and successfully completed developmental English and other college-level courses outside of STEM than comparison group students. For step two, we employed a nearest neighbor matching algorithm to conduct propensity score matching separately for each participating institution (Rosenbaum & Rubin, 1985). We selected this approach in order to preserve as many Statway students as possible  CARNEGIE STATWAY SUCCESS 9 while drawing their closest matches from a large batch of potential comparison group students. For each Statway student, we found up to five matches (5:1 ratio matching) in order to select the most proximal matches from the non-Statway population without sacrificing precision (Ming & Rosenbaum, 2000). To achieve this, we set a caliper distance of up to 0.2 as the maximum acceptable difference between matched students to mitigate the possibility of bad nearest neighbor matches based on recommendations in the literature (Austin, 2011; Rosenbaum & Rubin, 1985). For propensity score matching, we used the package MatchIt (Ho, Imai, King, & Stuart, 2011) in the statistics analytics system R (R Core Team, 2015). For more details regarding propensity score matching, see Yamada et al. (2016). The sample of comparison students was 82.165 prior to matching; 2,607 out of 2,706 Statway students were matched with 10,089 comparison group students across both years and all schools. As in the earlier analysis of Quantway (Yamada, Bohannon, & Grunow, 2016), we applied a four-level HLM model to estimate the effectiveness of Statway against that of traditional developmental mathematics programs. Our dichotomous outcome was defined as follows: successful students either completed Statway with a grade of C or higher within a single academic year (or as aforementioned, within a single academic term for students enrolled in accelerated Statway courses), or college-level mathematics with a grade of C or higher within two academic years (or as previously described, within a single academic year for non-Statway students matched with students enrolled in accelerated Statway courses). Matched clusters (level 1) were nested within Statway students (level 2), who were nested within their instructor classrooms (level 3), who were in turn were nested within their respective institutions (level 4). We used HLM 7 (Raudenbush, Bryk, Cheong, Congdon, & du Toit, 2011) for all of the HLM analyses. Analytic Approach Propensity Score Matching We utilized a two-level Bernoulli model and estimated its model parameters using maximum likelihood via adaptive Gaussian quadrature. ϕij represents the probability of student i enrolling in Statway in college j while ηij indicates the log-odds of this event, which may be formally expressed as: Level-1 Model (Student) Prob(SWil=1|βl) = ϕil log[ϕil/(1 - ϕil)] = ηil ηil = β0l + β1l(COV1il) + … + β37l(COV32il), Level-2 Model (College) β0l = γ00 + u0l, β1l = γ10, . . . Β32l = γ320,  CARNEGIE STATWAY SUCCESS 10 where SW is a dummy variable indicating whether students were in enrolled in Statway (coded as 1) or in a traditional developmental mathematics sequence (coded as 0), COV1…COV32 represent all covariates, i denotes a student, and j is a college. We matched a total of 5,853 comparison students to 1,353 Statway students for Year 3 and a total of 4,240 comparison students matched to 1,254 Statway students for Year 4.2 Thus, there were a total of 12,700 cases included in the analysis reported here. Table 1 compares the descriptive statistics on each covariate before and after matching to the Statway group. Table 2 documents the balance in propensity score college-by-college for Years 3 and 4. For both cohorts, there were no significant differences in mean propensity scores between the Statway and matched students in any of the colleges (see t-values in Table 2). Taken together, this provides strong evidence that comparability of the groups was achieved on the measured covariates. Estimating Statway Effects To estimate Statway’s effect on student success, we ran a four-level Bernoulli model and estimated model parameters using maximum likelihood via penalized quasi-likelihood estimation. ϕijkl represents the probability that student i in Statway student j’s matched group enrolled in instructor k’s course within college l successfully achieved college mathematics credit. Correspondingly, ηijkl is the log-odds of this outcome and formally expressed as: Level-1 Model Prob(SUCCijkl=1|πjkl) = ϕijkl log[ϕijkl/(1 - ϕijkl)] = ηijkl ηijkl = π0jkl + π1jkl*(SWijkl) + π2jkl*(PSijkl), Level-2 Model π0jkl = β00kl + r0jkl π1jkl = β10kl + r1jkl π2jkl = β20kl, Level-3 Model β00kl = γ000l + u00kl β10kl = γ100l + u10kl β20kl = γ200l, Level-4 Model γ000l = δ0000 + v000l γ100l = δ1000 + v100l γ200l = δ2000, where SUCC represents college mathematics credit achievement (1 for success and 0 for non- success), and SW is a dummy variable indicating whether the student was enrolled in Statway (coded as 1) or one of the matched comparisons (coded as 0). As a further safeguard, we 2 While we attempted to match five non-Statway students to each Statway student, we lowered our matching ratio at ten institutions in Cohort 3 and eight institutions in Cohort 4. These reduced matching ratios ranged from 1:1 to 4:1, as illustrated in Table 2. We also dropped select Statway students for whom we were unable to find appropriate matches from the original analytic sample.  CARNEGIE STATWAY SUCCESS 11 included individual students’ propensity scores, PS, as an additional adjustment variable.3 Results Overall, Statway students were significantly more likely to succeed at earning college level math credits compared to their peers. The findings, presented in Table 3, indicate that on average, Statway students demonstrated significantly higher odds of success, 4.96 (95% CI [3.42, 7.19]4), in achieving college-level mathematics credit than their matched counterparts. This translated into the estimated probabilities of success of 58% for the Statway group and 22% for the comparison group.5 In Table 3, -0.51 and -0.85 represent the estimated correlation coefficients between the intercept and slope at the college and faculty levels, indicating that the Statway effect is greater when the outcome of matched non-Statway students is lower. While this relationship was observed at both the college and faculty levels, it was stronger in the latter. Variation in Performance Statway had a positive effect across all institutions, as illustrated by Figure 4, in which the center line represents the average Statway effect across all students. The lines above and below the main line represent the upper and lower bounds of the average Statway effects and are positioned two standard errors away from the center line. Figure 4 also shows that Statway effects varied at the institutional level (with a variance component of 0.51). For instance, Colleges 11 and 12 were positioned below the lower bound of the center line, showing smaller Statway effects relative to other colleges. On the other hand, College 1 was above the upper bound and could be seen as a positive deviant. All faculty, with the exception of instructor 11_01, saw an overall positive effect of Statway on student outcomes. This finding is illustrated by Figure 5, in which the center line again represents the average Statway effect across all students. Similarly, the lines above and below the main line represent the upper and lower bounds of the average Statway effects and are positioned two standard errors away from the center line. Figure 5 also depicts faculty-level variation in Statway effect (with a variance component of 0.21). For example, the Statway effect tended to be homogeneous at College 1, where instructors consistently achieved student success within their respective classes. In contrast, College 16 exhibited the greatest faculty- level variation: some of its instructors demonstrated effects slightly below the NIC average whereas others saw effects well above average. Our findings suggest that it may be of interest to investigate ways to reduce the substantial variation in Statway effects both between and 3 We initially included binary covariates indicating institution type (2 year versus 4 year institutions), mode of adaptation (Statway offered as a standard two-course sequence versus an accelerated one-course sequence), and cohort (Year 3 versus Year 4) at the intercept and slope for levels 4 and 2. However, our initial model did not indicate that these factors had any significant effects on student success. To maintain a parsimonious model, we therefore excluded these non-significant covariates in our final analysis. However, these findings do not necessarily preclude the possibility of differences in student outcomes between different modes of adaptation or institutional types. 4 HLM 7 produces 95% confidence intervals of odds ratios. 5 We also conducted a sensitivity analysis (Hong & Raudenbush, 2005) on Statway effects on college math achievement. Results indicated that with adjustments for the largest potential hidden bias, none of the 95% confidence intervals for the new Statway effect estimates contained 0 or any negative values, thereby supporting the strong ignorability assumption. Thus, it is very unlikely that our general conclusion regarding the positive effects of Statway on the student outcomes has been influenced by the omission of unmeasured confounding factors.  CARNEGIE STATWAY SUCCESS 12 within colleges. Subgroup Analyses We constructed a four-level HLM to account for potential differences in the Statway effect between sex and race/ethnicity subgroups. In order to represent main and interaction effects on our outcome of interest, we utilized effect coding for each subgroup variable. The respective reference categories for sex and race/ethnicity groups were female and White, which were each coded as -1. We excluded students of unknown sex from the analytic sample. To enhance interpretability, we converted the model-based results to proportions of students successfully earning college mathematics credit, as displayed in Figure 6. We found large positive Statway effects for both female and male students, as well as for all race/ethnicity subgroups. Black females demonstrated the largest gain in college mathematics achievement relative to the baseline performance. Our results suggest that although Statway advances equitable outcomes for diverse groups of students, those from certain groups may benefit more from the program than others. Accordingly, future improvement efforts may focus on reducing the gap in Statway effects between various race/ethnicity subgroups. Discussion Our study provides an assessment of Statway’s effectiveness for post-secondary students in Years 3 and 4 of the initiative by implementing a propensity score matching technique (Rosenbaum & Rubin, 1983) within a hierarchical linear modeling framework (Raudenbush & Bryk, 2002). This rigorous analytic approach allowed us to examine the relationship between the Statway program and student success in achieving college-level mathematics credit. We maintained a high matching ratio (approaching 1:5) between Statway students and traditional developmental mathematics students while matching students across 32 covariates. We found that even when providing non-Statway students with twice the amount of time to achieve the same outcome as their Statway counterparts, Statway students exhibited significantly higher probability of success (often approaching odds that were twice as great). Additionally, we conducted a sensitivity analysis and found no evidence that the estimated effects were affected by confounding factors not previously addressed by our analyses. While Statway has both scaled within institutions and spread to new ones, our results replicated those from an earlier analysis on the first two years of Statway (Yamada & Bryk, 2016). Our findings held across all sex and race/ethnicity subgroupings, suggesting that Statway promotes equitable outcomes for all students, including females and students of color, who are historically at higher risk of failing developmental mathematics (Bailey et al., 2010; Carnevale & Desrochers, 2003; Chen, 2016; National Research Council, 2002). Strikingly, the results from our subgroup analysis indicate that Statway may be most beneficial to minority students. Statway’s multifaceted approach to developmental mathematics reform may be key to its evidenced ability to improve student success across diverse populations and conditions (Hodara, 2013). Notably, Statway’s accelerated format, which may be easier to navigate than traditional developmental mathematics sequences (Bailey et al., 2010), is bolstered by additional foci on productive persistence, a quality system of instruction, relevant content, support for faculty learning, and networked improvement. Students, especially those of color,  CARNEGIE STATWAY SUCCESS 13 may suffer from negative academic stereotypes and may question whether they truly belong or will be respected in a college setting (Center for Community College Student Engagement, 2010; Gardenhire-Crooks, Collado, Martin, & Castro, 2010; Walton & Cohen, 2011). Accordingly, Statway promotes productive persistence, which addresses a number of such psychosocial factors. These include developing a growth mindset toward learning, academic tenacity, good learning strategies, and a sense of classroom belonging—all of which have been demonstrated to empower student motivation, engagement, and therefore success (Beilock, 2011; Dweck, 2006; Stigler & Heibert, 1999; Yeager & Dweck, 2012). Furthermore, the program’s instructional system fosters quality learning by drawing upon the research-based learning principles of productive struggle, explicit connections among concepts, and deliberate practice (Boaler, 1998; Ericsson, 2008; Ericsson et al., 1993; Hiebert & Grouws, 2007; Pashler et al., 2007; Schmidt & Bjork, 1992). The initiative also addresses the needs of many traditionally disadvantaged students by providing language and literacy support as well as contextualized lessons and culturally relevant pedagogy (Gomez et al., 2013; Gomez et al., 2015; Morest, V.S., 2014; Muro, A., 2015). Not only does Statway advance learning at the student level, but it also equips instructors with ample training and professional development resources to ensure faculty success (Edwards et al., 2015; Grubb, 1999; Grubb & Grabiner, 2013). Moreover, Statway bridges administrators, practitioners, researchers, and improvement experts in a NIC to encourage collaboration, learning, and sharing of best practices across institutions (Bryk et al., 2011, Dolle et al., 2013). For example, several instructors across participating institutions have formed teams for the purposes of revising Statway curricula, developing a more user-friendly onboarding system for new faculty, iteratively testing change ideas for building productive persistence, and improving student assessments. One of the foundational principles of improvement science posits that achieving quality improvement requires reducing variation in performance. Accordingly, we investigated variation in Statway’s effect among participating faculty members and institutions. As mentioned above, we found positive Statway effects for all institutions and all but one faculty member. Significant variation in performance also existed both across and within colleges. Some instructors and colleges represented positive outliers that outperformed others, while other faculty members and institutions underperformed (relative to the rest). By leveraging the primary function of a NIC to collaboratively learn how to improve (Bryk et al., 2015), Carnegie Math Pathways team members plan on conducting qualitative interviews with faculty members at colleges that showed above average performance in this analysis to illuminate promising instructional and supportive practices that promote student success. For instance, positive deviant instructors within College 16 could potentially lead training sessions on how to implement additional homework interventions for underperforming colleagues, thereby improving instructional practices within their respective institutions. On the other hand, College 1, whose instructors were all positive deviants, may well have initiated practices at the institutional level to promote such success across the board. College 1 might have designed, prototyped, and refined additional faculty development opportunities that it could share with other Statway institutions. By testing and scaling innovative practices among its members, the NIC has a unique opportunity to continuously improve Statway’s design, implementation, and student outcomes.  CARNEGIE STATWAY SUCCESS 14 Although we did not find any single mode of adaptation or institution type to significantly affect Statway outcomes, there may still be differences between accelerated and standard Statway formats, as well as two-year and four-year institutions, that merit further investigation. Specifically, institution type may not have been significant due to a relatively small number of participating four-year institutions, (which composed only two of the twenty colleges in our sample). In addition, other analyses of relevant data suggest that four-year universities tend to have higher success rates than community colleges (Huang et al., 2016). As we continue to scale to four-year institutions, including large universities, we will have the opportunity to examine whether they owe high success rates to student body characteristics or distinct teaching and/or administrative practices. Descriptive data also indicate that students taking an accelerated one-course form of Statway generally exhibit success rates at least equal to their standard two-term Statway counterparts. However, one institution only offered accelerated Statway, rendering it difficult to disentangle college-level effects from those induced by mode of adaptation. In this case, it may be of interest to compare post-participation outcomes between students from the aforementioned institution and students who took Statway as a standard two-course sequence. Should accelerated Statway students go on to be just as successful as their standard counterparts in subsequent years, it may be worthwhile to look into whether a more intensive design could be established as an option for implementing Statway. Overall, we find that Statway students in the third and fourth years of the program were not only more likely to achieve college-level mathematics credit than their comparison group counterparts, but were also able to do so in half the time. These findings replicate those of a previous analysis conducted on the first two years of Statway. By leveraging a rigorous multi- level modeling technique with propensity score matching, our findings offer further evidence of the causal relationship between Statway and student success, suggesting that the program provides a viable alternative means of navigating the developmental mathematics sequence. Notably, Statway continues to benefit students even as the program spreads to a growing number of students and colleges. Our findings indicate that Statway’s unique and multifaceted approach to developmental mathematics reform enables students, instructors, and institutions to overcome impediments to student success across a diverse range of contexts.  CARNEGIE STATWAY SUCCESS 15 References American Association of Community Colleges. Community college fact sheet. (2016). Retrieved November 9, 2016, from www.aacc.nche.edu. Austin, P. C. (2011). Optimal caliper widths for propensity-score matching when estimating differences in means and differences in proportions in observational studies. Pharmaceutical Statistics, 10, 150-161. Bailey, T., Jeong, D. W., & Cho, S.W. (2010). Referral, enrollment, and completion in developmental education sequences in community colleges. Economics of Education Review, 29, 255–270. Beilock, S. (2011). Back to school: Dealing with academic stress. Psychological Science Agenda, 25(9). Blackwell, L. S., Trzesniewski, K. H., & Dweck, C. S. (2007). Implicit theories of intelligence predict achievement across an adolescent transition: A longitudinal study and an intervention. Child Development, 78, 246–263. Boaler, J. (1998). Open and closed mathematics: Student experiences and understandings. Journal for Research in Mathematics Education, 29, 41-62. Bryk, A. S., Gomez, L. M., & Grunow, A. (2011). Getting ideas into action: Building networked improvement communities in education. In M. T. Hallinan (Ed.), Frontiers in sociology of education (pp. 127-162). New York, NY: Springer. Bonham, B. S., & Boylan, H. R. (2011). Developmental mathematics: Challenges, promising practices, and recent initiatives. Journal of Developmental Education, 34(3), 2. Bryk, A. S., Gomez, L. M., & Grunow, A. (2011). Getting ideas into action: Building networked improvement communities in education. In Frontiers in sociology of education (pp. 127- 162). Springer Netherlands. Bryk, A. S., Gomez, L. M., Grunow, A., & LeMahieu, P. G. (2015). Learning to improve: How America's schools can get better at getting better. Cambridge, MA: Harvard Education Press. Cannata, M., & Nguyen, T. (2015). Consensus versus Clarity: Tensions in Designing for Scale. Paper presented at the annual meeting of the American Educational Research Association in Chicago, IL, April 16-21, 2015. Carnevale, A. P. & Desrochers, D. M. (2003). Standards for what?: The economic roots of K-16 reform. Princeton, NJ: Educational Testing Service. Center for Community College Student Engagement. (2010). The Heart of Student Success: Teaching, Learning, and College Completion (2010 CCCSE Findings). Austin, TX: University of Texas at Austin, Community College Leadership Program. Chen, X. (2016). Remedial coursetaking at U.S. public 2- and 4-year institutions: Scope, experiences, and outcomes (NCES 2016-405). U.S. Department of Education, Washington, DC: National Center for Education Statistics. Cohen, G. L., Garcia, J., Purdie-Vaughns, V., Apfel, N., & Brzustoski, P. (2009). Recursive processes in self-affirmation: Intervening to close the minority achievement gap. Science, 324, 400-403. Cullinane, J., & Treisman, P. U. (2010). Improving developmental mathematics education in