Playing for Success An Evaluation of its Long Term Impact (PDF)

Playing for Success An Evaluation of its Long Term Impact Caroline Sharp, Tamsin Chamberlain Jo Morrison & Caroline Filmer-Sankey National Foundation for Educational Research Research Report RR844 RESEARCH  Playing for Success An Evaluation of its Long Term Impact Caroline Sharp, Tamsin Chamberlain Jo Morrison & Caroline Filmer-Sankey National Foundation for Educational Research The views expressed in this report are the authors’ and do not necessarily reflect those of the Department for Education and Skills. © NFER Trading Ltd 2007 ISBN 978 1 84478 931 3 Research Report No 844   Acknowledgements The authors would like to thank the PfS Centre staff who kindly submitted data which fed into the statistical analysis. We would particularly like to thank the Centre staff, link teachers and Critical Friends who gave up their time to be interviewed as part of this evaluation. We are also grateful to our colleagues at NFER who contributed to the research, in particular Karen Lewis, Vanessa Woodley, Emma Scott and Lesley Kendall. Finally, we would like to thank members of the project steering group for their help and support throughout the evaluation: Alison Lockwood, Sarah Baker, Catherine Davie, Anna Chapman, Halina Gammie and Rex Hall.   Contents Executive Summary 1 Introduction 1 2 Aims and objectives 2 3 Methodology 4 3.1 Strand 1 study design 4 3.2 Strand 2 study design 8 4 Findings from Strand 1: Statistical analysis 11 Summary of findings from Strand 1 11 4.1 How well matched were the PfS and comparison samples? 11 4.2 Results of the multilevel modelling 15 4.3 Progress of lower attaining pupils 17 4.4 Did it matter when pupils attended? 22 4.5 Centre-level analysis 22 5 Findings from Strand 2: Case studies of good practice 25 Summary of findings from Strand 2 25 5.1 Background to Strand 2 26 5.2 Good practice before pupils attend the PfS programme 27 5.3 Good practice during pupils’ attendance at the PfS programme 34 5.4 Good practice after pupils have attended the PfS programme 44 5.5 Partnership working 50 5.6 Reflections on longer term impact from attending PfS 55 6 Discussion and conclusions 57 6.1 Why did this study find limited evidence of longer term effects? 57 6.2 Why did certain PfS pupils do better? 58 6.3 Are some PfS Centres more effective? 59 6.4 What does this study add to our understanding of effective practice in promoting longer term gains? 59 6.5 Conclusion 64 6.6 Recommendations 65 References 67 Appendix A Procedure and approach to quantitative data analysis 69 Appendix B PfS and comparison groups’ background characteristics 72 Appendix C Multilevel Models 75   Executive Summary i Executive Summary Introduction The Playing for Success (PfS) initiative is targeted on underachieving young people. It aims to contribute to raising educational standards, especially in numeracy and literacy, bringing the attainment levels of lower achieving pupils closer to the average expected for their age. Previous national evaluation studies have measured pupils’ performance at the start and end of their attendance at PfS Centres (Sharp et al., 1999, 2001, 2002, 2003). The results have shown clear evidence of significant improvements, especially in numeracy and information and communications technology (ICT). This study set out to consider whether there was any evidence of longer term changes in pupils’ performance in National Curriculum Assessments (NCA) associated with attending a PfS Centre. It also aimed to gather information on the strategies adopted by PfS Centres to encourage schools to capitalise on the learning gains achieved by pupils attending PfS. Key findings The statistical analysis indicated that: • In Key Stages 2, 3 and 4, low attainers who attended PfS Centres did better than expected and higher attainers did less well than expected in NCAs (except in respect of English in Key Stage 4, where no statistically significant difference was found). For example, in Key Stage 2 maths, one in four PfS pupils who performed below the expected level in Key Stage 1 maths went on to perform above the expected level at Key Stage 2, compared to one in five similar pupils that did not attend. • In Key Stage 4, pupils who had attended PfS made greater progress in maths when compared to similar pupils that did not attend. The difference was equivalent to one in seven PfS pupils attaining one higher grade in maths GCSE than expected. PfS pupils also made greater progress overall at GCSE (in terms of the average GCSE score, one in ten PfS pupils attained one grade higher than expected in all subjects they took). • In Key Stage 2, pupils who attended PfS made less progress in English (by 0.7 of a month). No statistically significant impact was found in Key Stage 3. • An analysis of the progress achieved by pupils attending each Centre identified eight PfS Centres that had performed significantly better than other Centres on at least two of six NCA outcome measures. The qualitative analysis of ‘more effective’ Centres indicated that: • Centres and partner schools were focussing on the details of liaison and good practice that provided a well targeted, high quality learning experience during the course and facilitated transfer of learning after the pupils had left.  An evaluation of the longer term impact of Playing for Success ii • Partnership working between Centres and schools was critical, before, during and after pupils attended the PfS programme. Establishing a good relationship with the host club was important, as was a good working relationship with local authority colleagues, Critical Friends and the central team at DfES. Background Playing for Success is a national initiative, established in 1997 by the Department for Education and Skills in partnership with the FA Premier League and their clubs, and local education authorities. Since then it has expanded to include a wide range of professional sports, including cricket and rugby. It aims to contribute to raising educational standards, especially in urban areas, by setting up Study Support Centres in professional football clubs and other sports venues. The initiative has expanded from three pilot Centres in 1997 to over 150 signed up in 2006. Playing for Success focuses on underachieving young people, mainly in Years 6 to 9, and places a strong emphasis on improving pupils’ attitudes and motivation to learn. Centres are managed by experienced teachers. They use the medium and environment of sport to support pupils’ work in literacy, numeracy and ICT. Pupils attend the Centres after school for around 20 hours during a period of about ten weeks. Methodology The study took place in 2006-7. Data were collected in two main strands. Strand 1 used quantitative methods to compare the academic performance at Key Stage 2, Key Stage 3 and Key Stage 4 of 14,541 pupils who had attended PfS Centres during academic years 2000/1 to 2004/5 (and took their NCAs in 2003, 2004 or 2005) with the performance of pupils who did not attend. In order to make fair comparisons, multilevel modelling was used to take account of pupil and school factors known to influence pupil progress (including such variables as prior attainment, gender, ethnicity, eligibility for free school meals, special educational needs status and school size). Strand 2 entailed a qualitative analysis of strategies adopted in ‘more effective’ Centres in relation to their longer term impact on pupil progress. Seven Centres were identified as more effective on the basis of the quantitative analysis. Two Centres were amongst those identified as more effective by the PfS Central team, based on the judgements of PfS Critical Friends and local evaluation results. Interviews were conducted with Centre Managers and two link teachers sending pupils to each Centre (a total of 27 interviewees).  Executive Summary iii Other findings The study investigated whether the time that had elapsed between a pupil’s attendance at PfS and their assessments made any difference to their progress. There was no indication of a consistent relationship between this variable and the progress of PfS pupils. The qualitative data from ‘more effective’ Centres demonstrated the importance of partnership working between Centres and schools. Best practice points included: • Emphasising the importance of selecting pupils who were most likely to benefit from PfS. These Centres took care of practical details and ensured that teachers, pupils and parents were well prepared for the experience. • Providing a positive ethos, where individuals were welcomed, encouraged and valued. Centres helped pupils to engage in individual target setting and ensured pupils were aware of what they needed to do to achieve their goals. • Emphasising personalised learning, devising individual programmes and providing constant feedback. These Centres offered practical activities, which enabled pupils to relate curriculum content to real life situations. They also ensured a good communication with schools during the course. • Helping pupils to recognise their achievements and encouraging them to continue to succeed after PfS. Centres ensured that pupils received public acknowledgement of their effort and attainment at end-of-course celebration events. They provided pupils with a portfolio of work that could be displayed at school and encouraged schools to build on the progress achieved. Some partner schools capitalised on PfS particularly well by adopting similar approaches and using pupils’ skills. Conclusions and recommendations The overall findings from the statistical analysis employed in this study present a mixed picture. However, rather than see this as evidence of a lack of impact at Key Stage 2 and 3, the contribution of the qualitative analysis suggests that longer term progress should be viewed as a shared responsibility between Centres and schools. PfS is not an inoculation against future underperformance, but it does give underachieving young people a new chance, as long as they receive recognition for their achievement, continued support and opportunities to succeed. The study of ‘more effective’ Centres found that staff were paying very detailed attention to liaison and partnership with schools, as well as to the quality of the programme provided at the Centre. The study identified specific strategies adopted before, during and after the pupils attended PfS that helped pupils to transfer their learning to the school environment.  An evaluation of the longer term impact of Playing for Success iv The evaluation findings have the following implications and recommendations for those managing PfS at local and national levels as well as for PfS Centres and their partner schools: • The government, local authorities and sponsors should continue to support PfS. Further consideration should be given to the opportunities for pupils to transfer their learning from PfS to other contexts (for example, through continued partnerships between Centres and schools, and through developing opportunities for PfS graduates to build on their achievements, both within and outside school hours). • Given that this study has found evidence of greater long term impact among lower-attaining pupils, Centres may wish to reconsider their selection criteria. • Local authorities and Centre Managers should consider carrying out their own studies of longer term progress. This would entail adopting systems to record which pupils had attended, selecting an appropriate comparison (control) group and analysing progress over time, including progress at Key Stage 4. • PfS should share information about best practice in contributing to longer term impact.  Introduction 1 1 Introduction Playing for Success (PfS) was established in 1997 by the Department for Education and Skills (DfES). The broad aim of the initiative is to contribute to raising educational standards, especially in numeracy and literacy. It is targeted on underachieving young people in Key Stages 2 and 3 and is particularly concerned with bringing the attainment of lower achieving pupils closer to the average expected for their age. Pupils attend Study Support Centres after school for around 20 hours during a period of about ten weeks (see Sharp et al., 2002a). The initiative began by establishing Centres in English professional football clubs. The number of Centres has grown from three pilot Centres established in 1997 to over 150 signed up in 2006. The initiative has also expanded to encompass sports other than football. Further information on PfS can be found on the website: www.dfes.gov.uk/playingforsuccess The National Foundation of Educational Research (NFER) was responsible for the national evaluation of PfS for four consecutive years (Sharp et al., 1999, 2001, 2002a, 2003). The evaluation studies measured pupils’ performance and attitudes at the start and end of their attendance at PfS Centres. The results showed clear evidence of significant improvements on several measures, especially numeracy and ICT, during the pupils’ time at the Centres. However, the previous national evaluation studies did not investigate the question of whether or not the initiative had led to longer term changes in pupils’ performance, after they had left the Centres. The longer term impact of PfS was considered in a preliminary study (Sharp et al., 2004), which set out to consider the issues involved in attempting to measure the subsequent impact of PfS on pupil performance. It attempted to find an appropriate method, using the National Pupil Database (NPD) and explored the potential usefulness of a statistical approach to investigate the impact of attending PfS on subsequent pupil performance in National Curriculum Assessments (NCAs). The evaluation reported here builds on the experience gained during the preliminary study.  An evaluation of the longer term impact of Playing for Success 2 2 Aims and objectives The aims of the evaluation were: • to establish the evidence for longer term gains among pupils who have attended PfS Centres • to consider the evidence for differential effects among different PfS Centres • to gather information on the strategies adopted by PfS Centres to encourage schools to capitalise on the learning gains achieved by pupils attending PfS • to identify the implications of the evaluation findings for PfS Centres and their partner schools. The evaluation sought to address the following research questions: • Are there any longer term gains for pupils who have attended PfS Centres once back in school, compared to pupils with similar attainment who have not attended a PfS Centre? • Which are the most effective Centres in producing longer term gains in attainment and how can best practice be spread? • How can this information be used to develop guidance for schools to enable them to support PfS graduates within schools during and after the intervention? Figure 1 shows the main factors that the research team considered likely to influence the longer term impact of PfS on pupil performance.  Aims and Objectives 3 Figure 1 Hypothesised relationships for longer term impact of PfS The hypothesis underlying the evaluation design was that both the PfS initiative as a whole and the individual Centre attended would influence pupils’ performance in basic skills during their time at the Centre and that these immediate effects might have a longer term impact on pupils’ performance in National Curriculum Assessment outcomes. The strength of influence of PfS on National Curriculum Assessment outcomes was thought to vary according to the amount of time that had elapsed between a pupil’s attendance at PfS and taking the tests. It was also hypothesised that the degree to which schools capitalised upon the PfS programme and the PfS-related support they provided to pupils would influence the longer term effects of PfS on pupil performance. PfS immediate effects on performance PfS Programme features PfS Centre programme & management NCA performance School liaison and support Time elapsed between PfS and NCA tests  Methodology 4 3 Methodology The methodology was split into two strands, dealing with quantitative and qualitative data. Strand 1 aimed to investigate whether there were any longer term gains for pupils attending PfS Centres once back in school. It also aimed to identify PfS Centres that were ‘more effective’ in producing longer term gains. Strand 2 of the research aimed to identify the best practice which could be used to develop guidance for schools to enable them to support PfS graduates. Both strands of the research are reported here. Section 3.1 outlines the study design for Strand 1 (statistical analysis) and Section 3.2 outlines the study design for Strand 2 (case study analysis) 3.1 Strand 1 study design This study made use of the National Pupil Database (NPD), held by the DfES. The NPD is a ‘data warehouse’ which brings together value-added national performance data with pupil-level information from the Pupil Level Annual Schools Census (PLASC). It links pupils’ performance in Key Stage 1, 2 and 3 assessments to GCSE/GNVQ results, thereby providing the means to identify pupil performance at a given point in time and progress from one Key Stage to the next, taking important pupil characteristics into account. The preliminary study of the longer term impact of PfS (Sharp et al., 2004) considered the academic attainment of pupils completing Key Stage 2 or Key Stage 3 in 2003. The study reported here, focused on pupils’ progress from Key Stage 1 to Key Stage 2, from Key Stage 2 to Key Stage 3 and from Key Stage 2 to Key Stage 4, using the datasets from 2003, 2004 and 2005. This is therefore a much larger study compared to the preliminary study. 3.1.1 Target population The study focused on the population of pupils who took their Key Stage 2, Key Stage 3 or Key Stage 4 assessments in the summer of 2003, 2004 and 2005. The population was divided into two main groups for the purposes of this study: those who were known to have attended PfS Centres (the PfS group) and a much larger population of pupils who did not attend PfS. A subset of non-PfS pupils was chosen as a comparison group, comprising pupils with the same background, school characteristics and prior attainment as the PfS group. The PfS group included nine cohorts of pupils (labelled A to G for the purposes of the study) in year 6, 9 and 11. These pupils took their National Curriculum Assessments in 2003, 2004 and 2005. Pupils completing Key Stage 2 attended PfS in years 5 or 6 and their attainment at the end of Key Stage 1 was used as the measure of prior attainment. Pupils completing Key Stage 3 and 4 attended PfS in