Policy makers in the US and elsewhere readily recognize the value of after-school programs and extended school day services in improving children’s academic, social, behavioral and health outcomes. But the evidence on their impact is actually rather mixed: some studies report developmental benefits; others uncover nothing much at all.

One reason for the variation is assumed to be an unevenness in the quality of after-school programs themselves. So researchers have started looking for evidence-based strategies for improving program content as a necessary preliminary to pronouncing on their benefits to children.

The pattern of what happens in schools outside working hours tends to differ structurally from what happens in the classroom, but there are enough similarities between the two environments to make researchers think it may be possible to adapt interventions successfully between settings.

Kathryn Hynes and colleagues from Pennsylvania State University have been studying the implementation of the Good Behavior Game (GBG), a classroom-based behavior management system, in eight US after-school programs.

All of the test sites were 21st Century Community Learning Center (CCLC) programs, an initiative that receives nearly $1bn in annual federal funding. CCLC is designed to boost academic achievement and foster healthy development among at-risk students.

The Good Behavior Game is an empirically supported application of behavior management that operates through a combination of positive peer pressure and positive reinforcement.

The research team considered two questions: first, what are the barriers to implementing the GBG in after-school settings; second, what kinds of approach are needed to be able to evaluate the effects of classroom-based interventions in after-school programs.

They team relied on qualitative interviews with staff, directors and intervention coaches, and on quantitative data collected during observation sessions.

Program staff, directors and implementation coaches were asked about obstacles to implementation, including motivational barriers, shortcomings in training and support, logistical issues, children’s lack of interest and questions surrounding the appropriateness of the Game to after-school activity.

Variations in quality from site to site was assessed using the Promising Practices Rating Scale (PPRS). Trained observers watched three activities at each after-school program and rated six aspects: supportive relations with adults, supportive relations with peers, over-control, chaos, student engagement and appropriate structure.

In relation to the first research question, there were few problems overall in implementing the Good Behavior Game. Despite some initial skepticism, most staff members said they liked it and many said they would use it in the future.

Where problems occurred that were particular to GBG, they tended to be about the role of the Game during less academic time, small group sizes, the involvement of older children and the use of teachers as program staff.

Staff struggled to strike a balance between meeting children’s academic needs and their entitlement to free play and mentoring.

Hynes and her colleagues conclude that interventions designed for school classrooms may need significant adaptation to work among activities that are normally more recreational.

Training materials are needed to show the Game being used in less academic surroundings, and implementation coaches must be prepared to help program make the transition, they write in the latest edition of Journal of Children’s Services.

Where staff and children associated the Game with the academic life of the school, there was resistance to using it during more informal time. But the barriers were not such that they should inhibit successful implementation.

The quantitative data on implementation fidelity and staff enthusiasm largely supported the qualitative finding that it was relatively easy for staff to implement GBG outside the classroom-based curriculum.

Overall, then, the authors were optimistic: afterschool programmers looking for evidence-based ways to improve quality are likely to find helpful tools inside mainstream education.

As to the second research question, the researchers uncovered few barriers to collecting data on the implementation of the GBG or on outcomes such as children’s knowledge of behavioral expectations or their observed behavior.

However, whereas a classroom typically consisted of a teacher and an aide, the after-school programs employed up to 15 staff, dealing by turn and turn about with a mobile group of children.

Results from the PPRS confirmed that there was considerable quality variation within-programs – and that it had several implications for evaluation.

Penn State has no easy answers to offer, but urges evaluators to be aware of the disparities when selecting evaluation methods. They highlight different options, from asking observers to provide a score for the whole program to recording one score for each staff member or for each activity observed and then aggregating the results.

The study conceded some limitations. First, the sample size was too small. Second, data on implementation fidelity was forthcoming from only four of the sites, making the findings on that score encouraging but hardly conclusive.

See: Hynes K, Smith E P and Perkins D (2009) “Piloting a classroom-based intervention in after-school programmes: a case study in science migration,” Journal of Children’s Services 4 (3), 4-20.