One of the advantages of implementing evidence-based programs with the help of an "operating system" is that community leaders have an opportunity to develop their own skills and resources as part of the process.

But does providing technical assistance actually improve a community's capacity to select, implement and evaluate evidence-based programs? And does this expertise grow over time?
 
A team from the Prevention Research Center at Penn State University, has been investigating these questions in an evaluation of the programing knowledge of 422 community leaders involved in a randomized controlled trial of the PROSPER operating system.
 
Developed by Richard Spoth and colleagues at Iowa State University with increasing Federal support, PROSPER seeks to cultivate sustainable partnerships between communities, schools and universities to promote the implementation of evidence-based interventions. [See: Federal cash puts PROSPER model on the road.]

Community leaders form local prevention teams, who are responsible for program selection and implementation. Throughout the project they get coaching, training and technical assistance, all designed to improve innovation-specific capacity.
 
In the research study, community leaders in both intervention (n=271) and control communities (n=151) were interviewed once a year over six years to assess their knowledge in four areas: sources of evidence-based programs, standards of evidence, ensuring fidelity of program delivery, and evaluation methods.

Researchers Max Crowley, Mark Greenberg and Mark Feinberg differentiated between participants' "expert" and "non-expert" responses. So, in the case of where they would look if a colleague asked for names of evidence-based programs, the expert answer needed to include reference to established lists of evidence-based programs, such as Blueprints or SAMSHA.
 
But here there was no statistically significant difference between the trial and control groups, and, disappointingly, the relationship did not change over time: in both groups the proportion of respondents giving "expert" answers stayed under 20% throughout.
 
As for standards of evidence, respondents were asked how they would know if a program was supported by good research. Expert answers included reference to evaluation research design, outcome data, or the program being on a published list of evidence-based programs.
 
In this case, there was a statistically significant difference. Among the PROSPER group 38% were able to give expert answers by the end of the project, compared with under 20% in the control group.
 
More than 40% of program group participants could give expert responses to the question of how to ensure program fidelity. Fewer than 20% of the control group mentioned the key ingredients of implementation monitoring, certification or technical assistance.
 
Over 30% of the program group said it was necessary to use a specific quality method to determine whether a program was working; only 15% or less in the control group said so.
 
In all four cases, the main differences between the two groups became apparent between baseline and the start of year two – following the most intensive input from the PROSPER team.
 
The research team concluded, therefore, that the expertise of community leaders involved in PROSPER could be enhanced – at least in relation to the selection, evaluation and implementation of evidence-based programs.

On the other hand, Crowley, Greenberg and Feinberg acknowledge, it was sobering to discover that the knowledge of both groups in the comparison remained so low throughout the study, and that, even after several years of intensive input, a significant proportion of community leaders struggled to give "expert" answers to critical questions.