In 2005, before the interest in “translational research” reached its present pitch, a team of University of Florida researchers led by Dean Fixsen set about clearing the ground for a new “implementation science”.

“Major gaps exist between what is known as effective practices (ie. theory and science) and what is actually done (i.e. policy and practice)” they wrote. “The review is concerned with establishing from the literature what we know about the gap and how to narrow it.”

They worked on a wide front, plowing agriculture, business and engineering, as well as child welfare, for empirical studies, meta-analyses and literature reviews.

They emerged with just under 750 articles that seemed to shed some light. Around half were considered significant – describing well-designed experimental studies, careful reviews of the implementation literature or good quality theoretical discussions of implementation factors.

But only 22 – medical studies included – reported the results of experimental analyses or meta-analyses of implementation variables.

Compounding that weakness, they found compelling evidence for what did not work, only reasonable evidence for what did – and a glaring lack of information about much else in between.

High on the list of methods widely used and reported were two shown to be signally ineffective: disseminating information about policies and programs and offering training support.

Common as they were, and regardless of the skill with which they were delivered, they were simply not sufficient to change the way people worked.

The better remedy was a “longer term multi-level approach”. The team identified certain core implementation components and proposed a rudimentary framework for organizing them.

The first element was staff selection – finding the right people. Most successful programs had selection criteria, listing the qualifications, experience and practitioner characteristics needed to deliver the program competently.

The second was pre-service training – how practitioners learned when, where, and with whom to use the new approaches and skills.

Next came consultation and coaching, during which staff received feedback on their performance and how far they were delivering the program as designed. This staff evaluation was a precursor to program evaluation, which assessed the organization's capacity to ensure that the core intervention components were being properly delivered.

The last two core implementation components were appropriate administrative support, including leadership and good data, and system interventions to ensure adequate financial, organizational and human resources.

The reviewers described these components as being necessarily “integrated and compensatory” – the elements needed to be elastic and adaptable.

The message on training and coaching was clear. Very few skills could be successfully transmitted from the training environment to the workplace without on-the-job coaching. One meta analysis of the effects of training and coaching on teachers’ implementation in the classroom found that none of the participants in training used their skills if the training consisted merely of theory and discussion. Where there was demonstration, practice and coaching, the equivalent figure was 95%.

The importance of measuring “fidelity” was confirmed, but there was little to support today’s conventional wisdom that if you deliver half of a program you get – not half – but zero effect. The studies they examined only identified the correlation between fidelity and outcomes. Experimental studies that might establish a causal link were missing.

There were gaps, too, in the findings on the value of administrative support and system interventions and on organizational level influences.

For example, the researchers investigated accounts of several measures for establishing the readiness of a site (school, community or business) to assimilate an evidence-based program, but there was nothing in the literature to say whether the tools were reliable.

Similarly, most programs acknowledged the need for community “buy-in,” but there was virtually no data to suggest which ways of achieving it were most likely to succeed. The obstacles were much more thoroughly documented.

Evidence-based programs needed be accommodated within the complex systems that provided services to the vulnerable. But such systems served large, disparate populations whose wide ranging difficulties frequently defied the inclusion and exclusion criteria of the typical evidence-based program.

Accordingly, professional staff – be they teachers, psychologists, nurses or social workers – were qualified to practice by virtue of their training and recognition by a professional body. Their practice knowledge was based on the accumulation of eclectic experience.

Just as the programs often did not fit the characteristics of the general, population, so the professional training model was largely out of tune with the operational needs of evidence-based practice, the Florida team concluded. Historically, the mismatch had been responsible for much ineffectiveness and some harm.

To compensate for the lack of common language and common framework in the field, they proposed a strategy drawing on the routines of computer programming and software development.

It had four elements:

• a source – an innovation (in this case a program) that had been found to work in real world conditions
• a destination – the practitioners and the organization who adopted, supported and funded the innovation
• a communication link – an individual or group of individuals who strove to implement the innovation with fidelity
• a feedback mechanism – a regular flow of reliable information about the performance of individuals, teams and organizations hosting or delivering the innovation.

Across the studies, the reviewers identified a common implementation sequence: first exploration, followed by the decision to adopt a program; next, installation and initial implementation, all preceding full implementation and a process of adjustment to guarantee long-term sustainability. The attempt was most likely to fail, they found, at the initial implementation stage.

Dean Fixsen’s review ended with recommendations for the key players: policy makers, researchers and “purveyors”.

Made with US context in mind, they urged that implementation knowledge should be infused into policy, that there should be investment in the use of implementation technologies, and that program funding should routinely cover start up and infrastructure costs.

A long list of research recommendations was intended for a scientific environment in which expenditure on implementation research remained negligible: 1% of the medical research budget compared to the 99% devoted to (too often) futile tests of effectiveness.

Across the tracts of virgin territory, they identified three main areas for action: identifying core components, determining the effectiveness of implementation procedures, and describing “hospitable” organizational and socio-political factors.

Purveyors were recommended to develop partnerships with skilled researchers, to establish communities of practice at implementation sites and to share lessons across purveyor teams.

They identified four conditions that appeared to contribute to successful implementation:

• carefully selected practitioners receiving coordinated training, coaching and frequent performance assessments
• organizations providing the infrastructure for timely training, skillful supervision, coaching and regular process and outcome evaluations
• communities and consumers fully engaged in selection and evaluation of programs and practices
• funding, policies and regulation that created a favorable environment for implementation and program operations.

•For more about implementation science and the work of Dean Fixsen, see Global search begins for missing link, Times for a new science of community change? and We can’t rely on wizards who are half Machiavelli, half saint

See: Fixsen D, Naoom S, Blase K, Friedman R and Wallace F (2005) Implementation Research: A synthesis of the Literature, Tampa, Florida, University of South Florida, Louis de la Parte Florida Mental health Institute, The National Implementation Research Network (FMHI Publication #231)