"We can’t rely on wizards who are half Machiavelli, half saint"

Communities, service designers and practitioners alike are abandoning the over-optimistic notion that hard work and good intentions are enough when it comes to improving child well-being, and they are acknowledging that they need more help.

“We cannot march program by program into the better future we seek, and assume we are just ‘a tinker away’ from resolving the problem,” says Lisbeth Schorr in a paper for the Brookings Institution.

A bolder approach that recognizes the complexities of the task is wanted – first, a broader conception of what works, second, greater awareness that improving child outcomes is a complex, multi-layered task.

One of the barriers to progress, she argues, is a lack of consensus in the research community. In one camp are the nihilists who pour scorn on attempts to find out if anything works; in another are the tight-lacing pragmatists who preach that a few programs work and that they just need replicating. Somewhere in the middle are proponents of the view that each community is unique, and it is difficult to generalize.

Schorr argues that what counts as credible knowledge must be modified. Most promising change initiatives “are complex efforts with multiple, interacting components that require constant mid-course correction and the active engagement of committed human beings”.

Thus, they are “deeply dependent on context” – and not amenable to randomized controlled trials. There is, she says, “a mismatch between what may be the ‘gold standard’ of effective interventions and the ‘gold standard’ of evaluation methodology. Traditional evaluation has excluded from consideration precisely those interventions that are most likely to have an impact. Only very slim pickings emerge when the methodological funnel is constructed as narrowly as it typically is today.”

The US Pathways Mapping Initiative [1], which Schorr heads up, is one manifestation of the shift of focus “from assessing the effects of individual programs or strategies” to “attempting to draw inferences from multiple sources of evidence, analyzed in the context of sturdy theory”.

The Initiative was established at Harvard in 2000 and is funded by the Annie E. Casey Foundation. It aims to give communities a coherent body of information that can be used to strengthen efforts to improve community-wide outcomes, on the basis that they will function best when they combine local wisdom and their understanding of local circumstances with accumulated knowledge, drawn from research, theory, and practice, about “what has worked elsewhere, what is working now, and what appears to be promising”.

Schorr is critical of much existing knowledge in the field on the grounds that it represents interventions designed and circumscribed primarily for the purposes of elegant evaluation.

In recommending her own “mental mapping” approach, she explains that groups of “highly knowledgeable, experienced individuals, including researchers and practitioners, who are steeped in their respective fields but diverse in their perspective and beliefs” draw on their accumulated wisdom to make explicit their “mental maps” of what works to reach the outcome under consideration.

Communities are encouraged to “custom-craft a coherent set of priorities that take into account their unique circumstances, including what is already in place and their own needs, resources, and opportunities”.

Another feature of the Mapping Initiative – and one that is shares with more conventionally positioned logic modeling – is that it makes explicit the links between proposed actions and outcomes. A pathway to school readiness, for instance, maps out how high quality health education and reproductive services contribute to “intended well-timed pregnancies,” which in turn lead to good health.

According to Schorr, possibly the greatest contribution of PMI may be in identifying the attributes of effective interventions and avoiding simplistic questions such as “Does home-visiting work?” It also tries to identify the elements of the community and systems infrastructure that contribute to effectiveness – because “context matters”.

“The heroic struggle of practitioners and managers to swim upstream, bending or breaking rules to make things work, is fine in pilots but not scale-up. We can’t forever rely,” Schorr suggests, “on wizards who can beat the bureaucracies and dysfunctional regulations and funding practices because they are some combination of Mother Teresa and Machiavelli.”

She looks forward to a move from a “predominant reliance on a knowledge base consisting of a few narrow, circumscribed change efforts – even when that knowledge is certain, final, and ‘true’” to providing people with “access to knowledge that is integrated, coherent, accessible and deep, and deals with a broad range of promising change efforts – even if some of that knowledge is tentative, contingent, and approximate”.

See: Schorr L B (2003) Determining ‘What Works’ in Social Programs and Social Policies: Toward a More Inclusive Knowledge Base, Washington DC: The Brookings Institution.