How do you measure what's been tailor-made?

Mental illness might appear to be the most private of struggles. However, the US government is investing over $1 billion in making these interior problems a more public concern and it is marshaling community resources to provide better care for children with mental health problems.

The 125 communities receiving the federal funds are developing co-ordinated “systems of care”. They can involve a range of community-based, culturally sensitive services for the mentally ill including diagnosis and evaluation, case management, outpatient therapy, 24-hour emergency services, intensive home-based care, therapeutic foster care and transition services.

Although the principles may be the same, each of the 125 tailors a combination of initiatives to its needs, with the inevitable result that it is difficult to know how far the “systems of care” philosophy, in general, is helpful to children with mental illness.

Indeed the predicament in which US evaluators have found themselves is rather reminiscent of the conundrum for English reseachers created by the diversity of Sure Start. Meaningful comparisons can become very difficult. [See: Learning the moral of the Sure Start story [1]].

The task of assessing the impact of the effort and at the same time accounting for the distinctiveness of each community and its particular system, has lately fallen to E. Michael Foster [2] of the University of North Carolina and colleagues at Macro International Inc. [3]

In their recent article in Children and Youth Services Review, Foster and his team acknowledge that the best way to assess the impact of the systems of care approach would have been to assign mentally ill children randomly either to a community that received funds to adopt the approach or to one that didn’t, and then to compare the psychological functioning of the two groups over time.

So another familiar conflict was encountered – between the essential nature of the program and the requirements of reliable scientific evaluation: children and families could hardly be uprooted from their home communities for the benefit of a control group.

Instead, the team had to compare the outcomes of children in funded communities to children in similar communities (in terms of geography and demographic and economic characteristics) which did not receive funds. They took the extra step of removing from the study any child who did not have a similar counterpart in the other community.

In all, the study involved 573 children receiving psychological services in Alabama and Nebraska. Children and their parents were interviewed over the course of a year.

In Alabama, children in "system of care" communities showed substantially greater improvement in their mental health status than did their counterparts in the comparison community. In Nebraska, however, there was no significant difference in the outcomes for children who experienced system of care versus those who did not.

The authors conclude that results from more of these types of comparisons are needed to assess the impact of the underlying philosophy. They also stress the unique circumstances of each community and suggest that a better of question than “does the system of care work in general or not?” might be “under what conditions does it work?”

Summary of “Can system integration improve mental health outcomes for children and youth?” by E. Michael Foster, Robert Stephens, Anna Krivelyova and Phyllis Gamfi in Children and Youth Services Review, Volume 29, Issue 10, (October 2007), pp1301-1319