Some thirty-five years before former UK Prime Minister Tony Blair made his last major speech on social exclusion, the then Conservative Minister for Social Services, Keith Joseph, spoke about what he called the “cycle of deprivation” – the way poverty and associated behaviors and attitudes seem to pass across generations.

The “remarkably similar content” of the two speeches is compared by public health specialist John Welshman as a basis for investigating continuities and discontinuities in this debate during the intervening years.

His book focuses on the origins of Joseph’s analysis, the direction of the government-funded research program that followed it and on the connections between ideas in the 1970s and New Labour’s approach to tackling poverty, social exclusion and anti-social behavior.

Welshman takes the perspective of a social historian, relying primarily on published documents, extensive archival research (for example, minutes of meetings and correspondence between ministers and civil servants) and interviews with numerous key players in Joseph’s 1970s program, including social scientists and civil servants.

He acknowledges that uncovering “what really happened” is doomed in a postmodern age but nevertheless he provides an in-depth case study of the political process from a variety of perspectives.

In it we witness the fluctuating fortunes of ideas about the nature, causes of and responses to poverty and attempts to get them embedded in policy. The influence of individuals stands out. For example, Keith Joseph’s concern with deprivation was fueled by guilt over his privileged background, while his choice of solution could be traced to his attraction to simplistic remedies to complicated problems.

Meanwhile, tensions between researchers and policy makers meant that the ensuing research program departed dramatically from the Joseph thesis. He wanted empirical evidence for ideas he cherished. Instead, the agenda shifted from a focus on behavioral (or individual) explanations of poverty to structural ones with an emphasis on discontinuities as opposed to continuities.

This deviation owed much to changes of personnel on the research committee and broader political shifts, notably the election of a Labour government in 1974. However, researchers also exerted their power. Many were attracted by the funding but actually saw the cycle of deprivation hypothesis as a red herring and consequently ignored or subverted it.

Some, such as Peter Townsend, had trained at the LSE and imbibed their tutors’ Fabian outlook, and were consequently reacting against stereotypes of problem families and an emphasis in social work on casework. The DHSS (Department for Health and Social Security) remained sceptical about the research, in part because of methodological differences but also because it anticipated conclusions that were “impractically broad and unrealistic”.

The important footnote to this story is that 30 years later, in a UK New Labour administration, the original and largely discredited idea took root in policy. Welshman helpfully provides evidence from numerous ministerial pronouncements and government documents on social exclusion that the “cycle of deprivation” thesis is alive and well. Even the language is often the same, reflected in the objectives of the early years program Sure Start.

In its early days, he argues, New Labour was careful to present anti-social and other undesirable behavior as responses or adaptations to adverse circumstances, but increasingly, he suggests, an emphasis on “problem families” in the context of a crackdown on anti-social behavior betrays just one idea: that the individual child or family are at the root of the problem and must therefore be the target for intervention.

There are important lessons in all of this for applied researchers working in children’s services but also other fields of inquiry. One is a reminder that policy making is far from being a linear process in which good ideas translate seamlessly into better practice. This is old news but it is frustrating today to watch a government implementing initiative after initiative with little evidence of effectiveness while at the same time ignoring programs that have been proven to work in real world settings in numerous countries. A challenge for researchers is to better understand why this happens so that we have a greater chance of improving the evidence-base of policy in the future.

Perhaps a more valuable lesson concerns the danger of focusing too much on either the individual or society when seeking to explain human development and behavior. In the 1970s research program, too many studies were interested in psychological factors only, or economic factors only. Indeed, a leaning towards the latter meant that human motivation and agency – the capacity to undertake the preferred action – were largely ignored in favor of more determinist explanations of disadvantage.

More recently there has been greater recognition of the dynamics of disadvantage, with its complex continuities and discontinuities, and an associated interest in understanding the mediating and moderating mechanisms operating between the individual, family and wider environment.

• From Transmitted Deprivation to Social Exclusion: Policy, Poverty, and Parenting by John Welshman is published by The Policy Press. ISBN 978 1 86134 835 7