By Jeremy Hardie
Everybody likes evidence-based policy – who could favour a policy that is not confronted with the facts? – but after twenty or more years trying to make it work, we have ended up with some quite strange results, at least in the US and the UK. For instance, with many social policy interventions, you have to show that your proposal is supported by a Randomised Controlled Trial (RCT). That sounds fine, if a bit tough. It at least means that policy is not, as Mrs. Thatcher’s Home Secretary Kenneth Baker once complained, determined by what she heard from her hairdresser yesterday – unless the hairdresser had an RCT to hand. And it is what drugs have to pass if they are to be approved by the National Institute for Health and Clinical Excellence (NICE) or the U.S. Food and Drug Administration (FDA).
However, it also means that policies which have been designed by professionals, exhaustively tested by experience in the field, and accepted by them and their clients as doing good, can be terminated because subsequent evaluation by an RCT shows no effect. Now, that may just be having to face the facts. Just as Mrs. Thatcher’s hairdresser may have to accept that rigorous testing shows her policy proposal to be wrong, so in the more respectable circumstances of a parenting programme in Wales – where most recently this problem has arisen – it may be that the professionals have come to accept as the conventional wisdom that their programme works, even though it does not.
Self deception and conservatism and vested interests do certainly exist. But maybe something else is going on. Some of the problems may be technical. There are difficulties if an RCT is designed not to test one outcome (such as ‘this drug reduces heart disease’), but several; so the parenting programme mentioned above helps with parent self confidence and child anxiety and truancy and…. If you ask people to take part in an RCT, you are saying that they have an even chance of being in the control group, and therefore not getting the help. If you, the parent, are worried about the family and can get help otherwise – from a GP or from friends – might you not then seek that help directly, rather than risk being part of the control group in the test? So, there is recruitment bias; your test population will tend to exclude those who have the more acute problem and those who might be most motivated to take part in the programme.
Another quite different worry is this. We want professionals to think seriously about what will work for their clients, and use their professional judgement and experience to do so. That is a key part of wanting to decentralise decisions, and get away from decisions made by ticking the boxes in a centrally mandated code of best practice. In the Welsh case, what nearly happened was that a programme which everybody involved approved of, including the professionals and the parents, might have been struck down by the application of a centralized rule – follow the RCT. If that were to happen a lot, then we have to say goodbye to decentralisation and the exercise of local expertise. That will be a good thing if indeed we can prove that again and again the locals are wrong. But can that be right?
Jeremy Hardie is now a Research Associate at the Centre for Philosophy of Natural and Social Science, London School of Economics. He was Chairman of the WHSmith Group from 1992 to 1999. He is co-author (with Nancy Cartwright) of Evidence-Based Policy: A Practical Guide to Doing It Better (OUP, 2012).
Image credit: Controlled test tubes. Photo by Armin Kübelbeck via Wikimedia Commons.