patrickmoriarty

A delightful – if unverifiable – rumour has it that top staff of ‘a leading philanthropic donor’ are not allowed to travel together on the same plane: the loss to the world of their collective brainpower in case of a sudden accident being too much to bear.  That said donor is a champion of ‘evidence based’ approaches makes the rumour priceless (if unverifiable!), demonstrating as it does the very human limits to the use of any kind of evidence. (in this case, the proper evidence based policy would be to INSIST that the entire senior management spend every waking and sleeping hour in flight on a reputable airline, preferably with mid-air refuelling and rapid (and heavily guarded) transit from plane to plane when it becomes necessary to land for routine maintenance).

Un-attributable – and quite possibly untrue (surely no one could really be this stupid?) – anecdotes to one side, there is a more serious point.  That the evidence based craze currently sweeping parts of the development world, fanned by a self-seeking sub-set of ‘academia’ who – for a (typically large) price – will provide development projects with the required ‘evidence base’.

What all of this ignores of course – as with so many bright development ideas – is that there is no real evidence for evidence based approaches.  And certainly no proper cost benefit analysis (no – evidence is not free!) to provide evidence that evidence based policy is better than the standard stick-a-wet-thumb-in-the-air type.  I’m a big fan of the Guardian’s bad science column, but like many people form a medical background Ben Goldacre seems oblivious in this recent article (“How can you tell if a policy is working? Run a trial” ) to the cost of collecting evidence through trials, and to the need for trials to be the carefully chosen end point of policy (or drug!) development processes – not a sort of random ‘wouldn’t it be cool if ….’ first step.  As he says: “If you don’t know which of two reasonable interventions is best, and you want to find out, a trial will tell you.”   The point of course being that you need to get to the ‘two reasonable interventions’ in the first place!

Buoyed by economically illiterate cheerleading we are increasingly being asked to accept that if only governments (and everyone else) in developing countries would adopt the gold standard doubly blinded randomised control trial as the starting point for every possible intervention from bed-nets to maternal mortality to the colour to paint the inside of the presidential palace …. everything would be so much better.    One more silver development bullet in a long line.

This not only ignores the fact that the use of ‘evidence’ in the developed world (where data has become relatively cheap) is vanishingly rare (OK – there is NICE) but in practice of course, most evidence is used to justify a decision that has already been taken (decision based evidence making as a clever person once told me).  It also makes a false analogy.  (Randomised) controlled trials are not so much a “gold standard” as a carefully chosen specialist surgical instrument: the end point of a long and (hopefully) painstaking process of identification and fine tuning of possible approaches that arrives at a usefully testable hypothesis (those “two different interventions”).  Like a finely honed scalpel, trials should only be used by skilled practitioners in tightly controlled environments – otherwise the results risk getting messy.

There’s a serious point to this rant (though of course the ranting is fun too and is the major point of blogging).  The use of evidence – as currently being pushed – runs two big risks.  In the short term it is becoming an excuse for stone-walling  ……… does this sound familiar?

Advocating NGO (AN): ‘we think you should change the way you spend your money in the water sector, don’t just spend it on capital investment, but make sure that some is kept to strengthen sector institutions and to pay for larger repair and maintenance’

Bored/Cynical Development Bureaucrat (BCDB) – for it is he: ‘oh, but do you have any evidence that this will lead to better outcomes?’

AN: ‘well not directly, but we have lots of evidence that only investing in building new systems leads to unacceptably high levels of breakdown and inefficiency – and hence to reduced impact.  And we have promising signs from talking to practitioners and from case studies, that what we’re proposing may improve things’

BCDB ‘yes, but we need evidence that what you are proposing will be better – otherwise I will never be able to convince my superiors to invest money in it, especially not in the current climate’

Increasingly frustrated AN ‘yes, but if no one will invest in doing things differently then where will we get the evidence from?’

BCDB ‘Zzzzzz’

In the longer term, as poorly designed and implemented trials (and evidence generally) go the way of other silver bullets, the lack of positive outcomes risks discrediting the entire approach (think participation, think gender …) – and let’s be very clear – that would be a really bad thing.

So where do we go from here?  Perhaps a first step would be to help ‘policy and decision makers’ to understand ‘evidence’.  That it takes many forms – from anecdote, through well documented case studies, to surveys and statistics and, indeed, in the end – trials.  That it can be positive or negative, that it can provide strong confirmation or softer suggestion.  That negative evidence is often the starting point ‘the way things are being done does not work’.  And that while such a finding implies the need to look for solutions, this process will of its nature be hesitant to start with and based on suggestive rather than confirmatory evidence – ‘this might work better’, ‘what they did in this other country looks a bit similar and might have something interesting to show us’.  And that each step along this route, each increase in the certainty of the finding and solidity of the evidence, requires a concomitant increase in the size of the sample and the amount spent on collecting data.

Perhaps most important to understand is that evidence from anything but full scale impact evaluation of large projects is highly unlikely to provide ‘robust’ findings from a policy perspective.  And that therefore if our BCDB wants robust findings she will have to cough up the money, make the intervention, do so in a testable manner, and have the patience to wait for the results (which may of course be negative!).  The sort of evidence that can be collected from a typical NGO pilot study – particularly of systemic interventions as opposed to those that target individuals – is highly unlikely to be strong or statistically representative.  Such experiments can at best provide strong suggestion and help to better form hypotheses for further testing.

As for those of us involved in action research aimed at bringing about systemic change, we can also help ourselves.  By being more honest (with ourselves, our donors and those policy makers we seek to influence) about the limits of what we can (and can’t) demonstrate, about the likely time-frames in which we can expect to see measurable change, about the place of our own work in the larger ‘research into policy impact’ scheme of things and finally by being clear about the sort of evidence we expect to be able to generate to support our propositions:  evidence based on anecdote; evidence based on case studies; evidence based on limited surveys …….