In this edition of Development drums, Owen talks to Rachel Glennerster, Director of the Poverty Action Lab, about rigorous evaluation of development programmes.
Rachel explains how rigorous evaluation techniques can give important insights into the effectiveness of development programmes. She explains the role (and limitations) of randomised controlled trials, and she addresses some of the criticisms of this kind of evaluation.
Podcast: Play in new window | Download
Subscribe: RSS
This is an interesting episode episode! I liked the illustrative examples provided by Rachel. But, in what situations would randomized evaluation not be advisable to conduct randomized evaluation? I agree with Rachel that the ethical concerns may not be that serious in countries where factors such as sectarianism and corruption have already heightened the probability of certain individuals not being reached by externally designed development interventions. Given that mindsets tend to either facilitate or impair progress towards desired development change, how do you handle it in randomization since the trait may not be randomly distributed in a given community?
I enjoyed the first part on the interview – RG’s comments on RCTs and evaluations all seemed quite measured and reasonable. However, a bit past half way through, once you started to discuss some of the criticisms of RCTs, she started going too far in her claims for what questions RCTs have answered and can answer.
Does she really think that RCT methodology based studies can answer most of the pressing questions about how to improve health services delivery in developing countries? If she does, as she says, then she must be un familiar with the health systems and pressing policy questions in developing countries, and also with the health services research that sheds the most light on the questions (Hint: mostly not RCTs). She subsequently states that the RCT studies documenting a kink in the demand curve around the zero price for preventative health products (e.g. demand drops significantly with smally increases in price from zero) provide sufficient grounds to recommend that no health services or products should have any positive price. This statement is dumbfounding. First off, can she not distinguish preventative from curative products? (NB: they have different demand curves). And, it is widely understood that you need to analyze them, and set their prices, taking the differences into account.
When you ask her about supply issues, she says yes, well, of course supply must be dealt with somehow. Uh huh. But she’s perfectly comfortable recommending zero prices for all the services even though there are no RCT studies that examine the supply effect (statically or over time) – cause they assume the problem away for their nice little studies. In developing country health systems, the problems with ensuring supply are just as profound as the problems with ensuring demand. And, policies can only be informed by analyzing the effect of a policy choice on BOTH at the same time.
In addition, she is assuming away one of the most pressing questions: given limited resources policy makers in developing countries have to decide which services to keep prices low or free on, and which services to allow charging for, to put their budgets on a sustainable footing. This is a question which RCT studies showing the slope of the demand curve don’t answer – they require study in a general equilibrium framework, and require dynamic analysis.
Perhaps this is why most social policy questions in developed countries rely extensively on non-RCT method studies, and a bit on RCT studies.
Oh, and about that de-worming result which she claims is “very generalizable”, I think she may have forgotten the other JPAL study which found the deworming intervention to be one quarter as effective (in increasing school attendence) in India as it was in Kenya. See this link to the graph http://www.denniswhittle.com/2011/06/randomized-trials-not-silver-bullet.html.
The external validity problem (e.g. failure of RCTs to shed light on the key context variables which influence the level of impact observed) is more serious than RG’s comments would lead a listener to believe.
Though RG acknowledges that other methods have their place and are needed to answer “some” questions, the evaluation training course JPAL runs only trains people on RCTs. (see course description http://www.povertyactionlab.org/course)
So how would their students be able to select the best methodology to suit the research question?
My guess? They won’t. They will just take their hammer, and soon find that every thing looks like a nail.
Owen, I would be super appreciative if you could find a non-randomista to interview.