Making your policy work for you

How can you predict whether a policy you develop will actually work? This was the question addressed by a public lecture at the London School of Economics yesterday evening, delivered by Nancy Cartwright, Professor of Philosophy in the Department of Philosophy, Logic and Scientific Method at LSE and at the University of California, San Diego.

Prof. Cartwright questioned how you can know whether a policy that is successful in one location will work in another. She suggested that in fact, based on success in one place you cannot assume that it will: “getting policy right is hard, but it is harder if you only rely on one tool”.

Prof. Cartwright criticised the enthusiasm of some economists to champion the significance of Randomised Controlled Trials (RCTs) to assess the efficacy of policies. As in RCTs for a medical intervention, policy treatments are assigned to some of a sample group and not to others, with the randomisation minimising any significant background or location differences between the controls. This allows a policy-maker to conclude that a significant improvement in the ‘treated’ group is due to the success of the policy intervention.

The mistake that enthusiastic policy developers then make is, Professor Cartwright suggested, to infer from the success of the policy in one location that the policy must then produce the desired outcome more widely. Several assumptions underlie this: that in the inital ‘treated’ policy population the policy had the causal effect due to ‘causal principle X’; ;that in the target population the policy will also have the desired effect due to this causal principle, and; that the support factors needed for successful policy implementation are in place in both populations.

Professor Cartwright demonstrated that causal principles have a number of problems, making it less than clear that a policy successful in one location will be successful in another. The first problem was described as ‘fragility’: the idea that a causal problem can easily be broken by pushing this too hard. One example of this is the relationship between employment and inflation. Inflation and rising prices will lead to reduced unemployment as businesses expand production. If a government tries to reduce unemployment by manipulating inflation however, businesses will not create jobs because they are aware that inflation is caused by government measures, not market forces. The second problem is one of ‘locality’, with causes dependent on local situations and particular conditions.

Professor Cartwright illustrated the problems of ‘fragility’ and ‘locality’ by an example; that of a child nutrition policy in Tamil Nadu, India, that provided mothers with nutritional advice and with supplementary baby food. An RCT proved that this could improve child nutrition. The conclusion was therefore that “mothers given better nutritional knowledge improve their child’s nutrition (when also given supplementary baby food)”.

However, when implemented in Bangladesh this policy did not work. Men in Bangladesh procure food for the household and in many extended families Mothers-in-law control food distribution between family members. Therefore in this case mothers were not able to influence their children’s nutrition. The causal principle underpinning the policy should in fact have been: “Better nutritional knowledge improves child nutrition when given to those who a) control what food is procured; b) control what food gets dispersed and c) hold the child’s interests as central to performing A and B”.

This example demonstrates that causal principles often have to be expressed in an abstract way if they are to be applied to any situation which differs from the original situation under test. The causal principle can be characterised as a rule which (very probably) holds true. It is vital to understand how the supporting factors in the new situation differ from that under test in order to make sure that the causal principle applies in a range of cases.

With thanks for this information to Evelyn Underwood, Policy Analyst at the Institute for European Environmental Policy</a.