A practical framework for evidence informed policy: addressing the generalizability puzzle

There has been a huge increase in the number of impact evaluations of different approaches to reducing poverty but despite this if you are a policy maker it is unlikely that there will be a rigorous impact evaluation that answers precisely the question you are facing in precisely the location you are facing it. How do you draw on the available evidence, both from your local context and from the global base of impact evaluations in other locations, to make the most informed decision?

In an article just published Stanford Social Innovation Review, Mary Ann Bates and I set out a practical generalizability framework that policy makers can use to decide whether a particular approach makes sense in their context. The key to the framework is that it breaks down the question “will this program work here?” into a series of question that come out of the theory behind a program. Different types of evidence can then be used to assess the different steps in the generalizability framework.

Here is a generalizability framework for providing small incentives to nudge parents to immunize their children. The first steps require a local diagnosis of the problem and need to be answered using local descriptive data as well as qualitative interviews and local institutional knowledge. The next steps are about general lessons of human behavior where studies from other contexts can be very valuable. The final steps are about local implementation where local process monitoring evidence is key.

In the article we discuss our experience working alongside policymakers around the world to apply this framework to solve practical policy problems. We also show how this approach enables policy makers to draw on a much wider range of evidence than they might otherwise use: for example, with only two published RCTs on the immunization program above there is a wealth of rigorous impact evaluation supporting the general behavioral lesson behind the program. With this article we seek to move the debate about generalizability of impact evaluations from its rather confused and unhelpful present to a more practical future.

See full article here.