So you want to do an RCT with a government: things you should know

/

David McKenzie’s blog about the arms race in RCTs raised the concern that there will be a reduction in the number of RCTs with governments and that it will get harder to publish them. I am not going to comment on the publication issue but instead on the pros and cons of doing an RCT with governments and some practical suggestions for those who do want to work with governments.

On the benefit side, governments:

  1. Have substantial resources to invest in large and expensive programs and evaluations;
  2. Cover large populations. When else could you randomize at the subdistrict level and have 1.8 million beneficiaries in an RCT, as in Olken et al. (2014)?;
  3. Collect a lot of data on individuals, such as test scores for children, earnings for adults, and encounters with the criminal justice system. Working with governments can help you get access to these administrative data which can reduce the cost and hassle of running an RCT;
  4. Can run a program in the way it would be scaled by a government;
  5. Are able to manipulate policies in areas other organizations cannot influence, such as how tax inspectors are rewarded (Khan, Khwaja, and Olken 2014) police officers are trained and rewarded (Banerjee et al. 2012); and how firms' emissions into the environment are regulated (Duflo et al. 2012). 

Some examples of RCTs that use these advantages of partnerships with government to good effect are Angrist et al. (2006), who were able to follow up winners and losers of a lottery for vouchers to attend private school in Colombia by linking winners to a centralized college-entry exam seven years after the vouchers were issued. In ongoing work, Bettinger et al. link the same voucher winners and losers to government tax and earnings data, 17 years after the lottery. Muralidharan and Sundararaman (2011) test the impact of teacher-incentive pay in a representative sample of rural schools across the state of Andhra Pradesh, meaning their results are valid across a population of 60 million. Banerjee, Hanna, Cohen, Sumarto, and Olken worked with the Government of Indonesia to test how providing individual ID cards to recipients of government-subsidized rice (which indicated the amount and price of rice they were eligible for) could reduce corruption in the distribution system. The results showed that the cards increased the subsidy received by targeted recipients by 25 percent, so the government scaled up the ID card program, reaching 66 million people. The time from evaluation design to scale-up was about a year.

With these benefits, however, come considerable costs:

  1. Governments can be slow-moving and less able or willing to test out-of-the-box solutions than NGOs.
  2. It may be particularly difficult to run more theory-oriented field experiments with governments. They tend to be less interested in answering an abstract question, the answer to which could inform many policies but would not be scaled up as a specific program.
  3. Governments can also find it harder than NGOs to provide services only to a limited group of needy citizens. Some governments have laws requiring them to treat citizens of equivalent need equally. When the Government of France wanted to test programs using randomized trials, they first had to change the constitution to make this possible. 
  4. Staff turnover in governments can be high as civil servants are transferred regularly. This makes it even more important to build support at different levels of government: if the RCT has support from the minister but not the bureaucrats, then it is likely to die with the next cabinet reshuffle.
  5. An election can lead to a dramatic change in policy priorities and personnel at the same time. It can also lead to paralysis for a period both before and after an election, even if the program being evaluated has bipartisan support. (An RCT I was involved in collapsed when none of the planned monitoring could take place because a newly elected government froze all nonessential expenditure while they thought through new priorities. In another instance a survey had to be suspended just as it was about to go into the field because of a national exchange rate crunch, which again led to a freeze. Government budget shortfalls and last minute crunches are not confined to developing countries.)
  6. Governments can renege on any agreement with impunity. There is not much a researcher can do when a government decides to fill a shortfall in a program budget with money set aside for, say, the endline.

Many of the strategies for working with partners discussed in my previous blog are just as relevant to working with governments. Government partners are in a powerful position vis-à-vis the researcher, so it is important to listen hard to what they want. They often work within short political timelines, so delivering intermediate products such as baseline reports can be key for keeping them engaged. 

There are also specific actions a researcher can take to help the often bumpy ride of partnering with governments. A more formal approach to partnership may be needed than in the case of working with NGOs. Governments often require a memorandum of understanding that sets out clear expectations for both parties. Discussions may be going well at the practical implementation level, but any final decision—even a relatively small one—is likely to require sign-off from someone senior. It is important to build extra time into the schedule to account for this.

Government procurement rules can also cause considerable delay. For example, if we decided that an intervention needs a leaflet to explain the study to participants, the government may require a competitive bid for the printing of the leaflet, leading to several months delay. Having some independent funding that does not run through the government can be very helpful in easing some of these constraints: a researcher can come in and offer to pay for a leaflet, or for additional monitoring, etc. Independent funding can also help keep the research going if the government faces short-run liquidity constraints.

Being the first to do something might be exciting for an NGO, but can make a government nervous about being exposed to criticism. Thinking through the optics of the experiment (i.e., how it would look on the front page of a newspaper) can help alleviate concern. Another strategy is to bring in an official from another department or country who has worked on an experiment before, preferably of a similar type.  It is much more reassuring for officials to talk to other officials than it is to hear from a researcher.

Policymakers often have a healthy skepticism of researchers who want to provide advice about how to measure or improve a program, especially those coming from another country, state or region. It is important for researchers to prove their relevance and their local knowledge. A mix of humility, a desire to learn from the policymaker, and a lot of homework about local conditions can help. I have seen policymakers visibly relax and start to engage when they hear from a researcher about their on-the-ground experience. A well-placed anecdote about a conversation with a farmer in Kenema or a teacher in Pittsburgh can be critical for building credibility.

The bottom line is that working with governments can be rewarding, but is very hard and takes enormous investment. It is also risky. As such it may be inadvisable for junior researchers who have to finish a PhD on time or get a paper published before a tenure decision to do an RCT with a government. Even senior researchers need to think hard about the commitment, risk, and reward before taking on an RCT with a government.

For more discussion on ways for researchers to make working with governments easier, see my next blog entry.