On December 12th I attended the annual meeting of the Berkeley Initiative for Transparency in the Social Sciences (BITSS). BITSS brings together economists, political scientists, biostatisticians, and psychologists to think through how to improve the norms and incentives to promote transparency in the social sciences. I was on a panel talking about preanalysis plans in which researchers specify in advance how they will analyze their data.
I have now been involved in writing four of these plans and my thinking about them has evolved, as has the sophistication of the plans. Kate Casey, Ted Miguel and I first wrote one of these plans for our evaluation of a Community Driven Development program in Sierra Leone (see the previous blog ). It was exactly the type of evaluation where pre-analysis plans are most useful. We had a large number of outcome variables with no obvious hierarchy of which ones were most important so we specified how all the outcomes would be grouped into families and tested as a group. While the outcomes were complex the randomization design was simple (one treatment, one comparison group).
The next case also included multidimentional outcomes: empowerment of adolescent girls in Bangladesh. However, now we had five treatments and a comparison group with different treatments targeted at different ages. The task of prespecifying was overwhelming and we made mistakes. It was extremely difficult to think through in advance what subsequent analysis would make sense for every combination of results we might get from the different arms. We also failed to take into account that some of our outcomes in a given group were clearly more important than others: we ended up with strong effects on years of schooling and math and literacy scores but the overall “education” effect was weakened by no or negative effects on indicators like how often a girl read a magazine. We hope, when we write the paper people will agree it makes sense to deviate from our plan and concentrate on the more important education results.
Learning from that example, the most recent pre-analysis plan was written in stages. Kelly Bidwell, Kate Casey and I evaluate the impact of screening debates between MPs in the Sierra Leone election (Kate did a webinar on the emerging results). We wrote our initial plan of action (when the survey was in the field) and then updated it as we revealed and analyzed different parts of the data sequentially. We also specified which outcomes were primary and which were secondary (i.e. would help us understand the mechanics of how the intervention had an impact but weren’t to be considered “success” on their own). We started by looking at data collected in treatment areas before and after the debates. Analyzing these data helped us update our hypotheses for the next version of the PAP. We next examined data from the comparison group and downgraded some outcomes as too hard to change (when turnout is 98% in the comparison group, debates are unlikely to increase turnout). With this information we updated our PAP for our individual level experiment in which different voters were shown different parts of the debate (we logged the history of how this PAP evolved over time). Finally, after analyzing these results we finalized our PAP for the main evaluation of the debates. All changes were redlined with dates on which the changes were made.
The other unusual part of this PAP was that for some outcomes we specified the use of one sided tests rather than the standard two sided tests. For example, we only tested whether debates increased knowledge. We increased our power to detect this effect by committing not to look at decreases in knowledge. This approach is only reasonable if there is no theory under which a negative effect could make sense: ie if you saw a negative effect you would assume it was an anomaly. Where a one sided test makes sense its important to commit to it in advance.
These issues are discussed in more detail in Module 8.3. For examples of published pre-analysis plans see resource links under Chapter 8. There will also be a panel to discuss pre-analysis plans at the AEA.