I've recently worked on a few pre-registrations and want to share my experiences with people who're thinking of doing one, and I've invited Theresa Wege, a visiting student in our lab to write a guest blog as well! We won't talk about what pre-registration is or the pros and cons of it, because there're already great discussions here and here. Instead, the two of us will reflect on our experiences and talk about a few things that we've learned through the pre-registration process.
The cool thing about this blog post about pre-registration is that we're coming from different stages in our career.
Thoughts from Theresa!
When you follow the discussions about open science and reproducibility, you often hear people say how they did science in the past and how they want to do it now. The “open science movement”, from my perspective, is framed as a change of your current habits.
And then there are people like me, who are still a „tabula rasa“. I just graduated in Psychology and am working on my very first research project with Pierina and Rebecca Merkley. Suddenly, I’ve become a researcher, and being in charge of a study has become overwhelming. When I started this project, blinded by enthusiasm, I underestimated how many decisions I would have to make and how many times a plan can change over time.
I didn’t know where to start, but the pre-registration has really helped me structure this ‘chaos’. Collectively, we pre-register to do better science, but on an individual level, I also want to give a home to all my thoughts related to the study. The pre-registration document has become a tool for me to develop my ideas. And although it’s supposed to be short, focused and finalized, for me, it has evolved into a much larger document that contains everything related to the study. I also find it very helpful for collaborating with my supervisors. Whenever I have questions or we meet to catch up, everything we need is in the pre-registration document. This ensures that all of us are always on the same page.
After my first experience with pre-registration this is something I can't imagine missing in my workflow. The „crisis“ of psychology might make those, like me, who have just started doing research in psychology feel insecure about how we can build our careers. But at the same time, we also have the amazing opportunity to grow up with good research practices and become „open science natives“.
Thoughts from me!
1) Pre-registration makes me think more carefully about my study design.
You know that feeling when you’ve gained new insights about a study when you start writing a manuscript? To some extent, pre-registration gives me that feeling.
You design a study to test a hypothesis, and you know what tasks, conditions, or trials to include. But wait -- before you collect your data, have you thought carefully about what you consider to be evidence for your hypothesis? (We always think we do. After all, there's a reason why we include this condition but not that condition, but I think we sometimes underestimate the number of decisions we have to make in the process.)
In developmental psychology, we sometimes ask when children begin to demonstrate some knowledge X, and we design a study such that if children demonstrate success on some tasks or trials such as A and B, then we take that as evidence for having kowledge X. But your job doesn’t end there. We’re still left with questions such as the following: Do you need to see success in both A & B, or are you willing to take as evidence for knowledge X if kids succeed on just A but not B, or if only some kids show success across both A & B? And what does it mean if you don't get your predicted outcome(s)?
In my own experience, in thinking about what I take as support for my hypothesis, I have decided to drop some trials because I realized they were redundant.
2) There is more than one way to analyze your data, so making up your mind early is good.
Figuring out how to analyze a dataset is not an easy task. Even as simple as a 2AFC task that’s commonly used in developmental psychology, there are still a number of ways to analyze it (e.g., what type of pairwise comparisons are you going to do? will you be collapsing across the levels of a variable? how do you quantify 'success'? will you analyze group performance only or are you also going to look at individual kids and categorize them based on some criteria, such as the number of kids who answer N trials correctly?). Just because you’ve designed a study to seek evidence for X doesn’t mean there’s only one way to show evidence for X in your data. There’re often multiple ways!
Working on the pre-registration has forced me to think about what I consider to be the strongest evidence, and whether I’m going to do secondary analyses or what they’re gonna be. Often times, we have a multi-step analysis process. Analysis 2 depends on the outcome of Analysis 1 and working through the pre-registration makes you think very carefully about what those analyses 2, 3 and 4 may be.
3) Write the methods section too!
The pre-registration form asks you to specify the variables in your study (e.g., between- or within-subjects variable, levels of factor variables). And I thought since I’m providing all this information on the pre-registration, I may as well write the methods section. And I did. I find it helpful to work on the methods section and the pre-registration side by side. The two documents are complementary. The pre-registration has elements that we don't always include in the methods section (e.g., justification for sample size, power analysis), and the methods section has study details that you may not include in the pre-registration (e.g., justification for the conditions / trials, details of counterbalancing, study procedures). With the analysis plan that will help you write your results section, you’re almost halfway done with a paper. Plus, our memory of study details fades over time, so writing up exactly how you plan to run your study beforehand is easier than writing about it after you're done data collection.
So far, we think there are more pros than cons for pre-registration. We’ll update you all on what our thoughts are when we're on the other side of the tunnel. It’s likely that we’ll have different thoughts when we're actually analyzing our data!