Note: For the sake of brevity, I have paraphrased Richard’s answers; what follows is not a verbatim script of the interview but it is an accurate summary of it.
As always, there are practical recommendations at the end.
TvdZ: “Why and how did you become a scientist?”
RM: “That’s a complicated story. As an undergrad I was a musician, and was not really that involved with science. When I moved to Missouri with my wife Candice, I had a wide range of odd jobs, from selling dog food to being a casino accountant. Through Candice I met Jeff Rouder, who inspired me to go into cognitive psychology, as well as taking a Masters in statistics. When we were working together on complex hierarchical models, we wrestled with some of the limitations of frequentist statistics, and started using Bayesian methods.
At that time, it was not that common to share code. Most mathematicians did not bother with creating and sharing software for others to use, which meant we had to program it ourselves. Luckily, I have always been fond of programming, so I could work around that problem. It didn’t take much time until others starting asking about other applications of Bayesian statistics. “How can I apply this to ANOVA’s? Which programs can I use?”. That’s why I started working BayesFactor package for R; to empower others to be able to do the same thing, even if you don’t have a strong background in both mathematics and programming. With JASP we took it one step further, as it doesn’t require any programming at all.”
TvdZ: “There is an increasing awareness that there is a crisis in the social sciences. How would you characterize this crisis?”
RM: “All of science depends on trust. A single scientist can’t do everything by him/herself. Only together can we accumulate a body of evidence. I think we have been too trustful. We relied on findings which turned out to be false; unreliable. While we depend on trust, the foundation of science is skepticism, and there is a great need to be skeptical. There is this image of scientist being objective heroes, who go out and explore the world to uncover truths. It doesn’t work like that; scientists are just human, and suffer from the same reasoning problems as everyone else. We have been paying too little attention to issues like wrong incentives. We don’t want to worry about these things, because that’s not what we got into science for. But we have to think about these issues, if only to not fool ourselves.”
TvdZ: “For many scientist it is a very daunting set of problems. It doesn’t help that most proposed solutions are meta-solutions: ‘change the publication process’, ‘offer better incentives’, ‘improve the education of new scientist’. For an individual scientists, especially early career scientist, it might feel like they have no influence over any of this.
What would you suggest individual (early career) scientists to do?”
RM: “Radical transparency. Be open. Share your work; share your data. In some ways, being radically transparent puts you at a disadvantage compared to those who do not take this effort. However, there is a growing awareness that radical transparency is not just a good thing, but that is critical for science. This is especially important for early career scientists; at the start of your career it is much easier to get into the habit of being so transparent. So make your data available. Share it on social media: tweet about it, blog about it. That can be a way of making your own research open, even if a specific publishing venue does not facilitate that.
I think it’s important that PhD students enjoy the process of learning about all of these things. In light of how much you are already learning, it won’t take you that much more time to actually learn open practices. In some way that’s your job: to be a sponge, to soak in that knowledge; and you have to enjoy that. Otherwise, science might not be the right career for you.
You have to be open to learn new things, and better ways of doing science. It has a lot of advantages as well: more awareness about your research, more citations, and less skepticism about your work.
Radical transparency is just going to be necessary in some point in the future. There is no doubt in my mind that these open practices will be required in a very short time, say 5 years. You don’t really have an option but to learn them
The zeitgeist is for openness, and that is not reversible.”
TvdZ: “One of your main topics of interest is evidence. How would you define evidence?”
RM: “Evidence is always in regard to a specific question. Evidence is the extent to which a rational person should change his beliefs one way or another. There is an important distinction between statistical evidence and scientific evidence. We spend too little time thinking about what we should be convinced by, and how we can be fooled.
We tend to regard evidence as being interpretable by itself; we read a paper and think that ‘the evidence’ as just being about that paper. But a lot of the problems we have seen in science stem from the lack of consideration of the context of that finding. For instance, a paper is part of a publication system which is subject to rampant publication bias. You can’t ignore this context when thinking about the evidential value of a paper.
This all goes back to the trust we put in other scientist; and we don’t like to think about these things. We just like to read a paper and think about its evidence outside of its context, but we are gradually learning that this is a bad idea. We have to think about the context, we have to think about people’s motivation to defend their theories. We have to be skeptical of one another.”
TvdZ: “How skeptical are you? If you would have to choose, would you say that most scientific findings are true, or that most findings are false?”
RM: “Most findings are not even wrong. There are right things, and wrong things, and many findings in Psychology are so couched in ridiculous theories it is hard to even classify them as right or wrong. Of course you can argue about ‘is this effect size really different from zero?’, but that’s a trivial way of being wrong or right. I think that many scientific conclusions are in this category of ‘not even wrong’.”
TvdZ: “Another dilemma: if you would have to choose: pre-registration or open data?”
RM: “I think making your data openly available is always good to do. However, pre-registration is not always necessary; it’s not for every paper. But almost everything can (and should be) open.
On the other hand, I think that if people start to pre-register their work, this will increase our understanding of the problems in science. I don’t think it is a solution, but it is a diagnostic tool. So I guess I will pick pre-registration, as it will help us fix the problems by uncovering them. Open data is great, especially if you have a healthy science. But if you don’t have a healthy science, it is not clear what exactly we can do with open data sets. So for the long-term health of the social sciences, pre-registration will help us diagnose some of the problems we are having.”
TvdZ: “In your opinion, what is more important to increase the quality of education: better training in statistical methods, or in philosophy of science?”
RM: “Philosophy of science, for sure. I think that tools, even good tools, in the hands of somebody who does not know how to use them, are dangerous. You will shoot yourself in the foot. That’s what statistics is: a tool. That’s one reason why I think we should not just teach either frequentist statistics, or only Bayesian methods. We should not enforce either upon anyone. Instead, by given students a solid introduction in both, they can think for themselves and decide which set of tools (or maybe both) best align with what they want to accomplish.”
TvdZ: “Last question: who is Dr. Primestein?”
RM: “Haha, I can’t tell you that!”
TvdZ: “Aha, so you know!”
RM: “I can tell you it’s not me.”
- Radical transparency: Ask your publishers to put your data (and materials, etc.) online, or do it yourself. Check out this website for an overview of journals which support open science.
- Pre-registration: Increase the quality of your own research, and reduce skepticism by pre-registering your studies. Here you can see which journals allow pre-registration. Remember that even if a journal you want to publish in does not have this option, you can still manually pre-register at the Open Science Framework. Mention that you do this in your publications (and when talking with other scientist) to increase the awareness of pre-registration
- Sign the PRO open science initiative: By signing this initative, you will join hundreds of other reviewers who will actively ask scientist to publish their data and materials and will not offer comprehensive review for, nor recommend the publication of, any manuscript that does not meet the following the Open Science quality criteria.
- Social Media: Social media is another great tool to make (your) science more open, networking, and discussing research methodology.
- Learn both frequentist and Bayesian methods: Why limit yourself to one set of tools? Try to get at least a basic introduction in both methods, so you can choose which one is most appropriate for your type of research. As most will already be familiar with frequentist statistics, here is a guide on becoming a Bayesian eight easy steps. Sadly, this discussion can sometimes results in a bitter fight with two camps slinging feces at each other; don’t get dragged down into this. We all have the same goal: high quality research.
- Most important of all: Think About Evidence: Evidence is at the core of our work. Realize how easy it is to fool ourselves. Be a Skeptical Scientist.