Why you should be a skeptical scientist

Don’t take my word for it, but being a scientist is about being a skeptic.

About not being satisfied with easy answers to hard problems.

About not believing something merely because it seems plausible…

.. nor about reading a scientific study and believing its conclusions because, again, it seems plausible.

“In some of my darker moments, I can persuade myself that all assertions in education:
(a) derive from no evidence whatsoever (adult learning theory),
(b) proceed despite contrary evidence (learning styles, self-assessment skills), or
(c) go far beyond what evidence exists.”
– Geoff Norman

Why you should be a skeptical scientist

The scientific literature is biased. Positive results are published widely, while negative and null results gather dust in file drawers (1, 2). This bias functions at many levels, from which papers are submitted to which papers are published (3, 4). This is one reason why p-hacking is (consciously or unconsciously) used to game the system (5). Furthermore, researchers often give a biased interpretation of one’s own results, use causal language when this isn’t warranted, and misleadingly cite others’ results (6). For example: close to 28% of citations are faulty or misleading, which typically goes undetected as most readers do not check the references (7).

everybody lies.jpgThis is certainly not all. Studies which have to adhere to a pre-registered protocol, such as clinical trials, often deviate from the protocol by not reporting outcomes or silently adding new outcomes (8). Such changes are not random, but typically favor reporting positive effects and hiding negative ones (9). This is not at all unique to clinical trials; published articles in general frequently include incorrectly reported statistics, with 35% including substantial errors which directly affect the conclusions (10, 11, 12). Meta-analyses from authors with industry involvement are massively published yet fail to report caveats (13). Besides, when the original studies are of low quality, a meta-analysis will not magically fix this (aka the ‘garbage in, garbage out’ principle). One such cause for low quality studies is the lack of control groups, or what can be even more misleading: inappropriate control groups which can incorrectly imply that placebo effects and other alternative explanations have been ruled out (14).

Note that these issues are certainly not restricted to quantitative research or (semi-)positivistic paradigms, but are just as relevant for qualitative research from a more naturalistic perspective (15, 16, 17).

Everybody lies

This list could go on for much longer, but the point has been made; everybody lies.

In the current system, lying and misleading is not only very simple, it is incentivized. Partly this is due to the publication system, which strongly favors positive findings with a good story. In addition to cultural aspects, individual researchers of course also play a fundamental role. However, what makes it especially tricky is that it is also partly inherent to many fields, especially those which do not have ‘proof by technology’. For example, if you claim you can make a better smartphone, you just build it. But in fields like psychology this is rarely possible. The variables are often latent, and not directly observable. The measurements are indirect, and it is often impossible to proof what they actually measure, if anything.

Bad incentives won’t disappear overnight. People are resistant to change. While there many who actively fight to improve science, it will be a long, if not never-ending journey.

And now what…

Is this an overly cynical observation? Maybe. Either way, it is paramount that we should be cautious. We should be skeptical of what we read. What is more, we should be very skeptical about what we do, about our own research.

This is perhaps the prime reason why I started this blog: I am wrong most of the time. But I want to learn and be slightly less wrong over time. We need each other for that, because it is just to easy too fool oneself.

In the upcoming blogs I will continue to focus on issues in science, but more importantly I will attempt to highlight better ways to do science and share practical recommendations.

Let’s be skeptical scientists.

Let’s become better scientist.

References

  1. Dwan, K., Gamble, C., Williamson, P. R., & Kirkham, J. J. (2013). Systematic review of the empirical evidence of study publication bias and outcome reporting bias—an updated review. PloS one, 8(7).
  2. Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502-1505.
  3. Coursol, A., & Wagner, E. E. (1986). Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias. Professional Psychology: Research and Practice, 17(2), 136-137
  4. Kerr, S., Tolliver, J., & Petree, D. (1977). Manuscript characteristics which influence acceptance for management and social science journals. Academy of Management Journal, 20(1), 132-141.
  5. Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of p-hacking in science. PLoS Biol, 13(3).
  6. Brown, A. W., Brown, M. M. B., & Allison, D. B. (2013). Belief beyond the evidence: using the proposed effect of breakfast on obesity to show 2 practices that distort scientific evidence. The American journal of clinical nutrition, 98(5), 1298-1308.
  7. Van der Zee, T. & Nonsense, B. S. (2016). It is easy to cite a random paper as support for anything. Journal of Misleading Citations, 33(2), 483-475.
  8. http://compare-trials.org/
  9. Jones, C. W., Keil, L. G., Holland, W. C., Caughey, M. C., & Platts-Mills, T. F. (2015). Comparison of registered and published outcomes in randomized controlled trials: a systematic review. BMC medicine, 13(1), 1.
  10. Bakker, M., & Wicherts, J. M. (2011). The (mis) reporting of statistical results in psychology journals. Behavior Research Methods, 43(3), 666-678.
  11. Nuijten, M. B., Hartgerink, C. H., van Assen, M. A., Epskamp, S., & Wicherts, J. M. (2015). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior research methods, 1-22.
  12. Nonsense, B. S., & Van der Zee, T. (2015). The reported thirty-five percent is incorrect, it is approximately fifteen percent. The Journal of False Statistics, 33(2), 417-424.
  13. Ebrahim, S., Bance, S., Athale, A., Malachowski, C., & Ioannidis, J. P. (2015). Meta-analyses with industry involvement are massively published and report no caveats for antidepressants. Journal of clinical epidemiology.
  14. Boot, W. R., Simons, D. J., Stothart, C., & Stutts, C. (2013). The pervasive problem with placebos in psychology why active control groups are not sufficient to rule out placebo effects. Perspectives on Psychological Science, 8(4), 445-454.
  15. Collier, D., & Mahoney, J. (1996). Insights and pitfalls: Selection bias in qualitative research. World Politics, 49(01), 56-91.
  16. Golafshani, N. (2003). Understanding reliability and validity in qualitative research. The qualitative report, 8(4), 597-606.
  17. Sandelowski, M. (1986). The problem of rigor in qualitative research. Advances in nursing science, 8(3), 27-37.

Note: I published an earlier version of this post here, which is the blog of my research group.

18 thoughts on “Why you should be a skeptical scientist

  1. I love being a skeptical scientist, and i think this is a very important characteristic for true scientists! Great post.

    Question: i read your post about “Registered Reports” published in Education (http://www.timvanderzee.com/list-of-pre-registered-education-research/) and wanted to see what a Registered Report and pre-registration looks like. One of your own articles is listed there. I have read your paper here: http://econtent.hogrefe.com/doi/full/10.1027/1864-1105/a000208 but i can’t find a link to the pre-registration information in the paper. Am i missing something?

    • Registered Reports work different than regular pre-registration as you work together with the journal. In the case of RR, you typically submit the Introduction + Method section to a journal, receive peer review, revise the Method, and then run the study. As such you don’t (need to) do a pre-registration at websites like the OSF. You still can do a pre-registration, but this process already ensures a much more thorough (and actually checked!) pre-registration so it seems somewhat redundant.

      In short: the Introduction + Method section of a RR is the pre-registration, because it was literally written before the study was run.

  2. Thanks for the reply!

    “In the case of RR, you typically submit the Introduction + Method section to a journal, receive peer review, revise the Method, and then run the study”

    *Starts rant*
    Hold up, wait, what!?!? So after decades of psychology reviewers, journals, and editors doing all kinds of pseudo-scientific stuff like blocking critical articles of certain theories, refusing to publish null-results, and telling authors to leave out conditions and/or measurements, you are now telling me that these same people will now be in charge of “pre-registrations” without anyone being able to check this ?!!?! Are you F#CKING kidding me?!!? Are you guys actually going along with this?! Is this in line with “transparency” and “open science” and “improvements” and being a “skeptical scientist”?!
    *Ends rant*

    More concretely, concerning your Registered Report and pre-registration: Did you submit an analysis-plan with that introduction + method? And if so:

    1) How is the reader able to check whether you adhered to that plan? How do we even know if the analysis-plan is really a useful plan, and not some vaguely described BS?

    2) Why wouldn’t the journal that uses Registered Reports mandate that you pre-register (at websites like the OSF) this same plan (takes about 5 minutes), and that you add a link to it in the paper, so people can actually check what the plan looked like and whether you adhered to the plan?

    3) How can you even value transparency and/or pre-registration without providing the reader with links to this information? Did the journal forbid you to do so?

    • Yes, it’s the full Introduction + Method section, including materials, analysis code, etc.

      1) Because they read the Method section (which describes, prior to when the study was run, what the plan was) and the Result section (which describes, after the study was done, what the results are)

      2) You could, and I agree that there is a (minor) added value in doing so. But like I said, you can actually check what the plan looked like because the Method section *is* the plan.

      3) Again, the Method section *is* the pre-registered plan. It’s not publicly time-stamped, so there is an added value in public time-stamped pre-registration.

      All in all, with a RR you do *much more* than with a mere pre-registration and a reader can actually be more confident that it wasn’t a big fishing expedition. I hope this clarifies your questions.

  3. “1) Because they read the Method section (which describes, prior to when the study was run, what the plan was) and the Result section (which describes, after the study was done, what the results are)

    2) You could, and I agree that there is a (minor) added value in doing so. But like I said, you can actually check what the plan looked like because the Method section *is* the plan.

    3) Again, the Method section *is* the pre-registered plan. It’s not publicly time-stamped, so there is an added value in public time-stamped pre-registration. ”

    Would you say the same about research of the past? If not, then i don’t see why because just like research of the past with Registered Reports performed like your article, the reader simply has to “trust” editors/journals/authors that no conditions and/or measures were left out of the method. And the reader simply has to “trust” editors/journals/authors that there were no further analyses performed than those that are reported in the results section. The fact that you don’t seem to think this is a problem worries me.

    “All in all, with a RR you do *much more* than with a mere pre-registration and a reader can actually be more confident that it wasn’t a big fishing expedition”

    As a skeptical scientist, i totally disagree. All else being equal, I have more confidence in a time-stamped pre-registration with link in the paper that i can actually check compared to the pre-registration black-box of Registered Reports for reasons i expressed above. The fact that you don’t seem to think this worries me.

    I find it remarkable that psychology and Registered Reports are able to mess up something so simple as pre-registration and make it completely useless.

    I thought i could awaken the skeptical scientist in you, but i now have given up.

    Thanks for the replies.

    • You also “simply have to trust” that someone didn’t have 10+ private pre-registrations and only made public the one that was most convenient after having checked the data. If you think that pre-registration and RR are meant to eliminate fraud I have some bad news for you. That’s not at all the point, although it makes it somewhat more difficult. With pre-registration you have to trust that a the author did not blatantly commit fraud. With RR you have to trust that the author, the reviewers, and the editor did not collude together to commit fraud. The latter is by definition less likely. But again, this isn’t about preventing fraud.

      I will be be posting a pre-registration of the current RR I am involved it. It’ll simply be the Introduction & Method section, and all the other materials that are already part of the peer-reviewed stage 1 RR. The design and evidential value of the study has already substantially increased (although it hasn’t even started yet) exactly because it was peer-reviewed before data collection. Feel free to remain skeptical of that.

      Registered Reports do not just somewhat limit the amount of researcher’s degrees of freedom (which is the only thing that pre-reg does), but limits it further, more strictly, and with more oversight. It also almost entirely eliminates publication biases based on specific results, prevents a wide range of low-quality methods (ranging from underpowered designs to flawed comparisons to inappropriate stimuli), limits tunnel vision as more people are involved in decided on the method, reduces the risk of editors making bad decision for ulterior motives, etc.

      If you want to cast aside one of the biggest improvements to peer-review since its inception based only on a lack of publicly available timestamps (there still are timestamps), then be my guest, but I remain skeptical of the legitimacy of your skepticism.

  4. This is a super concise overview of what we have discovered to be wrong with science. I am going to be teaching grad methods/stats this fall and I will start by simply reading this to class! Thanks!

  5. 1) “You also “simply have to trust” that someone didn’t have 10+ private pre-registrations and only made public the one that was most convenient after having checked the data. If you think that pre-registration and RR are meant to eliminate fraud I have some bad news for you”

    If i understood things correctly the “OSF” will make them known somehow after a few years time, and the site “as predicted” mentions that “The pre-registration of alternative plans for the same study can be handled through other means, including system design that prevents abuse, and algorithms that detect it. AsPredicted.org includes both types of protection”.

    I reason, pre-registration is, next to eliminating flexibility in analysis as you mentioned, about accountability and transparency so we keep the “trust”-issue to a minimum. Both accountability and transparency are sub-optimal at best, and non-existent at worst, in your model of Registered Reports.

    2) “With RR you have to trust that the author, the reviewers, and the editor did not collude together to commit fraud. The latter is by definition less likely. But again, this isn’t about preventing fraud.”

    Depending on what you view as “fraud” (e.g. QRP’s ?), this is partly about fraud. Perhaps it’s not that unlikely that authors, reviewers, and editors collude together to commit fraud. E.g. see http://arinabones.com/
    “Finally, a skeptic might counter that the JPSP authors could have conducted the studies, found results,
    dismissed inconsistent data, and then written the paper as if those were the results that they had anticipated all along. However, orchestrating such a large-scale hoax would require the coordination and involvement of thousands of researchers, reviewers, and editors. Researchers would have to selectively
    report those that “worked.” Reviewers and editors would have to selectively accept positive, confirmatory results and reject any norm violating researchers that submitted negative results. The possibility that
    an entire field could be perpetrating such a scam is so counter-intuitive that only a social psychologist could
    predict it if it were actually true.”

    3) “If you want to cast aside one of the biggest improvements to peer-review since its inception based only on a lack of publicly available timestamps (there still are timestamps), then be my guest, but I remain skeptical of the legitimacy of your skepticism.”

    If by improvements you mean reducing flexibility in analyses and publishing the results no matter the outcome, i reason they can be big improvements but not as being a complete black box like they are now in your model of Registered Reports. Perhaps they can even be seen as “bad”, given the simple fact that readers now possibly assume that “confirmatory analyses” reported are indeed all the “confirmatory analyses” performed without any possibility to check this for themselves. In a way, this is similar to trusting that things like peer-review and impact factors will result in more “credible” science, and we all know what that has lead to.

    Also see: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0059910

    “This survey revealed that only one-third of the peer reviewers surveyed examined registered trial information and reported any discrepancies to journal editors.”

    4) We can just keep it very simple: in this new world of psychology of improvements and transparency, would it be an improvement if Registered Reports would from now on require to provide a link to the pre-registration information in the actual paper to the reader?

  6. Sorry to bother you again, but since you are a fan of “Registered Reports” and have published one, i thought you could perhaps answer my questions, because i still don’t understand.

    I have asked the following questions on Chris Chamber’s blog, but i guess he’s to busy to answer them, so i thought i try here: http://neurochambers.blogspot.nl/2012/10/changing-culture-of-scientific.html

    “I don’t understand something and have a few questions: why is this called “Registered Reports”?

    Is there anything that is “registered” and if so, how is that done, what does this “registration” imply, where is it “registered”, and is this accessible to the reader?

    The only form of “registration” i know of is pre-registration which involves a time-stamped, frozen, description of the research and analysis-plan that can be included in the paper so readers can check this information. But it seems to me that this is not the kind of “registration” that is involved in Registered Reports.”

    I am asking these question because it seems to me that “Registered Reports” might be a misnomer, because there might be nothing “registered” at all. Am i not understanding things correctly?

    • No problem, I like your critical questions and you have brought up some good points. I’ll try to address them all. It seems that you only consider public pre-registration to be a form of pre-registration, and I get that. With the RR model you are not forced to have a public pre-registration, although this is common to do. I haven’t done it for my first RR, will be do it for the one I am working on now as soon as the stage 1 peer-review has finished and all authors, reviewers and the editor has agreed on the design of the study. I don’t think anyone will disagree with you that creating a public, time-stamped, and frozen pre-registration is a bad thing to do, as it indeed adds to the transparency of the process.

      Having said that, I want to clarify that the only added benefit of doing as such is the public time-stamp. That is indeed good and more transparent, but all the other benefits are already covered by the RR model itself. Lets take my RR as an example again (http://econtent.hogrefe.com/doi/full/10.1027/1864-1105/a000208). If I would have put it on OSF to get the public time-stamp, I would have just copy-pasted the entire Introduction and Method section. Why? Because I wrote it before the data collection. Once it was approved by the reviewers all I did was execute the study according to the protocol, write down the results, and I added some exploratory analyses. The pre-specified analyses are reported under ‘Confirmatory Results’ and the once that I hadn’t pre-registered are listed under the ‘Exploratory Analysis’ header. This is how it works for all Registered Reports.

      In your previous comment you cited a paper stating “This survey revealed that only one-third of the peer reviewers surveyed examined registered trial information and reported any discrepancies to journal editors.”. This is another advantage of the RR model: the reviewers are forced to check the pre-registration, twice. The first time is during the stage 1 reviews where they have to review the pre-registered methods. The second time is during the stage 2 reviews where they review the full paper.

      If I understand your criticisms correctly, they are all solved by either making the review process open (Open Peer Review) as you will get the time-stamped pre-data collection version of the manuscript, and/or when the authors put the Introduction + Method on OSF with a time-stamp. Do you agree that this would address all your concerns? If not, let me know what other concerns you might have.

  7. Thank you so much for your time and answers. I really appreciate it! You’re a shining example of what (i think that) true scientists behave like (not that i am the one who decides this, but i hope you get the gist of what i am trying to express).

    1) “With the RR model you are not forced to have a public pre-registration, although this is common to do”

    That this is not required is very strange to me. Perhaps it can be seen as even stranger when taking the name “Registered Reports” into account: this at least implies “registration” to me and my common sense.

    2) “Having said that, I want to clarify that the only added benefit of doing as such is the public time-stamp.(…) The pre-specified analyses are reported under ‘Confirmatory Results’ and the once that I hadn’t pre-registered are listed under the ‘Exploratory Analysis’ header. This is how it works for all Registered Reports.”

    I disagree that the only benefit is the public time-stamp. The whole point of pre-registration (and making this available to the reader) is that the reader can check this information and see if things in pre-registration align with the final paper. That is, in my reasoning, one of the main functions of pre-registration. “Registered Reports” have completely by-passed this crucial function of pre-registration, which is incomprehensible to me.

    3) “In your previous comment you cited a paper stating “This survey revealed that only one-third of the peer reviewers surveyed examined registered trial information and reported any discrepancies to journal editors.”.(…). The second time is during the stage 2 reviews where they review the full paper.

    The fact that reviewers have to check this information twice instead of once, doesn’t solve the possible issue i was trying to make clear that trusting reviewers to check this information is 1) sub-optimal at a minimum, and 2) that there already is evidence that trusting reviewers does not work. I repeat what i said above: The whole point of pre-registration (and making this available to the reader) is that the reader can check this information and see if things in pre-registration align with the final paper (and doesn’t have to rely/trust reviewers).

    4) “If I understand your criticisms correctly, they are all solved by either making the review process open (Open Peer Review) as you will get the time-stamped pre-data collection version of the manuscript, and/or when the authors put the Introduction + Method on OSF with a time-stamp. Do you agree that this would address all your concerns? ”

    Yep, i think both would work fine in light of transparency and the other functions that pre-registration has.

    I have tried to bring this all up at several places, at several times, but i feel silenced and ignored (e.g. see link to Chamber’s blog). I really appreciate you taking the time to address my questions and concerns. I’ll leave it in your and others hands now, as i have concluded that if you’re not part of the “in-crowd” in psychological science (i.c. someone like me) you simply get ignored and silenced. So, this is all i could do to try and improve matters. Thank you very much again, you have no idea how much i appreciate it. All the best!

    • Agreed.

      A note about the name ‘Registered Reports’. Registration does not necessarily entail public registration, although I understand that is what you think about given the increasing popularity of public pre-registration. For example, most studies are typically also reviewed and registered at a local Ethics Committee of an institution. These reports are rarely public, but we can still say that ethics approval was given. So while it would be more transparent to make those ethics approval publicly available, that does not take away that it was reviewed and approved.

      But yes, making it all even more transparent is indeed the way to go, also for Registered Reports. Thank you for your comments.

  8. Final (after-) thoughts:

    1) “4) “If I understand your criticisms correctly, they are all solved by either making the review process open (Open Peer Review) as you will get the time-stamped pre-data collection version of the manuscript, and/or when the authors put the Introduction + Method on OSF with a time-stamp. Do you agree that this would address all your concerns? ”

    Yep, i think both would work fine in light of transparency and the other functions that pre-registration has. ”

    I think it would be best if a simple link would be included in the paper. That way, when the paper gets shared in some way of form (e.g. pdf), the crucial pre-registration information is included in the paper. If i understand things correctly, open peer-reviews are typically put alongside the paper online somewhere, but when the papers gets downloaded and shared would not include the open-review (and consequently also not the pre-registration) information. A simple link in the paper would be best i reason.

    2) “But yes, making it all even more transparent is indeed the way to go, also for Registered Reports”

    I would say not “even more transparent”, but “transparent” period. If i understood things correctly, “Registered Reports” as we are talking about them here are the opposite of transparent. There is no transparency at all, which i find strange given that this initiative has been thought of, and is being promoted by, people and institutions who/which are supposed to be all in favor of transparency.

    Thank you again!!

  9. Final (hopefully) question:

    When submitting your stage 1 peer-review Registered Report proposal did you have to answer questions like “have you performed the research yet” and/or “have you analyzed the data yet”?

    • Yes. That is a key requirement. Also, because the method is peer-reviewed before the study takes place, the final agreed-upon method will be different than the initial submission (which impacts the kind of data you need to collect and how).

Leave a Comment