Academic dreams

UPDATE: To see how science should absolutely not been done, see our investigation of 4 papers which total over 150 errors.

I am but a phd student, only just starting to crawl my way into academia. Yet even the little ones are allowed to dream big, and this is my dream of how science will be done in the (hopefully near) future.

Everything will be open…

… no matter the publication method. All scientific articles will be freely accessible by anyone, because no matter how many subscriptions your university can or cannot afford, scientists should be able to read everything. Do you know how much money is spend on buying back the right to read the research we wrote? To you give an impression, a small country like the Netherlands (only 13 universities) pays over 42.000.000 euro per year. As science is not just a hobby of scientists: you should be able to access the scientific literature, even if you are not working for a research institute.

With this free access to scientific articles comes free and instant access to the data, analyses, and any other material which underlies the paper. The scientific method is nothing if it isn’t transparent and retraceable. How we have come so far while stubbornly keeping relevant information hidden from each other is beyond me. Furthermore, reviewers and editors will become accountable by making the reviews transparent. This will discourage low quality reviews and abuses of power, while encouraging strong reviews. In addition, good reviewers can be properly accredited for their efforts.

Good science flourishes with radical transparency, while bad science withers.

There will be fewer, but larger studies…

… because in science, size does matter. Small, under-powered studies are the hallmark of uninformative science. Currently, everyone seems to want to study their own novel idea, running many small studies. Absurd theories can be kept afloat through a publication system which picks up the few studies which appear informative, but are so by chance alone. Good science is slow science, with large studies which give us so much more informative than had the same amount of resources been split across several smaller studies. Given the limited amount of resources, larger studies means fewer studies. This has the benefit of reducing the workload for reviewers and editors, and lowering both the absolute and relative amount of bad papers. Investing more resources into fewer studies also makes it more feasible to, for example, use multi-level methods and various control groups when necessary.

Every study will be pre-registered…

… or at least to some extent. This will be done at journals, and the study methodology will be reviewed by experts before any data collection takes place. This will prevent countless weak study to ever take place, thereby increasing the overall quality of the published literature. With this single measure, publication bias will almost completely disappear, as publication can no longer be conditional on the outcome.

If there is one way to ruin the scientific literature it is to selectively publish positive findings while ignoring the rest. This is just absurd.

Pre-registration of studies can be done in phases, if necessary, as not every type of study can be fully scripted a priori. These phases, and the decisions made will be published alongside the article, such that the reader can get a good sense of the circumstances in which the data was gathered. Pilot studies can still be done of course, for purposes like testing practical and technical feasibility (but not for informing a power analysis).


There are so many other ways in which science can improve. Making grants smaller and more numerous. Having fewer phd students and more permanent higher level positions. Focusing on replications.

What are your dreams for how science should be done?

4 thoughts on “Academic dreams

  1. Hey there just wanted to give you a quick heads
    up. The words in your content seem to be running off the
    screen in Ie. I’m not sure if this is a format issue or something to do with
    web browser compatibility but I thought I’d post to let you know.
    The design look great though! Hope you get the problem resolved soon. Thanks

  2. Yes, love your dream!

    I have had a dream with several things being in common with your dream. I hope it’s okay for me to share my dream. I hope it makes sense, but i am not smart enough to decide whether it is. I’ll let the reader decide.

    Thank you for all your efforts in trying to help improve psychological science!

    1) Small groups of let’s say 5 researchers all working on the same theory/topic/construct perform a pilot study/exploratory study and at one point make it clear for themselves and the other members of the group to have their work rigorously tested.

    2) These 5 studies will all then all be pre-registrated and prospectively replicated in a round robin fashion.

    3) You would hereby end up with 5 (what perhaps often can be seen as “conceptual” replications depending on how far you want to go to consider something a “conceptual” replication) studies, that will all have been “directly” replicated 4 times (+ 1 version via the original researcher, which makes a total of 5).

    4) All results will be published no matter the outcome in a single paper: for instance “Ego-depletion: Round 1”. This paper then includes 5 different “conceptual” studies (probably varying in degree of how “conceptual” they are, e.g. see LeBel et al.‘s “falsifiability is not optional” paper), which will all have been “directly’ replicated.

    5) All members of the team of 5 researchers would then come up with their own follow-up study, possibly (partly) related to the results of the “first round”. The process repeats itself as long as deemed fruitful.

    Additional thoughts related to this format which might be interesting regarding recent discussions and events in psychological science:

    1) Possibly think how this format could influence the discussions about “creativity”, “science being messy” and the acceptance of “null-results”.

    Researchers using this format could each come up with their own ideas for each “round” (creativity), there would be a clear demarcation between pilot-studies/exploratory studies and testing it in a confirmatory fashion (“science is messy”), and this could also contribute to publishing and “doing something” with possible null-results concerning inferences and conclusions (acceptance of “null-results”).

    2) Possibly think about how this format could influence the discussion about how there may be too much information (i.c. Simonsohn’s “let’s publish fewer papers”).

    Let’s say it’s reasonable that researchers can try and run 5 studies a year (2 years?) given time and resources (50-100 pp per study per individual researcher). That would mean that a group of researchers using this format could publish a single paper every 1 or 2 years (“let’s publish fewer papers”), but this paper would be highly informational given that it would be relatively highly-powered (5 x 50-100 pp = 250-500 pp per study), and would contain both “conceptual” and “direct” replications.

    3) Possibly think about how this format could influence the discussion about “expertise” and “reverse p-hacking/deliberately wanting to find a “null-result” concerning replications.

    Perhaps every member of these small groups would be inclined to a) “put forward” their “best” experiment they want to rigorously test using this format, and b) execute the replication part of the format (i.c. the replications of the other members’ study) with great attention and effort because they would be incentivized to do so. This is because “optimally” gathered information coming from this format (e.g. both significant and non-significant findings) would be directly helpful to them for coming up with study-proposals for the next round (e.g. see LeBel et al.’s “falsifiability is not optional” paper).

    4) Possibly think about how this format could influence the discussion about “a single study almost never provides definitive evidence for or against an effect”, and problems if interpreting “single p-values”. Also see Fisher, 1926, p. 83: “A scientific fact should be regarded as experimentally established only if a properly designed experiment rarely fails to give this level of significance.”

    5) Possibly think about how this format could influence the discussion about the problematic grant-culture in academia. Small groups of collaborating researchers could write grant proposals together, and funding agencies would give their money to multiple researchers who each contribute their own ideas. Both things contribute to psychological science becoming less competetive and more collaborative.

    6) The overall process of this format would entail a clear distinction of post-hoc theorizing and theory testing (c.f. Wagenmakers, Wetzels, Borsboom, van der Maas, & Kievit, 2012), “rounds” of theory building, testing, and reformulation (cf. Wallander, 1992) and could be viewed as a systematic manner of data collection (cf. Chow, 2002)

    7) Finally, it might also be interesting to note that this format could lead to interesting meta-scientific information as well. For instance, perhaps the findings of a later “round” turn out to be more replicable due to enhanced accurate knowledge about a specific theory or phenomenon. Or perhaps it will show that the devastating typical process of research into psychological phenomena and theories described by Meehl (1978) will be cut-off sooner, or will follow a different path.

    • Thanks for sharing! I like your dream. When it comes to the considerations on the publication process (independent of the results, no p-hacking, etc) this aligns very well with Registered Reports (see cos.io/rr/, and http://www.timvanderzee.com/registered-reports/) which I of course fully agree with.

      I am also a big fan of doing more extensive testing of a finding before publishing. That will result in fewer papers, and each paper having more information/evidence. Of course, you could also extensively test a paper throughout multiple publications, which has the advantage of a more rapid research cycle and earlier knowledge dissemination, but the same could be done through pre-prints that you update along the way.

  3. “this aligns very well with Registered Reports ”

    Yes, my thought exactly!

    I already posted a similar post i did above on Chris Chamber’s blog a few days ago tying it to his criticism concerning “conceptual replications”, and “Registered Reports”. It hasn’t (yet) been posted though:

    http://neurochambers.blogspot.nl/2012/03/you-cant-replicate-concept.html

    Thank you for letting me share the dream, thank you for all your efforts in (improving) psychological science, and all the best!

Leave a Comment