Description

In this episode, we dive into the replication crisis—a major challenge shaping psychology and other scientific fields—with Dr. Bruno Sauce, Assistant Professor of Biological Psychology, and Dr. TuongVan Vu, Assistant Professor of Clinical Developmental Psychology, both from Vrije University Amsterdam.

We explore key topics such as publication bias, problematic research practices, and how the replication crisis affects public trust in science. Our guests share their experiences, insights, and actionable solutions for making research more transparent, reliable, and accessible. We also discuss the role of AI in tackling these challenges and offer advice for early-career researchers navigating today’s research landscape.

We hope you enjoy this engaging discussion on building a stronger, more trustworthy foundation for scientific research. Check the show notes below for additional resources and links to learn more about our guests and their work.

Below are the references to the studies/articles mentioned in the podcast: 

• Researcher degrees of freedom: The study on stats choices on racism in soccer. Many different results even without “p-hacking”!

https://journals.sagepub.com/doi/full/10.1177/2515245917747646

“Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 (Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship.”

• P-hacking: A special “degree of freedom” when we researchers have incentives to find a positive result (p < 0.05). Like trying different stats, collecting more data (or stopping the collection when p is favorable), trying many different predictors and conditions/treatments and only reporting the positive ones, rounding down p values, deciding to treat outliers after looking at the (unfavorable) p value, HARKing (Hypothesizing After the Results are Known).

https://journals.sagepub.com/doi/10.1177/0956797611430953

• Publication Bias: the outcome of an experiment or research study biases the decision to publish or otherwise distribute it.

https://www.cambridge.org/core/journals/psychological-medicine/article/cumulative-effect-of-reporting-and-citation-biases-on-the-apparent-efficacy-of-treatments-the-case-of-depression/71D73CADE32C0D3D996DABEA3FCDBF57

The “psychic” paper: https://psycnet.apa.org/record/2011-01894-001

Conference about meta-science: https://metascience.info/

Center for Open Science: https://osf.io/

Psychological Science Accelerator Network: https://psysciacc.org/

An exemplar Many Lab study: https://journals.sagepub.com/doi/10.1177/2515245918810225

Website of the Many Babies project: https://manybabies.org/

AI tool for literature review: https://www.litmaps.com/, https://www.researchrabbit.ai/

PNAS paper that use AI to quantify the replicability of thousands of papers across subfields of psychology: https://www.pnas.org/doi/10.1073/pnas.2208863120

A study analyzing 93 statistical papers using fMRI data from 2010–2021 found that only 15.1% were fully reproducible, emphasizing the need for better code and data sharing practices. https://www.tandfonline.com/doi/full/10.1080/00031305.2022.2131625

dr. TuongVan Vu - Vrije University Research Portal
dr. Bruno Sauce Silva - Vrije University Research Portal

Special thanks to Joost Canters and Vrije Universiteit Amsterdam for their support in producing this podcast episode.