Implementing FAIR Workflows: A Proof of Concept Study in the Field of Consciousness is a 3-year project funded by the Templeton World Charity Foundation. In this project, DataCite works with a number of partners on providing an exemplar workflow that can be used by researchers to implement FAIR practices throughout their research lifecycle. In this monthly blog series, the different project participants will share perspectives on FAIR practices and recommendations.
In this first blog post, we hear from Zefan Zheng, who is doing his Ph.D. at the Max Planck Institute for Empirical Aesthetics and is responsible for the neuroscience experiments in the project.
A true story
Ph.D. student J worked on a project that set out to find evidence of whether humans could understand high-level information unconsciously. At the time, the replication crisis of psychological research was widely known. Many replication studies had been published to challenge the reproducibility of seminal papers in some well-established domains of psychological science. Literature on the capacity of unconsciousness is one of them. Early works claimed to have found evidence that humans can understand sentences (Sklar et al., 2012), facial emotions (e.g., Dimberg et al., 2000), and complicated images (Mudrik et al., 2011) without consciously being aware of them, all of which were either found hardly replicable or subject to confounds in recent work (Rabagliati et al., 2018; Mudrik & Deouell, 2022; Schlossmacher et al., 2017; Moors et al., 2016).
However, null results were not favored in the publication system (Rosenthal, 1979). J’s supervisor was strongly convinced that the targeted unconscious effect is present and their data from five years ago proved that. It became the research team’s task to replicate the effect and, ideally, further examine the mechanism behind the effect. Unfortunately, their first two replication attempts were disappointing, as they got null results. Seeing the null results, the supervisor questioned the scientific competence of J and asked her to replicate it again. So she did another five times until a compatible result emerged. J was very excited, but after some scrutiny, it became obvious that it was a consequence of her excluding three “outliers” in the data that went against the expected direction. After this, J used the data to graduate with a Master’s degree after working four years in this lab but left academia.
The replication crisis in psychology and beyond
Unfortunately, this story is not uncommon, and I believe it is still happening in many psychology labs and beyond today. Sparse academic positions lead to high competition. Academic competence is traditionally calculated based on the number of publications, citations, and impact factors. As illustrated in the story above, bias to publish only significant results may ultimately lead to scientific misconduct (Rosenthal, 1979). Scientific misconduct jeopardizes science in general.
In the psychology domain, many highly influential effects that have been investigated for decades and cited thousands of times (Bargh et al., 1996; Baumeister et al., 1998; Bem, 2011; Strack et al., 1988) were found unreplicable using much larger sample sizes than the original study (Doyen et al., 2012; Vohs et al., 2021; Ritchie et al., 2012; Wagenmakers et al., 2016). Normally, such research was conducted using the traditional research workflow (Figure 1). Experiments with null results were discarded. And only those with positive results were published in peer-reviewed journals, typically without depositing the data and code that are necessary for reproducing the results. Unfortunately, the replication of such research comes on average 20 years later than the first publication.
The silver lining
Academia is a very competitive system and currently, the only thing that credits the scientific contribution of a researcher is publication in high-impact journals. Therefore, scientists are forced to prioritize novel, interesting, but maybe less robust findings over robust, solid, but maybe less eye-catching ones. An important step forward is to also give credit to those publishing research outputs like pre-registrations, data, code, and protocols that fundamentally foster reproducible research. The premise of this would be making all these research outputs findable, citable, creditable, and reusable. This is also exactly what we are working on in the FAIR Workflows project. This workflow suggests researchers pre-register and deposit the data and code of all experiments conducted (if possible), including those with null results that will not be described in the peer-reviewed publication (Figure 2).
In general, the FAIR Workflows project aims to build an exemplar FAIR and Open research workflow, by bringing together stakeholders to formulate, support, and put into practice steps and procedures that lead to successfully FAIRified research outputs. The good news is that most of the infrastructure and the necessary services and platforms are already available, and many of the practices have been gradually picked up by the researcher community, even though dispersed and often lacking planning or consistency. We want to show through this project, for the first time, what a FAIR consciousness research project will look like in reality, when it is planned to be FAIR from inception, and provide a concrete answer to the pitfalls of traditional research workflows.
Reference
Abrams, R. L., & Greenwald, A. G. (2000). Parts outweigh the whole (word) in unconscious analysis of meaning. Psychological science, 11(2), 118-124. https://doi.org/10.1111/1467-9280.00226
Bargh, J. A., Chen, M., & Burrows, L. (1996). Automaticity of social behavior: Direct effects of trait construct and stereotype activation on action. Journal of personality and social psychology, 71(2), 230. https://doi.org/10.1037//0022-3514.71.2.230
Baumeister, R. F., Bratslavsky, E., Muraven, M., & Tice, D. M. (1998). Ego depletion: Is the active self a limited resource?. In Self-regulation and self-control (pp. 16-44). Routledge. https://doi.org/10.1037//0022-3514.74.5.1252
Bem, D. J. (2011). Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. Journal of personality and social psychology, 100(3), 407. https://doi.org/10.1037/a0021524
Dang, J., Barker, P., Baumert, A., Bentvelzen, M., Berkman, E., Buchholz, N., … & Zinkernagel, A. (2021). A multilab replication of the ego depletion effect. Social Psychological and Personality Science, 12(1), 14-24. https://doi.org/10.1177/1948550619887702
Dijksterhuis, A., & Van Knippenberg, A. (1998). The relation between perception and behavior, or how to win a game of trivial pursuit. Journal of personality and social psychology, 74(4), 865. https://doi.org/10.1037//0022-3514.74.4.865
Dimberg, U., Thunberg, M., & Elmehed, K. (2000). Unconscious facial reactions to emotional facial expressions. Psychological science, 11(1), 86-89. https://doi.org/10.1111/1467-9280.00221
Doyen, S., Klein, O., Pichon, C. L., & Cleeremans, A. (2012). Behavioral priming: it’s all in the mind, but whose mind?. PloS one, 7(1), e29081. https://doi.org/10.1371/journal.pone.0029081
Greenwald, A. G., Draine, S. C., & Abrams, R. L. (1996). Three cognitive markers of unconscious semantic activation. Science, 273(5282), 1699-1702. https://doi.org/10.1126/science.273.5282.1699
Mudrik, L., Breska, A., Lamy, D., & Deouell, L. Y. (2011). Integration without awareness: Expanding the limits of unconscious processing. Psychological science, 22(6), 764-770. https://doi.org/10.1177/0956797611408736
Mudrik, L., & Deouell, L. Y. (2022). Neuroscientific evidence for processing without awareness. Annual Review of Neuroscience, 45, 403-423. https://doi.org/10.1146/annurev-neuro-110920-033151
Moors, P., Boelens, D., Van Overwalle, J., & Wagemans, J. (2016). Scene integration without awareness: No conclusive evidence for processing scene congruency during continuous flash suppression. Psychological science, 27(7), 945-956. https://doi.org/10.1177/0956797616642525
Rabagliati, H., Robertson, A., & Carmel, D. (2018). The importance of awareness for understanding language. Journal of Experimental Psychology: General, 147(2), 190. https://doi.org/10.1037/xge0000348
Ritchie, S. J., Wiseman, R., & French, C. C. (2012). Failing the future: Three unsuccessful attempts to replicate Bem’s ‘Retroactive Facilitation of Recall Effect. PloS one, 7(3), e33423. https://doi.org/10.1371/journal.pone.0033423
Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638
Schlossmacher, I., Junghöfer, M., Straube, T., & Bruchmann, M. (2017). No differential effects to facial expressions under continuous flash suppression: An event-related potentials study. NeuroImage, 163, 276-285. https://doi.org/10.1016/j.neuroimage.2017.09.034
Sklar, A. Y., Levy, N., Goldstein, A., Mandel, R., Maril, A., & Hassin, R. R. (2012). Reading and doing arithmetic nonconsciously. Proceedings of the National Academy of Sciences, 109(48), 19614-19619. https://doi.org/10.1073/pnas.1211645109
Strack, F., Martin, L. L., & Stepper, S. (1988). Inhibiting and facilitating conditions of the human smile: a nonobtrusive test of the facial feedback hypothesis. Journal of personality and social psychology, 54(5), 768. https://psycnet.apa.org/doi/10.1037/0022-3514.54.5.768
Vohs, K. D., Schmeichel, B. J., Lohmann, S., Gronau, Q. F., Finley, A. J., Ainsworth, S. E., … & Albarracín, D. (2021). A multisite preregistered paradigmatic test of the ego-depletion effect. Psychological Science, 32(10), 1566-1581. https://doi.org/10.1177/0956797621989733
Wagenmakers, E. J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams Jr, R. B., … & Zwaan, R. A. (2016). Registered replication report: strack, martin, & stepper (1988). Perspectives on Psychological Science, 11(6), 917-928. https://doi.org/10.1177/1745691616674458
This project was made possible through the support of a grant from Templeton World Charity Foundation, Inc. The opinions expressed in this publication are those of the author(s) and do not necessarily reflect the views of Templeton World Charity Foundation, Inc.
