Background & Summary

The question of what processes can take place without conscious awareness has generated extensive research (for reviews, see1,2,3,4,5). Yet despite this ongoing effort, the field has yet to converge on an agreed-upon account regarding the extent and scope of unconscious processing6. While some researchers argue for high-level, extensive unconscious processing7,8 others claim that it is rather limited9,10.

These highly divergent conclusions reflect the difficulty of integrating the accumulated data into a cohesive account. The results abound with discrepancies, which might stem from the great variability in the methods used in the field. These different methodologies have been suggested to yield differential results11,12,13,14,15. Indeed, when examining a related field within consciousness science – namely, support for neuroscientific theories of consciousness16 – our group has recently showed that the methodological choices researchers make can predict the conclusions of their study, even when the results of the experiments are unknown to the predicting algorithm17. We accordingly hold that the field of unconscious processing will greatly benefit from being able to examine the methodological choices taken by researchers and explore their potential effect on the results of studies.

Here, we present the UnconTrust database (https://osf.io/2jgsx; see also http://uncontrustdb.tau.ac.il)18 for studies exploring semantic processing of, and attentional capture by, stimuli that are not consciously perceived. The database allows researchers to obtain a bird’s eye view on the field with respect to these domains, and better understand the way processing of such stimuli has been empirically studied. With 426 experiments currently included, the database is a valuable asset for researchers who are interested in unconscious processes. It provides a useful overview of the research questions and methods of the field of unconscious processing. As such, it constitutes a new meta-scientific tool that can be used in many ways to get a sense of the methods in the field.

The dataset includes information about the methods and research questions of experiments probing unconscious processing, pertaining to the features like the way conscious perception has been manipulated and measured, the type of unconscious processes probed, the sample size, and analysis choices (e.g., trial and participant exclusion). At the time of writing, the included experiments are based on classifications collected as part of two unpublished meta-analyses, focusing on semantic processing and attention allocation without awareness. As a result, it currently only includes data from healthy human adult participants and focuses on behavioral evidence for processing of stimuli that are not consciously perceived. It also misses other types of unconscious processing (e.g., emotional, perceptual). However, we plan to gradually add those types, as well as incorporate additional sample types (e.g., children, animals) and data (e.g., neuroscientific); This expansion will be conducted both by systematically search for studies of unconscious affective processing and of learning without awareness, and also by extracting the data from new papers exploring unconscious processing, which will be routinely uploaded to the interactive website where the data is stored (see Usage Notes for more information on the website). Note also that because the database focuses on studies investigating processing of stimuli that are not consciously perceived, it does not include experiments on implicit learning, where participants are aware of the stimuli, but not of the influence that these stimuli have on their behavior (for reviews, see19,20,21,22,23). Yet already at this point, UnconTrust is the largest, most comprehensive dataset with meta-data and classifications of studies of unconscious processes, which are Findable, Accessible, Interoperable and Reusable (FAIR24). Below we describe the current dataset and the methods for creating it.

Methods

The data for the database has originally been gathered for the purpose of conducting two meta-analyses: the first focused on attentional capture by visual stimuli which are not consciously perceived and the second on semantic processing of such stimuli. Below we detail the process of collecting the data for both and then the process of preparing the dataset for the purpose of the UnconTrust database.

Information sources and search strategy

Included papers were detected in two ways (see Fig. 1). First, database searches were conducted in PsycInfo (for both meta-analyses) and also in Medline (for the semantic processing meta-analysis), Embase, Scopus and PubMed (for the attentional meta-analysis). Search strategies and key terms were developed by research librarians with expertise in systematic searching. With their help, the team composed a sensitive search utilizing keywords and subject headings for the databases. The search terms included strings that (1) limited the search to consciousness studies, specifying relevant stimuli and methods; (2) limited the search only to English language published papers, including ones in press; (3) filtered out any papers unrelated to psychology or neuroscience; (4) removed studies in which the participants were non-human or non-adult (for the full list of terms, see Appendix 1). Publication dates were defined to include papers from inception to December 2020 (for the semantic processing meta-analysis) or June 2020 (for the attentional meta-analysis).

Fig. 1
figure 1

PRISMA flow diagram of the process of selecting papers for (a) the meta-analysis of unconscious attentional processing; and (b) and the semantic meta-analysis of unconscious semantic processing, based on Page, et al.42. (license: https://creativecommons.org/licenses/by/4.0/).

Full size image

Second, we went over previously published meta-analyses25,26,27 and review papers1,28,29,30,31,32,33,34,35,36,37, and included all relevant papers. We further scrutinized all studies citing those meta-analyses and reviews to detect potentially missing papers; if such papers were detected, they were added to the list of papers that underwent the screening process.

Data management and screening process

The screening of the papers was conducted in two stages. Title and abstract screening to identify potentially eligible records was performed using the systematic reviews web apps Covidence and Rayyan QCRI. At this stage, papers in doubt were included (see Fig. 1). This stage was conducted by FS for the attentional meta-analysis (with only the first 100 records screened also by another coder), and by MS for the semantic meta-analysis.

For the semantic meta-analysis (Fig. 1b), there was an additional step; In the first stage, where the initial pool of identified papers was screened for relevance by title and abstract, we casted a wider net, and identified studies potentially involving unconscious processing in general. Out of these papers, we then focused on papers including the following keywords which are related to semantic processing: semantic, semantic processing, meaning, semantical, semantically, semantic priming, translation, synonym, lexical and reading. The papers that were chosen underwent a subsequent title and abstract screening aimed to remove all papers not involving semantic processing.

For both meta-analyses, after the initial screening of irrelevant papers, a second screening took place, where the remaining papers were subjected to full-text screening. This screening was conducted independently by two coders. At this stage, it was decided whether each paper would be included in the meta-analysis. Any disagreements were discussed, and if needed, resolved in a discussion with LM.

This second screening was based on the following eligibility criteria: (a) Design: Original studies (not re-analyses of already published data) that were controlled and randomized; (b) Participants: Healthy human adults (18 years or older). Studies that featured other samples as well (such as children, animals, etc.) were only included if the data for the two groups had been provided separately; (c) Interventions: All experiments that measured either variations in allocation of attentional resources to visual stimuli in the absence of awareness, or the influence of the meaning of unconsciously presented stimuli on behavioral response. Absence of awareness had to be established by some measure of awareness. (d) Comparators: In cases where participants were reportedly unaware of the stimuli, we compared behavioral measures between the critical experimental conditions; (e) Outcomes: Experiments where the measure chosen for the processing is behavioral (e.g., reaction times, accuracy, eye movements); (f) Language and publication date: All relevant studies published in English since inception to December 2020 (for the semantic processing meta-analysis) or June 2020 (for the attentional meta-analysis); (g) Publication status: Any study, published in a peer-reviewed journal, that was found eligible according to the above-mentioned criteria.

Data collection process

Data extraction forms were completed in parallel by two coders. Discrepancies between the forms were resolved together with LM. From each paper, the following information was retrieved: The paradigm; the suppression method used to render stimuli unconscious; the measure for awareness; the power of objective measure for awareness; the type, size and duration of the stimuli; the processing domain; the measures outcome; and the task participants are asked to perform. In order to compute effect sizes and variance and to interpret the results, the following variables were also extracted: experimental design, key statistical results, and the reported statistics. When studies included more than one experiment, we coded these variables independently for each experiment. Appendix 2 describes further information about the variables in the extraction forms, their definitions and their theoretical justifications.

Dataset preparation

For the data to be accessible, interoperable and reusable, we combined the two datasets (collected separately for each meta-analysis) into one and unified the coding and structure conventions. As a first step, we defined a list of variables and possible levels that should be included for each experiment in the database (see Data records below). As not all variables were coded in both original datasets, we completed missing values.

Data Records

The data is stored in an independent repository (https://osf.io/2jgsx)18 and also in an interactive website (see Usage Notes).

For each paper, meta-data has been collected (i.e., the authors of the paper, the country in which the study took place, the journal in which the paper was published). In addition, the database contains detailed information we extracted regarding various aspects of experiments in the field: which paradigm was used in the experiment (for the entire list of values, see Table 1); information about the samples (Table 2); which tasks participants were asked to perform (Table 3); information about the stimuli used in the experiment (e.g., Tables 4, 5); which suppression method was used in order to present stimuli without them being perceived consciously (Table 6); what was the processing domain of the suppressed stimuli (Table 7); information regarding the awareness measures used to demonstrate unawareness of the suppressed stimuli (Table 8), and information regarding the experiment’s findings (i.e., which behavioral outcome was measured to assess the unconscious processing; Table 9). Note that the levels could be expanded to encompass new values based on added papers.

Table 1 Variables in the dataset describing the paradigm used in the experiment.
Full size table
Table 2 Variables in the dataset describing the samples in the experiment.
Full size table
Table 3 Variables in the dataset describing the main task participants perform in the experiment.
Full size table
Table 4 Variables in the dataset describing the suppressed stimuli used in the experiment.
Full size table
Table 5 Variables in the dataset describing the non-suppressed stimuli used in the experiment.
Full size table
Table 6 Variables in the dataset describing the suppression method used in the experiment.
Full size table
Table 7 Variables in the dataset describing the processing domain of the suppressed stimuli.
Full size table
Table 8 Variables in the dataset regarding the awareness measures used in the experiment.
Full size table
Table 9 Variables in the dataset describing the experiment’s findings.
Full size table

Technical Validation

The data acquired so far was validated by peer-reviewing and a double classification procedure, in line with the recommended practices for meta-analyses38. For each experiment, the classifications were extracted by two independent coders and were then compared (see methods section above). Any disagreements were resolved in consultation with LM. In addition, the structure of the database, the inclusion criteria and the scope were approved by a steering committee that includes, in addition to the authors of this paper, leading researchers in the field (see acknowledgements). The committee thus served as an additional monitoring and validation mechanism.

To further validate the data, we performed some sanity check analyses, whose results are reported below. These were aimed at (1) making sure that most experiments used the masking procedure39, followed by Continuous Flash Suppression40. This expectation was based on the literature2,13,41; (2) validating that the probed processes relate to either semantic processing (of different types) or to attention allocation, in line with the proclaimed goals of the meta-analyses on which this database is based; (3) demonstrating that we find compatible values when the data is analyzed in two different ways; and (4) that we have a good coverage of the journals in which papers in this topic are typically published. Figure 2 describes the results of all these sanity check analyses, which confirmed that the data indeed complies with our expectations, providing further validation for its integrity.

Fig. 2
figure 2

Sanity check analyses conducted to validate the database. (a) Masking is the most used procedure for rendering stimuli invisible, followed by Continuous Flash Suppression. (b) As expected, the experiments probe different types of semantic processing (lingual, visual and numerical), followed by attention allocation experiments. (c) A comparison of the types of experiments confirms the integrity of the data: both analyses reveal 319 experiments reporting positive effects, 113 reporting negative, and 37 experiments reporting mixed. (d) A distribution of the experiments to journals. This nicely shows that the database covers all the expected journals in the field (and some non-professional ones).

Full size image

Importantly, we also established a validation procedure for new data that will be added to the database by future users. Each new entry will be vetted by at least one member of the steering committee. In addition, if the entry has not been added by the authors of the publication, we will send the classification to the authors for validation and approval.

Usage Notes

The database18 is hosted on an interactive website (http://uncontrustdb.tau.ac.il) that allows researchers to generate their own queries, to explore methodological trends in unconscious processing research, to examine co-occurrence of certain methodological decisions, to inspect how the results might be modulated by the methodological choices of the researchers and to upload their own papers to the database. If successful, it could serve as a blueprint for the development of similar initiatives in other fields where researchers would benefit from tools that enable an easy exploration of methodological choices. Thus, with time, the files downloaded there will include more papers beyond those reported here.