Abstract
Web-based testing of cognitive abilities allows for large-scale assessments without geographical constraints. Yet, the extent to which it can reach populations beyond the typical demographic groups recruited for cognitive studies is unclear. This study focused on comparing the characteristics of individuals from a general population study (HUNT4) who chose to participate in a cognitive study (HUNT4-Hjernetrim) with those who did not. Additionally, we investigated participants' engagement and user experience. We obtained data on socio-demographics, health conditions (both physical and mental), self-reported cognitive or learning difficulties, and lifestyle factors of Hjernetrim participants and non-participants from the HUNT4 database. Hjernetrim involved 13 cognitive tests, administered through the online platform Memoro. We used logistic regressions to assess participation biases and linear regressions to assess participants’ engagement and user experience. Of 65,851 HUNT4 participants invited via regular mail to Hjernetrim, 5634 (9.4%, aged 13–97, 54% women) participated. The best represented in the sample were 50–79-year-olds, women, tertiary educated, living alone, from urban areas, not occupationally active, and reporting memory complaints. Individuals who were aged 80+, had motor or vision impairments, and teenagers with learning disabilities, were underrepresented. Participants were more likely to have mental health problems, have or survived cancer and less likely to have cardiovascular disease. Participants logged on mainly during weekdays, the preferred time of day varied by age. On average, participants used 42 min and completed 78% of the tasks. Using PCs provided the most complete data. In terms of user experiences, 65% were positive while 14% were negative or reported technical difficulties. Overall, the study demonstrated that web-based methodology allowed for a relatively well-represented sample that included groups typically difficult to reach. The presence of somatic and mental diseases had a variable influence on participation. Participants finished most tests and reported positive experiences overall.
Similar content being viewed by others
Introduction
Web-based testing of cognitive functions allows for geographically unconstrained and large-scale assessment of cognition without the need for participants to install specific software, use a particular device, or travel to a testing location. The method has several advantages over in-person or phone-based assessments1. For instance, the methodology is significantly more cost-effective than traditional pen-and-paper testing2 and allows participants to complete tests at their convenience, in a location of their choice. Moreover, participants often prefer this administration form over traditional testing, stating they feel more in control and are free of being scrutinized by a test administrator3, leading to a more comfortable test experience. The increasing familiarity with desktop computers and mobile devices across all age groups4 along with the expanding access to the internet around the world (internetworldstats.com/stats4.htm, datareportal.com/reports/digital-2022-global-overview-report) sets the stage for the successful integration of web-based cognitive testing. This holds particular promise for epidemiological studies.
Epidemiological studies have been experiencing declining participation rates in recent decades5,6,7. The implementation of a web-based methodology may increase participation by providing more flexibility, but possibly also introduce new types of bias8,9. The latter could stem from factors such as willingness and/or ability to engage in self-administered testing and varied computer familiarity, as well as specific health problems.
Since selection bias is a concern for the representativeness of results from epidemiological studies, understanding how web-based data collection affects participation is a relevant and timely undertaking. Yet, few studies address selection bias in the context of web-based data collection. This issue is not unique to web-based methodologies, as exemplified by a census analysis of research articles from the Cancer Epidemiology, Biomarkers & Prevention journal, which revealed that only 44% of published studies documented response rates, approximately 10% compared participants to non-participants, and among longitudinal studies, only 3 out of 17 reported dropout rates10. The studies that have investigated participation biases show numerous differences between participants and non-participants. A previous study performed on the Trøndelag Health Study (The HUNT Study) sample showed participants to have higher levels of education and better health than non-participants11. Furthermore, the mode of assessment, e.g. web-based testing, can introduce bias, as indicated by studies that have compared web-based surveys to paper-based and mixed-mode surveys and identified variations in response rates and disparities in performance9,12. A specific concern related to web-based cognitive testing pertains to the administration of studies in non-laboratory settings. Participants use their own diverse devices at a time of their choosing, can discontinue at any time, and are not shielded from distractions.
In the current study, we used data from HUNT4, the fourth wave of the Trøndelag Health study (The HUNT study), a large general population study of the northern part of Trøndelag county and three districts of Trondheim, the largest city in the southern part of Trøndelag, Norway5,13, combined with information from the Hjernetrim substudy of HUNT4. In Hjernetrim, HUNT4 participants were invited by mail to take a self-administered, web-based cognitive test battery administered via Memoro3,14 assessing several aspects of cognitive abilities, including memory (pattern separation, working memory, spatial and verbal memory), perceptual speed, attention, and executive functions.
The current study has two main aims. Firstly, to characterize those who choose to participate in web-based cognitive testing. The objective is to compare the characteristics of individuals who participated in the HUNT4-Hjernetrim cognitive substudy with those who did not, with regard to sociodemographics, self-reported health, cognition, and lifestyle. We hypothesize that there are significant differences in the demographic, health, and lifestyle profiles between participants and non-participants.
Secondly, we aim to describe the participants’ engagement with the study. The objective is to assess: (1) their computer familiarity and preferred device and browser; (2) the time of day and week participants took the test; (3) the time spent completing the tasks; (4) the number of tasks completed and reasons for discontinuing; (5) participation at field station; and (6) participants’ self-reported user experience and frequency at which they accessed their performance results. We hypothesize that there are age-related variation in all these aspects of participant engagement.
Together, the results can be utilized by researchers to enhance their planning, design, and recruitment strategies, thereby improving participation rates and quality of data in future studies that employ web-based technologies.
Materials and methods
Study population and procedure
The HUNT4 study
The HUNT4 study is the fourth wave of the Trøndelag Health Study, one of the world's largest ongoing longitudinal population-based health studies13. The HUNT study is considered representative of Norway geographically and economically with age, morbidity, and mortality distribution similar to the national averages. HUNT4 data collection was approved by the Norwegian Data Inspectorate and Regional Committee for Research Ethics (REK-2016/801). Informed consent was obtained from all subjects and/or their legal guardian(s) and the study was performed in accordance with relevant guidelines and regulations, including the Declaration of Helsinki.
The research sample consisted of three geographical and age-based cohorts: the HUNT4-N (NT4) cohort consisted of all residents aged 20 years and older in the northern part of Trøndelag County, the Young-HUNT4 (YH4) cohort consisted of residents of the same area between ages 13 and 19, and finally the ST1 cohort consisted of adults aged 70 years and older living in three districts of Trondheim, the largest city in the region.
The three cohorts filled out mostly overlapping cohort-specific questionnaires, where some questions were adapted to the age group (e.g., adults were asked about cognitive issues while the youth were asked about learning disabilities). From the HUNT4 study the following variables were acquired: age (calculated from the date of birth to the day of invitation to NT4, YH4, or ST1), biological sex, education level, living alone or with a partner, work status, place of residence (rural vs. urban), general health issues and specific diagnoses, including scores on the Hospital Anxiety and Depression Scale (HADS), self-reported cognitive or learning problems, lifestyle satisfaction, and current smoking and alcohol use (Table 1). The scoring of the HADS was categorized into two groups: (1) scores below were considered within the normal range; (2) scores of 11 or greater indicated a potential clinical range of symptoms of anxiety and depression (fhi.no/globalassets/dokumenterfiler/rapporter/2016/maleegenskaper-ved-den-norske-versjonen-pdf.pdf). A detailed description of all variables included in the study can be found on the HUNT Databank website (hunt-db.medisin.ntnu.no/hunt-db).
In addition, the HUNT Databank includes data on age and sex from the Norwegian National Registry, on all those residing in the catchment area where the HUNT4 study was performed, i.e., also those who did not participate in HUNT4, which we obtained.
The HUNT4 Hjernetrim study
The HUNT4 Hjernetrim Study (hereafter called Hjernetrim) was accepted as a HUNT4 substudy by the HUNT4 planning committee in 2015. The data collection was approved by the Norwegian data inspectorate study as part of HUNT4, the Regional Committee for Research Ethics (REK-155024 HUNT4 Hjernetrim study), and the HUNT administration. The participants gave their informed consent online before the testing started. The study was performed in accordance with relevant guidelines and regulations, including the Declaration of Helsinki.
As required by the HUNT Research Centre, only participants who had taken part in the main HUNT4 study could be invited to Hjernetrim. An invitation letter for the Hjernetrim study was included in the feedback letter and sent by regular mail to participants from the three cohorts (NT4, YH4, ST1), totalling 65,851 invited individuals (54% women), of whom 5634 participated (54% women). The letter contained the results of the clinical assessment and physical activity accelerometer results collected in HUNT4 and personal health recommendations. The letter also included a half-fold 10.5 cm by 14.5 cm leaflet providing a brief description of the Hjernetrim study, its website address, and the participant-specific login credentials required to log on to the website. Participants were given instructions to manually enter this web address and their login credentials into their web browsers to access the web-based cognitive test platform, Memoro, for participation. Invitation letters were sent out continuously by regular mail as the results from the HUNT4 data collection were processed. The possibility of participating in Hjernetrim was additionally advertised in local newspapers, by flyers at shopping malls, and at the HUNT4 field stations.
Participants were told to set aside about 40 min to complete the tests, be in a quiet place, and preferably use a personal computer with the Google Chrome browser, although other platforms and browsers were also supported.
The Hjernetrim participation was conducted using our proprietary, validated, self-administered, web-based cognitive test platform Memoro3,14 to perform a selection of tests covering several domains. A total of 13 tests were administered in a fixed order: Simple Reaction Time, Pattern Separation, Visual Memory (Immediate Recall), Verbal Memory (Learning and Immediate Recall), Symbol-Digit Coding (Main Test), Symbol-Digit Coding (Recall of a Symbol-Digit Key), Digit Span Forward, Visual Memory (Delayed), Visual Memory (Recognition), Digit Span Backwards, Verbal Memory (Delayed Recall), Verbal Memory (Recognition), Complex Reaction Time.
Before cognitive testing started, the participants consented online and filled in a short questionnaire including birth year, gender, level of education, handedness, and computer familiarity assessed on a 5-point Likert scale (“How comfortable are you with computers?”). We used gender, age in years, and level of education (three levels, see above) to calculate normalized scores for participant-specific feedback on performance on the battery (see below).
The instructions for tasks were both written and verbally presented. The main tasks were preceded by short training sessions which participants needed to perform correctly in order to proceed to the main tasks. If training results indicated that task instructions were not understood, participants were automatically transferred to the next test.
For participants not familiar with using computers or not owning one, Hjernetrim field stations were arranged and advertised together with Hjernetrim information. Those who chose to participate in the field stations could use the computer equipment and technical assistance of the researchers available on-site and could log in using either their original credentials or, to keep the subject anonymous, use temporary credentials which were later connected back to their HUNT ID.
Participants who completed at least one test could access a feedback page that was available after completing or aborting the last task. Participants needed to log in anew to access the feedback. The mode of feedback was developed in collaboration with the HUNT administration and their user group before data collection started. The feedback page displayed details regarding the cognitive skills evaluated in Memoro, along with practical examples of how these abilities are employed in daily life. Participants could see how well they performed on the different tasks in comparison to others of the same gender, age, and education group rated from the participant’s best to the worst test performance and for each test whether the performance was above, at, or below the participant’s age, gender and education group average.
Engagement, user experience, and feedback
We collected metadata on Hjernetrim participation for each log-in. The metadata included device type, operating system, web browser type and version, number of logins, the date and time of starting and finishing each of the tasks, and the status of each task (started, finished, aborted, failed).
For participants logging more than once and using multiple devices, we defined the participant’s main device and web browser engine as the ones that were used to complete the highest number of tasks. For participants who completed an equal number of tasks on more than one device and/or web browser, we used the one that was used first.
The time and day of participation were registered at the first login to Memoro. To analyze them, we defined one day as the period between 4 and 4 am of the next day to separate the “night owls” from the “early birds”. If a participant completed the battery in more than one session, only the first session was considered.
We calculated battery completion time as the sum of the time taken to complete the training and main tasks from start to finish, measured in minutes, excluding the breaks that participants took between the tasks. Tasks in which participants spent less than 5 times the interquartile range (IQR) or more than 10 times the IQR were considered outliers and excluded from the analysis.
We defined the number of finished tasks as the sum of main tasks that were not aborted by participants or as a result of failed training, and that provided valid scores. Training tasks were not included in the battery completion statistics.
Participants who did not start all tasks were categorized as having discontinued the battery. Successful completion of all 13 tasks was not necessary to avoid being classified as having discontinued; participants only needed to initiate each task.
After the testing session concluded, participants were asked to indicate whether they had encountered any disruptions or interruptions during one or more tasks. They could do this by marking the affected tasks off on a provided list. They were also asked to share their overall experience with the testing in the form of an open question “Please, tell us about your experience”. The free text responses were subsequently stratified into the following categories: general positive experience (e.g. “Fun”); general negative experience (e.g. “Stressful”); comments about battery difficulty or own performance (e.g. “Struggled with word task, was better with numbers”); any technical issues with the platform (e.g. “Aborted a task by mistake”, “Sound problems”, “Images did not appear”); and other comments (typically about a specific task, like its length, instructions, or about being interrupted, e.g. "Disturbed by dad ", “Difficult, especially with a screaming baby”).
Contact information for the researchers (e-mail address and phone number) was available in the invitation mail and on the Memoro website for participants who experienced technical issues, wanted to participate in a field station, or had other inquiries.
Statistical analysis
Participants’ sociodemographic and health characteristics, as well as metadata from the web testing, are presented as frequencies and percentages or means and standard deviations, as appropriate.
Participation bias
To examine factors affecting the likelihood of participation in Hjernetrim, we compared those participating in HUNT4 but not in Hjernetrim (n = 60,217) and those participating in both studies (n = 5634) as the dependent variable in logistic regression. For each variable, the group with the largest number was used as the reference group when reporting the odds ratios (OR). First, we examined age groups in decades, sex, and education (stratified into primary, secondary, and tertiary). In all subsequent logistic regression models, age group, sex, and education were included as variables of no interest. We investigated other sociodemographic variables next, namely living alone or with a partner, work status, and place of residence (urban or rural). Subsequently, we investigated health-related variables: general health and impairment(s), and the presence of specific somatic and mental diagnoses/problems. Then we investigated the impact of cognitive and learning issues. Next, the impacts of satisfaction with own lifestyle, smoking, and alcohol use were assessed.
Finally, as age and sex were available for participants from all invited to HUNT4 (n = 119,558) including those who did not participate and thus were not invited to the Hjernetrim study, we used it to assess the effects of age and sex on the likelihood of participation in Hjernetrim among the general population. We used a logistic regression model with participation as a dependent variable and age group and biological sex as variables of interest.
Engagement in Hjernetrim
To assess engagement among those participating in HUNT4 Hjernetrim, we used linear regression models with the time of day and weekday of starting the battery, time spent in the battery, and number of finished tasks as the dependent variable in separate models with age or age group, sex, education, and, where appropriate, device used and computer familiarity, as independent variables.
To assess factors leading to discontinuation and leaving feedback, we used logistic regressions with discontinuation and leaving feedback as the dependent variable in separate models. In both models, the independent variables were age group, sex, education level, device used, and computer familiarity. For the model analyzing leaving feedback, we additionally included the number of tasks completed by the participants as an independent variable.
To assess the level of computer familiarity across Hjernetrim participants, we used ordered logistic regression with computer familiarity as a dependent variable and age group, sex, and education as independent variables.
For dummy variables, the most numerous subgroup from HUNT4 for each variable was used as the reference group when reporting odds ratios (OR).
Results
Sample characteristics
Hjernetrim data were collected during the HUNT4 study between November 2017 and February 2021, with most of the data (69%) collected in 2018 (Fig. 1a). Overall, 5,634 participants (3,254 women, 2,380 men, 13 to 97 years old, Fig. 1b) gave consent to participate in Hjernetrim, which translates into 8.6% of those invited and 4.7% of the general population (Fig. 2). An overview of the sociodemographic, health, cognitive, and lifestyle-related variables in the HUNT4-only participants and those who also participated in Hjernetrim are presented in Table 2.
(a) Date of participation (year-month) in HUNT4 Hjernetrim (frequency = n). (b) Age and sex distribution of the Hjernetrim participants.
Flowchart showing the number of Hjernetrim participants with the number of people invited to the substudy and entire cohorts.
Sociodemographic variables associated with HUNT4 participants partaking in Hjernetrim
Demographics
Age, sex, and education affected the likelihood of participation in Hjernetrim (Fig. 3). Compared to 50-year-olds, the 60–69-year group was the most likely to participate in Hjernetrim (OR = 1.18, 95% CI [1.08, 1.29]), whereas participants ≥ 80-year were the least likely (OR = 0.33, 95% CI [0.26, 0.40]). Sex distribution differed across age, with more women (62.9%) participating in Hjernetrim below age 65, but more men (56.3%) among those aged 65 and older (Fig. 1b). Overall, men were less likely to participate (OR = 0.87, 95% CI [0.83, 0.93]). The probability of participating was positively associated with educational attainment, with those with only primary education least likely (OR = 0.55, 95% CI [0.49, 0.61]) and those with tertiary education most likely to participate (OR = 2.14, 95% CI [2.02, 2.28]).
The odds ratio of age group, sex, and education on participation in Hjernetrim relative to participation in HUNT4. Fifty-year-olds, women, and those with secondary education were used as references as these were the most numerous subgroups. For details on variables and their frequency, see Tables 1 and 2.
Living alone increased the participation likelihood (OR = 1.13, 95% CI [1.04, 1.22]) compared to living with a spouse or partner. Not being occupationally active also increased OR of participation (OR = 1.10, 95% CI [1.01, 1.20]). Living in a rural area decreased the likelihood of participation (OR = 0.73, 95% CI [0.68, 0.78]).
Somatic and mental health variables associated with partaking in Hjernetrim
The likelihood of participation in Hjernetrim was lower among those reporting a chronic disease or injury (OR = 0.90, 95% CI [0.85, 0.96]), having had a myocardial infarction (OR = 0.81, 95% CI [0.67, 0.97]), motor impairment (moderately impaired: OR = 0.77, 95% CI [0.65, 0.91]; greatly impaired: OR = 0.64, 95% CI [0.48, 0.84]) and a vision impairment (moderately impaired: OR = 0.84, 95% CI [0.71, 0.98]). On the other hand, those who had or had had cancer (OR = 1.13, 95% CI [1.01, 1.26]) and mental health problems (OR = 1.23, 95% CI [1.13, 1.32]) had a higher likelihood of participating. Diabetes, stroke, HADS score of 11 or higher, and hearing impairment had no significant association with participation. Compared to those who reported good health (the most numerous group) at the time of HUNT4, those with not very good health were less likely (OR = 0.90 CI [0.84, 0.98]), and those with very good health were more likely (OR = 1.17, 95% CI [1.09, 1.25]) to participate in Hjernetrim. An overview of the ORs for all the aforementioned variables is presented in Fig. 4.
The influence of self-reported somatic and mental health problems, as well as learning disabilities and self-reported memory problems in HUNT4 on the probability of participating in Hjernetrim. Each color represents a separate model. Fifty-year-olds, women, and those with secondary education were used as references as these were the most numerous subgroups. For details on variables and their frequency, see Tables 1 and 2.
Self-reported cognitive variables associated with partaking in Hjernetrim
The likelihood of participation in Hjernetrim among teenagers was lower in those with learning disabilities (OR = 0.65, 95% CI [0.51, 0.85]), especially math disabilities (OR = 0.55, 95% CI [0.38, 0.80]). In adults, participation was lower in those reporting no memory problems (OR = 0.93, 95% CI [0.87, 0.99]). An overview of OR for participation relative to self-reported cognition is presented in Fig. 4.
Lifestyle satisfaction, smoking, and alcohol use
Compared to those who were satisfied with their lifestyle (diet, exercise, smoking and drinking habits), those who were very satisfied were more likely to participate (OR = 1.17, 95% CI [1.06, 1.28]) (Fig. 5). Compared to those who never smoked, ex-occasional smokers were more likely (OR = 1.20, 95% CI [1.10, 1.32]), whereas current daily smokers were less likely (OR = 0.71, 95% CI [0.62, 0.81]) to participate. Those who abstain from alcohol were less likely to participate (OR = 0.87, 95% CI [0.79, 0.95]).
The influence of self-reported lifestyle factors in HUNT4 on the odds ratio of participating in Hjernetrim. Each color represents a separate model. Fifty-year-olds, women, and those with secondary education were used as references as these were the most numerous subgroups. For details on variables and their frequency, see Tables 1 and 2.
Demographics of Hjernetrim participants compared to the general population
This analysis compared the distribution of age and sex in those who participated in Hjernetrim and everybody else in the catchment area, i.e., both those participating in the HUNT4 study and not. These results were quite consistent with the above results, showing the 60–69-year group to be the most likely (OR = 1.20, 95% CI [1.11, 1.31]), and the ≥ 80-year-olds to be the least likely (OR = 0.20, 95% CI [0.16, 0.24]) to participate in Hjernetrim. Men were less likely to participate (OR = 0.71, 95% CI [0.67, 0.75]) than women. The distribution of age and sex among Hjernetrim participants and those in the catchment area who did not participate is presented in Supplementary Table S1. The full results of logistic regression are presented in Supplementary Table S2.
Hjernetrim engagement
Computer familiarity, hardware, and software
The majority of Hjernetrim participants rated themselves as familiar with computers (Table 2, Supplementary Table S3), with 90.6% of men and 89.2% of women reporting being rather comfortable or very comfortable using them.
Being younger, male, and having more education were associated with higher computer familiarity (Supplementary Table S4).
Personal computers running Windows were the most frequently used devices, accounting for 52.1–73.6% of devices across all age groups. The frequency of the other devices in descending order was: iPads (5.9–18.8% across age groups), Macbooks (5.1–20.7%), Android devices (3.3–9.4%), iPhones (0.7–12.0%), and other devices (0.0–1.7%). The distribution of devices used varied with age (Fig. 6a).
Distribution of devices (a) and browser engines (b) used to log on to the web-based platform, subdivided by age group.
Blink (used in Google Chrome, Opera, and recent versions of Microsoft Edge) was the most frequent browser engine across all the age groups (Fig. 6b). Blink dominated the most in the 13–19 age group (59.4%) but was less frequent in the 70–79-year group (34.1%). The frequency of the other browser engines in descending order was: WebKit (Safari, 21.9–29.9% across age groups), Trident (Internet Explorer, 3.4–17.4%), EdgeHTML (older versions of Microsoft Edge, 2.8–17.9%), and Gecko (Mozilla Firefox, 2.6–10.2%). The relative popularity of browser engines other than Blink and WebKit increased with age (Fig. 6b).
Day and time of participation
Thursdays were the most popular days to participate in most age groups, with 17.6% of users starting the battery on that day, followed by Tuesdays (16.8%), Mondays (16.1%), Wednesdays (15.1%), Fridays (13.2%), Sundays (12.2%), and Saturdays (9.1%).
The mean time of starting the battery was 14:10, with 90% of participants starting between 7:33 and 20:42. Preferred starting times varied widely across different age groups. The data showed that younger participants generally started in the afternoon, those in working age tended to start in the evenings, while older adults preferred starting in the mornings (Fig. 7).
The distribution of Hjernetrim participants’ logging into the battery during the day by age group. The Y axis illustrates the participant density of each age group.
Time spent
For participants who completed all the tasks, the mean duration was 47 min, with a median of 46 min and a range from 27.6 to 96.7 min (Fig. 8a). The time to complete the whole battery increased linearly with age (t = 18.35, p < 0.001), with a mean increase of time of about 1 min per 7 years of age. When including participants who discontinued, the average time spent was 42 min, with a range from 0.2 to 96.7 min and a median of 44 min (Fig. 8b).
Time spent in Hjernetrim in (a) those who completed all tasks and (b) regardless of battery completion. The Y axis illustrates the participant density of each age group.
Completion and discontinuation of test battery
The completion rate was high with 58% (n = 3248) completing all tasks and an additional 22.5% (n = 1270) completed 7 tasks or more. A total of 13.2% (n = 743) participants completed < 7 tasks, while 6.7% (n = 373) did not complete any tasks. The mean number of completed tasks was 10.3 out of 13 tasks. Several factors influenced task completion (R2 = 0.06, F(16, 5313) = 21.28, p < 0.001), including education, age, and device (Table 3). Participants with tertiary education were more likely to complete all tasks compared to those with secondary education. The number of completed tasks was lower in older adults. Sex and computer familiarity were not related to the number of tasks completed, but using an Android device, an iPhone or an iPad negatively influenced completion rates (Fig. 9).
The mean number of completed tasks in Hjernetrim by device and age group. Frequencies below 5 and observations of unidentified hardware were not included in the figure.
Likewise, those with tertiary education were less likely to discontinue the battery (OR = 0.72, 95% CI [0.59, 0.88]), while participants who used Android devices (OR = 1.63, 95% CI [1.10, 2.11]) and iPhones were more likely to discontinue (OR = 2.17, 95% CI [1.49, 3.17]). Sex and computer familiarity were not related to discontinuation.
The test associated with this greatest discontinuation was the learning trials of the Verbal Memory task (n = 249, corresponding to 41% of all such cases and 4% of all participants).
Participation at field stations
Overall, 23 (0.4%) participants performed the battery in field stations. Their mean age was 73 years (49.9–85.6 years), which is significantly higher than the rest of the sample (t = 6.06, p < 0.001). Those participants completed fewer tasks on average compared to the entire sample (8.6 vs 10.3), but the difference was not statistically significant (t = − 1.07, p = 0.284), likely due to the low number of field station participants and large variability of the number of tasks they completed. Participants at the field stations spent an average of 52.7 min (19.5–89.9 min) on Memoro tasks, over 10 min longer than those who solved them at home (t = 3.10, p = 0.002).
User experience and feedback for participants
A total of 27.6% reported being interrupted or disturbed on at least one task. A total of 58.1% answered the open question about their experience with the test battery. Participants were more likely to leave a comment if they finished all the tasks compared to those who finished most (OR = 0.65, 95% CI [0.56, 0.76]), less than half (OR = 0.006, 95% CI [0.003, 0.010]), or none of the tasks (OR < 0.001, 95% CI [< 0.001, 0.001]), and the probability increased with age between 20 and 80 years. Conversely, the group aged 13–19 was significantly more likely, while the 80 + age group was significantly less likely to provide comments on their experience compared to their neighboring age groups. Sex and education did not influence the likelihood of leaving comments. An overview of the comments stratified by topics is presented in Table 4.
Only 20.4% (n = 1147) logged back into the Memoro website to access their performance results and an additional 0.8% (n = 44) contacted us directly to receive their results.
Discussion
The study had two main objectives. The first objective was to compare the characteristics of individuals who participated in the HUNT4-Hjernetrim cognitive substudy with those who did not, focusing on sociodemographics, self-reported health, self-reported cognition, and lifestyle. The second objective was to evaluate participants' engagement with the study with regards to their computer familiarity and preferred device and browser; the time of day and week they took the test; the time spent completing the tasks; the number of tasks completed and reasons for discontinuing; participation at field stations; and their self-reported user experience and the frequency with which they accessed their performance results.
To our knowledge, this is the first study to examine participation bias and engagement in web-based cognitive testing in a general population. Our study provides insight into sociodemographic, health, self-reported cognition, and lifestyle characteristics of those opting in and completing web-based cognitive testing. The overall participation rate was only 8.6% of those invited and 4.7% of the general population, suggesting substantial room for improvement in how participants are invited, re-invited and reminded of such studies. Importantly, the field stations were an ineffective means of recruitment. Those self-selecting into web-based testing in HUNT4 were predominantly between 50 and 80 years old and had tertiary education. Web-based testing appeared to appeal more to participants living alone and not working, and those with self-reported good health, certain diagnoses (e.g., cancer, mental health) and subjective experienced cognitive issues among adults, but not to teenagers with learning problems. To reach all age groups, web-based, unsupervised cognitive testing needs to be available on a multi-device platform. Finally, when participants did log in and engage, most completed all tasks and reported a positive experience. The number of completed tasks was lower and the risk of discontinuation was higher among older participants, those without tertiary education, and those using phones or tablets rather than PCs. Weekdays were optimal for participation.
Characteristics of Hjernetrim participants
Sociodemographics
Hjernetrim participants were between 13 and 97 years of age, illustrating that web-based testing can be used to recruit participants of all age groups. The age distribution in Hjernetrim participants was comparable to those who only participated in HUNT4, but with the highest participation rate among those aged 50–69 years, whereas participants above 80 years were relatively underrepresented in Hjernetrim.
Hjernetrim participation rates were higher with age until the late sixties in women and seventies in men and lower thereafter in both sexes compared to only HUNT4 participation. This result differs from previous health-related studies, for example in heart failure RCTs and a study of self-selection for digital health research participants tended to be younger than non-participants15,16,17. The higher participation rate among middle-aged and older (but not oldest old) in Hjernetrim could be attributed to more leisure time at these ages18. Another explanation might be that people become more invested in their cognitive health as they age, which makes them more likely to view involvement in this type of research as advantageous for themselves or their age group at large19. Our finding of a lower participation rate among those who reported having no memory problems supports this notion.
Nevertheless, the least likely to participate in Hjernetrim were those aged 80 years and above. This might be due to the general decrease in engagement in leisure activities despite more free time, observed in the oldest old20, which could have affected Hjernetrim more than the main study because of the latter’s somatic health testing with feedback and visit to a test clinic with its social aspect. The relatively low participation rate among the older participants could also be due to another substudy, HUNT4 70 + in which MoCA and other cognitive tests were performed21. The number of 80- and 90-year-olds in Hjernetrim compared to HUNT4 could also be due to the recruitment procedure. Many of the older participants in HUNT4 have taken part in previous HUNT waves and are familiar with this part of the study while web testing is something new to them. The computer skills assumed to be required to participate could have been another barrier. The high computer familiarity observed in Hjernetrim participants (see Hjernetrim engagement) supports this notion. Hjernetrim was initially developed and validated for older adults, and they were able to follow instructions irrespective of computer familiarity level3. However, we cannot rule out that the need to enter the website address and login credentials manually, which was required here, could have discouraged individuals from participating. Using passwordless authentication, for example by sending an electronic invite via text or e-mail to participants that contains a participant-specific Magic Link allowing direct access to the battery without manually entering login credentials, would likely increase participation rates across all ages22.
Men were less likely to participate than women. This is a well-known bias in epidemiological studies23,24,25 and even more pronounced than in this study in other self-administered and self-recruiting cognitive test batteries such as MindCrowd26, BAM-COG27, and CFT28. The reason behind this sex difference is unclear, but it has been attributed to women being more engaged in volunteering, as shown in the previous HUNT waves5,13. Intriguingly, the trend was reversed in older adults partaking in Hjernetrim, with more men than women participating after the age of 65. A potential explanation is lower computer usage and computer self-efficacy in women over the age of 65, reported by Statistics Norway (ssb.no/en/statbank/table/11124) which was also reflected in the Hjernetrim participants (Supplementary Table S2). If the same was true for the entire cohort, it could be assumed that less familiarity with computers discouraged older women from participating in the study. As younger generations show high computer self-efficacy and no sex differences29, we predict increasingly higher rates of participation in web-based studies among elders, including women, in the future. It is interesting that although women across all age groups indicated lower familiarity with computers compared to men, still a greater number of women under the age of 65 participated in the Hjernetrim study. This implies that factors not accounted for in the study may have contributed to the observed differences in participation between men and women.
The Hjernetrim participants were highly educated, with approximately 54% having tertiary education. This percentage was higher than in the HUNT4 study and consistent across age and sex. High educational attainment is a common selection bias across study designs11,17,24,25,30,31, and also reported in other large web-based cognitive studies such as MindCrowd where the majority had between 14 and 16 years of education32. Highly educated people tend to exhibit different engagement patterns (as discussed below) and achieve higher cognitive scores33 compared to less educated participants. This overrepresentation might limit the generalizability and could skew the average test performance, task engagement, and user experience, impacting the study's overall results. This bias could have been minimized by sending reminders to those receiving letters and not participating, which we were not allowed as per the HUNT Research Centre regulations.
Participants living alone were more likely to participate in Hjernetrim. Reaching those people is especially important in cognitive ability studies since it was demonstrated that those who are lifelong single or widowed have a higher risk of dementia34 and that singlehood was associated with negative outcomes such as somatic symptoms, anxiety and insomnia, severe depression, and romantic loneliness35.
Living in urban areas significantly increased the probability of participation and the effect was significant across age and education groups. It is not clear why living in rural areas decreased the probability of participation given that the opposite effect was observed for participation in the HUNT4 study13.
Not being occupationally active increased participation in the study, particularly among individuals between 50 and 70 years of age. This trend is likely due to the more leisure time, which could facilitate greater participation. These findings indicate that middle-aged individuals who are not engaged in occupational activities represent a key demographic for web-based cognitive studies. Additionally, this group is known to be at a heightened risk for various somatic and mental health issues36,37,38. Our study suggests that online research methodologies could be an effective approach to engage and study this demographic, potentially addressing their unique health challenges.
Somatic and mental health
The odds ratios for participation were impacted by different health conditions in varying ways. The lack of influence of diabetes, stroke, hearing impairment, and HADS score over 11 on participation is probably important to note as it shows that somatic and mental health challenges do not necessarily impair the ability to take part in web-based cognitive testing. This is further exemplified by the increased participation odds in those reporting cancer, or mental health problems they sought help for. Those conditions have been connected to cognitive problems or complaints39,40,41,42. One could speculate that facing such challenges would make these groups more conscious about their cognitive health and thus more likely to participate, suggesting that web-based testing is a suitable method for studies targeting those groups, although previous studies showed otherwise17. The higher participation rates among those with mental health problems they sought help for, provide support for this reasoning. On the other hand, we observed lower participation among groups of individuals who reported “not very good” health or experienced impairment due to chronic illness or injury, myocardial infarction, or motor and vision impairments. Depending on the type of injury or chronic disease, participants could need physical therapy or treatments, and thus not have enough time or possibility to participate in this study. Likewise, participants with motor or vision impairment could have difficulties using the devices or perform the tasks. Implementation of universal design will be important to meet the needs of those with sensory and motor deficits in the future. A shorter test battery might have been better as it would reduce the overall load on participants and facilitate their participation.
Overall, in a study targeting the general population, no clear pattern emerged suggesting that diseases or impairments necessarily lead to reduced participation. Instead, the data suggest a more complex picture, with some diseases, disorders or impairments associated with higher participation rates and others associated with lower participation rates, similar to a previous study examining HUNT participation43. Additionally, some groups previously considered hard to reach, such as those living alone44 and those not occupationally active appear to have a high participation rate in a study with a web-based design.
Self-reported cognition
Teenagers with learning disabilities, especially in mathematics, participated less often. Since learning difficulties negatively impact academic well-being45, students with learning disabilities could have felt discouraged from participating in cognitive testing. On the other hand, adults who self-reported having no memory problems were less likely to participate, which again could indicate that worrying about one’s health increases participation odds. It should be noted that subjective memory complaints in this cohort were connected with lower scores on the verbal list learning test46, indicating correspondence between subjective and objective measures of cognition. The dichotomy in participation between teenagers and adults with self-reported cognitive issues indicates that different strategies might be needed when recruiting different age groups to cognitive testing.
Lifestyle satisfaction, smoking, and alcohol use
Being very satisfied with one´s lifestyle (diet, exercise, smoking and drinking habits) increased participation. This is a largely expected outcome given that those who tend to lead a healthier lifestyle and are satisfied with life show other characteristics associated with higher participation in our study, such as higher education, urban place of residence, and less or no chronic diseases47.
Individuals who formerly smoked occasionally were more likely to participate, whereas current daily smokers and those who abstain from alcohol were less likely to take part. These findings resemble those of the UK Biobank24, in that in both studies, participants were less likely to be current smokers and never drinkers compared to nonrespondents from the general population. This pattern has also been seen in a phone-based Stockholm Health of the Population Study48.
Taken together, those who self-selected to Hjernetrim constitute only a small part of the general population and are not fully representative of the general population with regard to several sociodemographic attributes, somatic and mental health, cognition, and lifestyle factors. The relatively low participation rate was partly caused by the fact that only those who participated in the main HUNT4 study were invited and by regular mail. Still, the rate was comparable to other general population studies such as the UK Biobank24. Among the largest and expected sources of bias were age, educational attainment, self-reported health and certain health conditions such as impaired motor ability. However, contrary to the popular “healthy user bias”7,49,50,51, certain conditions such as cancer and self-reported mental health problems actually increased participation. Moreover, most of the assessed factors, including life-altering ones such as vision and hearing impairment, had relatively small to no effect on participation. Lastly, it is important to note that while self-selection bias is a widespread concern in health research, it does not necessarily compromise the generalizability of the results, as long as its sources are clearly understood and factored into the interpretation of the study's findings7,44,52. It can also be accounted for by using certain statistical approaches, for example by using inverse probability weighting53.
Hjernetrim engagement
Computer familiarity, hardware, and software
In computerized studies such as Hjernetrim, computer familiarity influences participation54,55. Our results confirm that the vast majority of those who self-select to web-based studies were comfortable or very comfortable with using computers. This was expected since 95% of people in Norway use the internet daily or almost every day, including more than 75% of those ages 65–79 (ssb.no/en/statbank/table/11124). However, computer familiarity was lower in women and decreased with age, which could have affected the results56. Furthermore, since computer familiarity was high in people younger than the late seventies, this issue was limited. An increase in computer familiarity is expected in the future, and a Dutch study reported that less than 5% of older participants did not use a computer frequently57. A Malaysian study found that 77% of the sample of older adults had computer-related expertise and only 3% of the sample had none58. Similarly, internet usage in Norway has consistently increased among people of all age groups over the last decades. For example, the percentage of individuals aged 55–74 who reported using the internet in the past 3 months surged from 47.8% in 2005 to 99.1% in 2023 (stats.oecd.org/Index.aspx?DataSetCode = ICT_HH2). Overall, computer familiarity seems not to be an issue for most participants and will likely become even less of a concern in the future.
As recommended, most (74%) of the participants used PCs (Windows PC or Macbook) to partake in Hjernetrim. Among those who did not follow this advice, younger participants predominantly chose phones, whereas older participants were more inclined to use tablets. This highlights the need for a web-based platform to be accessible and optimized for all devices to prevent the loss of potential participants. We cannot rule out that we lost participants who did not or could not participate on a computer. Importantly, performing the tests on a phone was associated with a lower rate of completed tasks and a higher likelihood of discontinuation compared to a Windows PC. Participants will experience more disruptions using their mobile devices as the web-based format does not hinder pop-ups or calls. Additionally, conducting tests on smaller screens might present more challenges. Considering that mobile devices are the most commonly used platform for internet access in general (perficient.com/insights/research-hub/mobile-vs-desktop-usage; gs.statcounter.com/platform-market-share/desktop-mobile-tablet) and become the preferred computing device across various settings59,60, it is crucial to implement methods for obtaining cognitive data of the same quality irrespective of the type of device.
The Blink browser engine, integral to the recommended Google Chrome browser, emerged as the most popular choice among all age groups, again implying that participants generally adhered to the guidelines. The selection of a browser is primarily a concern for the participant’s security61,62; thus, educating participants about this aspect could enhance compliance. However, there may be reluctance among some participants, particularly older adults, to download or use an unfamiliar browser. Our results support this as the use of older and non-standard browsers increased with age. A hesitancy to update the browser could potentially impact participation rates.
Day and time of participation
The preferred time of participation appeared to depend on the convenience of a certain time, i.e., finding a day and time when it was possible to devote ≥ 40 min to taking part in the online testing, and when the different age groups feel their mental capacity peaks at different times during the day, morning for older adults, later in the day for teenagers, and evenings for working age participants. Importantly, while participation at a preferred time does not affect the results in all age groups, it can help older adults perform to the best of their ability63,64. Web-based methodology is optimal in this respect. Interestingly, there was a significant variation in participation rates throughout the week. Thursdays and Tuesdays emerged as the most preferred days for participation, while Saturdays and Sundays saw the lowest engagement across almost all age groups. This finding is crucial for planning future web-based study promotions, indicating that advertising efforts might be more effective if concentrated on the weekdays. This conclusion is inconsistent with some previous studies on the impact of invitation day on response rate to online questionnaires65,66, but aligns with insights from studies conducted by marketing and survey platforms, as evidenced by the data and recommendations shared on their respective blogs (such as checkmarket.com/blog/survey-invitations-best-time-send, getresponse.com/blog/best-time-to-send-email-infographic, zendesk.com/blog/maximize-survey-response-rates, mailchimp.com/resources/insights-from-mailchimps-send-time-optimization-system).
Time spent
The average time participants spent completing the tests was approximately 46 min, exceeding the 40-min duration outlined in the informational leaflet. This extended duration can be partially attributed to the high performance of participants46,67. For example, successfully recalling more words in a verbal memory test results in a longer completion time, while failing a training task could result in skipping the main part of that task. The fact that most participants completed the majority or all of the tasks and provided predominantly positive feedback suggests that a duration of over 40 min is generally acceptable. Yet, it is notable that some individuals, particularly older adults, spent considerably more time than expected, which can lead to participant fatigue, even if it does not decrease performance67,68. Therefore, future studies should aim to strike a balance between the length of the tests and the number of cognitive domains assessed, possibly across multiple sessions, to optimize both participation rates and completion rates without inducing fatigue.
Completion and discontinuation of test battery
Overall, the completion rate for all tests was high for Hjernetrim participants. It is important to note that participants who completed fewer tests or discontinued the battery shared some demographic characteristics with those who were less likely to participate in the study overall, such as lacking tertiary education and, in the case of those who completed fewer tests, being in the oldest age groups. This suggests that successfully recruiting harder-to-reach individuals for web-based studies does not guarantee complete data. A similar pattern was observed in a previous web-based depression intervention, where lower education levels increased the risk of participants dropping out before completing most of the study modules. This risk was further heightened by younger age and being male69. However, we did not observe this in our study, possibly due to differences in the study topics. Additionally, longer or more demanding tasks, such as verbal list learning, appeared to cause relatively high dropout rates. This finding is consistent with previous research that demonstrated an increase in participant discontinuation with longer survey lengths in several web-based psychological questionnaire studies involving undergraduate students70. Other verbal memory tests such as verbal paired associates might be a better choice.
Participation at field stations
Field stations turned out not to be very popular. Less than 0.5% of all participants participated at a Hjernetrim field station despite advertisements in newspapers, shopping malls, and other HUNT field stations. The participants who chose field stations were predominantly older and not very comfortable with computers, which was most likely the reason they signed up. In theory, field stations could help to recruit important, underrepresented groups, but we were not able to harness such potential. Other solutions, like having the field stations available at the same time as the main HUNT4 data collection was performed, collaborations with senior centres, or help over a remote connection, might have been more efficient.
User experience and feedback for participants
Approximately 28% of participants reported disruptions during one or more tasks. Despite this, the average rate of disruption per task was approximately 5%, which we consider relatively low. This suggests that testing participants in non-controlled environments likely had a limited impact on results, a conclusion supported by previous studies suggesting that web-based methodology does not negatively impact the quality of data and study validity or reliability3,71,72,73.
A positive experience was reported by approximately 65% of participants. Negative experience or technical issues were reported by less than 3% of participants, despite the use of a wide range of devices and software. This suggests that most participants found web-based cognitive testing quite enjoyable, in line with previous Hjernetrim study findings3. A caveat here is that participants who provided feedback were mainly those completing all tasks.
Finally, most participants did not log back in for feedback on their performance. A potentially more rewarding and effective approach for feedback could be to provide immediate results or send the results via regular mail like the other results in HUNT4.
Conclusion
This study is the first to examine the characteristics of individuals who volunteered to participate in web-based cognitive testing versus those invited who did not participate and to identify sources of participation bias while also showing in detail participants’ engagement in the study. Our study found that groups underrepresented in prior research, such as individuals with mental health concerns, subjective memory complaints, the unemployed and those living alone, were adequately represented in our sample. Differences in factors like educational attainment, age, and various health conditions contributed to varying selection bias. By recognizing and addressing these sources of bias, we can ensure the representativeness and reliability of our future research results. Moreover, the level of participant engagement observed in our study indicates that web-based cognitive testing is effective, even when the test battery is extensive and time-consuming. Finally, the majority of participants reported having a positive experience with the testing process. The patterns of participant engagement can help refine planning, design, and recruitment strategies for future studies that utilize web-based technologies, ultimately boosting participation rates, data quality and participation experience.
Data availability
The data that support the findings of this study regarding factors affecting the likelihood of participation in Hjernetrim compared to HUNT4 are available on request from the HUNT Databank. HUNT Databank website (https://hunt-db.medisin.ntnu.no/hunt-db) can be used to apply for a research project or order variables. Data on engagement patterns are available from the authors. Access to the data will require review by the regional ethics committee and permission from the HUNT Databank to ensure adherence to GDPR as well as institutional regulations.
References
Galea, S. & Tracy, M. Participation rates in epidemiologic studies. Ann. Epidemiol. 17, 643–653 (2007).
Ebert, J. F., Huibers, L., Christensen, B. & Christensen, M. B. Paper- or web-based questionnaire invitations as a method for data collection: cross-sectional comparative study of differences in response rate, completeness of data, and financial cost. J. Med. Internet. Res. 20, e24 (2018).
Hansen, T. I., Haferstrom, E. C. D., Brunner, J. F., Lehn, H. & Håberg, A. K. Initial validation of a web-based self-administered neuropsychological test battery for older adults and seniors. J. Clin. Exp. Neuropsychol. 37, 581–594 (2015).
Kennedy, G. E., Judd, T. S., Churchward, A., Gray, K. & Krause, K.-L. First year students’ experiences with technology: Are they really digital natives? Aust. J. Educ. Technol. 24, 1 (2008).
Krokstad, S. et al. Cohort profile: The HUNT study Norway. Int. J. Epidemiol. 42, 968–977 (2013).
Morton, L. M., Cahill, J. & Hartge, P. Reporting participation in epidemiologic studies: A survey of practice. Am. J. Epidemiol. 163, 197–203 (2006).
Nohr, E. A., Frydenberg, M., Henriksen, T. B. & Olsen, J. Does low participation in cohort studies induce bias?. Epidemiology 17, 413–418 (2006).
Mayr, A. et al. Web-based data collection yielded an additional response bias—but had no direct effect on outcome scales. J. Clin. Epidemiol. 65, 970–977 (2012).
Zinn, S., Landrock, U. & Gnambs, T. Web-based and mixed-mode cognitive large-scale assessments in higher education: An evaluation of selection bias, measurement bias, and prediction bias. Behav. Res. 53, 1202–1217 (2021).
Demark-Wahnefried, W., Bowen, D. J., Jabson, J. M. & Paskett, E. D. Scientific bias arising from sampling, selective recruitment, and attrition: The case for improved reporting. Cancer Epidemiol. Biomarkers Prev. 20, 415–418 (2011).
Honningsvåg, L.-M., Linde, M., Håberg, A., Stovner, L. J. & Hagen, K. Does health differ between participants and non-participants in the MRI-HUNT study, a population based neuroimaging study? The Nord-Trøndelag health studies 1984–2009. BMC Med. Imaging 12, 23 (2012).
Sax, L. J., Gilmartin, S. K. & Bryant, A. N. Assessing response rates and nonresponse bias in web and paper surveys. Res. High. Educ. 44, 409–432 (2003).
Åsvold, B. O. et al. Cohort Profile Update: The HUNT Study, Norway. 2021.10.12.21264858. https://doi.org/10.1101/2021.10.12.21264858 (2021).
Hansen, T. I. Memoro—Evaluation of core psychometric properties and usability of a new self-administered web-based neuropsychological test platform. (NTNU, Trondheim, 2016).
Harrison, J. M. et al. Refusal to participate in heart failure studies: Do age and gender matter?. J. Clin. Nurs. 25, 983–991 (2016).
Heiat, A., Gross, C. P. & Krumholz, H. M. Representation of the elderly, women, and minorities in heart failure clinical trials. Arch. Internal Med. 162, (2002).
Poli, A., Kelfve, S. & Motel-Klingebiel, A. A research tool for measuring non-participation of older people in research on digital health. BMC Public Health 19, 1487 (2019).
Breivik, G. & Hellevik, O. More active and less fit: Changes in physical activity in the adult Norwegian population from 1985 to 2011. Sport Soc. 17, 157–175 (2014).
Akmatov, M. K. et al. Motivations for (non)participation in population-based health studies among the elderly—comparison of participants and nonparticipants of a prospective study on influenza vaccination. BMC Med. Res. Methodol. 17, 18 (2017).
Agahi, N., Ahacic, K. & Parker, M. G. Continuity of leisure participation from middle age to old age. J. Gerontol. Ser. B 61, S340–S346 (2006).
Gjøra, L. et al. Current and future prevalence estimates of mild cognitive impairment, dementia, and its subtypes in a population-based sample of people 70 years and older in Norway: The HUNT study. J. Alzheimers Dis. 79, 1213–1226 (2021).
Prasad, A. A comparative study of passwordless authentication. Preprint at https://doi.org/10.36227/techrxiv.171560547.71979752/v1 (2024).
Eagan, T. M. L., Eide, G. E., Gulsvik, A. & Bakke, P. S. Nonresponse in a community cohort study: Predictors and consequences for exposure–disease associations. J. Clin. Epidemiol. 55, 775–781 (2002).
Fry, A. et al. Comparison of sociodemographic and health-related characteristics of UK biobank participants with those of the general population. Am. J. Epidemiol. 186, 1026–1034 (2017).
Glass, D. et al. A telephone survey of factors affecting willingness to participate in health research surveys. BMC Public Health 15, 1017 (2015).
Talboom, J. S. et al. Two separate, large cohorts reveal potential modifiers of age-associated variation in visual reaction time performance. Npj Aging Mech. Dis. 7, 1–18 (2021).
Aalbers, T., Baars, M. A. E., Olde Rikkert, M. G. M. & Kessels, R. P. C. Puzzling With Online Games (BAM-COG): Reliability, validity, and feasibility of an online self-monitor for cognitive performance in aging adults. J. Med. Internet Res. 15, e270 (2013).
the Cognitive Function Test. Trustram Eve, C. & de Jager, C. A. Piloting and validation of a novel self-administered online cognitive screening tool in normal older persons. Int. J. Geriatr. Psychiatry 29, 198–206 (2014).
Rohatgi, A., Scherer, R. & Hatlevik, O. E. The role of ICT self-efficacy for students’ ICT use and their achievement in a computer and information literacy test. Comput. Educ. 102, 103–116 (2016).
Pani, J. et al. Effect of 5 years of exercise intervention at different intensities on brain structure in older adults from the general population: A generation 100 substudy. Clin. Interv. Aging 16, 1485–1501 (2021).
Stensvold, D. et al. A randomised controlled study of the long-term effects of exercise training on mortality in elderly people: study protocol for the Generation 100 study. BMJ Open 5, e007519 (2015).
Talboom, J. S. et al. Family history of Alzheimer’s disease alters cognition and is modified by medical and genetic factors. eLife 8, e46179 (2019).
Sokołowski, D. R. et al. 5 years of exercise intervention did not benefit cognition compared to the physical activity guidelines in older adults, but higher cardiorespiratory fitness did. A generation 100 substudy. Front. Aging Neurosci. 13, 742587 (2021).
Skirbekk, V. et al. Marital histories and associations with later-life dementia and mild cognitive impairment risk in the HUNT4 70+ study in Norway. J. Aging Health 8982643221131926. https://doi.org/10.1177/08982643221131926 (2022).
Adamczyk, K. Voluntary and involuntary singlehood and young adults’ mental health: An investigation of mediating role of romantic loneliness. Curr. Psychol. 36, 888–904 (2017).
Saito, T., Murata, C., Saito, M., Takeda, T. & Kondo, K. Influence of social relationship domains and their combinations on incident dementia: A prospective cohort study. J. Epidemiol. Community Health 72, 7–12 (2018).
Stankunas, M., Kalediene, R., Starkuviene, S. & Kapustinskiene, V. Duration of unemployment and depression: a cross-sectional survey in Lithuania. BMC Public Health 6, 174 (2006).
Wilson, S. H. & Walker, G. M. Unemployment and health: A review. Public Health 107, 153–162 (1993).
Abramovitch, A., Short, T. & Schweiger, A. The C Factor: Cognitive dysfunction as a transdiagnostic dimension in psychopathology. Clinical. Psychol. Rev. 86, 102007 (2021).
Hall, N. M., Kuzminskyte, R., Pedersen, A. D., Ørnbøl, E. & Fink, P. The relationship between cognitive functions, somatization and behavioural coping in patients with multiple functional somatic symptoms. Nord. J. Psychiatry 65, 216–224 (2011).
Janelsins, M. C., Kesler, S. R., Ahles, T. A. & Morrow, G. R. Prevalence, mechanisms, and management of cancer-related cognitive impairment. Int. Rev. Psychiatry 26, 102–113 (2014).
Pendergrass, J. C., Targum, S. D. & Harrison, J. E. Cognitive impairment associated with cancer. Innov. Clin. Neurosci. 15, 36–44 (2018).
Langhammer, A., Krokstad, S., Romundstad, P., Heggland, J. & Holmen, J. The HUNT study: participation is associated with survival and depends on socioeconomic status, diseases and symptoms. BMC Med. Res. Methodol. 12, 143 (2012).
Nilsen, R. M. et al. Self-selection and bias in a large prospective pregnancy cohort in Norway. Paediatr. Perinat. Epidemiol. 23, 597–608 (2009).
Korhonen, J., Linnanmäki, K. & Aunio, P. Learning difficulties, academic well-being and educational dropout: A person-centred approach. Learn. Indiv. Diff. 31, 1–10 (2014).
Kljajevic, V. et al. Female advantage in verbal learning revisited: A HUNT study. Memory 31, 831–849 (2023).
Phulkerd, S., Thapsuwan, S., Chamratrithirong, A. & Gray, R. S. Influence of healthy lifestyle behaviors on life satisfaction in the aging population of Thailand: A national population-based survey. BMC Public Health 21, 43 (2021).
Boström, G. et al. Socioeconomic Differences in Smoking in an Urban Swedish Population: The bias introduced by non-participation in a mailed questionnaire. Scand. J. Soc. Med. 21, 77–82 (1993).
Pinsky, P. F. et al. Evidence of a healthy volunteer effect in the prostate, lung, colorectal, and ovarian cancer screening trial. Am. J. Epidemiol. 165, 874–881 (2007).
Shrank, W. H., Patrick, A. R. & Alan Brookhart, M. Healthy user and related biases in observational studies of preventive interventions: A primer for physicians. J. Gen. Intern. Med. 26, 546–550 (2011).
Struijk, E. A. et al. Mortality and cancer incidence in the EPIC-NL cohort: Impact of the healthy volunteer effect. Eur. J. Public Health 25, 144–149 (2015).
Pirastu, N. et al. Genetic analyses identify widespread sex-differential participation bias. Nat. Genet. 53, 663–671 (2021).
Seaman, S. R. & White, I. R. Review of inverse probability weighting for dealing with missing data. Stat. Methods Med. Res. 22, 278–295 (2013).
Baek, N. J., Park, G. I., Byun, Y. S., Jeon, M. J. & Sakong, J. Effects of computer familiarity and computer type on the performance of Korean computerized neurobehavioral test. Ann. Occup. Environ. Med. 28, 44 (2016).
Toscos, T. et al. Selection biases in technology-based intervention research: Patients’ technology use relates to both demographic and health-related inequities. J. Am. Med. Inf. Assoc. 26, 835–839 (2019).
Iverson, G. L., Brooks, B. L., Ashton, V. L., Johnson, L. G. & Gualtieri, C. T. Does familiarity with computers affect computerized neuropsychological test performance?. J. Clin. Exp. Neuropsychol. 31, 594–604 (2009).
Rijnen, S. J. M. et al. Evaluation of normative data of a widely used computerized neuropsychological battery: Applicability and effects of sociodemographic variables in a Dutch sample. Assessment 27, 373–383 (2020).
Vanoh, D. et al. Development and assessment of a web-based intervention for educating older people on strategies promoting healthy cognition. Clin. Interv. Aging 13, 1787–1798 (2018).
Armakolas, S., Lora, C. & Waligóra, A. Increased mobile phone use in high school students as a social problem under the dimension of addiction. Adv. Mob. Learn. Educ. Res. 4, 911–919 (2024).
Barton, J. J., Shumin Zhai & Cousins, S. B. Mobile Phones Will Become The Primary Personal Computing Devices. in Seventh IEEE Workshop on Mobile Computing Systems & Applications (WMCSA’06) 3–9 (IEEE, Orcas Island, WA, USA, 2006). https://doi.org/10.1109/WMCSA.2006.17.
Franken, G. Security and Privacy Policy Bugs in Browser Engines. Beveiligings- en Privacybeleidsbugs in Browser Engines (2024).
Radivojevic, K., Clark, N., Klempay, A. & Brenner, P. Defending novice user privacy: An evaluation of default web browser configurations. Comput. Secur. 140, 103784 (2024).
Intons-Peterson, M. J., Rocchi, P., West, T., McLellan, K. & Hackney, A. Age, testing at preferred or nonpreferred times (testing optimality), and false memory. J. Exp. Psychol. Learn. Mem. Cogn. 25, 23–40 (1999).
Robison, M. K. & Unsworth, N. Do participants differ in their cognitive abilities, task motivation, or personality characteristics as a function of time of participation?. J. Exp. Psychol. Learn. Mem. Cogn. 42, 897–913 (2016).
Howard, C., Greaves, L. M., Osborne, D. & Sibley, C. G. Is there a day of the week effect on panel response rate to an online questionnaire email invitation?. J. Survey Stat. Methodol. 12, 94–104 (2024).
Lynn, P., Bianchi, A. & Gaia, A. The impact of day of mailing on web survey response rate and response speed. Soc. Sci. Comput. Rev. 42, 352–368 (2024).
Ackerman, P. L. & Kanfer, R. Test length and cognitive fatigue: An empirical examination of effects on performance and test-taker reactions. J. Exp. Psychol. Appl. 15, 163–181 (2009).
Jensen, J. L., Berry, D. A. & Kummer, T. A. Investigating the effects of exam length on performance and cognitive fatigue. PLoS ONE 8, 1 (2013).
Karyotaki, E. et al. Predictors of treatment dropout in self-guided web-based interventions for depression: An ‘individual patient data’ meta-analysis. Psychol. Med. 45, 2717–2726 (2015).
Hoerger, M. Participant dropout as a function of survey length in internet-mediated university studies: Implications for study design and voluntary participation in psychological research. Cyberpsychol. Behav. Soc. Netw. 13, 697–700 (2010).
Germine, L. et al. Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments. Psychon. Bull Rev. 19, 847–857 (2012).
Hansen, T. I., Lehn, H., Evensmoen, H. R. & Håberg, A. K. Initial assessment of reliability of a self-administered web-based neuropsychological test battery. Comput. Hum. Behav. 63, 91–97 (2016).
Ramsey, S. R., Thompson, K. L., McKenzie, M. & Rosenbaum, A. Psychological research in the internet age: The quality of web-based data. Comput. Hum. Behav. 58, 354–360 (2016).
Acknowledgements
The HUNT Study is a collaboration between the HUNT Research Centre (Faculty of Medicine and Health Sciences, Norwegian University of Science and Technology), Nord-Trøndelag County Council, Central Norway Health Authority, and the Norwegian Institute of Public Health. The HUNT Hjernetrim was funded by the Liaison Committee between the Central Norway Regional Health Authority and the Norwegian University of Science and Technology, and MiDT National Norwegian Research Center, St. Olav’s University Hospital, Trondheim. The authors would like to express their sincere gratitude to Henning Hoel Rise, Line Skarsem Reitlo, and Hallvard Røe Evensmoen for their invaluable assistance with data collection, which greatly contributed to the success of this study. Additionally, we would like to extend our appreciation to Jørn Søberg Fenstad for his invaluable help with the HUNT4 database and for answering our questions regarding HUNT4.
Funding
Open access funding provided by Norwegian University of Science and Technology.
Author information
Authors and Affiliations
Contributions
DRS:organized data collection and quality controlled data, performed the statistical analysis and drafted the manuscript. JP: organized data collection and quality controlled data. TIH: developed the test battery, collected data, and followed up participants. AKH: developed the test battery, applied for funding, organized data collection, and oversaw data collection, organization and analyses.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Sokołowski, D.R., Pani, J., Hansen, T.I. et al. Participation and engagement in online cognitive testing. Sci Rep 14, 14800 (2024). https://doi.org/10.1038/s41598-024-65617-w
Received:
Accepted:
Published:
Version of record:
DOI: https://doi.org/10.1038/s41598-024-65617-w
