Introduction

Substance use disorders (SUD) represent a global health crisis. They result in significant individual and societal costs and, in fact, lead to premature mortality due to mental health disorders1. As such, it is not surprising that SUD and its treatment have been the subject of much interest from the scientific community in recent decades. However, despite the attention that it has received, the effectiveness of SUD treatment is still far from satisfactory, with relapse rates ranging between 40 and 60% (e.g., Brandon et al.2).

The relapsing nature of SUD, along with data on associated lasting neural changes, has helped in the conception of these disorders and addiction in general as a chronic brain disease3,4. The successful management of chronic illnesses depends on factors that may not be present in the treatment of acute illnesses5. Continuing care has been addressed by proposing different treatment models that include some type of follow-up intervention as well as self-management support6,7. The move towards a more patient-centred and comprehensive care approach has meant a greater emphasis on the multifactorial nature of SUD and the individual needs and level of functioning of each patient. Addressing multiple dimensions known to influence the development and recovery of these disorders (e.g., biological, cognitive, psychological, social) can lead to improved treatment outcomes8.

One dimension that has long been associated with SUD is cognitive dysfunction. Cognitive deficits in a number of cognitive domains, such as memory and executive function, have been shown to have a bidirectional relationship with substance use9. Cognitive impairment can be a risk factor for substance use initiation and addiction development. Conversely, substance use can directly cause cognitive deficits9. More importantly, cognitive dysfunction appears to be linked to increased disease severity, while recovery of cognitive function is known to be predictive of increased treatment adherence and decreased relapse rates. Given its impact on treatment outcomes and prognosis, cognitive functioning has been increasingly considered an important target for SUD treatment9,10.

In a recent systematic review11, we found that 69% of the studies on the effectiveness of cognitive training in improving memory, executive functioning, and processing speed in individuals with SUD reported cognitive improvement, indicating its potential as a useful intervention for SUD treatment. However, the effectiveness of cognitive training is not undisputed. Many authors have questioned the possibility of “far-transfer” - where training in one cognitive domain leads to improvements in unrelated domains - resulting from cognitive training12,13,14, questioning whether these effects can lead to generalized cognitive benefits. Considering both the lack of scientific agreement on the topic and the potentially significant impact that cognitive training could have on the recovery of patients, it is important that these programs continue to be studied, considering newer research as well as advances in technology.

While most cognitive training programs already make use of new technologies (e.g., computerized, smartphone-based), few have utilized one of the most recent technologies – virtual reality (VR). By providing an immersive environment, VR appears to enhance motivation and engagement in its users15, both of which are essential ingredients for the effectiveness of any intervention. A recent review16of randomized controlled trials studying fully immersive VR-based cognitive training in individuals with neurological and psychiatric disorders reported promising results. Jahn and colleagues16 reported that VR-delivered cognitive training led to significant improvements in cognitive domains such as executive function and attention.

To our knowledge, only two studies have investigated the effectiveness of VR cognitive training in individuals with SUD. Both studies presented encouraging results, with significant improvements reported for attention17,18, memory18, and cognitive flexibility17. However, despite categorizing their interventions as cognitive training, both programs seem to be more focused on improving everyday functioning, making use of tasks that mirror real-world activities (e.g., going to the supermarket and selecting the items on a list). It is our belief that, in our target population, this type of task would not be very motivating and could lead to low adherence.

In this study, we describe the development and test the usability of VRainSUD, which was designed as an add-on intervention for SUD treatment. VRainSUD targets cognitive domains known to be affected by substance use as well as influence recovery, namely, memory and executive functioning. VRainSUD includes two delivery media to be used in distinct stages of treatment—a fully immersive VR platform and a mobile follow-up application—for the purpose of this study.

Development of the VRainSUD virtual reality platform

The development of the cognitive training program VRainSUD, in which the VR platform is integrated, was based on The Medical Research Council (MRC) guidelines for complex interventions, namely, the design of complex interventions19.

VRainSUD comprises a VR platform and a mobile follow-up application. The cognitive training sessions through the VR platform are the main component of the program, and the mobile application only aims to maintain the cognitive gains once the patients leave treatment. VRainSUD includes 18 VR training sessions (3 times per week, for 6 weeks, 30 min each session) during which the patient can complete a selection of the six available cognitive training tasks according to their needs. These tasks target the following cognitive domains: memory, executive functioning, and processing speed.

Since VRainSUD was created to be a personalized program, the structure of the sessions will depend on the patient’s baseline cognitive functioning, identified deficits, and progress throughout the program. A brief description of the VR platform cognitive training tasks can be found in Table 1.

Table 1 Brief description of VR platform cognitive tasks.
Full size table

Following the MRC guidelines for complex interventions, VRainSUD is currently in the second phase, which focuses on assessing feasibility, acceptability, and design. The feasibility of VRainSUD was evaluated within the context of its potential further implementation in inpatient SUD treatment.

Methods

Participants

Participants included 17 patients receiving inpatient treatment at VillaRamadas International Treatment Center in Portugal, selected through a convenience sampling method. Participants were required to be at least 18 years old and have a diagnosis of SUD. Individuals with concurrent gaming addiction or neurological conditions were excluded from the study.

Eligible patients were approached by their therapists and asked about their interest in participating in the study. Those who agreed to participate signed a human ethics term and informed and free consent to participate, in accordance with the Declaration of Helsinki and the Oviedo Convention.

Procedure

The experiment was conducted in a room with enough space for comfortable VR use. Two researchers were present, one with an assisting role and the other with an observer role. The assisting researcher assumed the role of clarifying and guiding the participant during the testing process. The observer researcher registered the actions performed by participants, such as the way they used the physical resources of the VR platform, their ability to follow the instructions provided, and the time they took to complete each of task included in the script. Each participant received a brief explanation about the VR platform as well as the purpose of the usability tests. A familiarization period with the VR headset and controllers was included to address any questions. After completing all the intended tasks, participants answered an online questionnaire that included a brief survey designed to evaluate their perception of the platform and a recognized measure of usability - the Post-Study System Usability Questionnaire (PSSUQ).

Survey

The survey was developed by the research team to assess the platform’s usability. Questions 1–5 were closed-ended with yes/no responses, while questions 5.1 and 6 were open-ended to gather more detailed qualitative feedback. The survey included the following questions:

  • 1 Did you consider the platform to be intuitive?

  • 2 Do you consider that the platform has easy-to-use navigation?

  • 3 Was it easy to select the tasks included in the platform?

  • 4 Was it easy to complete the tasks on the platform?

  • 5 Did you feel difficulties executing the tasks?

  • 5.1. If so, what difficulties did you feel?

  • 6 What suggestions for improvement of the platform do you have?

Poststudy system usability questionnaire (PSSUQ)

The PSSUQ is an instrument that aims to assess the perceived satisfaction of users when using computer systems20. It consists of 19 statements evaluated on a 7-point Likert scale between 1 and 7, where 1 corresponds to “Strongly Agree” and 7 corresponds to “Strongly Disagree”21. Mean scores can be calculated for the total scale and each of the three subscales focusing on system usefulness, information quality, and interface quality. Lower scores correspond to greater perceived satisfaction and usability.

Software and hardware

Regarding the hardware used in the early stage of development, the Oculus Quest 2 headset22was selected, as it enables the end user to move freely and have a more seamless experience inside the VR platform. This headset uses the Android 10 operating system and enables Wi-Fi and Bluetooth access over the traditional HDMI connectivity23. Unreal Engine version 4.27.2 (Epic Games, 2022; available at https://www.unrealengine.com/en-US/download) was selected for the software development as it offers tools to develop interactive and visually engaging VR environments24. This platform enabled easy access to several resources that boosted the initial project development. In addition to implementing the business logic for each task, it allowed the creation of virtual environments according to the needs of each intervention objective. As a scripting language, Blueprints was used, ensuring the platform’s scalability in a sustainable manner since it allows the logic to be compartmentalized in small, structured blocks25.

Statistical analysis

IBM SPSS version 28 was used to perform the quantitative statistical analyses. Descriptive statistics were performed to characterize the sample and assess the observations of the script tasks and the participants’ responses to the final survey and PSSUQ.

The sample distribution was assessed using the Kolmogorov‒Smirnov test, and normality was confirmed. As such, ANOVA tests were used to compare the mean PSSUQ scores and mean completion times according to sociodemographic variables.

Results

Sociodemographic characteristics

An overview of the sociodemographic data is presented in Table 2.

Table 2 Sociodemographic and professional characteristics.
Full size table

Observation

Script task A – navigate the main menu room

Of the 17 participants, 16 (94.1%) completed script task A. The mean task completion time was 37.75 s, ranging from 18 to 76 s. Only one participant revealed difficulties in performing the task due to not being familiar with the VR controllers and clicking the wrong shortcuts for exploring the environment. This participant did not use the expected path to perform script task A and required additional explanations regarding the VR controllers before completing the task and proceeding to the following ones.

Script task B – start cognitive task, pause it, and return to the main menu

Script task B was completed by all participants (100%). The mean task completion time was 24.65 s, ranging from 11 to 79 s. However, one participant did not use the expected path to perform the task in question. This participant clicked the wrong sticks on the command, entered the task, and exited when instructed to pause the task. The participant recognized his mistake and completed the task independently.

Script task C – start cognitive task and return to the main menu

All participants (100%) were able to complete script task C. The mean completion time of task C was 6.12 s, ranging from 3 to 14 s.

Script task D – cognitive task 4, cube memory – complete 4 levels

Script task D was completed by all participants (100%). The mean completion time of task D was 60.18 s, ranging from 38 to 112 s. The participant who took the longest to complete the task showed initial difficulties in understanding the purpose of the task. However, after a few seconds of exploring the environment, the participant understood the task and was able to complete it.

Script task E – cognitive task 5, working maze – complete 1 st level

Fifteen participants (88.2%) completed script task E without difficulty, while two participants (11.8%) did not complete it. The mean completion time was 86.80 s, ranging from 14 to 302 s, showing the greatest variation among the participants. The two participants who did not complete the task struggled with understanding the instructions, not with navigating the VR environment. Another participant, contrary to the previous two, revealed initial difficulties using the commands, which delayed task completion.

Script task F – cognitive task 3, passing balls – gather 700 points

Fifteen participants (88.2%) successfully completed script task F, while two participants (11.8%) did not complete it. The mean completion time was 212.73 s, ranging from 112 to 347 s. Participants understood how to perform the tasks but failed to complete the task due to the point structure of the game. One participant suggested a louder auditory cue for the change in instructions, while another felt the instruction panel for the avoidable ball was too small and went unnoticed.

Script task G – cognitive task 2, number sequence – gather 300 points

Script task G was successfully completed by all participants (100%), with a mean completion time of 35.24 s, ranging from 15 s to 85 s. Although all participants successfully completed the task, two participants took longer than the average (72 and 85 s, respectively) because they initially struggled to understand the objective of the task. This instruction was not available because one of the goals of the test was to understand whether it was easy for participants to understand the objective of the task on their own. A bug was also identified in this task.

Script task H – cognitive task 1, labyrinth navigation – complete 1 st level

Script task H was successfully completed by all participants (100%). The mean completion time was 25.53 s, ranging from 12 to 45 s.

Script task I – cognitive task 1.1., remembering space

Of the 17 participants, 16 (94.1%) successfully completed script task I. The mean completion time of the task was 39.24 s, ranging from 21 to 71 s. One participant was unable to pick up the object needed to carry out the task and did not complete it.

Poststudy system usability questionnaire (PSSUQ)

The PSSUQ results for each item are presented in Table 3. According to the mean total score, participants were generally satisfied with the usability of the VR platform (2.72 ± 1.92). The System Usefulness Subscale presented the lowest mean score (1.76 ± 1.37), ranging from 1.35 to 1.88, indicating the most satisfactory score. Conversely, the Information Quality Subscale presented the highest mean score (3.00 ± 1.95), ranging from 2.07 to 2.46, indicating the most unfavourable rating. The Quality Interface Subscale also presented relatively low mean scores (2.05 ± 2.02), ranging from 1.88 to 2.07.

Table 3 PSSUQ results (%) for each item.
Full size table

Variables that may impact usability

Regarding the mean PSSUQ scores, participants with secondary education had significantly greater scores than participants with other education levels (p =.011) (see Table 4). No significant differences in mean PSSUQ scores were found between gender and previous experience with VR (Table 4).

Table 4 Comparisons of mean PSSUQ scores based on sociodemographic variables.
Full size table

No significant differences on mean completion times between gender, education level and experience with VR were observed (see Table 5).

Table 5 Mean completion times by sociodemographic variables.
Full size table

Survey

Of the 17 participants, 94.1% (n = 16) considered the platform intuitive. Additionally, all participants (100%) mentioned that navigation was easy and tasks were simple to select and complete. When asked about difficulties in performing tasks, 88.2% (n = 15) reported none, while 11.8% (n = 2) experienced difficulties. Of the two participants who encountered difficulties, one experienced initial difficulty in understanding the controls (“did not know right away how to use the controllers”), while the other had difficulties pressing the button with the hammer in task 4 (“With task 4, if you press the button with the hammer, sometimes it does not work. Overall, it was a nice experience”).

To improve the platform’s usability, participants were asked to provide suggestions for improvement (Question 6). The most relevant recommendations included increasing the visibility of instructions in task 3 to better indicate how to avoid the ball, allowing users to practice two or more tasks beforehand to familiarize themselves with the required actions, and incorporating an arrow within the environment to indicate when an object needs to be picked up.

Discussion

The main objective of the present study was to describe the development and test the usability of the VR platform incorporated in the VRainSUD cognitive training program. Participants were observed while completing the scripted tasks, with their completion times, difficulties and/or comments systematically recorded. Afterwards, participants completed a survey that included the PSSUQ and open-ended questions, allowing them to provide qualitative feedback and suggestions. Both the quantitative and qualitative results indicated that the VR platform demonstrated high usability during testing.

Regarding observation, participants did not show difficulties completing most of the script tasks focused on general platform navigation (i.e., using manual controllers, navigating the main menu, entering and leaving cognitive tasks, pausing, or stopping a cognitive task, and returning to the main menu). Nonetheless, some cognitive tasks required more instructions than expected and suffered alterations following the usability test, including the addition of information at the start of each task.

On cognitive task 1.1. (Remembering Space) and cognitive task 3 (Passing Balls), participants did not intuitively understand that they had to look down to see and pick up the necessary object to complete the task (a hammer in cognitive task 1.1. and a racket in cognitive task 3). Likewise, some participants did not intuitively understand the objective of cognitive task 2 (number sequences) and therefore struggled to complete it without additional verbal instructions. Cognitive task 5 (working maze) presented the highest variability of results, with some participants completing it easily and others struggling to understand the verbal instructions and taking much longer than expected. It became clear that the marked arrows on the floor of the environment were not enough to adequately inform the participants when to turn and follow the initially memorized sequence of movements.

Female participants had shorter completion times averaging 39.27 s less than male participants. Although this difference was not statistically significant, it aligns with existing literature indicating that women often perform better on visual working memory tasks26.

The quantitative results (PSSUQ) support what was recorded through observation. Most participants considered the platform to be intuitive and easy to use (system usefulness) and to enjoy the graphic interface (interface quality). However, when asked if the VR platform included all the expected functions and capabilities, some participants deemed the question not applicable. This finding may reflect the limited experience most participants had with VR platforms in general, or with cognitive training in particular.

The dimension of the platform with the lowest scores was the information provided within it (information quality). A considerable number of participants (59%) considered that the information (such as on-screen messages) was not clear, although 71% still considered that the information provided was easy to find and understand. These results are consistent with normative data from the PSSUQ. For all three subscales, the Information Quality Subscale frequently presents the highest mean (least satisfactory) score20. For a novel technology such as VR, it is possible that this disparity is even more marked, as the lack of experience may result in higher informational demands. These findings were crucial for improving the platform. Following the usability test, we not only revised the already included information to make it more accessible but also incorporated on-screen instructions based on participants’ specific feedback.

The mean PSSUQ score was impacted by the educational level. Participants with secondary education presented significantly higher (less satisfactory) scores than participants with other levels of education. These results may be explained by a confounding variable not controlled during the study. The mean PSSUQ score was also impacted by the previous experience with VR. Those without previous experience with VR presented higher (less satisfactory) scores, although the difference was not statistically significant.

The participants’ level of familiarity with VR technology was also a contributing factor in their qualitative feedback and suggestions. The two participants who reported having had difficulties performing the script tasks mentioned not knowing how to use the manual controllers or struggling to perform a specific command with them. In the given suggestions, beyond referring to the quality of the on-screen information/instructions, participants considered that it would be useful to be able to perform a trial at the beginning of the cognitive tasks so that they could better understand what they were being asked to do. This feedback was considered throughout the development of the cognitive training program. Considering the purpose of the usability tests, the participants were not given prior access to the platform or the opportunity to explore it, as suggested. However, once the cognitive training program is implemented Session 0 will be included, allowing participants to familiarize themselves with the manual controllers, explore the virtual environment, and practice the cognitive training tasks. Regardless of participants’ previous VR experience, these findings reinforce the potential of VR as a viable tool in clinical settings.

Several limitations should be considered. First, although two researchers were present during data collection, only one was responsible for taking observation notes, which may introduce observer bias. Additionally, the lack of control over factors such as medication use, treatment duration, and SUD severity may have influenced outcomes like completion times, which should be addressed in future studies. Finally, while the small sample size helped identify key usability issues, it may limit the generalizability of our findings.

Future work should aim to study of the effectiveness of VRainSUD in individuals with SUD. Additionally, subsequent research should extend the evaluation to include other populations, thereby broadening the applicability of this VR-based cognitive training program and improving its generalizability.

Conclusion

Cognitive training programs and VR have both been used as therapeutic tools. In particular, highly immersive VR has been shown to be associated with greater motivation and engagement in different activities27. VRainSUD was created to offer engaging cognitive training exercises in an immersive virtual environment.

Overall, participants considered the VR platform easy to use and useful. The usability testing led to the addition of on-screen written instructions at the beginning of certain cognitive tasks. Despite the target population’s limited familiarity with VR, participants quickly learned to use the controllers and navigate the environment. Provided that patients will have the opportunity to explore the VR platform before starting any cognitive training (i.e., Session zero), this technology can be integrated into clinical treatment without significant difficulties. Moreover, participants were actively engaged during the tests and expressed interest in having VRainSUD as an add-on intervention to the standard treatment they were receiving.