Introduction

Background

Functional electrical stimulation (FES) and exoskeletons are rehabilitation technologies used to restore functional abilities following spinal cord injury (SCI). FES is the stimulation of two or more muscles to produce a functional FES-movement pattern. When FES is off, the movement pattern is no longer there. Whereas an exoskeleton is an external device (passive or with motor) that provides a functional movement pattern. Without the exoskeleton movement is no longer there. Clinical trials may quantify the immediate benefits that people may experience while wearing exoskeletons and FES systems, as well as the therapeutic effects, that are sustained when the devices are not utilized.

Therapeutic effects of FES and exoskeletons when the devices are turned off or removed may include recovery of movement, increase in muscle strength, improvement in physical functioning and interjoint coordination, among other outcomes [1,2,3]. Although the assessment of recovery of movement and muscle strength is relatively straightforward, assessment of improvements in physical functioning in SCI is a bit more challenging due to shortcomings in existing clinical outcome assessments (COA), including the ordinal nature of most response scales, presence of both ceiling and floor effects, multi-dimensional constructs, and inability to distinguish between motor recovery and compensatory movement strategies used for function [4,5,6]. The ability of a COA to distinguish between recovery and compensation is important when assessing therapeutic effects of devices. Owing to the number of clinical trials of FES and exoskeletons that are interested in not only the immediate benefits of FES and exoskeletons while they are used but also the potential therapeutic effects they have, it is important to understand what COAs are currently used to assess therapeutic effects and the degree to which they distinguish between recovery of movement and compensatory movement strategies used for function.

Moreover, most measures of physical function utilized in clinical trials are ordinal and multidimensional. The use of ordinal scales is suboptimal for clinical trials because the differences between item scores are unknown or unequal, whereas interval scales offer equal, known, and objectively scored differences between items [7]. There is a need for a performance-based SCI clinical trial outcome measure that has the measurement properties recommended for clinical trials [7,8,9].

Study aim

The aim of this review was to examine COAs that are used to measure the therapeutic effects of exoskeletons as an initial step to determining the need for a new COA that evaluates movement and distinguishes from compensation. To our knowledge, there were no other reviews addressing our guiding questions, or similar questions.

The guiding questions were as follows:

  1. 1.

    What COAs are used in intervention studies that test the therapeutic effects of FES only, exoskeletons only, and combined FES/exoskeletons in SCI for upper and lower limbs?

  2. 2.

    What are the National Institute of Neurological Disorders and Stroke SCI Common Data Elements (NINDS CDE) recommendation levels for each COAs?

  3. 3.

    Among the COAs identified, which ones distinguish between recovery of movement and compensation strategies used for function?

Methods

Study design

An integrative review methodology was used to identify, analyze, and synthesize the results from studies. An integrative review provides a synthesis of content knowledge through a systematic process of preparing a guiding question(s), searching the literature, obtaining and analyzing content of manuscripts, discussing results, and presenting the review for application [10]. The inclusion of studies using designs beyond the randomized controlled trial provides a comprehensive understanding of the literature and captures emergent phenomena and trends in the field [10, 11].

Search strategy

This literature review was conducted by two occupational therapy (OT) doctoral students with consultation from a library scientist, using the PubMed and Cochrane Library databases. The keywords that were used within the search strategy were ā€œspinal cord injuriesā€, ā€œfunctional electrical stimulationā€ or ā€œexoskeletonā€, and ā€œtraining effectā€. The search was completed using the databasesā€™ thesaurus and relevant Medical Subject Heading terms (Fig.Ā 1).

Fig. 1
figure 1

Search Strategy: Medical Subject Heading terms.

Full size image

Two OT doctoral students completed the search and initial selection of studies using the following inclusion criteria: peer-reviewed research; English-language full-text publication; human studies; at least 50% of study participants had SCI; intervention was FES and/or exoskeleton; intervention involved repeated training with FES/Exoskeleton; use of COAs to assess therapeutic effect of FES/exoskeleton on either or both the upper and lower extremity; and studies with a primary or secondary study endpoint being motor and/or physical function. The students collaborated with a research librarian throughout the integrative review process. Studies were excluded if the interventions were not clearly defined, used electrical stimulation only as an exercise modality, and if COAs were only administered to evaluate the immediate effects of FES, exoskeleton or FES/exoskeleton and not to evaluate the therapeutic effects when the devices were off. Grey literature (dissertation, thesis), abstracts, review papers, position papers, and white papers were excluded as well. There were no limits on the date range.

As illustrated in the PRISMA flowchart (Fig.Ā 2), 1006 manuscripts were identified in PubMed (nā€‰=ā€‰744), Cochrane Library (nā€‰=ā€‰261), and by Citation Searching (nā€‰=ā€‰1) with 168 duplicate records removed. The two OT doctoral students, blinded to each otherā€™s assessment, separately performed a title and abstract screen on 838 records, then met to determine consensus on articles that did not meet agreement, resulting in 700 records failing the initial screen with 138 records advancing to full text screen (Fig.Ā 2). Of the 138 articles, 134 articles could be retrieved for the subsequent full-text screen.

Fig. 2
figure 2

PRSIMA Flowchart.

Full size image

The OT doctoral students were not blinded during the full-text screen but independently reviewed the articles, met to obtain consensus, and reviewed the data from one anotherā€™s work. During the final screen for eligibility, manuscripts were excluded if: they were a duplicate with another included manuscript; did not describe COAs that assessed motor or physical function; had poorly described interventions or if the intervention did not involve FES or exoskeleton and did not meet one or more of the inclusion criteria. A total of 56 manuscripts met the inclusion for full review and data extraction.

Analysis

The level of evidence of studies was defined according to the Oxford Centre for Evidence-Based Medicine Levels of Evidence Working Group [12]. To examine the three guiding questions for this review, COAs were organized by type of COA ā€“ namely performance-based outcome (PerfO) and patient-reported outcome (PRO) measures. PerfOs are administered by a trained rater to measure participant functions/functional ability through participant completion of standardized tasks that must be performed according to specific instructions and scored with standardized guidelines (e.g., gait speed, word recall memory test), whereas PROs are completed by the participant or by an interviewer filling in the participantā€™s responses without any changes or interpretation to better understand a participantā€™s perspective or experience (e.g., rating scales, counts of events) [13]. The COAs were further examined for level of measurement (nominal, ordinal, interval, ratio), ability to distinguish between motor recovery and compensatory motor strategies, and the NINDS SCI CDE designation. The Rehab Measures Database (RMD) and Physiopedia were used as primary resources to garner information about the measurement construct of COAs [14, 15]. If these databases did not have sufficient information, peer-reviewed manuscripts and Stroke Engine were used as supplements.

If it was unclear whether the COA discriminates between motor recovery and compensation, the following decisions adopted were: (1) patient-reported outcomes (PROs) do not differentiate; (2) performance-based outcomes (PerfOs) that only evaluated ā€œtimeā€ (e.g., to walk, to complete task) do not differentiate; (3) for PerfOs that evaluated constructs other than ā€œtimeā€, the administration guidelines, (when available) test items, and response categories were examined, and discussed until consensus was reached among the authors; and (4) if there was insufficient information to confirm the ability to differentiate between recovery and compensation, the COA was marked as ā€œnot enough informationā€.

Results

Of the 56 studies that met inclusion criteria, 16 studies were level II (randomized trial or observational study with dramatic effect), 24 studies were level III (non-randomized controlled cohort/follow-up study), and 16 studies were level IV (case-series case-control or historically controlled studies) [12]. There were 13 studies on upper limb, 42 studies on lower limb, and 1 study that focused on both the upper and lower limb (seeĀ supplementary material). Further, 31 studies involved exoskeletons, 23 studies involved FES, and 2 studies involved both FES and exoskeleton (TableĀ 1). There were 38 COAs identified across all studies, including multiple versions of the same COA that were recorded as separate measures (for example, Walking Index for Spinal Cord Injury I [WISCI I] and Walking Index for Spinal Cord Injury II [WISCI II] were considered two separate COAs) (TableĀ 1). Of these 38 COAs, 24 were PerfOs and 7 were PROs. One COA (Spinal Cord Independence Measure [SCIM]) that has been validated as a PRO and PerfO was administered as a PerfO in 2 studies and PRO in 12 studies.

Table 1 Total COAs identified across all studies (nā€‰=ā€‰38).
Full size table

Of the 38 COAs, 11 (28.95%) were used in more than one study (Fig.Ā 3). Seventeen (44.74%) of the 38 COAs were included in the NINDS SCI CDE, with only 4 designated as supplemental, highly recommended, and the remaining as supplemental or exploratory (TableĀ 2). Only 3 of the 38 (7.89%) COAs either disallows compensation (Capabilities of the Upper Extremity Test [CUE-T]), or accounts for compensation in scoring (Graded Redefined Assessment of Strength Sensibility and Prehension [GRASSP], Spinal Cord Functional Ambulation Inventory [SCI-FAI]), all ranking supplemental on the NINDS CDE (TableĀ 2).

Fig. 3: Most Frequently Used COAs (nā€‰=ā€‰11).
figure 3

Other COAs were used inā€‰ā‰¤ā€‰3 studies (out of 56 studies). ARAT action research arm test, 2MWT 2 minute walk test, JTHFT jebsen-taylor hand function test, GRASSP graded redefined assessment of strength sensibility and prehension, FIM functional independence measure, BBS berg balance scale, SCIM I-III spinal cord independence measure I-III, TUG timed up and go, WISCI I&II walking index for spinal cord injury I&II, 6MWT 6 minute walk test, 10MWT 10 meter walk test.

Full size image
Table 2 COAs recommended by the NINDS CDE.
Full size table

When considering the most frequently used COAs, the studies evaluating FES and exoskeleton systems often utilized a combination of walking tests and COAs. Studies that focused on assessing therapeutic effects of exoskeleton systems for walking tend to use a combination of walking tests and COAs (Timed Up and Go [TUG], WISCI I & II, 6-min walk test [6MWT], 10 meter walk test [10MWT]). Those studies that only studied FES systems included SCIM I-III more frequently than studies that examined exoskeleton systems or studies that examined both FES and exoskeleton.

Discussion

This integrative review described COAs used in SCI studies to evaluate therapeutic effects of FES and exoskeleton interventions. Of the 38 COAs identified across studies, the majority (23/38 or 60.5%) were PerfO. PerfOs may be more appropriate for measuring physical function in the context of SCI than PROs as they include a direct assessment of the participant performing certain standardized tasks and may distinguish recovery of function from compensation, albeit they may not measure a meaningful aspect of health or have as much relevance in real-world situations. Although PROs are often used to measure effectiveness of an intervention, they provide a subjective assessment from the participants on their experiences and functioning without interpretation from anyone else [16, 17]. PROs may not capture therapeutic effects of FES or exoskeleton on movement and movement recovery, albeit their relevance to real world experiences.

It is important to note that for studies with walking as an endpoint of interest, many use the same COA. Specifically, most studies utilized 10MWT (28/56 or 50% of studies), 6MWT (20/56 or 35.7% of studies), TUG (13/56 or 23.2% of studies) or the WISCI-II (15/56 or 26.8% of studies). Except for the WISCI, the walking tests are on a ratio level scale, and designated by NINDS CDE as supplemental, highly recommended (10MWT, 6MWT) or supplemental (TUG). The WISCI-II is also designated as supplemental, however uses an ordinal scale, where the difference between WISCI levels is unknown. COAs on an interval and ratio level of measurement are more precise because they are quantifiable with known differences between possible scores [7]. Given the endorsement of NINDS for the 10MWT, 6MWT, and TUG, it is highly recommended that they are considered when selecting COAs to assess therapeutic effects of FES and exoskeletons on walking.

Unlike walking, there are no COAs that are consistently and repeatedly used to measure endpoints of upper extremity function or physical function. SCIM (versions I-III) was used in 14 of the 56 studies (25%) and is designated as supplemental by NINDS SCI CDE but has known limitations that diminish its effectiveness to detect therapeutic effects [18,19,20]. The Jebsen-Taylor Hand Function Test (JTHFT), GRASSP, and Box and Blocks Test (BBT) were used in five (8.9%), five (8.9%), eight (14.3%) of the 56 studies, respectively, and two (JTHFT and GRASSP) are designated by NINDS CDE as supplemental. All three are narrow in scope (hand function/grasp/release) and have not been widely adopted. Recommendations for selection of COAs for measuring therapeutic effect on upper extremity and physical function are needed.

Only 44.74% of the COAs had NINDS SCI CDE designation. Four (10MWT, 2MWT, 6MWT, SCIM-III) are designated as supplemental, highly recommended and, of those, three (10MWT, 2MWT, 6MWT) evaluated gait. Given that many rehabilitation technologies aim to improve functional ability beyond gait, there is a clear need for robust COAs to evaluate endpoints other than gait in SCI. The non-gait COA with NINDS CDE designation of supplemental, highly recommend is the SCIM-III, which has known limitations, especially with respect to ceiling and floor effects [4, 21,22,23] and inability to distinguish between recovery and compensation [23, 24]. Until recently [25], the SCIM-III lacked standardized guidelines for set-up and administration, which led to variability in administration and scoring [18, 19]. This variability poses a risk to the reliability of a clinical trial and may result in an overestimation or underestimation of function [25]. Implementing the standardization guidelines [26] in studies evaluating therapeutic effects is recommended.

To determine which COAs were able to differentiate between recovery of function and compensation, the team reviewed literature to identify COAs that may distinguish recovery vs. compensation [4, 27,28,29], considered the construct, items, and response scale for each COA, and obtained team consensus. CUE-T, GRASSP, and SCI-FAI were determined as able to distinguish between recovery and compensation due to their consideration of and ability to identify compensations during their administration process and/or response scale [4, 5]. The number of COAs that can distinguish between recovery of function and compensatory movement in SCI are limited. Those that can differentiate are narrow in measurement constructs. The measurement constructs of the GRASSP and CUE-T are sensorimotor function of the hand (wrist and fingers/thumb) and upper limb function (shoulder to hand), respectively, while the SCI-FAIā€™s concept of interest is ambulation and balance. Further, the GRASSP and CUE-T use an ordinal scale with the SCI-FAI having items that are nominal, ordinal, and ratio. Given that these measurement properties are not ideal for precise measurements, these COAs have inherent limitations for measuring therapeutic effects.

Integrative reviews, although less rigorous than systematic reviews, are more comprehensive given the inclusion of studies that are not randomized controlled trials. This allowed for the inclusion of a wide variety of COAs with constructs measuring motor and physical functioning. However, the inclusion of studies with different methodologies and study designs may have led to issues with inaccuracy, bias, or rigor [30, 31].

When extracting data, we only reported on the entire COA rather than on each separate subtest/subscale. If a study only used certain domains or subtests of a COA (e.g., self-care of SCIM I-III), this nuance was not captured in our extraction. Moreover, the mode of administration reported by the studies was what was presented in the study results. For example, if a study reported the SCIM I-III mode of administration as PRO, then that is what was documented when extracting and organizing study data, even though the SCIM I-III may be administered as either PerfO or PRO. If the COA was known to have multiple modes of administration and the author did not report which mode was used, then it was marked as ā€œnot reportedā€.

To obtain information about the measurement construct or response categories for COAs, the Rehab Measures Database (RMD) and Physiopedia were used as primary resources. If these databases did not have the information, we used peer-reviewed articles or Stroke Engine. When reporting on COAs, it is imperative that studies include information about the COA, such as measurement construct and response categories, and specify how the COA was administered and scored (e.g., mode of administration, total score versus subscale scores). This is important when considering study outcomes and for study replication.

Limitations

There are several limitations of this study that may have impacted the comprehensiveness and accuracy of our search. Our initial search strategy included specific companies and brand names to include keywords that are as specific as possible to the literature we were seeking to find. Because the list of specific company names for this search strategy was not comprehensive, some manuscripts may have been missing. Further, we used the NINDS CDE designation as the primary guide to examine the COA and the REHAB Database, Physiopedia, Stroke engine and published studies to understand the measurement construct and response categories of the COAs. There are additional resources that we did not use that may have provided additional/different information about the COA. Lastly, if the team was uncertain about how to proceed with certain articles, consensus was obtained. These consensus positions may not be entirely accurate.

Implications and recommendations

FES and exoskeletons are rehabilitation technologies that provide direct functional benefits when they are used, with potential therapeutic effects when they are not used. These therapeutic effects may be due to motor recovery, compensation, or a combination of both [4, 5, 32]. Thus, when therapeutic effect is of interest in a study of FES or exoskeletons it is imperative to select COAs that can differentiate motor recovery from compensatory strategies. Of the 38 COAs used in the studies included in this review, only three (CUE-T, GRASSP, and SCI-FAI) either disallowed or accounted for compensation of movement; all are designated as supplemental within the NINDS SCI CDE structure.

The results of this review catalyzed several suggestions for the field. First, it is suggested that all SCI studies, not only ones funded by NIH, consider the NINDS SCI CDE designations when selecting COAs. Second, when reporting on COAs that are validated as both PerfO and PRO, studies should make explicit the mode of administration (e.g., PRO, PerfO) used. Finally, most of the SCI COA do not account for compensatory movement strategies, which can lead to both underestimating and overestimating the therapeutic effects of interventions that target motor recovery. When recovery of movement is a study endpoint, prudence is warranted when utilizing COAs that do not account for compensation.

Conclusion

There are four COAs that are commonly used to evaluate the therapeutic effects of FES and exoskeletons on gait; three are ratio (10MWT, 2MWT, 6MWT) and one is ordinal (WISCI-II) levels of measurement; each are designated as supplemental, highly recommended (10MWT, 2MWT, 6MWT) or supplemental (WISCI-II) by NINDS SCI CDE structure. Robust COA to measure the therapeutic effects of FES and exoskeletons on upper extremity and overall physical functioning are lacking. Those that do exist are either narrow in scope, have known limitations, and/or do not differentiate between recovery and compensation. There is an opportunity to expand our measurement toolkit with more robust and precise COAs that disallow and/or account for compensatory movement.