Method for EEG signal recognition based on multi-domain feature fusion and optimization of multi-kernel extreme learning machine

Abstract

In response to the current issues of one-sided effective feature extraction and low classification accuracy in multi-class motor imagery recognition, this study proposes an Electroencephalogram (EEG) signal recognition method based on multi-domain feature fusion and optimized multi-kernel extreme learning machine. Firstly, the EEG signals are preprocessed using the Improved Comprehensive Ensemble Empirical Mode Decomposition (ICEEMD) algorithm combined with the Pearson correlation coefficient to eliminate noise and interference. Secondly, multivariate autoregressive (MVAR) model, wavelet packet decomposition, and Riemannian geometry methods are used to extract features from the time domain, frequency domain, and spatial domain, respectively, to construct a joint time-frequency-space feature vector. Subsequently, kernel principal component analysis (KPCA) is employed to fuse and reduce the dimensionality of the joint features, resulting in a reduced-dimensional fused feature vector. Finally, these feature vectors are input into a Radius-incorporated multi-kernel extreme learning machine (RIO-MKELM) for classification. The experimental results indicate that through multi-domain feature fusion and the incorporation of radius in a multi-kernel extreme learning machine, feature selection can be performed more effectively, eliminating redundant or irrelevant features and retaining the most useful information for classification. This approach enhances classification accuracy and other evaluation metrics, with the final classification accuracy reaching 95.49%, sensitivity at 97.88%, specificity at 98.12%, recall at 97.88%, and F1 Score at 96.67%. The findings of this study are of significant importance for the development and practical application of brain-computer interface (BCI) systems.

Introduction

Motor Imagery (MI) is a psychological process that involves the mental simulation or imagination of a movement without actually performing it. In the fields of neuroscience and BCI, motor imagery is used as a method to generate brain activities that can be detected and interpreted by external devices. However, due to the complexity of EEG signals and inter-individual differences, extracting effective motor imagery features has become key to improving the performance of BCI systems.

In order to improve the classification accuracy of BCI systems, researchers have proposed various feature extraction methods and classification algorithms. For example, Xue et al.¹ used the MVAR model as a time-domain feature and achieved a classification accuracy of 86.4% using a radial basis function neural network classifier; Jin et al.² extracted frequency-domain features through bicoherence analysis and achieved a classification accuracy of 83.6% using a SVM classifier; Ang et al.³ used the FBCSP-based method to extract spatial features and achieved a classification accuracy of 83.3% using an SVM classifier.

Through the studies of the above literature, they were found that most researchers extract features from only a single domain, such as the time domain, frequency domain, or spatial domain, and then use different classifiers to classify the extracted features. However, feature extraction from different domains has its own advantages. Whether combining the strengths of features from each domain can improve the recognition accuracy of motor imagery signals is a question worth studying. Therefore, Li et al.⁴ proposed a multi-domain feature fusion method that combines wavelet packet energy and hierarchical fuzzy entropy, and achieved a classification accuracy of 88.26% using an SVM classifier, verifying that the multi-domain feature fusion method has better classification performances. Cai et al.⁵ extracted linear and nonlinear features from EEG signals of each mode and used linear combination techniques to fuse EEG features of different modes, constructing a global feature vector, and achieved a classification accuracy of 86.98% using a KNN classifier. Liu et al.⁶ extracted multiple features from the frequency and time domains, and used sparse representation methods for feature fusion, achieving a classification accuracy of 90.875% with the Sparse Representation based Discriminant Analysis classifier ( SRDA ) classifier. These studies have already proven that multi-domain features are beneficial for more accurate classification and recognition.

In the classification phase of motor imagery EEG (MI-EEG) data analysis, nonlinear classifiers and algorithms from machine learning, including K-Nearest Neighbors (KNN), Linear Discriminant Analysis (LDA), and Support Vector Machines (SVM), have seen extensive application. Recently, neural network models have shown superior performance in handling nonlinearity and generalization compared to conventional machine learning approaches. Extreme Learning Machine (ELM)⁷ represents an efficient algorithm for single-layer feedforward neural networks, capable of expediting the learning process for nonlinear challenges by means of randomly assigning parameters to hidden layer nodes and simultaneously calculating the weights of the output layer. Nevertheless, conventional ELM implementations may grapple with overfitting issues stemming from the immutable nature of the parameters connecting the input and hidden layers. To combat this flaw, Iosifidis et al.⁸ introduced the Kernel Extreme Learning Machine (KELM) as a solution. To avoid directly setting the hidden layer parameters, KELM used kernel tricks to map input data into a high-dimensional feature space, thereby improving the generalization capability of the model. However, for more complex tasks, a single kernel may not be sufficient to capture the multi-dimensional features of the data. Therefore, inspired by Multiple Kernel Learning (MKL), Liu et al.⁹ proposed the Multi-Kernel Extreme Learning Machine (MKELM), which enhances the model’s generalization capability and performance through automatic learning and optimization of the combination of kernel functions. The hyperparameters in MKELM include kernel parameters and kernel weight coefficients, and the selection of these parameters has a direct impact on the performance of the model¹⁰. Therefore, it is necessary to use optimization algorithms to find the optimal configuration of hyperparameters, such as Grey Wolf Optimization (GWO)¹¹, Whale Optimization Algorithm (WOA)¹², Harris Hawk Optimization¹³, and other heuristic learning algorithms that have shown strong optimization capabilities in recent years. The goal of this study is to further explore the application of MKELM in EEG signal recognition and to enhance the model’s performance through multi-domain feature fusion and hyperparameter optimization.

As BCI technology advances, there is a growing need to move beyond basic left-right hand motor imagery tasks to more sophisticated control of individual joints. The classification of single joint motor imagery (MI) tasks is particularly daunting due to the less distinct event-related desynchronization/synchronization (ERD/ERS)¹⁴ patterns they generate. To enhance the accuracy of classifying multiple tasks involving single joint MI, and considering the richer brain information afforded by multi-domain feature fusion combined with the superior denoising and classification capabilities of the MKELM, this investigation employs multi-domain feature fusion and a RIO-MKELM to analyze multi-task MI-single joint EEG signal maps. The objective is to improve the classification accuracy of MI-EEG signals and enhance the usability of the system. During preprocessing, the Pearson correlation coefficient is utilized to sift through the Intrinsic Mode Functions (IMFs) derived from Improved Complete Ensemble Empirical Mode Decomposition (ICEEMD), retaining components with strong correlations to reconstruct the EEG signal. Furthermore, to ensure the robustness of the classification model, an algorithm that incorporates radius information is applied to optimize its hyperparameters. This study offers a structured approach to EEG signal classification within BCI systems, and by contrasting the methodologies employed here with other approaches, the significance of this research is underscored, contributing valuable insights to the field of single joint multi-task MI research.

The contributions of this paper include the following points: First, an improved Comprehensive Ensemble Empirical Mode Decomposition (ICEEMD) algorithm combined with the Pearson correlation coefficient is used for preprocessing EEG signals. After preprocessing, the accuracy rate of motor imagery recognition is improved by 14.07% compared to EMD, providing cleaner data for subsequent feature extraction. Second, a multi-domain feature extraction method: A multi-domain feature extraction method that combines time-domain MVAR model, frequency-domain wavelet packet decomposition, and spatial-domain Riemannian geometry is proposed. The average classification accuracy of the feature vectors constructed by this method is improved by at least 2.16%. Third, feature fusion and dimensionality reduction technology: Kernel Principal Component Analysis (KPCA) is used to fuse and reduce the dimensionality of multi-domain features. The dimensionality after feature fusion is reduced by 88.1%, while maintaining a classification accuracy of over 95%. Fourth, an optimized multi-kernel extreme learning machine classifier: The Radius-Incorporated Multi-Kernel Extreme Learning Machine (RIO-MKELM) is introduced for classification. On the test set, the classification accuracy of RIO-MKELM reaches 95.49%, which is at least 3.51% higher than other optimized extreme learning machines.

The subsequent chapters of this paper are organized as follows: Sect. 2 provides a detailed account of the data collection experiment, including the design and execution of the experimental process. Section 3 outlines the system framework of this study, with a comprehensive introduction to the data preprocessing, time-domain, frequency-domain, and spatial-domain feature extraction algorithms, as well as the strategies for feature fusion and dimensionality reduction, and the classification recognition methods. Section 4 reports the experimental results and explains the theoretical and practical rationale behind the selection of algorithm parameters. Section 5 engages in an in-depth discussion and comparative analysis of the experimental results, aiming to highlight the effectiveness and performance differences among various methods. Section 6 concludes the study, offering insights into potential future research directions and discussing the application prospects and limitations of this work.

Data collection experiment

In this study, a total of 6 right-handed subjects participated in the experiment, aiming to explore the recognition capability of our method for different motor imagery (MI) tasks. The experiment was designed to include three types of motor imagery tasks: shoulder abduction, extension, and flexion. All subjects were in good health, ensured adequate sleep before the experiment, and were prohibited from taking medication or consuming stimulating beverages such as alcohol, tea, coffee, etc., within the 48 h prior to the experiment. All participants were informed of the purpose of the study and volunteered to participate after understanding the details. Everyone signed an informed consent form before the experiment, and the experiment was approved by the Research Ethics and Technical Safety Committee of Northeast Electric Power University.

In this experiment, the Emotiv Epoc + device was used to collect EEG signals for motor imagery. The device consists of 14 electrode channels (AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, AF4) and 2 reference electrodes (CMS, DRL). The signal sampling frequency is 128 Hz, and the electrode placement follows the international 10–20 system.

During the experiment, the subjects naturally placed their hands, maintained relaxation, and tried to minimize body or head movements. They performed motor imagery in response to external cues. The experiment collected EEG data for shoulder abduction, extension, and flexion motor imagery tasks. Each subject conducted 60 sets of experiments, with 20 sets for each type of motor imagery task. A schematic of the experimental procedure is shown in Fig. 1, and the experimental setup and details are shown in Fig. 2.

Related studies¹⁵ have shown that the brain areas associated with limb motor imagery are mainly concentrated around the four channels F3, F4, FC5, and FC6, which are located in the central area of the brain. Therefore, this paper uses the EEG signals from these four channels for the recognition of the three types of motor imagery tasks.

Fig. 1

Schematic Diagram of the Experimental Process.

Fig. 2

Specific Experimental Scenarios.

Methodology

This study proposes an EEG signal recognition method based on multi-domain feature fusion and RIO-MKELM. Figure 3 shows the overall framework of this method. The method can be summarised into four steps: Firstly, preprocess the EEG signals to remove noise and artifacts. Secondly, use the multivariate autoregressive model, wavelet packet decomposition, and Riemannian geometry methods to extract features from the time domain, frequency domain, and spatial domain. Thirdly, employ kernel principal component analysis (KPCA) to reduce the dimensionality of the feature matrix and generate a fused feature vector after dimensionality reduction. Finally, these feature vectors are input into the RIO-MKELM for classification, to achieve the recognition of EEG signals.

Fig. 3

Architecture of a Multi-Class Motor Imagery Recognition Model Based on Multi-Domain Feature Fusion.

Preprocessing

An 8th-order Butterworth filter is used to filter the raw MI-EEG signals from 4 channels to a frequency range of 8–30 Hz. The 8th-order Butterworth filter can effectively remove various types of noise and interference, thereby enhancing the signal-to-noise ratio of the EEG signals. Furthermore, the common average reference (CAR) is used to filter out noise from the subjects’ raw EEG signals. The MI-EEG after CAR processing for each channel can be expressed as:

$$x_{i}^{{CAR}}(t)={x_i}(t) - \frac{1}{4}\sum\limits_{{j=1}}^{4} {{x_j}(t)}$$

(1)

In the formula, ${x_i}(t)$ is the original MI-EEG of the ith channel, and c is the number of channels.

The purpose of applying baseline removal¹⁶ in electroencephalogram (EEG) signals is to eliminate non-cerebral slow-varying components caused by electrode drift, skin electrical activity, environmental noise, etc., thereby analyzing brain electrical activity more accurately. The following are the specific steps of baseline removal in EEG signals:

Determining the Baseline Period: Select a period of relatively stable signal without obvious brain electrical activity as the baseline period. This is usually at the beginning of the experiment or a period of time before the stimulus event.

Calculating the Baseline Level: Average the signal within the baseline period to obtain an estimate of the baseline. Sometimes, more complex methods, such as a low-pass filter, can be used to estimate the baseline trend.

$$y_{i}^{{CAR}}(t)=x_{i}^{{CAR}}(t) - B$$

(2)

Subtracting the Baseline Value: Subtract the average value or estimated trend of the baseline period from the entire EEG signal to remove the baseline drift.

$$B=\frac{1}{{256}}\sum\limits_{{j=1}}^{{256}} {{x_j}(t)}$$

(3)

The ICEEMD algorithm¹⁷ improves the accuracy and efficiency of decomposition by introducing adaptive white noise and an improved ensemble averaging strategy. It also includes additional steps to reduce residual noise, ensuring that the decomposed IMFs are purer. The ICEEMD algorithm is used to decompose the aforementioned filtered EEG signals, obtaining the IMFs for different motor imagery tasks from all subjects.

The formula for the ICEEMD algorithm is as follows: by adding I group high-frequency white noise to $y_{i}^{{CAR}}(t)$, a new signal $y_{i} ^{{CARI}} \left( t \right)$ is generated.

$${y_i}^{{CARI}}\left( t \right)={y_i}^{{CAR}}\left( t \right)+{\beta _k}{w^I}$$

(4)

Where, $w^{I}$ is a set of white Gaussian noise variables ${\beta _k}={\varepsilon _k}std({r_k})$ is the k-th residual term, ε_k takes a constant value of ε₀, and its calculation process is the reciprocal of the signal to noise ratio of the added noise (usually set to 0.2).

Calculate the Pearson correlation coefficient between each IMF component and the original signal, and select the components with stronger correlation for signal reconstruction. The formula for screening using Pearson correlation coefficient is as follows:

$${\rho _{_{{{y_i}(t),IM{F_{ij}}}}}}=\frac{{\operatorname{cov} \left( {{y_i}(t),IM{F_{ij}}} \right)}}{{{\sigma _{_{{{y_i}(t)}}}}{\sigma _{_{{IM{F_{ij}}}}}}}}=\frac{{E\left[ {\left( {{y_i}(t) - {\mu _{_{{{y_i}(t)}}}}} \right)\left( {IM{F_{ij}} - {\mu _{_{{IM{F_{ij}}}}}}} \right)} \right]}}{{{\sigma _{{y_i}(t)}}{\sigma _{_{{IM{F_{ij}}}}}}}}$$

(5)

Where $IM{F_{ij}}$ represents the j-th IMF component of the i-th channel, $\rho _{{_{{y_{i} (t),IMF_{{ij}} }} }}$ represents the Pearson correlation coefficient between original channel ${y_i}(t)$ and $IM{F_{ij}}$, $\operatorname{cov} \left( {{y_i}(t),IM{F_{ij}}} \right)$ represents the covariance between ${y_i}(t)$ and $IM{F_{ij}}$,${\sigma _{{y_i}(t)}}$ represents the standard eviation of${y_i}(t)$, E(·) represents the mathematical expectation, and ${\mu _{_{{IM{F_{ij}}}}}}$ represents the average amplitude of $IM{F_{ij}}$.

These IMFs are considered to be components related to specific activities or patterns within the EEG signals, thereby allowing for further filtering and denoising of the EEG signals.

Time-domain feature extraction based on MVAR model

The MVAR model¹⁸ is an extension of the univariate AR model and is used to extract time-domain features from the reconstructed EEG signal matrix. In EEG analysis, the MVAR model can be used to capture the interactions and synchrony between different EEG channels. The MVAR model regards the current value of each channel as a linear combination of all channel values at previous time points, as well as the influence of possible input signals and noise. A p-th order MVAR model can be described as follows:

$$Y^{\prime} (n) = \mathop {\mathop \sum \nolimits^{} }\limits_{{i = 1}}^{p} \left[ {\begin{array}{*{20}c} {a_{{1,1}}^{\prime } (i)} & \ldots & {a_{{1,m}}^{\prime } (i)} \\ \vdots & \ddots & \vdots \\ {a_{{m,1}}^{\prime } (i)} & \cdots & {a_{{m,m}}^{\prime } (i)} \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {y_{1} (n - i)} \\ \vdots \\ {y_{m} (n - i)} \\ \end{array} } \right] + \left[ {\begin{array}{*{20}c} {e_{1} (n)} \\ \vdots \\ {e_{m} (n)} \\ \end{array} } \right]$$

(6)

The above equation can be simplified to:

$$Y^{{\prime}} (n) = \mathop {\mathop \sum \nolimits^{} }\limits_{{i = 1}}^{p} A^{{\prime}} (i)Y(n - i) + E(n)$$

(7)

Where Y(n) is a $({m^ * }1)$order column vector representing the value of a multichannel signal at moment n,${A^{^{\prime }}}(i)$ is a predictive coefficient matrix of$({m^ * }m)$, and$E(n)={({e_1}(n), \ldots ,{e_m}(n))^T}$ is the prediction error matrix.

$$E(n) = \mathop {\mathop \sum \nolimits^{} }\limits_{{i = 0}}^{p} A(i)Y(n - i)$$

(8)

Where:

$$A(0)=I,A(i)= - A^{\prime}(i)$$

(9)

$$A(i)=\left[ {\begin{array}{*{20}{c}} {{a_{1,1}}(i)}&{{a_{1,2}}(i)}& \cdots & \cdots &{{a_{1,m}}(i)} \\ \vdots & \vdots & \vdots & \vdots & \vdots \\ \vdots & \vdots & \vdots &{{a_{u,v}}(i)}& \vdots \\ \vdots & \vdots & \vdots & \vdots & \vdots \\ {{a_{m,1}}(i)}& \cdots & \cdots & \cdots &{{a_{m,m}}(i)} \end{array}} \right]$$

(10)

Using the above steps to solve the order of the MVAR model, which is denoted as p, construct a matrix with coefficients of size $({m^2} * p)$ as $[A^{\prime}(1), \ldots ,A^{\prime}(p)]$, Since this study employs 4-channel EEG signals, each coefficient matrix is then concatenated by rows to form the reconstructed signals for n participants, resulting in a 4*4*p order time-domain feature vector $\mathop {{f_{m\operatorname{var} }}}\limits^{ \to }$ as follows:

$$\mathop {f_{{m\text{var} }} }\limits^{ \to } = [a_{{1,1}}^{\prime } (1),...,a_{{1,4}}^{\prime } (1),...,a_{{4,1}}^{\prime } (1),...,a_{{4,4}}^{\prime } (1),...,a_{{1,1}}^{\prime } (p),...,a_{{1,4}}^{\prime } (p),...,a_{{4,1}}^{\prime } (p),...,a_{{4,4}}^{\prime } (p)]$$

(11)

As shown in the Fig. 4, taking the 13th experimental data of subject A’s shoulder forward movement as an example, Fig. 4 describes the relationship between the matrix coefficients and the MVAR coefficients within the 1st to 4th order (i.e., matrix coefficients ranging from 1 to 16), serving as its time-domain features.

Fig. 4

The relationship between MVAR coefficients and matrix coefficients in the 13th experimental data of subject A’s shoulder flexion.

Frequency domain feature extraction based on wavelet packet

Wavelet packet al.gorithms are a signal processing technique that extends the wavelet transform, allowing for more detailed time-frequency analysis of signals. Compared to standard wavelet transforms, wavelet packet transforms not only decompose the low-frequency part of a signal but also the high-frequency part, thereby providing a more flexible method for time-frequency localisation analysis. Wavelet packet al.gorithms are effective in extracting frequency domain features of non-stationary signals, such as electroencephalogram (EEG)¹⁹.The wavelet packet decomposition method is used to extract frequency domain features from each channel of the preprocessed EEG signals. At the i-th level, 2i sub-bands are obtained. In this paper, 4 sub-bands are used, i = 2, corresponding to the calculation of the energy spectrum energy for each frequency component:

$$f(t) = \mathop {\mathop \sum \nolimits^{} }\limits_{{j = 0}}^{{2^{i} - 1}} f_{{i,j}} (t_{j} ) = f_{{i,0}} (t_{0} ) + f_{{i,1}} (t_{1} ) + \cdots + f_{{i,2^{i} - 1}} \left( {t_{{2^{i} - 1}} } \right)$$

(12)

In the formula, $j=0,1,2, \ldots ,2i - 1,{f_{i,j}}({t_j})$ represents the reconstructed signal at node $(i,j)$ in the wavelet packet decomposition at the i-th level, and the energy spectrum of $f(t)$ is:

$$E_{{i,j}} \left( {t_{j} } \right) = \mathop \smallint \limits_{\Gamma } \left| {f_{{i,j}} \left( {t_{j} } \right)} \right|^{2} dt = \mathop {\mathop \sum \nolimits^{} }\limits_{{k = 1}}^{n} \left| {x_{{j,k}} } \right|^{2}$$

(13)

In the formula, ${E_{i,j}}({t_j})$ represents the band energy of the EEG signal $f(t)$ after wavelet packet decomposition at node $(i,j)$; ${x_{i,j}}(j=0,1,2, \ldots ,2i - 1;k=1,2 \ldots n)$ is the discrete amplitude value of the reconstructed signal ${f_{i,j}}({t_j})$; n is the number of signal sampling points.

Repeat the above steps cyclically, traversing other channels, and concatenate the elements of the frequency domain features of each channel by rows. Construct the frequency domain feature $\mathop {{f_{wpd}}}\limits^{ \to }$ based on the energy spectrum energy, and obtain the 16-order frequency domain feature vector of the reconstructed signal for a single participant as follows:

$$\mathop {f_{{wpd}} }\limits^{ \to } = \left[ {E_{{1,1}} , \ldots ,E_{{1,2i}} , \ldots ,E_{{m,1}} , \ldots ,E_{{m,2i}} } \right] = \left[ {E_{{1,1}} , \ldots ,E_{{1,4}} , \ldots ,E_{{4,1}} , \ldots ,E_{{4,4}} } \right]$$

(14)

As shown in the Fig. 5, taking the 13th experimental data of subject A’s shoulder forward movement as an example, Fig. 5 describes the wavelet packet energy of four frequency bands across four channels, serving as its frequency-domain features.

Fig. 5

The wavelet packet energy in the 13th experimental data of subject A’s shoulder flexion.

Spatial domain feature extraction based on Riemannian geometry

Riemannian manifold algorithm is a signal processing method based on geometry that uses the concepts of Riemannian geometry to analyze and process data. In EEG signal processing, the Riemannian manifold algorithm can be used to extract spatial features of the signal, especially when the data distribution of the signal is on a nonlinear manifold. This method takes into account the intrinsic geometric structure of the data, which helps to uncover complex patterns and dynamics in the signal.

Using Riemannian manifold for spatial domain feature extraction²⁰, this study employs the covariance matrix of EEG signals as the statistical feature of the data, resulting in the spatial covariance matrix of EEG signals for a single participant as:

$$P=\frac{1}{{n-1}}Y{Y^T}$$

(15)

Concatenate the covariance matrices from 20 experiments to form {P₁…P₂₀}, and calculate the Riemannian mean $\varphi$ for the 20 experiments using the following formula, which represents the Riemannian center $\varphi$ :

$$\varphi _{{i + 1}} = \varphi _{i}^{{\frac{1}{2}}} \exp \left( {\frac{1}{{20}}\mathop {\mathop \sum \nolimits^{} }\limits_{{i = 1}}^{{20}} \log \left( {\varphi _{i}^{{ - \frac{1}{2}}} P_{i} \varphi _{i}^{{ - \frac{1}{2}}} } \right)} \right)\varphi _{i}^{{\frac{1}{2}}}$$

(16)

Using the AIRM metric²¹, the 4th-order Riemannian local difference vector is calculated using the following formula to represent the deviation of a single covariance matrix from its Riemannian center:

$$\zeta \left( {{P_1},\varphi } \right)=2P_{1}^{{\frac{1}{2}}}\log \left( {P_{1}^{{ - \frac{1}{2}}}\varphi P_{1}^{{ - \frac{1}{2}}}} \right)P_{1}^{{\frac{1}{2}}}$$

(17)

That is, the spatial domain feature vector:

$\mathop {{f_{rm}}}\limits^{ \to } =\zeta \left( {{P_1},\varphi } \right)$

$$=\left[ {\begin{array}{*{20}{c}} {{\zeta _1},{\zeta _2},{\zeta _3},{\zeta _4}} \end{array}} \right]$$

(18)

As illustrated below, using the 13th experimental data of subject A’s shoulder protraction movement as a case study, Fig. 6 depicts the Riemannian geometric difference vectors for the four channels, which are considered as their spatial domain features.

Fig. 6

The Riemannian geometric difference vectors of the 13th experimental data for subject A’s shoulder flexion movement.

Dimensionality reduction of fusion features based on KPCA

Concatenate the time-domain feature vector $\mathop {{f_{m\operatorname{var} }}}\limits^{ \to }$, the frequency-domain feature vector $\mathop {{f_{wpd}}}\limits^{ \to }$, and the spatial domain feature vector $\mathop {{f_{rm}}}\limits^{ \to }$ to obtain the joint feature vector $\mathop a\limits^{ \to }$:

$$\mathop a\limits^{ \to } =[\mathop {{f_{m\operatorname{var} }}}\limits^{ \to } ,{\text{ }}\mathop {{f_{wpd}}}\limits^{ \to } ,{\text{ }}\mathop {{f_{rm}}}\limits^{ \to } ]$$

(19)

The joint feature vector $\mathop a\limits^{ \to }$ is a t-dimensional feature vector extracted from the same sample data using different signal processing methods. It is then used to construct a label vector for subsequent classification tasks.

Repeat all the aforementioned steps to extract the time-frequency-space joint feature vectors for all samples of various motor imagery tasks, constructing a time-frequency-space joint feature matrix T. Each row of the matrix represents the joint feature vector of a sample:

$$T=[\mathop {{a_1}}\limits^{ \to } ;\mathop {{a_2}}\limits^{ \to } ;{\ldots};\mathop {{a_b}}\limits^{ \to } ]$$

(20)

Perform KPCA²² on the joint feature matrix T, projecting it into a linearly separable high-dimensional space using a kernel function for KPCA. The obtained principal component coefficient matrix $Coeff$ is as follows:

$$Coeff={\left[ {\begin{array}{*{20}{c}} {k{p_{1,1}}}&{k{p_{1,2}}}&{.}&{k{p_{1,t}}} \\ {k{p_{2,1}}}&{k{p_{2,2}}}&{.}&{k{p_{2,t}}} \\ {.}&{.}&{.}&{.} \\ {k{p_{t,1}}}&{k{p_{t,2}}}&{.}&{k{p_{t,t}}} \end{array}} \right]^T}$$

(21)

During principal component analysis, the sample covariance matrix can be obtained, which is composed of eigenvalues as column vectors and arranged in descending order. This yields eigenvalues ${\lambda _1} \geqslant {\lambda _2} \geqslant \cdots \geqslant {\lambda _t}$ and their corresponding eigenvectors. The ratio ${\lambda _k}/\mathop \sum \limits_{{k=1}}^{t} {\lambda _k}$ reflects the contribution rate of the k-th principal component in the generated feature space. According to the formula for cumulative contribution rate of eigenvalue variance, the number of kernel principal components c is determined. The calculation formula is as follows:

$$E = \frac{{\mathop \sum \nolimits_{{i - 1}}^{s} \left| {\lambda _{i} } \right|}}{{\mathop \sum \nolimits_{{i - 1}}^{t} \left| {\lambda _{i} } \right|}} \ge 85\%$$

(22)

Select the number of principal components based on the principle that the cumulative contribution rate exceeds 85%. When the number of principal components is c, the cumulative contribution rate exceeds 85%, so the number of principal components is determined to be c. Select the first c principal components to construct the principal component coefficient matrix $t*c$:

$$Coef{f^{\prime}}={\left[ {\begin{array}{*{20}{c}} {k{p_{1,1}}}&{k{p_{1,2}}}& \cdots &{k{p_{1,c}}} \\ {k{p_{2,1}}}&{k{p_{2,2}}}& \cdots &{k{p_{2,c}}} \\ {\ldots}&{\ldots}&{\ldots}&{\ldots} \\ {k{p_{t,1}}}&{k{p_{t,2}}}& \cdots &{k{p_{t,c}}} \end{array}} \right]^T}$$

(23)

By projecting each sample’s constructed joint feature vector onto the principal component coefficient matrix, the reduced-dimensional fusion feature vector $\mathop {{a^{\prime}}}\limits^{ \to } \in {R^{1*c}}$ can be obtained.

$$\mathop {{a^{\prime}}}\limits^{ \to } =\mathop a\limits^{ \to } *Coef{f^{\prime}}=\mathop a\limits^{ \to } *\left[ {\begin{array}{*{20}{c}} {k{p_{1,1}}}&{k{p_{1,2}}}& \cdots &{k{p_{1,c}}} \\ {k{p_{2,1}}}&{k{p_{2,2}}}& \cdots &{k{p_{2,c}}} \\ {{\ldots}}&{{\ldots}}&{{\ldots}}&{{\ldots}} \\ {k{p_{t,1}}}&{k{p_{t,2}}}& \cdots &{k{p_{t,c}}} \end{array}} \right] \in {R^{1*c}}$$

(24)

RIO-MKELM

Radius Information Optimization (RIO) is an optimization method that takes into account the relative distances between samples. In RIO-MKELM, this optimisation method is used to adjust the parameters of the kernel function and can also be used to modify the activation function of hidden layer neurons to better capture the local structure of the data.

RIO-MKELM⁹ combines the flexibility of multi-kernel learning with the training speed of ELM, while enhancing the model’s generalization capability and prediction accuracy through radius information optimisation. This algorithm is suitable for application scenarios that require fast training and high performance of the model.

The formula for the Radius-incorporated MK-ELM is as follows:

$$\mathop {\min }\limits_{{\beta ,\xi }} \frac{1}{2}\left\| \beta \right\|_{F}^{2} + \frac{C}{2}\mathop \sum \limits_{{i = 1}}^{n} \left\| {\xi ._{i} } \right\|^{2} s.t.\beta ^{T} \phi (x_{i} ) = y_{i} - \xi _{i} ,\forall i$$

(25)

$\{ ({x_i},{y_i})\} _{{i=1}}^{n}$ is a training set, $\phi ({x_i})(i=1, \ldots ,n)$ is a feature mapping for ${X_i}$.$\beta \in {R^{|\phi ( \cdot )| * T}}$ is the output weight. $\xi \in {R^{T * n}}$ is the training error matrix on the training data.${\xi _{ \cdot i}}={[{\xi _{1i}},{\xi _{2i}},{\ldots},{\xi _{Ti}}]^T}(1 \leqslant i \leqslant n)$ is the first column of ξ.${y_i}={[0,{\ldots},0,{1_t},0,{\ldots},0]^T} \in {\{ 0,1\} ^T}$If ${X_i}$ belongs to the t-th class. n and T are the numbers of training samples and classes, respectively. C is a regularization parameter that balances the norm of the output weights and the training error matrix. $\parallel \cdot {\parallel _F}$ is the F-norm.

Input the initial parameters, including various types of kernel functions, class labels, and a regularization parameter that balances the norm of the output weights and the training error matrix. Initialize the weight parameters of the kernel combination function and the number of iterations, and calculate the radius R_p of the Maximum Envelope Ball (MEB) in the induced feature space using the kernel function. Cycle through iterations to calculate the combination parameters of the kernel functions and the types of kernel functions, updating the structural parameters $\alpha$ of ELM and the combination weights $\gamma$ of the kernel functions, with the formulas as follows:

$$\alpha ={\left( {K( \cdot , \cdot ;\gamma )+\frac{1}{C}} \right)^{ - 1}}{Y^T}$$

(26)

$${\gamma _p}=\frac{{\parallel {{\tilde {\beta }}_p}{\parallel _F}}}{{{R_p}\mathop \sum \nolimits_{{p=1}}^{m} {R_p}\parallel {{\tilde {\beta }}_p}{\parallel _F}}},\forall p,$$

(27)

Until the maximum difference between the sum function combination weights $\gamma$ is less than or equal to 10^− 4, output all the optimal parameters to obtain the best kernel extreme learning machine classification model.

Results

In this study, the computer system configuration used is as follows: the operating system is Windows 10 64-bit, equipped with an Intel^® Core i5-1155G7 processor, and 16.0 GB of random access memory (RAM). In terms of programming language, we chose MATLAB 2023a version for development and used auxiliary tools such as the EEGlab 7.0 toolbox in the data analysis process. Additionally, the device is also equipped with an RTX 3050 graphics card and a 512GB solid-state drive (SSD).

Performance of preprocessing denoising

To illustrate the efficacy of the preprocessing technique adopted in this research, it was juxtaposed against various decomposition techniques such as EMD, VMD and LMD. Figure 7(a), (b), (c) and (d) display the decomposition outcomes of these four methodologies when applied to a consistent segment of EEG data. A comparative analysis of the decomposition results depicted in these figures reveals that the preprocessing technique utilized in this study is superior in capturing subtle fluctuations and nuances, minimizing endpoint effects and mode mixing, and isolating IMFs of higher purity. In the context of EEG signal processing, the preprocessing approach demonstrated in this study demonstrates a higher degree of adaptability to variations in signal morphology and amplitude than other adaptive mode decomposition techniques, leading to a reduction in decomposition errors, an enhancement of noise attenuation, and an overall improvement in the robustness and fidelity of the decomposition outcomes.

Fig. 7

The Outcomes of These Four Methodologies When Applied to a Consistent Segment of EEG Data.

Noise reduction based on ICEEMD

The individual differences in MI-EEG signals and the variations in experimental conditions may result in inconsistent numbers of IMFs obtained through the ICEEMD algorithm across different situations. However, statistical analysis of the experimental results shows that starting from the third IMF component, the Pearson correlation coefficient with the original signal is generally below 0.2, as indicated by studies²³, suggesting a low correlation between these components and the original signal. In contrast, the Pearson correlation coefficients for the first two IMF components range between 0.6 and 1.0, showing a higher correlation.

Taking the data from 20 experiments of shoulder abduction motor imagery from the first subject as an example, Fig. 8(a), (b), (c) and (d) show three-dimensional line graphs of the Pearson correlation coefficients between the IMF components and the original data for the four electrode channels.

Fig. 8

Line Chart of Pearson Correlation Coefficients for IMF Components of Channels F3, F4, FC5, and FC6.

Selection of the order of the MVAR model

This study uses the normalized local weighted regression (LWR) algorithm to estimate the coefficients of the 1st to 10th order MVAR, and selects the optimal model order p by minimizing the Akaike Information Criterion (AIC). The smaller the AIC value, the better the model fits the data, and at the same time, the lower the complexity of the model. Through the calculation and analysis of experimental data, we found that among different orders p, the AIC value is smallest when p takes the values of 3, 4, and 5. Figure 9 illustrates the change in AIC coefficients with the order of the MVAR model for the 20th trial of shoulder extension for the second subject. However, in all experimental calculations, the proportion of cases where the AIC value is smallest when choosing p = 4 exceeds 90%. Therefore, we choose p = 4 as the final order of the MVAR model.

Fig. 9

Variation of AIC Coefficients with the Order of the MVAR Model for the 20 Trials of Shoulder Extension for Subject 2.

Performance of multi-domain feature fusion

As shown in Fig. 10, there is a higher degree of confusion between shoulder extension and shoulder abduction motor imagery. The classification accuracy of shoulder extension motor imagery is relatively high, reaching 95.49%. In addition, the feature extraction method proposed in this paper can significantly reduce the error rate of the three types of motor imagery. This indicates that through the method of multi-domain feature fusion, it is possible to extract more discriminative fusion features from the original EEG signals, thereby improving the recognition rate of various motor imagery categories.

Fig. 10

Confusion Matrix for EEG Category Based on Different Feature Extraction Methods.

Fig. 11

Flexicon’s ROC curve, Extension’s ROC curve and Abduction’s ROC curve.

As shown in the three Fig. 11(a), (b), and (c) above, the ROC curves for Flexicon, Extension, and Abduction are displayed, respectively. The X-axis represents the False Positive Rate (FPR), ranging from 0 to 1. The Y-axis represents the True Positive Rate (TPR), also known as recall or sensitivity, with the same range from 0 to 1. The actual ROC curve illustrates the performance of the classifier at different threshold settings. The closer this curve is to the top-left corner, the better the performance of the classifier. The red dots mark the coordinates of specific points. The AUC value is the area under this ROC curve and is commonly used to quantify the overall performance of the classifier. A higher AUC value indicates better classifier performance. Overall, these three graphs show a ROC curve with a high AUC value, suggesting that the classifier has a strong ability to distinguish between positive and negative cases.

Here is a table comparing the accuracy of different methods. The Table 1 lists three different methods and their combinations, and compares their accuracy on a certain task. Each method has a corresponding accuracy percentage, which is indicated in the respective row. Through these data, one can visually see the performance differences of each method and compare which single or combined method performs the best.

Table 1 The preparation rates of MVAR, Wavelet Packet, and Riemannian geometry methods, as well as the accuracy rates of their combined methods.

As shown in Table 1, the table lists the classification accuracy rates of the time-domain, frequency-domain, and spatial-domain algorithms used in this paper, as well as their combinations. Among them, when using a single domain, the average classification accuracy rates for motor imagery are 91.26%, 81.22%, and 90.24%, respectively. When using any combination of two domains, the accuracy rates are 90.24%, 92.35%, and 92.87%, respectively, which are higher than those of the single-domain accuracy rates. When using the fusion of time-frequency-spatial multi-domain features, the average accuracy rate is 95.49%, which is higher than the accuracy rate of combining two domains and significantly higher than that of a single domain. Therefore, the feature extraction method of multi-domain feature fusion can greatly improve the classification accuracy.

Fusion features constructed based on KPCA

Figure 12 illustrates the distribution of feature points after fusion and dimensionality reduction using KPCA, showing that the data points exhibit a certain regularity, are more dispersed, and the separation between classes has been significantly improved. This indicates that KPCA can effectively distinguish the principal component features of shoulder flexion, shoulder extension, and shoulder abduction, thereby enhancing the recognition effect.

Through the analysis of visualized images, it can be concluded that KPCA performs well in feature fusion and dimensionality reduction.

Fig. 12

Three-Dimensional Scatter Plot After KPCA Fusion and Dimensionality Reduction.

The results of KPCA feature fusion

To demonstrate the specific improvements of KPCA on fusion features, this study employed the Calinski-Harabasz index (CH) to evaluate the fusion features before and after processing with PCA, t-SNE and KPCA.The evaluation results, as shown in Table 2, indicate that the t-SNE, PCA and KPCA algorithms all significantly improve the Calinski-Harabasz index value of the fusion features. Among these, KPCA demonstrates the most significant improvement in the fusion features, with the largest increase in the Calinski-Harabasz index, further confirming the excellent performance of KPCA in fusion dimensionality reduction.

Table 2 Calinski-Harabasz index for all subjects’ Fusion features before and after Fusion Dimensionality reduction with t-SNE, PCA and KPCA.

Optimization of performance with incorporation of radius information optimization algorithm

In order to assess the proficiency of Radius Information Optimization (RIO) in parameter optimisation, it was benchmarked against the Grey Wolf Optimization (GWO) and the Whale Optimisation Algorithm (WOA). To mitigate any potential comparative bias and to maintain a level playing field, the evaluation utilized two distinct training datasets of varying sizes (N = 30 and N = 60). Moreover, all optimisation methods were initialised with identical parameter configurations and subjected to the same number of iterative cycles. As a case study, subject S9 was selected, and Fig. 13 (a) and (b) present the iterative performance of the MKELM model when optimised by different algorithms, using both small and large training datasets for MI-EEG tasks.

It is suggested by the analysis of Fig. 13 (a) and (b) that the RIO-MKELM approach consistently achieves more rapid convergence and lower fitness values across different training data volumes, outperforming the GWO and WOA methods. Additionally, a side-by-side comparison of these figures reveals that while expanding the training dataset size does accelerate the convergence of both GWO and WOA, the effect on the RIO is notably less pronounced. This observation suggests that variations in training data have a minimal impact on the optimization efficacy of RIO-MKELM, underscoring its stability and effectiveness in tuning MKELM parameters.

Fig. 13

Fitness change curves of three optimization algorithms with N = 30, 60.

Table 3 Corresponding hyperparameter setting values of MKELM for each trial participant.

As shown in Table 3, C represents the regularization parameter, λ₂, λ₃, and λ₄ are coefficients of the weight decay term, also known as the L₂ regularization weights. γ is associated with the Radial Basis Function (RBF) kernel and determines the width of the RBF kernel. The larger the γ, the narrower the RBF kernel; the smaller the γ, the wider the RBF kernel. σ is related to Gaussian noise, and β represents the scale factor.

The classification performance of RIO-MKELM

The extracted time, frequency, and spatial features are fused to obtain the fusion features, which are then used to train the constructed RIO-MKELM model, the selection of the base kernel for RIO-MKELM is based on the research in²⁴. Due to the small scale of the dataset collected, in order to prevent overfitting of the model, the collected data was used to train the established model multiple times. In addition, in order to select the optimal model and further improve the recognition effect of the model, five-fold cross-validation was adopted during the training process (where RIO-MKELM-Kn represents the RIO-MKELM model corresponding to each fold, n = 1,2,…,5). Figure 14 illustrates the training process of each model corresponding to each fold of cross-validation, where each model undergoes 50 rounds of iteration with all the training data. From this figure, it can be observed that after multiple rounds of training under each fold of cross-validation, the recognition effect of each model has reached 85% or more and tends to stabilize, indicating that the model has been trained successfully and possesses good recognition performance. Based on this, the structural parameters of each model corresponding to each fold are summed and averaged to enhance the robustness of the model.

Fig. 14

Average Classification Accuracy of Six Subjects.

Table 4 The accuracy, sensitivity, specificity, recall rate, and F1 score of all participants, as well as the average of all participants.

The Table 4 is a summary of the evaluation metrics including accuracy, sensitivity, specificity, recall rate, and F1 score for all participants, along with their averages. From the chart, it can be observed that the average accuracy, average sensitivity, average specificity, average recall rate, and average F1 score for the six participants are 95.49%, 97.88%, 98.12%, 97.88%, and 96.67% respectively. Taking shoulder abduction as an example, each cell in the Table 4 represents the average of 20 trials of shoulder abduction movements for each participant.

Discussion

Accuracy comparison

To validate the performance of the feature extraction method in this study compared to multi-domain feature extraction methods in other literature, this study conducted a comparison of accuracy rates. Under the condition of keeping other factors unchanged, only the multi-domain feature extraction algorithm was changed. From the comparison results in Table 5, it can be seen that the method proposed in this topic is superior in terms of accuracy compared to other multi-domain feature fusion methods. This indicates that the feature extraction method in this study has good performance in handling multi-domain data.

Table 5 Comparison of classification accuracy between the Method proposed in this paper and other Multi-feature Fusion methods.

Comparison of classifier classification

To assess the accuracy of the proposed model (MKELM) across various motor imagery tasks, it was compared with models such as KNN²⁵, LDA²⁶, ELM²⁷, SVM²⁸ and LSSVM²⁹.These comparisons are aimed at further testing the superiority of our method in the classification of MI-EEG signals.

For all experimental samples of each subject, this study used 70% of the samples for training and the remaining 30% for testing. Through the application of a suite of evaluation criteria, encompassing recognition accuracy, F1-Score and time efficiency, a thorough performance analysis of the model was conducted.

Fig. 15

Histogram of Average Recognition Accuracy and Average F1-Score for MKELM and Other Classification Models (with Error Bars).

For imbalanced datasets, the F1-Score is typically a better indicator of model performance than a single measure of accuracy. Figure 15 (a) and (b) respectively present histograms for the MKELM model and other classification models in terms of average recognition accuracy and average F1-Score, including error bars to represent statistical uncertainty. In these two figures, we can observe the performance of recognition accuracy and F1-Score for six different models.

By comparing the results in Fig. 15 (a) and (b), it is clearly observed that, in terms of recognition accuracy and F1-Score, the MKELM model significantly outperforms traditional models such as KNN, LDA, LSSVM and ELM. This result indicates that in the task of EEG signal classification, the MKELM model exhibits superior classification performance. In addition, it was found that the MKELM model exhibited the smallest fluctuations in both recognition accuracy and F1 score, indicating that the model has strong robustness and is better able to adapt to complex data distributions, thereby achieving superior performance in classification tasks.

Through comparison with traditional models such as KNN, LDA, ELM, SVM and LSSVM, MKELM has shown outstanding performance in terms of recognition accuracy and F1-Score, confirming its advantage in classification performance. Figure 16 shows the average time consumption of the six models. It can be seen from the figure that the time consumption of MKELM is relatively close to that of KNN, LDA, ELM, SVM and LSSVM. The running time of MKELM remains within an acceptable range, making it attractive for practical applications in brain-computer interface edge computing scenarios.

Fig. 16

Average Time Consumption of Six Models.

The limitations and future prospects of this study

The limitations of this paper lie in the fact that the process of feature fusion and optimization of the multi-core extreme learning machine may require considerable computational resources, especially when dealing with large amounts of data; although this method is suitable for real-time monitoring, the technical challenges of real-time processing of EEG signals and rapid response still exist. The future prospects are that we plan to reduce computational costs and complexity through optimization algorithms to ensure operation even on devices with limited resources, and utilize cloud computing and distributed computing resources to handle large-scale data. At the same time, we are committed to improving the real-time performance of EEG signal processing to meet the needs of real-time monitoring and feedback systems, researching low-latency signal processing and machine learning algorithms. We believe that through these efforts, our research results will provide more enriched information and offer a clearer background for the significance and practical application of the study.

Generalization validation

Leave-One-Out Cross-Validation (LOOCV) is a method of cross-validation, primarily used to evaluate the performance of a model on a limited dataset. In Leave-One-Out Cross-Validation, each time one sample is used as the validation set, and the rest of the samples are used as the training set. This process is repeated once for each sample, so if there are N samples, there will be N iterations of training and validation³⁰.

Leave-One-Out Cross-Validation (LOOCV) operates as follows: for each sample in the dataset, it is used as the validation set, while the remaining N-1 samples form the training set; then, the model is trained on the training set and its performance is evaluated on the validation set; this process is repeated once for each sample, and the overall performance of the model is estimated by calculating the average performance of all N evaluations. LOOCV is particularly suitable for situations where the number of samples is limited, although it is more computationally expensive than other methods such as k-fold cross-validation, as it requires training a model for each individual sample. However, since it uses almost all the data to train the model, LOOCV can provide an unbiased estimate of the model’s performance.

Therefore, in our experiment, we sequentially use one of A-F as the test set, with the remainder used as the training set. The 1–6 rounds in the Table 6 represent the results of us using one of A-F as the test set in turn, with the rest as the training set (one-to-one correspondence, A corresponds to 1, B corresponds to 2, …, 6 corresponds to F).

Table 6 The accuracy rate for each round and the standard deviation for all rounds.

As shown in the Table 6, the results range from 95.08 to 95.84%, with a standard deviation of 0.259, which is also a very small number. Therefore, the dependence of the data on the subjects can be neglected, proving that the method proposed in this paper has good generalizability.

The application of this study to healthcare

In applying the EEG signal recognition method based on multi-domain feature fusion and optimized multi-core extreme learning machines to the healthcare field, we take the following steps: First, extract multi-domain features related to motor imagery from EEG signals, including time-domain, frequency-domain, and time-frequency domain features; then, fuse these features to enhance the information content and interpretability of the signals; next, train the fused features using a multi-core extreme learning machine and improve the model’s recognition accuracy and generalization capability through optimization algorithms; finally, apply the trained model to clinical scenarios such as assistive communication, rehabilitation training, and disease diagnosis. Through these steps, the method can help healthcare providers gain a deeper understanding of patients’ brain activities, thereby offering more precise and personalized treatment options for patients.

Conclusion

This study proposes an EEG signal recognition method based on multi-domain feature fusion and optimized MKELM, which has been tested on a collected dataset of three types of shoulder joint motor imagery and compared with current classification models. The results demonstrate the method’s strong recognition ability, which is significant for advancing the practical application of single-joint MI-BCI. The contributions of this study are as follows: Firstly, a time-frequency-space multi-domain feature extraction method combining MVAR, wavelet packet, and Riemannian manifold is introduced, which not only comprehensively captures the multifaceted information of brain activity during motor imagery, including temporal dynamics, frequency energy distribution, and spatial activity patterns, but also, as evidenced by experimental data, the classification accuracy of the fusion method used is higher than that of some previous methods. Secondly, a KPCA fusion dimensionality reduction method is proposed to enhance data separability, making it easier to distinguish data points even in high-dimensional space, which is very beneficial for subsequent clustering and classification tasks. Lastly, an algorithm incorporating radius optimization is proposed to optimize the hyperparameters of MKELM, which, as compared with other algorithms under different training scales, has proven to be efficient in optimization and effectively avoids the random impact of manual selection of MKELM hyperparameters.