Background & Summary

The continuous advancement of BCI technology has sparked a growing interest in providing alternative communication solutions for individuals with speech impairments. Existing devices, such as P300 spellers and steady-state visual evoked potential (SSVEP) spellers, convert neural activity into computer commands, enabling patients to communicate. However, these technologies have several limitations, including restricted freedom of movement, lengthy training periods, the necessity for users to concentrate on specific tasks, and communication rates that are significantly slower than natural speech1,2,3. These challenges emphasize the need for more efficient and natural communication methods, leading to the development of speech-brain-computer interfaces. Speech BCIs decode neural signals associated with various modes of speech production, such as overt, silent, and imagined speech4. Overt speech BCIs capture neural activity during actual vocalization, often utilizing signals like EEG or ECoG, and may produce real-time speech or text output5,6. Silent speech BCIs focus on interpreting neural signals linked to subvocalization, typically using EEG, although MEG can also be used7,8. Imagined speech BCIs decode neural signals related to thought-based speech, which are often more complex to interpret and generally require advanced signal processing techniques, such as deep learning, to achieve accurate results9,10. Speech production involves five stages: cognitive language processing, encoding, conduction, vocalization generation, and perceptual feedback11,12. In the overt speech process, the brain’s language areas generate language content, which is translated into motor commands and sent through the nervous system to the vocal organs, resulting in sound production via airflow and vocal cord vibrations13,14. Similar brain regions are activated during silent and imagined speech processes, but speech-related signals are produced without vocalization, involving neural activity linked to internal speech or mental rehearsal of speech15,16.

The speech process involves neural signals, muscle activity, and speech generation. Decoding speech involves extracting meaningful patterns from neural or muscular signals, integrating acoustic information, and significantly enhancing decoding accuracy and robustness. In this study, we designed a specific speech paradigm and developed a publicly available dataset that includes multi-dimensional physiological signals across three speech conditions. Each mode offers unique insights into the brain’s speech processing: overt speech involves actual vocalization, silent speech focuses on subvocalization, and imagined speech investigates the neural activity underlying thought-based speech. By transferring relevant findings from overt speech research to silent speech and imagined speech, it is possible to overcome challenges related to data accuracy and significantly improve decoding performance.

Neural signals are divided into invasive and non-invasive categories. Invasive signals include electrocorticogram (ECoG) and intracranial electroencephalogram (iEEG); non-invasive signals include functional near-infrared spectroscopy (fNIRS), magnetoencephalography (MEG), and electroencephalography (EEG), etc. Invasive signals, collected via electrodes implanted on or near the cortex, offer high signal-to-noise ratios and strong spatial resolution but require surgical implantation, which presents high risks and limits their applicability17,18. In contrast, though lower in signal quality and spatial resolution, non-invasive signals are highly safe, easy to use, and non-invasive, making them attractive for practical applications19,20. EEG, in particular, stands out for its low acquisition cost, high temporal resolution, and strong portability, making it a popular choice for speech BCI research21.

Surface electromyography (sEMG) is frequently selected in speech-related research due to its non-invasive nature, ease of use, and ability to capture muscle activity involved in speech production. It is particularly valuable for monitoring facial, tongue, and throat muscle movements during overt and silent speech22,23. Despite its sensitivity to noise from surrounding tissues, advancements in signal processing techniques have greatly improved sEMG’s accuracy, making it a practical tool for speech-related technologies24,25. When combined with neural signals, such as those from EEG, sEMG provides a powerful approach to enhance speech decoding accuracy, particularly in silent speech interfaces and prosthetic speech systems26,27,28. This integration of neural and muscular data improves the precision of brain-computer interfaces (BCIs), enabling more effective communication solutions for individuals with speech impairments29,30,31. Furthermore, sEMG-based systems have demonstrated significant progress in real-time speech recognition and prosthetics, with studies reporting the development of multi-modal interfaces that utilize both sEMG and neural signals for more robust performance32,33. These advances highlight sEMG’s growing role in assistive technologies that can enhance communication for individuals with speech disorders34,35.

Speech is a conduit between neural processing and motor execution, converting linguistic cognition into audible signals. Neural activity in brain regions such as Broca and Wernicke areas generates motor commands for articulators like the tongue, lips, and vocal cords. These movements produce sound waves with distinct acoustic properties, such as frequency, amplitude, and temporal dynamics, that carry linguistic content and speaker-specific characteristics36,37,38. Critically, speech also provides real-time feedback via auditory and tactile mechanisms, allowing speakers to adjust and refine their output. For example, the auditory feedback loop ensures that the acoustic output aligns with the intended speech, enabling adaptability to noise and disruptions39. From a neuroscience perspective, speech perception and production rely on multiple functional regions of the brain cortex, with significant overlap in neural activity between overt speech, silent speech, and imagined speech40,41. During imagined speech, the planning of vocal actions primarily occurs in the premotor cortex. However, due to the absence of actual vocalization, the flow of information typically stops at the primary motor cortex42,43.

This dataset44 provides substantial advantages for advancing speech decoding research, particularly within the context of BCI systems. Combining non-invasive data acquisition with multi-modal signals is a valuable resource for exploring diverse decoding strategies. Furthermore, it focuses on Mandarin Chinese speech, offering a culturally relevant foundation for research. The dataset facilitates the integration of neural and muscular signals with acoustic features, enhancing decoding accuracy and robustness across various speech production modes. Researchers can utilize this resource to investigate the neural correlates of overt, silent, and imagined speech, thus contributing to a more nuanced understanding of speech perception and production mechanisms.

Compared to existing datasets, ours uniquely fills a critical gap by focusing on Mandarin Chinese—a tonal language under-represented in current resources. Prior datasets such as the English-centric ZuCo 2.045, DREAMER46, and DECAF47 provide high-density EEG for semantic or affective tasks, yet none address tonal encoding. Recent Chinese releases, ChineseEEG48 and its extension ChineseEEG-249, advance Mandarin coverage but remain limited to silent reading EEG without accompanying audio or electromyography. Similarly, VocalMind-sEEG50 offers Mandarin speech production conditions yet lacks sEMG and acoustic recordings, preventing a unified view of articulatory and neural dynamics.

By simultaneously integrating EEG, sEMG, and high-quality speech recordings across overt, silent, and imagined speech modes, our dataset offers a complete view of speech-related activity. The integration of muscle activity and the emphasis on imagined speech offer a new avenue for researchers to explore novel aspects of speech decoding. This dataset’s multifaceted nature enhances our understanding of speech-related neural processes. It facilitates developing and benchmarking more sophisticated algorithms for real-time speech decoding, accelerating advancements in BCI systems and speech processing research. Additionally, it provides a valuable foundation for exploring the neural encoding of tonal languages, investigating tone-related brain dynamics, improving assistive communication strategies, supporting cross-linguistic speech processing research, and contributing to data-driven neural speech decoding technology innovations.

Methods

Participants

A total of 30 participants were recruited, aged between 17 and 25 years, with an average age of 21 years, including 14 females. As our data also includes relevant data of three minors, S20, S24, and S28, we have fully explained the research purpose and data usage to their guardians separately, and obtained their written consent and signed informed consent. The data from subjects S01-S03 represent the pre-experimental phase. During this phase, the experimental paradigm was identical to the subsequent formal experiment, with only the total number of trials adjusted. The pre-experimental data from these three subjects are included in the dataset, serving as a preliminary test before formal data collection. This step was essential to assess and ensure the feasibility and reliability of the experimental setup and protocols. All participants were native Chinese speakers, one being left-handed and the others right-handed, and none had speech impairments or neurological disorders. The experiment in this study was approved by the Ethics Committee of Tianjin University, with the ethics approval number TJUE-2024-378. Partially de-identified, but data contain the participants’ real voices, and participants are permitted to share their data openly as part of the informed consent process. All participants knew the research procedure and signed the informed consent form. The information on all subjects is shown in Table 1.

Table 1 Subject information.
Full size table

Experimental setup

The experimental design includes three modes (Fig. 1a): In the overt speech experiment, participants are instructed to read the corresponding text in Mandarin Chinese at a normal pace after the prompt material appears (only once). In the silent speech experiment, participants must perform the corresponding vocalization movements without producing any sound when the prompt appears. In the imagined speech experiment, participants must silently read the text in their minds at a normal pace in Mandarin Chinese (only once), again without making any sound.

Fig. 1
figure 1

Experimental Framework. (a) Three modes of experimentation are imagined speech, silent speech, and overt speech. (b) Example of the experimental process of overt speech. (c) The data collection site, EEG, sEMG, and speech recordings were gathered from the subjects to construct a comprehensive dataset.

Full size image

The experimental materials are selected based on the General Standard Chinese Characters Table51, which consists of 8105 characters divided into three levels. Level one includes 3500 commonly used characters, meeting the basic needs of education and cultural communication. Level two comprises 3000 characters, fulfilling the general requirements of publishing, printing, dictionary compilation, and information processing. Level three includes 1605 characters, such as those used in surnames, personal names, place names, scientific and technical terms, and classical Chinese texts in primary and secondary school textbooks. Based on the clinician’s guidance and the characteristics of high-frequency use in the daily life of speech-disordered patients, 5 groups of Chinese characters (2 characters per group) were selected, totaling 10 Chinese characters. These selected characters all rank within the top 1% of the SUBTLEX-CH film subtitle corpus and cover 82.3% of basic imperative and declarative sentences in natural dialogue52. The specific materials used in the experiment and their corresponding descriptions are shown in Table 2. To reduce visual fatigue during the experiment, each material is presented with white Chinese characters centered on a gray background. The relevant code has been uploaded; see the Code availability section for detailed information.

Table 2 Materials of Mandarin Chinese and their description.
Full size table

Experimental procedure

The experimental procedures of the three speech modes are the same, and only the speech activity process is different. The experimental method is outlined using the overt speech mode as an example, as shown in Fig. 1b. Before the formal start of the experiment, participants were asked to sit in a chair and maintain a comfortable posture. After wearing an EEG cap and attaching muscle activity collection devices, they were instructed to rest for approximately 20 seconds to prepare for sensory cues. The participants were seated approximately 80 cm from the display screen. Ten materials were grouped, each subjected to 50 random, non-continuous repetitions. Participants were required to perform the corresponding speech activity following the visual cue. Subsequently, a fixation cross appeared at the center of the screen. The speech duration for each material was 1000 ms. During the experiment, participants were required to focus their attention and remain as still as possible. Each participant’s session lasted approximately 48 minutes.

To mitigate potential order effects, we adopted a block-wise design in which each experimental block contained trials from only one speech modality (imagined, silent, or overt); no intermixed trials were employed. This choice was made on pragmatic grounds: during preliminary equipment validation with two volunteers who did not participate in the main study, an intermixed-trial arrangement noticeably increased participant-reported fatigue and produced more movement-related EEG artefacts. Given the differing difficulty and mental demands of speech modes, we sequenced experiments to optimize data quality. Participants began with imagined speech, the most abstract and cognitively demanding, and then progressed to silent speech, which is less demanding but still requires focus. Overt speech, being the most natural and least taxing, was done last. Doing this lets participants adapt gradually, reducing fatigue and maintaining data quality. Participants were given the option to rest after completing one block to minimize cognitive load, which helps reduce the effects of EEG time-related correlations on the results. Consequently, the experiment was structured as three separate, modality-homogeneous blocks executed in the fixed order imagined → silent → overt. Imagined speech, the most cognitively demanding condition, was therefore collected first, when participants were maximally alert; silent and overt speech followed in ascending order of motor engagement.

EEG acquisition

The experimental setup and dataset composition are shown in Fig. 1c. The experimental interface was implemented using Psychtoolbox. EEG data were acquired with a NeuroScan SynAmps2 system, configured with 64 electrodes following the international 10–20 placement system. Electrode impedance was maintained below 5 kΩ, and data were sampled at 1000 Hz. Raw EEG recordings were collected using Curry 9.0 software and exported in the specific format (.cdt) for further analysis.

sEMG acquisition

For sEMG data collection, a NeuSen WM series wireless electromyography system with 6 channels was utilized. The grounding electrode was placed at the subject’s collarbone. Sampling was conducted at 1000 Hz. Anatomical studies of the face, neck, and speech-related nerves identified the critical muscles involved in speech production. Double-button electrodes (44*22 mm) were strategically placed on specific muscle groups, including the levator labii superioris, risorius, zygomaticus, mentalis, anterior digastric, and mylohyoid muscles. Electrode polarity and orientation were adjusted according to muscle alignment to optimize signal capture. The sEMG acquisition sensor placement area and corresponding muscle distribution are shown in Fig. 2. The raw sEMG data were exported using Neuracle acquisition software. Figure 2 was based on human facial anatomy53 and was drawn using AI tools.

Fig. 2
figure 2

The sEMG acquisition sensor placement area and corresponding muscle distribution. The distance from the center line of the nose tip quantifies the placement of channels 1–653.

Full size image

Speech recordings acquisition

Speech recordings were conducted using Adobe Audition 2023 software, a YAMAHA UR22C sound card, and a microphone. The microphone is placed approximately 20 cm directly in front of the subject’s lips, and the sound collection position faces the subject during the sound collection. The microphone was tilted downward at a 10-degree angle to minimize breath interference and adjusted to align horizontally with the corners of the mouth. The chamber walls were lined with sound-absorbing materials. The setup facilitated the capture of speech signals and monitoring of subject behavior during overt speech tasks, specifically recording vocalization onset and duration. The setup verified the absence of audible sounds for silent and imagined speech tasks. All audio data were sampled at a rate of 44100 Hz.

Multimodal Synchronization Protocol

Hardware triggering for EEG/sEMG: Visual stimulus onset events triggered TTL pulses via MATLAB’s lptwrite() function (Parallel Port: 0x3FF8). Identical event codes (1–10 corresponding to stimulus materials) were simultaneously sent to both NeuroScan SynAmps2 (DIN port) and NeuSen WM (digital input) systems using the custom TriggerBox class (see Code Availability). Hardware triggering achieved sub-millisecond synchronization precision between EEG and sEMG systems, consistent with manufacturer specifications (NeuroScan SynAmps2 DIN latency: <0.1 ms; NeuSen WM digital input latency: <0.5 ms). Actual synchronization precision was quantified by analyzing TTL pulse timing across all trials, revealing a median inter-system latency of 0.23 ms.

Audio alignment: Due to equipment constraints, speech recordings were temporally aligned post-hoc using sEMG activity as a physiological anchor. Audio segments were defined from 350 ms before detected speech onset (via spectrogram energy thresholding) to 900 ms post-stimulus, based on known neurophysiological latencies of articulatory muscle activation preceding vocalization54.

Data Records

The dataset is named T-MSPD: Tianjin University Multidimensional Speech Patterns Dataset and can be accessed at https://cstr.cn/31253.11.sciencedb.2441644. The publicly accessible data provided via the supplementary upload link is released under the data usage agreement. This dataset repository comprises four types of data: experimental materials, raw data, preprocessed EEG and sEMG, and speech recordings, along with the relevant code and instructions. This structured organization facilitates efficient data access and processing. This dataset comprises data from 30 participants, with approximately 24 hours of EEG data, 24 hours of sEMG data, and 8 hours of speech recordings in all. The overall dataset structure is depicted in Fig. 3:

  1. 1.

    Materials: Contains materials used in the speech experiment and their sources.

  2. 2.

    Raw Data: Comprises unprocessed data, including EEG data (in ‘.cdt’ format), sEMG data (in ‘data.bdf’ and ‘evt.bdf’ formats), and speech data (in ‘.wav’ format).

  3. 3.

    Preprocessed Data: Includes EEG and sEMG data that have been preprocessed and saved in ‘.set’ and corresponding ‘.fdt’ formats. Preprocessed speech data is also saved in ‘_pre.wav’ format, with ‘.mat’ files available for further post-processing. The file named ‘Subject ID_recording.mat’ contains speech data that has been manually annotated using the synchronization labels in the overt speech folder. We also provide the lexical information of each subject for different materials. We also provide spectrograms of different materials for each subject in overt speech mode.

  4. 4.

    Other Instructions: Some supplementary information, such as the dataset structure files and other auxiliary materials. The dataset structure files contain all the necessary information for the experiments, enabling other users to use this dataset efficiently.

Fig. 3
figure 3

The overall dataset structure and the composition of EEG-BIDS. (a–d) represent the directories of four folders, and (e) is the directory structure of the EEG-BIDS file, including subject-specific subfolders and metadata. (f,g) are the raw data and the data acquisition device, preprocessed and aligned data, respectively.

Full size image

Technical Validation

EEG data preprocessing

The onset of the EEG data was defined as 0 ms, with segments from −100 ms to 0 ms utilized for baseline correction. Consequently, the duration of the data spans from −100 ms to 900 ms. Preprocessing of EEG data was conducted using EEGLAB. First, irrelevant electrodes were excluded based on their spatial location. Specifically, M1 and M2 were not used as reference electrodes, and CB1 and CB2, which are associated with cerebellum localization, were excluded from the analysis. The power spectrum of each electrode signal was computed, and electrodes with energy exceeding twice the standard deviation of the average power spectrum were identified as bad channels. These were corrected using spatial interpolation, replacing their data with the average of adjacent electrodes. At the trial level, two measures were computed: the median variance of each electrode and the median deviation of each electrode from its mean within a trial. Trials in which either measure exceeded twice the standard deviation were flagged, and the union of these flagged trials was excluded. The reference for EEG data was recalculated as the average across all channels. Data were then filtered using a 0–120 Hz bandpass filter55 and a 50 Hz notch filter to remove powerline noise. Following this, data were down-sampled to 256 Hz, segmented according to labels, and subjected to baseline correction. Segments identified as bad were removed. Finally, independent component analysis (ICA) was applied to remove artifacts caused by muscle activity or unexpected movements, ensuring cleaner EEG data for subsequent analysis. As shown in Table 3, we provide the statistical results of the EEG data of 27 subjects in the formal experiment under different speech modes that were accepted and rejected (27*500*3). The average percentage of rejection trials to total trials was 3.9%, proving that our EEG data collection process is reliable.

Table 3 The statistical results of accepted and rejected EEG data trials under different speech modes.
Full size table

In this work, we focused on comparing brain activity across three speech production modes, overt, silent, and imagined, to investigate how neural dynamics vary across these modes. We did not examine feature differences between individual speech materials within each mode, as this approach allowed us to concentrate on how each speech mode engages distinct brain regions, avoiding unnecessary complexity.

EEG source localization

We applied source localization methods to reverse-engineer the origin of EEG signals, revealing the brain regions activated during different speech modes. This not only further validates the effectiveness of the EEG data but also helps us understand the brain’s working mechanisms during the execution of various speech activities. In this work, we selected the cortex surface as the source space and OpenMEEG BEM56 as the forward modeling method. The cortex surface, constructed from high-resolution MRI, provides an anatomically precise model of the brain’s cortical surface with approximately 15000–20000 source points. For forward modeling, OpenMEEG BEM uses a three-layer head model (scalp: 0.33 S/m, skull: 0.0042 S/m, brain: 0.33 S/m), accurately reflecting the electrical conductivity differences between head tissues. Finally, the sLORETA57 was employed for source localization, leveraging its standardized approach to produce unbiased and reliable cortical activation maps. Together, this methodological framework ensures high spatial accuracy and robust interpretability of brain dynamics under different speech tasks. The average source localization results of all subjects are shown in Figs. 4–6.

Fig. 4
figure 4

The temporal variation source analysis results under the overt speech mode. Taking the EEG data source reconstruction of all participants in the material ‘wo 我 (I)’ under three different speech modes as an example, we performed source localization analysis on each epoch (−100 to 900 ms) with a 100 ms sliding window, obtaining the time-varying results of the source reconstruction.

Full size image
Fig. 5
figure 5

The temporal variation source analysis results under the silent speech mode. The time window division and analysis methodology are identical to those described in Fig. 4.

Full size image
Fig. 6
figure 6

The temporal variation source analysis results under the imagined speech mode. The time window division and analysis methodology are identical to those described in Fig. 4.

Full size image

The distinct neural activation patterns observed across the three speech modes validate the reliability and richness of the dataset. In overt speech, activation progresses from motor planning regions, including the supplementary motor area (SMA), the primary motor cortex (M1), and the anterior cingulate cortex (ACC), in the early stage, to motor execution and auditory feedback areas, namely M1 and the auditory cortex (A1), in the middle stage, and finally to feedback integration regions, such as the dorsomedial prefrontal cortex (dmPFC) and the ventral prefrontal cortex (vPFC), in the late stage, reflecting a complete speech production process. Silent speech, while lacking phonation, activates the Broca area and the premotor cortex, with reduced motor and auditory cortex involvement, highlighting reliance on internal language processing. In imagined speech, cognitive control regions, including SMA and the prefrontal cortex (PFC), dominate across all stages, with minimal motor or sensory feedback activation, demonstrating the absence of overt production54. These consistent and stage-specific activation patterns across modalities confirm the robustness and potential for studying speech mechanisms in varying conditions.

EEG relative power spectrum analysis

Since the relative power of EEG data across different frequency bands reflects the brain’s functional activity and neural dynamics in specific states, after preprocessing the EEG data and extracting signals corresponding to various labels from 30 participants. Using the Yule-Walker method within the autoregressive (AR) model58, we computed the average relative power spectral density (PSD) for five commonly studied EEG frequency bands (δ: 1–4 Hz, θ: 4–8 Hz, α: 8–12 Hz, β: 12–30 Hz, γ: 30–120 Hz) across three speech modes: overt, silent, and imagined. The average relative power topography of the five EEG frequency bands at the Cz electrode under different speech modes is shown in Fig. 7. It can be observed that overt speech and silent speech primarily activate regions in the left frontal lobe, parietal lobe, and motor-related areas. In contrast, imagined speech predominantly activates areas in the medial temporal lobe, bilateral inferior frontal gyrus, right superior frontal gyrus, and anterior cingulate gyrus59. These regions correspond to Broca and Wernicke areas associated with speech activity60.

Fig. 7
figure 7

Topographic maps of EEG relative power in different frequency bands during three speech modes: (a) overt speech, (b) silent speech, and (c) imagined speech. The color bar indicates the relative power as a percentage, representing the contribution of each frequency band’s power to the total power across all frequency bands.

Full size image

Additionally, activity in the delta band gradually decreases across all three speech modes, while activity in the theta, alpha, and beta bands increases, indicating that the frequency bands closely related to imagined speech are concentrated in the 4–30 Hz range. This conclusion is also consistent with the findings of previous studies55. We verified the close correlation between the collected EEG data and speech activity through classic EEG data feature analysis methods. This conclusion provides a reliable data basis for future research, allowing researchers to further explore the specific features of different speech modes according to their own needs.

EEG data classification verification

In this work, we further assessed the reliability of the EEG data from a classification perspective. Specifically, we constructed and trained a classification model using the EEGNet61, which effectively extracts the spatiotemporal features of EEG signals through CNN (Fig. 8a), thereby validating the stability and effectiveness of the data. We divided the preprocessed EEG data from all participants into training, validation, and test sets in a 6:2:2 ratio. To improve the stability of model evaluation, we used 5-fold cross-validation during the classification process. The classification results for the 30 participants are shown in Fig. 8b, with an overall average accuracy of 45.73%. Although the classification accuracy for some participants was lower than the average, it was still significantly higher than the 10% random guessing probability, indicating that the classification model effectively distinguishes differences in EEG signals. Furthermore, the performance evaluation results of the classification model provided strong support for the reliability of the EEG data, ensuring the credibility of subsequent analyses and applications.

Fig. 8
figure 8

The EEGNet architecture and the results of classification. (a) The overall visualization of the EEGNet architecture. (b) The classification accuracy and average accuracy of all participants in the overt speech mode using EEGNet on EEG data.

Full size image

sEMG data preprocessing

The collected sEMG data underwent basic preprocessing, including removing irrelevant electrodes (channel 7 and channel 8), 1–300 Hz bandpass filtering, and 50 Hz notch filtering. In overt and silent speech modes, the sEMG data labels fully correspond to the EEG data labels. Therefore, we also set the data duration to −100 ms to 900 ms for each segment.

Mean absolute value of sEMG

To evaluate the reliability of sEMG data, we selected the Mean Absolute Value (MAV) of the EMG signal as a key time-domain feature. MAV effectively reflects the overall amplitude level of the signal, offering advantages such as simplicity in computation and strong robustness. Additionally, it demonstrates excellent discriminative capability in various motion pattern classification tasks. After data preprocessing, the sEMG signals were segmented into fixed-length, non-overlapping windows. The MAV was calculated based on Eq. 1 for each segment, generating feature vectors. Here, x[n] represents the signal amplitude at the n-th sampling point, and N denotes the length of the signal segment:

$${\rm{MAV}}=\frac{1}{N}\mathop{\sum }\limits_{n=1}^{N}|{x}[n]|$$
(1)

We focused on overt speech due to its clear articulatory sEMG signals and well-defined temporal structure, which allows precise trial alignment and ensures that variability in the sEMG data is primarily due to the recording system. As shown in Fig. 9, for the same subject, the sEMG recorded multiple times using the same material exhibited a high level of consistency. The statistical analysis revealed no significant differences between the trials, thereby strongly supporting the reliability of the sEMG data.

Fig. 9
figure 9

The sEMG segments from the same material in overt speech modes. (a) Preprocessed sEMG data of S07 for the material ‘不 bu(no)’. (b) MAV values of the same material in different channels under different trials. (c) Statistical analysis results. X-axis: Trial Index; Y-axis: MAV (μV).

Full size image

To assess the classification performance of sEMG data during overt speech tasks, we applied EEGNet using the same methodology for the EEG data. The classification results for 30 participants are shown in Fig. 10, with an overall accuracy of 71%. Despite some participants having accuracy below the cohort average, the results were still significantly above the 10% random guessing probability, demonstrating the model’s ability to distinguish differences in sEMG signals. These results strongly support the reliability of the sEMG data, confirming its suitability for subsequent classification tasks.

Fig. 10
figure 10

The classification accuracy and average accuracy of all participants in the overt speech mode using EEGNet on EMG data.

Full size image

Raw speech recording data evaluation

The speech recording data was intercepted, removing the irrelevant parts before and after the experiment. We comprehensively assessed speech quality by evaluating multiple objective metrics, including Signal-to-Noise Ratio (SNR), Energy Contrast, and Spectral Noise Proportion. The audio was segmented into 1-second frames, and the Short-Time Energy (STE) was computed for each frame. Signal energy was assumed to be the energy of the current frame, and noise energy was estimated using silent frames with low energy. The SNR was calculated as the ratio of signal energy to noise energy in decibels (dB), with the result normalized to a range of [0, 1] by assuming an ideal SNR of 30 dB. Energy Contrast was determined by calculating the logarithmic ratio of the mean energy over all frames to the noise energy, with an ideal energy contrast of 10 dB. This score was normalized to [0, 1]. Spectral Noise Proportion was assessed by calculating the energy ratio between high-frequency (>4000 Hz) and total spectral energy, with a lower ratio indicating cleaner speech. This value was inverted to provide a higher score for cleaner speech. The final speech quality score was derived by aggregating these metrics using weighted summation, with weights of 0.5 for SNR, 0.3 for energy contrast, and 0.2 for spectral noise proportion. The score was then normalized to a range of [0, 1], where higher scores indicate better quality. The results were visualized in a radar plot, where the normalized scores for SNR, energy contrast, and spectral noise proportion were represented on the axes, providing an intuitive way to assess speech quality across different dimensions. The quality assessment scores of the raw speech data for all participants are summarized in Table 4. As an example, the visualized results of the average speech quality assessment for S30 are presented in Fig. 11.

Table 4 Multidimensional quality assessment scores of raw speech recording data for each subject.
Full size table
Fig. 11
figure 11

Visualization of speech features and quality assessment scores for S01. (a) Waveform: Displays the original waveform of the speech signal, reflecting the changes in audio data over time. (b) Short-time energy map: Displays the short-time energy changes of the speech signal in different periods. (c) Spectrum: Displays the intensity distribution of the speech signal at different frequencies and times. (d) Radar chart: Visualizes the relative performance of the three indicators of signal-to-noise ratio, energy contrast, and spectrum-to-noise ratio. The higher the score, the farther the point of the radar chart is from the center, indicating that the speech quality is better. The score range of the radar chart is 0 to 1, where 1 indicates ideal speech quality, and 0 indicates poor speech quality.

Full size image

We combined both time-domain and frequency-domain metrics to provide a detailed and comprehensive assessment of speech quality. The weighting for each metric was carefully assigned based on its relevance to overall speech quality, with a higher weight given to the SNR due to its direct impact on speech clarity. At the same time, energy contrast and spectral noise proportion were also included to account for variations in signal strength and spectral characteristics. This approach ensures a balanced and robust evaluation. Additionally, the code for speech quality assessment has been uploaded with the dataset for transparency and reproducibility.

Label and temporal alignment of multimodal data

In this dataset, we provide multimodal neurophysiological data across three speech modes: overt, silent, and imagined. These data are initially analyzed independently to explore the similarities and differences in the neural mechanisms underlying each mode. Including overt speech data is particularly valuable in addressing the data scarcity challenge when training silent and imagined speech models, thereby advancing transfer learning in speech decoding. Although speech data is collected synchronously with EEG and sEMG signals during overt speech production, automatic annotation of the speech data is not feasible.

We developed an efficient method for aligning data labels to tackle this challenge. The speech data is first segmented based on the experiment’s start and end times during preliminary preprocessing. It is crucial to note that muscle activity typically precedes sound onset in speech production, reflecting the neuromuscular transmission process involved in the formation of sound 57]. Relying solely on spectrogram-based segmentation may introduce alignment errors. To address this, we visually examined the EMG data from the same channels across all participants, segmented by different labels. The significant changes in sEMG signal amplitude between 300 and 400 ms are depicted in Fig. 12 for all six channels, providing a physiological anchor point for temporal alignment. To provide a more detailed illustration, we have selected the sEMG data from Channel 1 for further analysis. Based on this neurophysiological evidence, we implemented the alignment formula:

$${{\rm{audio}}}_{{\rm{start}}}={{\rm{sEMG}}}_{{{\rm{onset}}}_{{\rm{time}}}}-350$$
(2)
Fig. 12
figure 12

Single-trial sEMG data for different materials across all participants. (Take the result of channel 1 as an example.).

Full size image

Which leverages the consistent sequence of articulator movement → vocalization. Validation against manual annotation showed this method achieves a mean absolute alignment error of 42 ± 18 ms across subjects. To ensure precise alignment between speech recordings, EEG, and sEMG data, we adjusted for this time lag by setting the beginning of each speech segment 350 ms before the first appearance of the spectrogram. This approach guarantees that both the initial speech and subsequent segments remain accurately synchronized with the EEG and sEMG data. This method facilitates precise alignment for analyzing speech production and ensures that the multimodal data—EEG, sEMG, and speech recording—can be used together as complementary inputs in classification and decoding studies.

Next, the preprocessed speech data is divided into trials, enabling precise extraction of experiment-related speech segments. Finally, these speech segments are aligned with the labels using the synchronized markers from the EEG and sEMG signals. This allowed us to identify the corresponding segments of the speech data accurately. At the same time, we align the three types of data according to the labels, as shown in Fig. 13. Manual segmentation and the addition of synchronized labels further refined the alignment, ensuring the completeness and accuracy of the speech data and providing precise temporal alignment of multimodal data. This provides a solid foundation for subsequent multimodal analysis and the development of speech decoding models.

Fig. 13
figure 13

Label alignment data. (a–c) represent the alignment data of EEG, sEMG, and speech recording, respectively. The alignment process ensures precise temporal correspondence across modalities.

Full size image

Usage Notes

The experiment and data analysis codes have been released together with the dataset. The repository contains four categories of resources: experimental materials, raw data, pre-processed EEG and sEMG, and speech recordings, along with the corresponding code and documentation. Because speech recordings are inherently sensitive and can directly identify individuals, the dataset is distributed under a restricted-access model that requires prior acceptance of the data usage agreement. To apply, users should register an account at Science Data Bank (https://cstr.cn/31253.11.sciencedb.24416)44, click the “Request Access” button, fill in the affiliation and project details, and submit the request; approval is normally granted within 5–7 business days. The data usage agreement permits the use of data for scientific research and technical development (including non-commercial and commercial purposes), prohibits any attempt to re-identify participants or redistribute the data without written permission from the dataset authors, and is publicly available on the dataset page before application.