Introduction

Rolling bearings are common components in rotating machinery, designed to reduce friction during rotation and thereby enhance the safety of the equipment. Studies in the industrial sector indicate that approximately 40%-70% of mechanical failures are caused by bearing faults1,2,3,4. However, traditional bearing fault detection methods are time-consuming. In the current era of artificial intelligence, it is crucial to use deep learning techniques for fault detection tasks. Training an efficient and accurate bearing fault detection model can not only improve detection efficiency but also significantly reduce economic losses5.

Bearing fault diagnosis is typically based on acoustic signals and bearing vibration signals6. Bearing faults cause abnormal vibrations and sounds, making it possible to detect faults by diagnosing these anomalies. In practical detection tasks, sensors are usually installed on machine tools to capture the bearing’s sound signals.

There are various approaches to vibration fault detection based on sound signals and bearings, including methods using deep belief networks (DBN)7,8,9, support vector machine(SVM)10, convolutional neural network (CNN)11,12,13,14, deep autoencoders (DAE)15,16,17, generative adversarial networks (GAN)18,19,20, and deep transfer learning (DTL)21,22,23,24. What’s common among these models is that they differ only in the way neural networks are constructed and training strategies, while the data type remains the same. Additionally, they all require a large amount of data.

Over the past 5 years, bearing fault detection tasks have commonly used models such as Convolutional Neural Networks (CNN)11,12,13, Recurrent Neural Networks (RNN)25,26,27, and Long Short-Term Memory Networks (LSTM)28,29,30,31. Experiments have shown that lightweight models fail to achieve the desired results, with the number of model parameters affecting the final outcome. For instance, LEFE-Net32, WDCNN33, and MCNN-LSTM31 models can achieve an accuracy of over 98.50% on the CWRU34,35 dataset, but their trainable parameters exceed 50,000. In contrast, the CLFormer36 model has 4,980 parameters but an accuracy of less than 95%.

Heavyweight models are usually able to achieve higher accuracy due to having more parameters, benefiting from their stronger fitting ability. However, this also comes with a higher inference overhead. In contrast, lightweight models, although slightly lacking in accuracy, significantly reduce inference costs and are suitable for resource-constrained application scenarios. The key to whether a model can be applied in industry lies in its lightweight nature, high accuracy, and robustness.

Models based on acoustic signals typically transform time-domain signals into frequency-domain signals using techniques such as Short-Time Fourier Transform (STFT)37, Empirical Mode Decomposition (EMD)38,39, and envelope spectrum analysis. Neural network models are then built to extract features, and finally, classification is performed through fully connected layers to achieve bearing fault diagnosis. Based on this process, we have constructed a lightweight bearing fault detection model.

In this paper, first, we trained a Teacher model with a large number of parameters, and then trained a lightweight Student model using DKD40. This Student model has only one convolutional layer, one pooling layer, and one fully connected layer. The Student model, guided by the Teacher model during training, has a significantly lower number of parameters. However, we found that the accuracy of the Student model was 2% lower than that of the Teacher model. To address this, we introduced a Low-Rank Adaptation (LoRA)41 to fine-tune the Student model, which improved accuracy by 1.5% with a relatively short training time. Compared to traditional knowledge distillation and fine-tuning methods, the combination of DKD and LoRA fine-tuning ensures model performance while significantly reducing the number of training parameters.

We propose a model for industrial application in rolling bearing fault detection and contributions are fourfold:

  • We developed a lightweight rolling bearing fault detection model based on DKD, characterized by a single-layer neural network, It is compressed by 90.20% compared to the teacher model (6-layer neural network).

  • We improved the model’s performance after knowledge distillation by using a LoRA fine-tuning method, addressing the performance degradation issue.

  • Compared to other lightweight models, our approach demonstrates superior performance on the CWRU dataset, with average accuracy, precision, recall, and F1-Score all higher than those of other models. Our F1-Score reached 99.50%, and the trainable parameters of our model are only 6,838.

  • Compared to the SOTA model our model improved the F1-Score by 0.58%.

Related work

Bearing fault detection. The fault detection of rolling bearings typically involves checking whether the three components of the bearing (outer race, inner race, rolling elements) are faulty. The structure of the bearing is visualized as shown in Fig. 1. There are 10 categories of bearing sounds42, where one category represents health components, three categories represent damaged rolling elements, three categories represent damaged inner races (IR), and three categories represent damaged outer races (OR). Once abnormal vibrations occur in the bearing, it indicates a fault.

Fig. 1
figure 1

Structure of rolling bearings.

Full size image

Convolutional neural network. There has been extensive work on bearing fault detection models based on convolutional neural networks. The Adaptive Deep Convolutional Neural Network (ADCNN)43 utilizes the distribution of key information in discrete frequency bands to diagnose the health status of rotating components. The 2D LeNet-544 is an improved bearing fault detection model based on LeNet, which performs one-dimensional convolution and pooling operations directly on the raw vibration signals without any preprocessing. The lightweight multi-scale CNN45 introduces depthwise separable convolution in a multi-scale CNN to reduce the model’s storage and computational costs. The hybrid deep neural network model integrated with Principal Component Analysis (PCA)46 based model combines RNN and CNN, using bidirectional long short-term memory (BiLSTM) to enhance the extraction of time-series data features, and finally employs an Attention mechanism to improve the model’s accuracy. KDSCNN47 is a CNN model based on KD and model parameters are only 5890. BearingPGA-Net48 is a lightweight model trained with DKD and applied on an FPGA module. MCNN-LSTM31 is a model based on CNN and LSTM, which first uses CNN to extract features and then processes classification according to the time series information from LSTM. FaultNet49 and WDCNN33 are CNN models based on convolutional kernels of different scales.

While these models have achieved commendable performance, none of them offers a low-parameter model that combines high efficiency and accuracy.

Model compression. Neural network compression involves various techniques, including model pruning, parameter quantization, low-rank decomposition, knowledge distillation (KD), and lightweight model design50. Model pruning51 can reduce the accuracy and lead to irregular structures. Parameter quantization51 requires extensive training and fine-tuning. Low-rank decomposition52 involves decomposing and training layer-by-layer parameters, which can be challenging for highly complex models. KD53 is a method where a teacher model guides the training of a student model. The teacher model is typically complex, while the student model can have a simple structure, such as a single-layer network. This method achieves significant compression with minimal accuracy loss. Lightweight model design is challenging to combine with other compression methods54. If improperly combined, it can lead to lower performance and poor generalization.

All in all, KD, on the other hand, can significantly reduce model parameters while maintaining high accuracy. In fact, the algorithm mentioned in this article shows an accuracy drop of less than 2%.

DKD40 is an algorithm that addresses the inefficiency problem of traditional knowledge distillation. It enhances the knowledge distillation process by decomposing and separately optimizing different components of the knowledge transfer from the teacher model to the student model. DKD divides traditional knowledge distillation into two independent parts: Target Class Knowledge Distillation (TCKD) and Non-Target Class Knowledge Distillation (NCKD), and defines separate loss functions to optimize the model. DKD is already a mature algorithm and has been widely applied in various tasks such as large language models LLMs55, LVM56, and object detection57. It is highly efficient for model compression.

Fine-tuning. LoRA was first proposed for fine-tuning LLMs41 for downstream tasks, such as fine-tuning GPT for grammar correction58, fine-tuning Llama for fire safety training59, and fine-tuning Stable Diffusion to enhance image generation for specific tasks60. LoRA achieves this by decomposing the weight matrices into low-rank matrices. This approach significantly reduces the number of parameters that need to be fine-tuned, decreases the storage requirements of the model, and lowers the computational complexity, while enabling fast learning with less data61. As a result, both inference and training become more efficient.

In CNNs, LoRA can be used to compress the model by applying low-rank decomposition to the convolutional kernels62, thereby reducing the number of parameters to be trained. This approach reduces the model’s storage and computational costs. After fine-tuning a CNN model with LoRA, both inference speed and training time are greatly improved. Using low-rank decomposition for object detection63 enhances the model’s performance while reducing its parameters. Compared to the original CNN model, a LoRA fine-tuned model can operate under lower hardware resource conditions.

Our idea is to propose a model based on DKD training and LoRA fine-tuning. And model is not only fast in reasoning but also possesses high accuracy.

Method

CNN and classification

A CNN consists of n (\(n \in \{0, 1, 2, \ldots , n\}\)) convolutional layers and m (\(m \in \{0, 1, 2, \ldots , m\}\)) pooling layers. Convolutional layers use multiple filters to convolve over feature maps, extracting features. The extracted features are then passed through activation functions for non-linear transformations, helping to mitigate issues like vanishing gradients during training. Pooling layers are used to reduce the number of features, thus decreasing computational complexity.

The output value \(a_j^l\) of the \(j-th\) unit of the convolutional layer l is given by Eq. (1).

$$\begin{aligned} a_j^l=f(b_j^l+\sum _{i\in M_j^l} a_i^{l-1}*k_{ij}^l) \end{aligned}$$
(1)

The activation value \(a_j^l\) in the pooling layer l is given by Eq. (2).

$$\begin{aligned} a_j^l=f(b_j^l+\beta _j^l down(a_j^{l-1},M^l)) \end{aligned}$$
(2)

where \(down(*)\) represents the pooling function. Common pooling functions include average pooling, max pooling, min pooling, and stochastic pooling. \(b_j^l\) denotes the bias, \(\beta _j^l\) represents the multiplier residual, and \(M^l\) is the size of the pooling window used in the l-th layer.

After the model extracts features through convolutional layers, it needs to classify the feature data. Generally, classification tasks uses a cross-entropy (CE)64 loss as a objective function, the expression for CE is as in Eq. (3). By optimizing the cross-entropy loss, the model’s classification accuracy is improved.

$$\begin{aligned} L_{CE} = \sum ^N_{n=1}-y_{n}log(p_n),\ p_n = \frac{e^{z_n}}{\sum ^{N}_{i=1}e^{z_i}} \end{aligned}$$
(3)

where N denotes the number of categories, \(y_n\) denotes true label, \(log(p_n)\) denotes natural logarithm of the predict probability of n-th label from model, \(z_n\) is raw scores from model output of n-th label, \(p_n\) predicted probability for the n-th label.

Decoupled knowledge distillation (DKD)

Although there are various types of knowledge distillation, we use DKD for improvement our model. First, in the classical KD, the logit, \(l_i\), computed for each class is converted to a probability \(p_i\) by using the \(softmax(*)\) function as shown in Eq. (4).

$$\begin{aligned} p_{i}=\frac{\exp \left( l_{i}\right) }{\sum _{j=1}^{N} \exp \left( l_{j}\right) } \end{aligned}$$
(4)

where N denotes the number of classes.

$$\begin{aligned} b = [p_t, p_{\lnot t}] \in \mathbb {R}^{1 \times 2} \end{aligned}$$
(5)

Then, we use binary probabilities \(b = [p_t, p_{\lnot t}] \in \mathbb {R}^{1 \times 2}\) to distinguish between predictions related and unrelated to the target class, where \(p_t\) denotes the target class and \(p_{\lnot t}\) denotes the non-target class, calculated as shown Eq. (6).

$$\begin{aligned} p_{t} = \frac{\exp (l_t)}{\sum _{j=1}^N \exp (l_j)}, p_{-t} = \frac{\sum _{d=1, d \ne t}^N \exp (l_d)}{\sum _{j=1}^N \exp (l_j)}. \end{aligned}$$
(6)

Meanwhile, we use \(\tilde{p}_i = \frac{p_i}{p_t}\) to denote the probability between non-target categories (i.e., without considering the target category t) calculated as Eq. (7).

$$\begin{aligned} \tilde{p}_i = \frac{\exp (l_i)}{\sum _{j=1, j \ne i}^{N} \exp (l_j)} \end{aligned}$$
(7)

Classical KD uses KL-Divergence as the loss function, and further, we re-represent KD using the binary probability b and the non-target class \(\tilde{p}\), T and S stand for teacher and student, respectively. represented as Eq. (8).

$$\begin{aligned} \begin{alignedat}{2} \text {KD}&= \text {KL}(\textbf{p}^T \parallel \textbf{p}^S) = p^T_t \log \left( \frac{p^T_t}{p^S_t}\right) + \sum _{i=1, i \ne t}^N p^T_i \log \left( \frac{p^T_i}{p^S_i}\right) \end{alignedat} \end{aligned}$$
(8)

Simplifying, we can rewrite Eq. (8) as Eq. (9) X by Eq. (4) and Eq. (7).

$$\begin{aligned} \begin{aligned} \text {KD} =&p^T_t \log \left( \frac{p^T_t}{p^S_t}\right) + p^T_{\lnot t} \sum _{i=1, i\ne t}^{D} \tilde{p}_i^{T} \left( \log \left( \frac{\tilde{p}_i^{T}}{\tilde{p}_i^{S}}\right) \right. + \left. \log \left( \frac{p^T_{-t}}{p^S_{-t}}\right) \right) \\ =&p^T_t \log \left( \frac{p^T_t}{p^S_t}\right) + p^T_{\lnot t} \log \left( \frac{p^T_{-t}}{p^S_{-t}}\right) + p^T_{\lnot t} \sum _{i=1, i\ne t}^{D} \tilde{p}_i^{T} \log \left( \frac{\tilde{p}_i^{T}}{\tilde{p}_i^{S}}\right) \end{aligned} \end{aligned}$$
(9)

Simplifying, KD can then be rewrite as Eq. (10).

$$\begin{aligned} \text {KD} = \text {KL}(b^T \parallel b^S) + (1 - p^T_t) \text {KL}(\tilde{p}^T \parallel \tilde{p}^S) \end{aligned}$$
(10)

where \(\text {KL}(b^T \parallel b^S)\) denotes the similarity between teacher and student probabilities in the target class, which we refer to as Target Class Knowledge Distillation (TCKD), and \(\text {KL}(\tilde{p}^T \parallel \tilde{p}^S)\) denotes the similarity between teacher and student probabilities in the non-target class, which we refer to as Non-Target Class Knowledge Distillation (NCKD), and thus we can rewrite KD Eq. (11).

$$\begin{aligned} \text {KD} = \text {TCKD} + (1 - p^T_t) \text {NCKD} \end{aligned}$$
(11)

Observing the latest KD formulation, we find that on the one hand NCKD is coupled with \((1 - p^T_t)\), which would suppress NCKD for well-predicted samples. On the other hand, the weights of NCKD and TCKD are coupled in the classical KD framework, which does not allow to change the weights of each term in order to balance the importance. Therefore DKD introduces two hyperparameters \(\alpha\) and \(\beta\) as weights for TCKD and NCKD, respectively. Thus the loss function of DKD can be written as Eq. (12).

$$\begin{aligned} \mathscr {L}_{DKD} = \alpha * \text {TCKD} + \beta * \text {NCKD} \end{aligned}$$
(12)

where \(\beta\) replaces \((1 - p^T_t)\) to prevent inhibiting the effectiveness of the NCKD, and secondly, \(\alpha\) and \(\beta\) can be allowed to be adjusted to achieve a balance of importance. By optimizing this decoupling loss, the knowledge gained by the teacher model is more easily transferred to the student model, thus improving the performance of the student network.

Low-rank adaptation (LoRA)

LoRA is a method for efficiently fine-tuning pre-trained models on specific tasks. This algorithm reduces the fine-tuning parameters using a low-rank approach while enhancing the model’s performance on the given task. Fine-tuning with fewer parameters can achieve over 90% of the performance of full fine-tuning. For a pre-trained model with a weight parameter matrix \(W_0 \in \mathbb {R}^{d \times k}\), \(\Delta W \in \mathbb {R}^{d \times k}\) represents the fine-tuning parameters for a specific task. \(\Delta W\) is a lower-dimensional parameter matrix that can be expressed as \(B \times A\), where \(B \in \mathbb {R}^{d \times r}\) and \(A \in \mathbb {R}^{r \times k}\), with \(r \ll k\). Thus, the parameter count of \(\Delta W\) is smaller than that of \(W_0\). The LoRA algorithm as show in the Eq. (13), with its key idea being to decompose the parameter matrix using a low-rank matrix decomposition.

$$\begin{aligned} h=W_0x+\Delta Wx=W_0 x+BAx \end{aligned}$$
(13)

Our DKDL-Net model

The DKDL-Net model is based on the DKD approach, where a Teacher model guides the training of a Student model. The model framework is illustrated in Fig. 2. The Teacher model is a large-scale model with a substantial number of parameters, and its increased depth enhances accuracy in bearing fault detection. However, the large parameter size of the Teacher model results in slow inference speed, making it unsuitable for efficient industrial tasks. Therefore, we trained the Student model using the DKD method. This model is a single-layer neural network, meaning it has fewer parameters and faster inference speed. However, since the Student model is derived from significant parameter compression, its accuracy decreases. In simple experimental analyses of bearing fault detection, the Student model’s accuracy is approximately 2% lower compared to the Teacher model.

Fig. 2
figure 2

Architecture of the Model. (top) Teacher Model, (below) Student Model.

Full size image

Through extensive research, we found that we can further fine-tune the Student model using the LoRA approach. The model framework is illustrated in Fig. 3. Typically, LoRA involves low-rank decomposition of the model’s convolutional and fully connected layers. In this task, we cannot reduce the model’s parameters further, as experiments have shown that this would decrease the model’s accuracy. Therefore, we integrated the LoRA module into the single-layer network.

In the DKDL-Net network, we copied the parameters from the Student model. The parameters of the A matrix in the LoRA module are initialized with data following a normal distribution \(N(0, \sigma ^2)\), while the B matrix is initialized to 0. This approach allows us to enhance the model’s accuracy by adding only a small number of parameters.

Fig. 3
figure 3

Architecture of the DKDL-Net Model.

Full size image

Regarding the training of the model, we utilize CE Loss as the objective function for training the Teacher model, defined mathematically as Eq. (3). For training the Student model, we employ a combination of TCKD Loss, NCKD Loss, and CE Loss as the loss functions. It is essential to balance CE Loss and DKD Loss during the training process, where DKD Loss is the sum of TCKD Loss, CE Loss and NCKD Loss, defined mathematically as Eq. (14).

$$\begin{aligned} \mathscr {L} = (1 - \gamma ) \underbrace{L_{CE}}_{\text {CE loss}} + \gamma \underbrace{\left( \alpha *\underbrace{\text {TCKD}}_{\text {TCKD Loss}} + \beta *\underbrace{\text {NCKD}}_{\text {NCKD Loss}} \right) }_{\text {DKD Loss}} \end{aligned}$$
(14)

where \(\gamma\) is a learnable parameter to balance CE and DKD Loss.

Finally, incorporating the LoRA plug-and-play module into the Student model, we fine-tune it using the CWRU dataset with CE Loss as the loss function, mathematically defined as Eq. (3). The pseudocode for the DKDL-Net algorithm is presented in Algorithm 1.

Algorithm 1
figure a

The training phase of DKDL-Net model.

Full size image

Structural configuration of the model

The framework of the Teacher model is shown in Fig. 2 (top), and the relevant parameter configurations for the model’s input, output, and convolutional kernel size are presented in Table 1.

Table 1 Teacher model parameters applied.
Full size table

The Student model is a single layer network and framework shown in Fig. 2 (below), and the relevant parameter configurations for the model’s input, output, and convolutional kernel size are presented in Table 2.

Table 2 Student model parameters.
Full size table

The DKDL-Net model is a single layer network and framework shown in Fig. 3, adding LoRA module before convolutional and fully connected layers, and the relevant parameter configurations for the model’s input, output, and convolutional kernel size are presented in Table 3.

Table 3 DKDL-Net model.
Full size table

Experiments and results

Experimental configurations

Environment configuration. All experiments for this model were conducted on a Windows 11 system with an Intel Core i7-9850H CPU at 2.60GHz and an NVIDIA GeForce GTX 1650 with Max-Q Design 4GB GPU. The code was run in an environment with Python 3.10.13 and PyTorch 2.0.1+cu117.

During the training of DKDL-Net, we utilized the Adaptive Moment Estimation (Adam) optimizer with a learning rate (LR) of 0.005 and a weight decay coefficient of 0.0001. We employed the cross-entropy loss function evaluate the loss between true and predicted labels.

Baseline. We selected MCNN-LSTM, FaultNet, BearingPGA-Net, KDSCNN, and WDCNN models as our baseline models. Among these, BearingPGA-Net and WDCNN are SOTA (state-of-the-art) models. However, BearingPGA-Net is more of a lightweight model, whereas WDCNN is a relatively large-scale model. KDSCNN is also a lightweight model and is comparable to our model.

Benchmark. Our benchmark is based on the CWRU dataset, curated by the Case Western Reserve University Bearing Data Center. The machine to generate the CWRU dataset is shown in Fig. 4. This dataset includes ten categories, comprising nine types of faulty bearings and one healthy bearing. Vibration data are collected at 12 kHz and 48 kHz. The fault types and labels of the CWRU dataset are shown in Table 4, waveforms as shown in Fg. 5.

Fig. 4
figure 4

Data collection machine tools.

Full size image
Table 4 The labels in the CWRU dataset and their corresponding fault types.
Full size table
Fig. 5
figure 5

Bearing abnormal and normal waveform format.

Full size image

Evaluation metrics. We use Accuracy, Precision, Recall and F1-Score to evaluate the performance of our model, which is calculated as in Eqs. (15) (16) (17) (18).

$$\begin{aligned} \text {Accuracy} = \frac{TP + TN}{TP + FP + FN + TN} \end{aligned}$$
(15)
$$\begin{aligned} \text {Precision} = \frac{TP}{TP + FP} \end{aligned}$$
(16)
$$\begin{aligned} \text {Recall} = \frac{TP}{TP + FN} \end{aligned}$$
(17)
$$\begin{aligned} \text {F1-Score} = \frac{2 \times \text {Precision} \times \text {Recall}}{\text {Precision} + \text {Recall}} \end{aligned}$$
(18)

where TP represents True Positive, TN represents True Negative, FP represents False Positive, FN represents False Negative.

Bearing fault detection experimental results

We conducted experiments on the CWRU dataset, and as shown in Table 5, the ratio of test data and training data is divided into 33.3% and 66.7%, the F1-Score of the DKDL-Net model is higher than that of the MCNN-LSTM, FaultNet, BearingPGA-Net, KDSCNN, and WDCNN models. Additionally, compared to the state-of-the-art (SOTA) model, our model achieved an improvement of 0.58%. Despite having only 6838 parameters, our model has a higher accuracy than BearingPGA-Net by 0.58%, with only 4008 more parameters. Compared to the KDSCNN model, our model has 948 more trainable parameters, yet it outperforms KDSCNN by 0.98%. In summary, the DKDL-Net (our) model achieves higher accuracy while maintaining fewer parameters and lower Flops.

Table 5 Comparison of F1-Score, model size and computational cost among different bearing fault diagnosis algorithms.
Full size table

As shown in Table 6, we evaluated the F1-Score, Precision, and Recall of the DKDL-Net model on the test dataset of CWRU. The DKDL-Net model outperforms the BearingPGA-Net, FaultNet, and MCNN-LSTM models in all three metrics. Compared to the best-performing BearingPGA-Net (SOTA) model, we achieved an improvement of nearly 0.55% across all three metrics. In conclusion, our model is the best-performing model on the CWRU dataset.

Table 6 Comparison of bearing fault diagnosis precision, Recall and F1-Score with different algorithms.
Full size table

As shown in Table 7, under the same configuration, our student model trained using Decoupled Knowledge Distillation (DKD) has 2,830 parameters. Compared to the teacher model, the student model’s trainable parameters are reduced by approximately 95.93%, but its F1-Score, Precision, and Recall decrease by 2.07%, 1.92%, and 2.08%, respectively. This indicates that while the DKD model can compress model parameters, its accuracy significantly decreases.

Table 7 Comparison of the parameters and assessment metrics of the teacher model, student model and DKDL-Net model.
Full size table

On the other hand, the DKDL-Net model, based on DKD compression and LoRA fine-tuning, has 6,838 trainable parameters. Compared to the teacher model, the DKDL-Net model’s trainable parameters are reduced by approximately 90.20%, indicating that DKDL-Net can significantly reduce model complexity and resource requirements. The F1-Score, Precision, and Recall decrease by only 0.09%, 0.12%, and 0.12%, respectively. This demonstrates that the accuracy loss caused by the DKDL-Net model compared to the teacher model is negligible. Therefore, the DKDL-Net model effectively compresses parameters while maintaining high accuracy.

In summary, our model achieves a compression ratio of 90.20% with a negligible decrease in accuracy compared to the Teacher model, making it highly efficient in terms of compression while maintaining high accuracy.

Finally, we computed the confusion matrices for the DKDL-Net model as shown in Fig. 6b, Student model as shown in Fig. 6a, and Teacher model as shown in Fig. 6c on the CWRU dataset, with 2,500 test samples, 250 samples per class.

Fig. 6
figure 6

Confusion matrix for student model and DKDL-Net model on CWRU dataset.

Full size image

We plotted the ROC curves for both the DKDL-Net model and the student model show in Fig. 7. As shown in Fig. 7b for the DKDL-Net model and Fig. 7a for the student model, it can be observed that the ROC curve for the student model is more jagged, while the ROC curve for the DKDL-Net model is smoother. Additionally, the area under the curve (AUC) for each class in the DKDL-Net model is generally larger than that of the student model. This indicates that our algorithm outperforms the student model’s algorithm, demonstrating better performance.

Fig. 7
figure 7

ROC curves for student and DKDL-Net models on the CWRU dataset.

Full size image

We tested the DKDL-Net model on the CWRU bearing fault detection dataset. Under the same training configuration as see Section Experimental Configurations, we tested 2500 samples, result as Shown in Table 8, and the DKDL-Net model required an average of 1757 \(\mu\)s per sample. DKDL-Net model with teacher model have 1x faster inference. This figure also demonstrates the high efficiency of our model.

Table 8 Inference time for the DKDL-Net model.
Full size table

Overall, the DKDL-Net model outperformed the Student model, and its performance was comparable to that of the Teacher model. Therefore, our DKDL-Net model can maintain good results even under high compression.

Ablation experiment. We fixed the \(\beta\) and \(\gamma\) parameters and analyzed the influence of different alphas on the model. The results are presented in Table 9.

Table 9 Accuracy (%) under different values of \(\alpha\), where \(\gamma\) and \(\beta\) is fixed.
Full size table

In terms of the results in Fig. 8, we fixed the alpha at 0.3 and analyzed the influence of beta and gamma on the model. We analyzed that when beta is greater than gamma, the performance ability of the model will improve. The experimental results show that the model is able to exhibit strong performance when \(\beta\) is larger than \(\gamma\).

Fig. 8
figure 8

Performance comparison under different combinations of \(\gamma\) and \(\beta\).

Full size image

Discussion

In this article, we propose a CNN model named DKDL-Net, which is based on decoupled knowledge distillation training and Low-Rank Adaptation fine-tuning. DKDL-Net is a single-layer neural network with only 6,838 parameters and an inference speed of 1,767 \(\mu\)s. It achieved an F1-Score of 99.50% on the CRWU dataset, representing a 0.60% improvement in F1-Score compared to the state-of-the-art (SOTA) models. Therefore, our model is highly efficient in detection while maintaining high accuracy. Moreover, the model is extremely lightweight, making it suitable for practical industrial applications.

(limatations and future work)While our model excels on the CWRU dataset, its generalizability to other datasets (e.g., Paderborn or SEU) remains untested. Additionally, the 0.5% residual accuracy gap compared to the teacher model suggests room for improvement in distillation efficiency. Future work could explore hybrid compression techniques (e.g., pruning + DKD) or adaptive LoRA ranks to further narrow this gap. Expanding evaluations to heterogeneous data sources and investigating cross-domain transferability would also validate broader applicability.