A computational intelligence approach for classifying dental caries in X-ray images using integrated fuzzy C-means clustering with feature reduction and a weighted matrix scheme

Abstract

Classifying dental caries in X-ray images poses several challenges that must be addressed to ensure accurate diagnosis and effective treatment planning. However, dental X-rays often have low contrast, making it difficult to distinguish between healthy and decayed regions, especially in the early stages of caries. This paper uses computational intelligence to present a new approach to classifying dental caries in X-ray images. The traditional Fuzzy C-means (FCM) segmentation algorithms have limitations, such as treating all feature components equally important. This can lead to misclassification in X-ray images due to the imbalance between caries-related and healthy features. To solve this problem, a computational intelligence approach is presented for the first time in dental X-ray analysis using the Feature Reduction and Weighted Scheme (FRWS). This method automatically calculates individual feature weights and reduces irrelevant features using a feature-reduction strategy and a feature-weighted scheme. By incorporating a feature-weight matrix, the algorithm can dynamically adjust the significance of dental caries and background features across different clusters and accurately determine the centers of smaller clusters in multidimensional feature spaces with imbalanced data. In addition to the algorithm, the MDOT (morphological dilation with optimal thresholding) technique is used to refine the classification of dental caries regions. The proposed method was evaluated on a dataset of 890 dental X-ray images, achieving an average accuracy of 91.62%, precision of 90.89%, specificity of 91.26%, sensitivity of 91.78%, a Dice coefficient of 90.74% and IoU of 83.90%. These results, particularly the high accuracy and precision, reassure the audience of the method’s superior ability to classify caries lesions, instilling confidence in its performance. We demonstrate the feasibility of classifying dental caries in X-ray images using integrated fuzzy C-means clustering, feature reduction, and a weighted matrix scheme.

Introduction

Dental caries are a common dental condition that affects people of all ages. Left untreated, it can lead to tooth loss, infections, and severe health complications. Early diagnosis of dental caries is crucial to prevent these adverse outcomes. There are several methods for diagnosing dental caries, ranging from visual inspection during routine dental examinations to more advanced imaging techniques. While visual inspection can identify caries lesions in some cases, specific lesions may be too small or subtle to be detected by the naked eye. In such instances, radiographic imaging plays a crucial role. X-ray imaging, in particular, enables dentists to accurately assess the extent and severity of caries lesions, thereby facilitating the selection of the most appropriate treatment plan¹. However, conventional radiography has limitations in detecting caries lesions because it relies on two-dimensional (2D) representations of the affected areas. This 2D nature can lead to the loss of critical diagnostic information, potentially underestimating the lesion’s extent^2,3. Advanced X-ray technologies have been used as a powerful diagnostic tool in dentistry to address these limitations. These technologies offer significant advantages over traditional radiography, including the ability to reconstruct images in multiple planes, greater precision, and reduced patient radiation exposure. By using X-rays, dentists can obtain highly detailed three-dimensional (3D) images of specific areas of concern, greatly enhancing diagnostic accuracy and the effectiveness of treatment planning. Consequently, X-ray technology has become an indispensable component of modern dental practice, particularly in diagnosing and managing dental caries lesions^4,5. X-ray imaging in dental diagnostics has many advantages, but several challenges must be addressed to improve its effectiveness. One major issue is excessive noise in X-ray images, which can be caused by factors such as patient movement, imaging equipment quality, and medical staff experience. This noise can create artifacts in the images, making it difficult to interpret the diagnostic information. Dental caries lesions pose unique challenges in medical imaging due to their small size. This makes it hard to distinguish the lesions from the surrounding tissue, especially when the background intensity is higher than the lesions’. This contrast difference makes it particularly challenging to detect dental caries, especially in the early stages when the lesions are subtle. Furthermore, dental caries is a progressive disease that starts in the advanced stages of the dentin and pulp cavity of the tooth. As the disease progresses, the boundaries of the lesions become less defined, making early detection more challenging. This gradual progression often leads to unclear lesion margins, further complicating accurate early-stage diagnosis^6,7,8.

In hospitals across Northeastern Thailand, many general hospitals still use X-ray imaging as the primary diagnostic tool. This is because they have limited resources and funding to acquire more advanced equipment, despite advances in medical imaging technology. Relying on X-ray imaging can delay accurate diagnosis and treatment, especially for patients with complex conditions that require high-precision imaging. Addressing these issues and enhancing healthcare infrastructure in resource-constrained areas is crucial to ensuring patients receive timely, accurate diagnoses and improved health outcomes.

Literature review

This literature review examines recent studies on the application of computational intelligence techniques to enhance the accuracy and precision of dental caries detection. A systematic selection process was used to identify relevant articles that explicitly focused on detecting dental caries lesions. A total of 189 articles were identified in central electronic databases, including ScienceDirect, IEEE Xplore, and PubMed. Only articles published between 2020 and 2024 were considered to ensure the relevance and currency of the findings. ScienceDirect was the source of 61 articles (32.28%), reflecting its extensive coverage of dental caries detection research. IEEE Xplore provided 79 journal papers (41.80%), demonstrating its specialization in distributing scholarly literature on computational intelligence and its applications in medical imaging. The remaining 49 articles (25.92%), including journal papers and conference proceedings, were obtained from PubMed, demonstrating its crucial role in disseminating research findings in dental and medical sciences.

In recent years, advances in dentistry have improved the classification of dental cavities in X-ray images. These methods fall within traditional computational intelligence approaches and have contributed to progress in automated dental diagnostics. Duong et al.⁹ introduced a computational technique for automatically detecting dental caries lesions on tooth surfaces using smartphone images. Their method utilizes a Support Vector Machine (SVM) method and features based on the International Caries Detection and Assessment System (ICDAS) to identify caries accurately. This approach is a significant step in making dental caries detection more accessible, especially in settings where conventional dental imaging equipment may not be available. Yu et al.¹⁰ proposed a new technique called the Unified Caries Lesions Segmentation and Assessment Framework (UCLSA), which integrates advanced deep learning architectures, specifically a Residual Network with Feature Pyramid Network (ResNet-FPN) and Fully Convolutional Network (FCN). This framework is designed to diagnose dental caries lesions in the first permanent molars of children. The system uses network feature pyramids and anchor boxes to identify and assess caries lesions autonomously, demonstrating high effectiveness in pediatric dental diagnostics. Additionally, the authors developed a dedicated image database, “Child-OID,” containing 1368 annotated images of primary school children, thereby establishing a valuable resource for standardized diagnosis and further research in this area. Geetha et al.¹¹ significantly contributed by developing an algorithm to segment and diagnose areas of dental caries in digital radiographs. The algorithm uses neural networks with Laplacian filtering, morphological operations, and backpropagation to improve the accuracy of dental caries diagnosis. The algorithm achieved high accuracy by training an artificial neural network with a dataset of 105 images, demonstrating its potential as a critical tool in digital dental diagnostics. Cantu et al.¹² focused on deep learning methodologies for detecting caries lesions in bitewing radiographs. In their study, four dentists carefully annotated the lesions at the pixel level, providing a reference standard for evaluating the performance of a trained U-Net neural network. The U-Net model was evaluated by comparing its performance with the diagnoses made by seven independent dentists, using tooth-level accuracy metrics. This approach highlights the growing role of deep learning methods in X-ray imaging, demonstrating their potential to complement and enhance traditional diagnostic processes. Vinayahalingam et al.¹³ proposed a deep-learning algorithm to classify pixels of dental caries in panoramic radiographs. They used a convolutional neural network (CNN) based on the MobileNet V2 architecture, trained on a dataset of 400 panoramic images to identify carious lesions in third molars. The MobileNet V2 model was evaluated on 100 panoramic radiographs to classify dental caries, with a focus on third molars. Lee et al.¹⁴ introduced a method for identifying early caries lesions on back teeth using bitewing radiography. They developed a CNN technique based on the U-Net architecture to improve clinicians’ diagnostic performance. The model was trained on 304 radiographs and evaluated on an additional 50 radiographs to assess its performance. The U-Net-based CNN demonstrated improved ability to detect early caries lesions, suggesting its potential as a supplementary tool in clinical practice. Mao et al.¹⁵ presented a solution for extracting individual teeth using image enhancement techniques combined with a CNN. Their model aimed to automatically identify caries lesions and restorations from bitewing images, aiding diagnosis and treatment planning. The introduced AlexNet method achieved high accuracy rates of 95.56% for restoration detection and 90.30% for caries lesion identification, indicating its reliability in automating diagnostic processes in dental care. Bayraktar et al.¹⁶ investigated caries lesion detection using a CNN, namely YOLO (You Only Look Once). This model was tested on 200 radiographs, of which 13.89% showed caries lesions. The YOLO-based CNN achieved 94.59% accuracy, 72.26% sensitivity, and 98.19% specificity. Additionally, the model showed a positive predictive value (PPV) of 86.58% and a negative predictive value (NPV) of 95.64%. The area under the curve (AUC) was 87.19%, indicating predictive solid capability. Kühnisch et al.¹⁷ conducted a study with 2417 anonymized photographs of permanent teeth, classifying them into caries-free, noncavitated lesions, or caries-related cavitation. They developed a convolutional neural network (CNN) using image augmentation and transfer learning. The CNN achieved an overall accuracy of 92.5% in detecting caries (sensitivity 89.6%, specificity 94.3%, AUC 0.964) and a classification accuracy of 93.3% for caries-related cavitation (sensitivity 95.7%, specificity 81.5%, AUC 0.955). The study found that the model was especially effective in detecting cavities in molars compared to premolars. Vimalarani et al.¹⁸ presented a combination of image processing and neural networks to segment dental caries lesions in bitewing radiographs. Their approach utilized a deep learning method, specifically the deep Gradient Ent-based LeNet, to classify dental images effectively. The process involved several stages: input image capture, enhancement, segmentation, feature extraction, and final classification using the deep learning model. This method has demonstrated considerable success in accurately detecting dental caries lesions, further showing the potential of deep learning techniques to improve dental diagnostics. Zhu et al.¹⁹ developed a diagnostic tool using Faster R-CNN to predict the number and locations of caries regions from X-ray images. They also created an open web platform to support this diagnostic approach, comprising two steps: collecting clinical samples and training and testing a caries-detection model. This method marks a significant advancement in the use of artificial intelligence for precise dental diagnostics. Ramana et al.²⁰ aimed to enhance diagnostic accuracy in radiology by utilizing neural networks to analyze large dental images. Their study presented a novel deep learning method that accurately identifies and segments dental caries in dental images. This approach integrates contrast enhancement, noise filtering, and clustering algorithms for precise segmentation. The segmented images are then post-processed using morphological operations, and caries detection is performed using a meta-heuristic-based ResNeXt model combined with a recurrent neural network (RNN). The study demonstrates the potential of integrating advanced neural networks with innovative preprocessing techniques to improve caries detection accuracy. Imak et al.²¹ proposed an advanced approach for automatically diagnosing dental caries lesions using periapical images. Their method involves a multi-deep CNN model that aggregates input from various sources. The method achieved 99.13% accuracy on a dataset of 340 periapical images, indicating its potential as a highly effective caries-detection tool in clinical settings. Park et al.²² proposed a study on using deep learning algorithms to detect caries lesions by segmenting tooth surfaces in intraoral photographic images. The study involved 445 participants and collected 2348 images. They used CNN architectures, including U-Net, ResNet-18, and Faster R-CNN, for image segmentation and caries detection. The CNN-based segmentation significantly improved the algorithm’s accuracy and the area under the receiver operating characteristic curve, showing the effectiveness of deep learning in enhancing diagnostic precision in dental imaging. Kim et al.²³ proposed an automated diagnostic support system, DeNTNet, to accurately identify and analyze periodontal bone loss in panoramic dental radiographs. By integrating advanced deep learning techniques with clinical prior knowledge, they achieved an F1 score of 0.75 on the test set, outperforming the average performance of dental clinicians. DeNTNet was validated on a large dataset of 12,179 images, demonstrating its robustness and potential to improve periodontal diagnostics. Furthermore, Hung et al.²⁴ used machine learning to identify the most significant variables associated with the presence or absence of root caries lesions. Their study employed various machine learning algorithms, with the SVM achieving the highest accuracy of 97.10%. The method also demonstrated a precision of 95.10%, a sensitivity of 99.60%, a specificity of 94.30%, and an area under the curve (AUC) of 0.997. The analysis revealed that age was the most influential factor associated with root caries lesions, underscoring the utility of machine learning in identifying critical risk factors in dental health. Lastly, Abdulaziz et al.²⁵ introduced an innovative approach to caries lesion identification, using a hybrid graph-cut technique to delineate the oral cavity and its structures. By combining CNN-based deep learning with graph cut methods to analyze dental images, they achieved an accuracy of 97.07%. The results demonstrate the effectiveness of hybrid approaches that include advanced image processing techniques and deep learning to enhance the accuracy and reliability of dental diagnostics. These studies collectively showcase the rapid advancements in dental caries detection through deep learning methods. The integration of cutting-edge technologies, such as CNNs, Faster R-CNN, YOLO, and hybrid methods, has significantly improved diagnostic accuracy, efficiency, and the ability to identify and manage dental caries lesions. As these technologies continue to develop, they promise to transform dental diagnostics and improve patient outcomes globally.

Limitations in X-ray imaging for dental caries detection

Recent studies have shown that machine learning and deep learning algorithms can effectively detect dental caries lesions. However, several limitations must be addressed to improve segmentation and classification accuracy. These limitations, as outlined in Table 1, pose challenges that could hinder the full potential of machine learning in dental diagnostics. Addressing these issues will ensure that computational intelligence algorithms can accurately and reliably classify dental caries lesions in X-ray images.

Table 1 Presents a clear overview of the references, the algorithms used, their performance metrics (Acc; accuracy, Sen; sensitivity, Spec; specificity, Pre; precision, and F1-score), and the respective limitations associated with each study.

Summary of the challenges in dental caries detection

This section provides a comprehensive overview of the challenges in identifying regions of dental caries in X-ray images, drawing on findings from previous studies summarized in Table 1. Segmentation of dental caries has become an increasingly important area of research, with machine learning and deep learning emerging as promising technologies. However, machine learning and deep learning have specific challenges^10,11. One of the main challenges in machine learning is the need for high-quality data to effectively train models. This is particularly difficult in dental imaging, where high-quality images are often unnecessary. Machine learning models can sometimes produce false positives or negatives, leading to incorrect diagnoses and unnecessary treatments. Moreover, implementing these models can be costly because they require significant computational resources^{12,13,14,15,16,17,18,19}. Deep learning also faces challenges, including the need for substantial volumes of annotated data for training. This requires considerable time and effort for manual labeling and annotation, which can be costly. Overfitting and interpretability are other challenges associated with deep learning models, leading to poor performance on unseen data and difficulty in understanding their segmentations^{20,21,22,23,24,25}. Despite these challenges, researchers continue to explore the potential of machine learning and deep learning in dental imaging. Before implementing them in practical applications, the limitations and challenges associated with these techniques must be considered. Nonetheless, the insights gained from these challenges contribute to the continuous improvement and innovation in the field, inspiring researchers to refine existing algorithms and explore novel methodologies. The evolving landscape of artificial intelligence and image processing technologies further fuels this drive, offering new avenues to enhance the accuracy and reliability of dental caries detection.

Contributions of this paper

Despite significant advances in using computational intelligence and machine learning to segment dental caries lesions, several knowledge gaps remain. One of the primary gaps is the limited availability of large, high-quality annotated datasets, which are essential for training robust and generalizable computational intelligence models. The limited availability of these datasets hinders the creation of models that can accurately apply across a wide range of patient groups and variations in imaging conditions. Additionally, while deep learning models have shown impressive capabilities in caries detection, they provide little insight into their decision-making processes. This lack of interpretability presents a challenge in clinical settings, where transparency and understanding of diagnostic decisions are crucial for clinician trust and adoption. Furthermore, existing models are often used to detect caries lesions early, particularly those that are subtle or located in complex anatomical regions, leading to potential underdiagnosis or misclassification. Implementing these advanced models often demands significant computational resources, which can be a barrier in clinical settings with limited resources. This constraint can impede the accessibility and scalability of these technologies in healthcare.

This study makes several key contributions to dental caries detection using advanced machine learning:

• 1. We propose an enhanced Fuzzy C-Means (FCM) algorithm with a Feature-Reduction and Weighted-Scheme framework (FCM-FRWS), which automatically estimates feature weights while suppressing irrelevant or weakly contributing components. By embedding a feature-weight matrix into the objective function, the method explicitly accounts for feature imbalance between caries and non-caries regions. Features assigned persistently low weights are progressively eliminated, thereby refining the feature set, improving computational efficiency, and strengthening the algorithm’s ability to handle heterogeneous feature distributions. In addition, a refined segmentation stage incorporating morphological dilation and optimal-thresholding (MDOT) is applied to enhance lesion delineation and improve the reliability of dental caries classification.

• 2. Our method reduces the dependency on large, high-quality annotated datasets by combining advanced image processing techniques, making it more adaptable to varied clinical settings. This approach moderates the challenges posed by the scarcity of high-resolution dental images and provides a more accessible solution for dental practitioners.

• 3. The proposed FCM-FRWS algorithm incorporates elements that allow a better understanding and interpretation of the segmentation process, addressing the “black box” nature of many existing machine learning models. This transparency is crucial for increasing clinician trust and facilitating the adoption of computational intelligence technologies in dental diagnostics.

• 4. The study emphasizes the early classification of dental caries, improving the algorithm’s ability to identify subtle lesions that other methods may miss. This focus is critical for enabling timely intervention and improving patient outcomes.

We will start by outlining our research methodology and describing the dataset utilized in this study. Next, we will discuss the preprocessing techniques applied to the X-ray images and the coarse segmentation method used. Following that, we will explain the enhanced FCM-FRWS method integrated with MDOT. We will also focus on refined segmentation, which includes analysis of connected components and measures of compactness. Additionally, we will discuss the evaluation metrics and methodologies used to assess the performance of our approach. We will then present our experimental results and conduct a comparative analysis against existing methods. Finally, we will comprehensively discuss our findings and the conclusions drawn from this study.

Research methodology

The following section outlines the research methodology used in the study. It focuses on an enhanced FCM approach with FRWS, which automatically calculates feature weights and reduces irrelevant components to improve the coarse segmentation of dental caries in X-ray images. The FCM-FRWS process introduces a new procedure to eliminate irrelevant features with small weights, refining the feature set. This approach reduces the computational complexity of the FCM algorithm and enhances its ability to manage feature weights. Additionally, the study integrates a refined segmentation algorithm for dental X-ray images using MDOT to improve the accuracy of segmented regions. The methodology encompasses dataset preparation, image preprocessing, algorithmic design, and performance evaluation. Figure 1 provides an overview of the proposed method’s structure, offering detailed insights into each process stage and the dataset used, as highlighted in the subtitles.

Fig. 1

An overview of the proposed FCM-FRWS with the MDOT segmentation framework by highlighting the three processes (preprocessing, coarse to fine segmentation, and performance evaluation).

Data collection

The X-ray dataset was divided into two groups: adult and children’s teeth. This extensive study comprised 890 cases, including 89 children and 701 adults. Among these, 490 images illustrated dental caries lesions, while an additional 400 images showcased the pristine condition of healthy teeth. These invaluable X-ray images were collected from Kantarawaichai Hospital over several months, from February to November 2024. Each scan boasted an impressive axial resolution of 700 × 519 pixels, capturing intricate details and nuances. The data stemmed from a robust cohort of 678 patients aged 11–79, reflecting a wide age range. Although the dataset covered a wide age range, the age and demographic distributions were not uniform across groups, with a larger proportion of adult patients than children. This imbalance may introduce subtle sampling bias, particularly in lesion characteristics that vary across age or developmental stages. In addition, all radiographs were obtained from hospitals in Northeastern Thailand, which may limit the model’s external validity in populations with different oral health profiles, imaging protocols, or clinical practices. Future work will therefore focus on validating the method on multi-center, cross-regional datasets and incorporating demographic-stratified performance analysis to strengthen generalizability. The X-rays were captured using the advanced PHOT-XII 505 devices manufactured by Takara Belmont Crop. Japan and the images were meticulously compiled in the DICOM format. Before their treatment and the ensuing data collection, each participant provided informed consent by signing a form authorizing the use of their anonymized medical data for non-commercial academic communication. Participants were informed that their de-identified information might be used for further academic purposes, including the potential sharing of anonymized data in scholarly forums. The ethics committee thoroughly reviewed the dataset and granted formal approval for its publication. It underwent rigorous scrutiny and was sanctioned by Mahasarakham University (IRB number: 223-201/2568). Throughout the research process, every procedure adhered strictly to ethical guidelines, ensuring the utmost protection of participants’ privacy and confidentiality. This commitment to ethical integrity emphasizes the importance of participant welfare in medical research. To ensure the dataset’s accuracy and reliability, all images were captured under the supervision of five experienced dentists. The X-ray images were captured in grayscale and varied in size to reflect the diversity of clinical scenarios. This variation in image size and content contributes to a comprehensive dataset that accurately represents the conditions under investigation. Figure 2 displays a representative subset of these X-ray images, illustrating the dataset’s diversity and characteristics.

Fig. 2

Example of an X-ray image illustrating dental structures and caries analysis: (a) The original X-ray image offers a detailed view of the dental anatomy, (b) The segmented image, color-labeled by a dentist, highlights dental components for in-depth analysis: caries is marked in red, enamel in blue, and pulp in yellow, with the background uniformly represented in black.

X-ray images for dental caries classification

Various structural and imaging factors, including color, brightness, lesion size, the presence of restorative materials, and lesion location, influence the segmentation of carious lesions in dental radiographic images. These factors make dental caries diagnosis complex and challenge clinicians to accurately detect all regions within a single examination. This study emphasizes the need for precision and sensitivity at each diagnostic stage to enhance caries segmentation accuracy. The proposed methodology comprises two key stages: (1) preprocessing and (2) a coarse-to-fine image segmentation approach. Each component is carefully designed to enhance the robustness and reliability of caries segmentation, resulting in more accurate diagnostic outcomes.

Preprocessing of X-ray images

To ensure consistent image quality and optimize the diagnostic accuracy of dental caries detection, a comprehensive three-step preprocessing framework was implemented to normalize intensity variations, suppress unwanted noise, and enhance the structural continuity of X-ray images before segmentation and classification. The first stage, intensity normalization, used histogram matching^26,28 to adjust each image’s intensity distribution to that of a carefully selected reference image characterized by clear caries features and strong contrast. This transformation is mathematically defined as $\text{s}=\text{F}_{\text{s}}^{{-1}}\left[ {{\text{F}_\text{r}}(\text{r})} \right],$ where ${\text{F}_\text{r}}(\text{r})$ and ${\text{F}_\text{s}}(\text{s})$ are the cumulative distribution functions (CDFs) of the input and target images, respectively, and $\text{F}_{\text{s}}^{{-1}}$ represents the inverse CDF of the reference image. The process enhances visual quality, equalizes brightness across datasets, and amplifies important anatomical details such as enamel boundaries and lesion textures. In the second stage, to counteract noise amplification caused by intensity adjustment, a linear averaging filter²⁹ was applied to the enhanced image using the expression ${\text{g}_{\text{i}\text{j}}}={\sum\nolimits_{{\text{k}=-\text{m}}}^{\text{m}} {\sum\nolimits_{{\text{k}=-\text{m}}}^{\text{m}} \text{w} } _{\text{k}\text{l}}}{\text{f}_{\text{i}+\text{k},\text{j}+\text{l}}},$ where w_kl are the filter weights,${\text{f}_{\text{i}+\text{k},\text{j}+\text{l}}}$represents neighboring pixel values, and m defines the half-width of the filter window. A 5 × 5 kernel (m = 2) was used to effectively reduce random and Gaussian noise while preserving important edge information and caries contours, ensuring that the filtered image remained sharp and diagnostically reliable. Finally, in the third stage, morphological smoothing was performed using the closing operator³⁰, defined as A • B = (A ⊕ B) ⊖ B, where A denotes the input image, and B is a rectangular structuring element of size 6. This operation combines dilation (⊕) and erosion (⊖) to fill small gaps, remove dark noise patches, and produce smoother, continuous object boundaries. The closing operator effectively enhances dental region integrity and improves the delineation of carious areas by smoothing surface irregularities while preserving significant morphological details. Altogether, the integration of histogram-based normalization, adaptive linear filtering, and morphological closing ensures that X-ray images exhibit consistent contrast, reduced noise artifacts, and enhanced structural clarity, providing standardized, high-quality input for subsequent segmentation, feature extraction, and deep learning-based classification tasks. Figure 3 presents the normalized results, demonstrating improved contrast and feature visibility.

Fig. 3

The results of histogram matching: (a) the reference image is carefully selected as the standard for intensity normalization, (b) the original image from the dataset displays its inherent intensity characteristics, (c) the normalized image demonstrates adjusted pixel intensities that align with those of the reference image, (d) histogram comparison of the three dental X-ray images.

Coarse segmentation of dental caries lesions

Coarse segmentation is the preliminary step in identifying and isolating dental caries lesions. The primary objective of this stage is to broadly delineate regions that potentially contain caries, laying the groundwork for more refined analysis in subsequent processing stages. This process involves carefully executed steps that contribute to the effective segmentation of caries-affected areas. The following sections provide a detailed exploration of the coarse segmentation methodology used in this study.

The FCM algorithm (base-line)

The FCM is a widely used algorithm and an effective segmentation technique for X-ray images³¹. This study employs the FCM algorithm to segment dental caries lesions below.

• 1. Initialize the FCM algorithm with a predefined number of clusters c. The clusters often correspond to the number of regions segmented (e.g., dental caries and background). Each pixel in the image is assigned a membership grade for each cluster based on its intensity value using Eq. (1).

  $$\text{U}=\left[ {{\text{u}_{\text{i}\text{j}}}} \right]{\text{ where }}{\text{u}_{\text{i}\text{j}}} \in \left[ {0, 1} \right]{\text{ and }}\sum\limits_{{\text{j}=1}}^{\text{m}} {{\text{u}_{\text{i}\text{j}}}=1} {\text{ }}\forall \text{i}$$

  (1)

  where $\text{U}=\left[ {{\text{u}_{\text{i}\text{j}}}} \right]$ represents the membership matrix in the FCM algorithm. Each element u_ij in the matrix corresponds to the degree of membership of the ith data point to the jth cluster. It is a membership value between 0 and 1, and the condition states that the sum of the membership values for a given data point i across all clusters j must equal 1. This ensures that each data point’s total degree of membership is fully distributed among all clusters.

• 2. Calculate the cluster’s centroids based on the weighted average of the pixel intensities, considering the membership values. The centroids are computed from the pixel intensities using Eq. (2).

  $${\text{v}_\text{j}}=\frac{{\sum\limits_{{\text{i}=1}}^{\text{n}} {{{\left( {{\text{u}_{\text{i}\text{j}}}} \right)}^{\text{m}}} \cdot {\text{x}_\text{i}}} }}{{\sum\limits_{{\text{i}=1}}^{\text{n}} {{{\left( {{\text{u}_{\text{i}\text{j}}}} \right)}^{\text{m}}}} }}$$

  (2)

  where v_j is the centroid of the jth cluster, x_i is the ith data point, m is the fuzziness parameter, and n is the total number of data points.

• 3. Update the membership values based on the distance between the pixel intensities and the cluster centroids as Eq. (3).

  $${\text{u}_{\text{i}\text{j}}}=\frac{1}{{\sum\limits_{{\text{k}=1}}^{\text{c}} {{{\left( {\frac{{\left\| {{\text{x}_\text{i}} - {\text{v}_\text{j}}} \right\|}}{{\left\| {{\text{x}_\text{i}} - {\text{v}_\text{k}}} \right\|}}} \right)}^{\frac{2}{{{\text{m-1}}}}}}} }}$$

  (3)

  where u_ij is the membership value indicating the degree of belongingness of the ith data point to the jth cluster, $\left\| {{\text{x}_\text{i}} - {\text{v}_\text{j}}} \right\|$ represents the Euclidean distance between the ith data point and the centroid of the jth cluster, and k is an index that runs across the cluster from 1 to c. The objective is to minimize the weighted sum of squared errors between the pixel intensities and the cluster centroids, as defined in Eq. (4).

  $${\text{J}_\text{m}}=\sum\limits_{{\text{i}=1}}^{\text{n}} {\sum\limits_{{\text{j}=1}}^{\text{c}} {{{\left( {{\text{u}_{\text{i}\text{j}}}} \right)}^\text{m}}} } \cdot {\left\| {{\text{x}_\text{i}} - {\text{v}_\text{j}}} \right\|^2}$$

  (4)

  where J_m represents the objective function of the FCM algorithm, the algorithm’s goal is to minimize this function.

• 4. The algorithm iterates over steps 2 and 3 until the following conditions are met using Eq. (5).

  $$\left\| {{\text{U}^{(\text{t}+1)}} - {\text{U}^{(\text{t})}}} \right\|<\varepsilon$$

  (5)

  where U^(t+1) and U^(t) are the membership matrices at iterations t + 1 and t, and ɛ is a small positive number that serves as a threshold for stopping the algorithm.

The Fuzzy C-Means (FCM) algorithm is used in dental X-ray analysis due to its ability to handle data ambiguity and preserve anatomical boundaries. FCM assigns membership values between 0 and 1 to each pixel for all clusters, enabling soft partitioning of image data and making it suitable for grayscale images such as radiographs, where region boundaries are often indistinct. In this study, FCM was applied for dental caries detection with the number of clusters (c) ranging from 2 to 5 to evaluate its capability in differentiating anatomical structures (tooth, cavity, background, and root canal). The segmentation performance for different values of c was evaluated under consistent parameter settings. The fuzzification coefficient (m), which controls the degree of fuzziness, was set to 2.0 to balance hard and soft clustering. The convergence threshold (ε) was defined as 1^e−5, allowing the algorithm to terminate when the objective function change falls below this limit. The maximum number of iterations was set to T = 15, which was empirically sufficient for convergence in dental X-ray images. The distance metric used was Euclidean distance, ensuring pixel-level spatial sensitivity. Pixel features (intensity, spatial coordinates) were normalized before clustering to stabilize convergence. When c = 2, segmentation distinguished only the broad foreground (tooth) and background regions. At c = 3, early signs of caries and root separation became more distinguishable. Increasing c to 4 better delineated the cavity region, and root area structures emerged more clearly. At c = 5, fine segmentation was achieved with distinct zones for enamel, dentin, cavity, and soft-tissue artifacts. However, higher cluster counts may lead to over-segmentation or increased sensitivity to noise. Overall, the analysis demonstrates that selecting an appropriate number of clusters is crucial for optimal clinical interpretation. Values c = 3 provided the best balance between structural clarity and segmentation simplicity for dental diagnostic tasks (As shown in Fig. 4).

Fig. 4

Segmentation results of dental X-ray using FCM at varying cluster numbers (c = 2 to 5), (a) c = 2, (b) c = 3, (c) c = 4, (d) c = 5.

The Fuzzy C-Means (FCM) algorithm has several limitations that hinder its performance in complex tasks like dental X-ray analysis. Fundamentally, FCM is an intensity-based clustering technique that lacks spatial awareness. It evaluates each pixel individually, based solely on its intensity features, without considering the spatial relationships between neighboring pixels. In dental X-rays, where delicate structures such as enamel boundaries, carious lesions, and overlapping tissues are common, this limitation can lead to fragmented segmentations and poor anatomical consistency. A notable drawback of FCM is its sensitivity to noise and grayscale inhomogeneity, which are frequently encountered in X-ray imaging due to variability in acquisition conditions and patient anatomy. Without built-in noise suppression or context modeling, FCM may misclassify noisy or low-contrast regions, leading to artifacts and boundary leakage in the segmentation output. Furthermore, FCM requires the user to specify the number of clusters (c). This is problematic in clinical applications, as the optimal number of anatomical or pathological regions can vary significantly from case to case, and trial-and-error tuning is inefficient and error-prone.

Proposed FCM with FRWS

Clustering algorithms for high-dimensional data often require careful feature selection to filter out irrelevant components. This is crucial because datasets may contain irrelevant features, making feature selection an essential step. Therefore, the X-ray images outlined an FCM-FRWS algorithm that introduces feature selection and weighting into clustering for high-dimensional data. The objective is not just to improve but to revolutionize the FCM algorithm by using feature weighting and entropy to eliminate irrelevant features, thereby progressively enhancing the clustering process. The FCM-FRWS algorithm assigns a weight w_j to each feature and updates it iteratively based on its relevance. The algorithm eliminates features with low weights, thereby improving clustering performance³². The FCM-FRWS objective function is given by Eq. (6).

$$\text{J}\left( {\text{U},\text{V},\text{W}} \right)=\sum\limits_{{\text{i}=1}}^{\text{n}} {\sum\limits_{{\text{k}=1}}^{\text{c}} {\sum\limits_{{\text{j}=1}}^{\text{d}} {\text{u}_{{\text{i}\text{k}}}^{\text{m}}} \cdot {\delta _\text{j}} \cdot {\text{w}_\text{j}} \cdot {\text{d}_{\text{i}\text{j}\text{k}}}} } +\frac{\text{n}}{\text{c}}\sum\limits_{{\text{j}=1}}^{\text{d}} {{\text{w}_\text{j}}\log \left( {{\delta _\text{j}}{\text{w}_\text{j}}} \right)}$$

(6)

where $\sum _{{\text{i}=1}}^{\text{n}}\sum _{{\text{k}=1}}^{\text{c}}\sum _{{\text{j}=1}}^{\text{d}}$are triple summations over the clusters k (from 1 to c), data points i (from 1 to n), and features j (from 1 to d), $\text{u}_{{\text{i}\text{k}}}^{\text{m}}$is the degree of membership of data point x_i in cluster k, m is the fuzziness parameter,${\delta _\text{j}}$denote a feature weighting factor used to control feature weights, w_j represents the weight of feature j, d_ijk represents the quared Euclidean distance between feature j of point i and cluster center k, and $\frac{\text{n}}{\text{c}}\sum\limits_{{\text{j}=1}}^{\text{d}} {{\text{w}_\text{j}}\log \left( {{\delta _\text{j}}{\text{w}_\text{j}}} \right)}$ is an entropy-based regularization that penalizes the feature weights w_j to avoid overly large or small weights, which ensures that irrelevant features are eliminated over time. The sum of membership values for each data point must equal 1, meaning that the total membership degree for a point across all clusters must be 1, as defined in Eq. (7).

$$\sum\limits_{{\text{k}=1}}^{\text{c}} {{\text{u}_{\text{i}\text{k}}}=1, 0 \leqslant {\text{u}_{\text{i}\text{k}}} \leqslant 1}$$

(7)

The sum of feature weights must also equal 1, ensuring all feature weights are normalized using Eq. (8).

$$\sum\limits_{{\text{j}=1}}^{\text{d}} {{\text{w}_\text{j}}=1, 0 \leqslant {\text{w}_\text{j}} \leqslant 1}$$

(8)

The FCM-FRWS algorithm is solved using an iterative optimization procedure. First, compute the membership function by taking the partial derivative of the Lagrangian for u_ik and setting it to zero³³. This leads to an equation that updates the fuzzy membership values. This ensures that the new values of u_ik satisfy the constraints using Eq. (9).

$${{{\text{u}_{\text{i}\text{k}}}={{\left( {\sum\limits_{{\text{j}=1}}^{\text{d}} {{\delta _\text{j}} \cdot {\text{w}_\text{j}} \cdot {{\left( {{\text{d}_{\text{i}\text{j}\text{k}}}} \right)}^2}} } \right)}^{ - \frac{1}{{\text{m}-1}}}}} \mathord{\left/ {\vphantom {{{\text{u}_{\text{i}\text{k}}}={{\left( {\sum\limits_{{\text{j}=1}}^{\text{d}} {{\delta _\text{j}} \cdot {\text{w}_\text{j}} \cdot {{\left( {{\text{d}_{\text{i}\text{j}\text{k}}}} \right)}^2}} } \right)}^{ - \frac{1}{{\text{m}-1}}}}} {\sum\limits_{{\text{k}=1}}^{\text{c}} {\left( {\sum\limits_{{\text{j}=1}}^{\text{d}} {{\delta _\text{j}} \cdot {\text{w}_\text{j}} \cdot \left( {{\text{d}_{\text{i}\text{j}\text{k}}}} \right)} } \right)} }}} \right. \kern-0pt} {\sum\limits_{{\text{k}=1}}^{\text{c}} {\left( {\sum\limits_{{\text{j}=1}}^{\text{d}} {{\delta _\text{j}} \cdot {\text{w}_\text{j}} \cdot \left( {{\text{d}_{\text{i}\text{j}\text{k}}}} \right)} } \right)} }}$$

(9)

After updating the membership values, the algorithm fixes U = Û and W = Ŵ, minimizing the objective function for the cluster centers v_kj. This is done by setting the partial derivative of the Lagrangian with respect v_kj to 0 using Eq. (10).

$${{{\text{v}_{\text{k}\text{j}}}=\sum\limits_{{\text{i}=1}}^{\text{n}} {{\text{\varvec{\upmu}}}_{{{\text{ik}}}}^{{\text{m}}}{\text{x}_{\text{i}\text{j}}}} } \mathord{\left/ {\vphantom {{{\text{v}_{\text{k}\text{j}}}=\sum\limits_{{\text{i}=1}}^{\text{n}} {{\text{\varvec{\upmu}}}_{{{\text{ik}}}}^{{\text{m}}}{\text{x}_{\text{i}\text{j}}}} } {\sum\limits_{{\text{i}=1}}^{\text{n}} {{\text{\varvec{\upmu}}}_{{{\text{ik}}}}^{{\text{m}}}} }}} \right. \kern-0pt} {\sum\limits_{{\text{i}=1}}^{\text{n}} {{\text{\varvec{\upmu}}}_{{{\text{ik}}}}^{{\text{m}}}} }}$$

(10)

Finally, the algorithm fixes U = Û and V = $\hat {\text{V}}$and minimizes the objective function for the feature weights w_j. This is done by setting the partial derivative of the Lagrangian for w_j to 0, as defined in Eq. (11).

$$\sum\limits_{{\text{k}=1}}^{\text{c}} {\sum\limits_{{\text{i}=1}}^{\text{n}} {{\text{\varvec{\upmu}}}_{{{\text{ik}}}}^{{\text{m}}}} {\delta _\text{j}}{{\left( {{\text{d}_{\text{i}\text{j}\text{k}}}} \right)}^2}+\frac{\text{n}}{\text{c}}\left( {\log {\delta _\text{j}}{\text{w}_\text{j}}+1} \right)+{\lambda _2}=0}$$

(11)

As the algorithm iteratively updates the feature weights, features with minimal weights are eliminated from the clustering process^32,33. This leads to a more efficient clustering, as irrelevant features are ignored after a few iterations. The update to the feature weights w_j is given by Eq. (12).

$${\text{w}_\text{j}}=\frac{1}{{{\delta _\text{j}}}}\exp \left( { - \frac{\text{n}}{\text{c}}\frac{{\sum\nolimits_{{\text{k}=1}}^{\text{c}} {\sum\nolimits_{{\text{i}=1}}^{\text{n}} {{\text{\varvec{\upmu}}}_{{{\text{ik}}}}^{{\text{m}}}{\delta _\text{j}}{{\left( {{\text{d}_{\text{i}\text{j}\text{k}}}} \right)}^2}} } }}{\text{n}}} \right)$$

(12)

The transition from Eq. (11) to (12) highlights the role of the exponential term in irrelevance suppression. Features whose distances from the cluster centers remain persistently large exhibit exponential decay in their associated weights over iterations. This decay mechanism enables the algorithm to progressively downweight and eventually discard features that contribute little to the clustering objective, thereby enhancing its ability to detect and filter out irrelevant features in a principled, data-driven manner. This weight update function exponentially reduces the weight of features whose values are far from the cluster center, effectively discarding irrelevant features over iterations. To ensure the constraint $\sum\nolimits_{{\text{j}=1}}^{\text{d}} {{\text{w}_\text{j}}=1}$is maintained, the weight values are normalized as in Eq. (13).

$${\text{w}^{\prime}_\text{j}}=\frac{{{\text{w}_\text{j}}}}{{\sum\limits_{\text{p}}^{\text{d}} {\text{w}_{\text{p}}^{{\left( {\text{n}\text{e}\text{w}} \right)}}} }}$$

(13)

This normalization step adjusts the feature weights so that their sum equals 1, preserving each feature’s relative importance while maintaining overall balance. In the FCM-FRWS objective function (Eq. 11), the term $\frac{\text{n}}{\text{c}}\sum\nolimits_{{\text{j}=1}}^{\text{d}} {{\text{w}_\text{j}}} \log {\delta _\text{j}}{\text{w}_\text{j}}$controls the regularization of feature weights. The X-ray image shows that this term must be carefully managed to avoid driving too many feature weights close to zero. If the sum of weighted distances between data points and cluster centers becomes too large, the weight updates may shrink too quickly, discarding too many features. Thus, the factor $\:n/c$is introduced as a normalization term that balances the contribution of the feature-weight regularization with respect to the number of samples $\:n\:$and clusters $\:c$. Without this scaling, the entropy-based penalty may dominate the objective when $\:n\:$is large, leading to excessive shrinkage of feature weights. By normalizing with $\:n/c$ the regularization term remains proportional to the average contribution per cluster, thereby stabilizing the learning of feature weights and preventing premature elimination of informative features. The algorithm uses the variance-to-mean ratio (VMR) to identify irrelevant features. The reciprocal of the VMR, the mean-to-variance ratio (MVR), is used in this context. The MVR is defined as Eq. (14).

$${\text{MV}}{{\text{R}}_{\text{j}}}=\frac{{{\text{mean}}\left( {{{\text{x}}_{\text{j}}}} \right)}}{{{\text{var}}\left( {{{\text{x}}_{\text{j}}}} \right)}}$$

(14)

This ratio is used to identify features with a small dispersion. A smaller dispersion (higher MVR) means the feature is closer to the cluster center, making it more relevant. In comparison, a larger dispersion (lower MVR) indicates the feature is farther from the cluster center, suggesting irrelevance. Then, the FRFCM algorithm proposes using VMR to measure the importance of each feature and to control the update process for ${\delta _\text{j}}.$ The learning procedure for${\delta _\text{j}}$which controls the feature weights are defined as Eq. (15). Equation (15) governs the adaptive update of the feature weights during the optimization process. Features that exhibit high dispersion with respect to the cluster centers are progressively penalized, whereas compact and discriminative features retain higher weights. This adaptive learning mechanism encourages the algorithm to emphasize features that consistently contribute to cluster separation, while gradually suppressing noisy, redundant, or weakly informative components. As a result, the feature-weighting process improves both robustness and clustering stability over successive iterations.

$${\delta _\text{j}}=\left( {\frac{{{\text{mean}}\left( {{{\text{x}}_{\text{j}}}} \right)}}{{{\text{var}}\left( {{{\text{x}}_{\text{j}}}} \right)}}} \right)$$

(15)

By applying VMR to each feature, the algorithm identifies low-variance features and eliminates high-variance ones. Based on this metric, the feature weighting is updated to reduce the impact of less relevant features. Finally, the FCM-FRWS algorithm introduces a threshold to determine which features to discard during clustering. A common approach is to discard features whose weights fall below a threshold that depends on the number of features d. The algorithm suggests using the following threshold, defined by Eq. (16).

$$\frac{{\text{1}}}{{\sqrt {{\text{nd}}} }}$$

(16)

This threshold balances the total number of features d with the dataset size n, enabling efficient discarding of irrelevant features. For example, we are implementing the FCM-FRWS algorithm for X-ray images, using feature reduction and weighted schemes. Suppose we have four features, x₁, x₂, x₃, and x₄, and we initialize the weights as W = [0.25, 0.25, 0.25, 0.25] (balanced data) and W = [0.1, 0.7, 0.1, 0.1] (imbalanced data). Figure 5 compares the clustering results and the progression of feature weights for the two initial weight scenarios. The plot illustrates the data being clustered into two distinct groups, represented by blue and red points. Figure 5a shows the clustering result using initial balanced weights (W = [0.25, 0.25, 0.25, 0.25]). The clusters appear well-separated, allowing the algorithm to assess each feature’s relevance neutrally. The clustering result using initial imbalanced weights (W = [0.1, 0.7, 0.1, and 0.1]) is shown in Fig. 5b. Despite starting with imbalanced initial feature weights, the algorithm found the relevant clusters. However, the initial emphasis on certain features can affect the initial clustering path before the algorithm converges to an optimal solution. The clustering is also booming, but the initial focus on feature x₂ may have influenced the initial clustering path before the algorithm adjusted the feature importance. Table 2 shows the progression of feature weights over 10 iterations. It demonstrates how the algorithm adjusts the weights over 10 iterations, starting from equal importance. Weights for less relevant features (x₁, x₂, x₃, and x₄) gradually decrease. This table indicates that even with an initial emphasis on x₂, the algorithm reduces the weights of x₁, x₃, and x₄ over iterations. The final weight of x₂ is higher, reflecting its initial priority and the algorithm’s recognition of its relative importance. In both cases, the FCM-FRWS algorithm successfully reduces the influence of less relevant features over iterations. The key difference is the rate at which the weights adjust, influenced by the initial weights. While both scenarios result in distinct clustering, the imbalanced initial weights may lead to an initial clustering that is more strongly influenced by the prioritized feature (x₂), which is corrected over iterations. This demonstrates the robustness of the FCM-FRWS in adjusting feature importance regardless of starting conditions. The convergence speed of the FCM-FRWS algorithm over 10 iterations for two different initial weight scenarios: balanced [0.25,0.25,0.25,0.25] and imbalanced [0.1,0.7,0.1,0.1] initial weights are shown in Fig. 6.

Fig. 5

The visualizations of the clusters for both balanced and imbalanced initial weights across three stages: early (Iteration 1), mid (Iteration 5), and late (Iteration 10).

Table 2 The progression of feature weights over 10 iterations, where the initial feature weights were set to a balanced and an imbalanced distribution.

Fig. 6

The convergence speed of the FCM-FRWS algorithm over 10 iterations for two different initial weight scenarios (balanced and imbalanced), across 10 different runs.

According to Fig. 6, suppose an X-ray image dataset has 46,225 data points. Many of these features may be related to subtle texture variations that do not aid in identifying regions of dental caries. Features that do not show a strong relationship with the cluster centers will receive lower weights over iterations, effectively reducing their impact on the final clustering outcome. If the number of data points reduced through feature weighting is limited, every feature becomes relatively more important, and the algorithm’s effectiveness in identifying irrelevant features is constrained. Therefore, every data point becomes relatively more important, and the algorithm’s ability to identify irrelevant features is constrained. When applying the FCM-FRWS to the X-ray image, we will follow a specific process and then analyze the outcomes. We set FCM-FRWS with m = 2 (fuzziness parameter) and c = 3 (number of clusters). The experimental results are illustrated in Fig. 7.

Fig. 7

Coarse segmentation process for dental caries, (a) the normalized image represents the preprocessed X-ray image, ensuring consistent intensity levels across the image, (b) the closed image, generated during the preprocessing steps, further refines the image by eliminating small artifacts and smoothing the boundaries, (c) the coarse segmentation result, obtained using an enhanced FCM-FRWS algorithm with c = 3 clusters, significantly improves segmentation accuracy, effectively delineating regions of dental caries within the X-ray image.

Classification of dental caries using otsu’s thresholding

Otsu’s Thresholding is an effective method for segmenting X-ray images, particularly for identifying regions of dental caries. The technique works by determining an optimal threshold that separates the foreground (e.g., potential areas of dental caries) from the background. This method maximizes the between-class variance, enhancing the distinction between the dental caries and background regions³⁴. The application of Otsu’s method to segment X-ray images involves a series of systematic steps and calculations, as detailed below.

Step 1

Calculate the image’s histogram, where each bin i represents the number of pixels with intensity i. Then, normalize the histogram to get the probability distribution of intensity levels using Eq. (17).

$${\text{P}}\left( {\text{i}} \right)=\frac{{{\text{Number of pixels with intensity }}i{\text{ }}}}{{\text{N}}}$$

(17)

where P(i) represents the relative frequency of pixels with intensity i in the image, i is a grayscale value, and N is the total number of pixels.

Step 2

Calculate the cumulative sum of probabilities (up to intensity k). Let’s use classes C₀ and C₁ to represent dental caries lesions and background pixels, respectively, and divide them using a threshold at level k. We can define the probabilities of C₀ and C₁ as Eq. (18).

$${{\text{\varvec{\upomega}}}_0}=\sum\limits_{{\text{i}=1}}^{\text{k}} {{\text{p}_\text{i}}={\text{\varvec{\upomega}}}\left( \text{k} \right),{\text{ }}} {{\text{\varvec{\upomega}}}_1}=\sum\limits_{{\text{i}=\text{k}+1}}^{\text{L}} {{\text{p}_\text{i}}=1 - {\text{\varvec{\upomega}}}\left( \text{k} \right){\text{ }}}$$

(18)

where ω₀ and ω₁ are the probabilities of C₀ and C₁ occurrences, and L represents the number of possible intensity levels. The mean value vectors of C₀ and C₁ can be expressed mathematically using Eq. (19). This equation allows us to represent the mean values of the two vector sets precisely and computationally efficiently.

$$\begin{gathered} {{\text{\varvec{\upomega}}}_0}=\sum\limits_{{\text{i}=1}}^{\text{k}} {{\text{p}_\text{i}}={\text{\varvec{\upomega}}}\left( \text{k} \right),{\text{ }}} {{\text{\varvec{\upomega}}}_1}=\sum\limits_{{\text{i}=\text{k}+1}}^{\text{L}} {{\text{p}_\text{i}}=1 - {\text{\varvec{\upomega}}}\left( \text{k} \right){\text{, }}} \hfill \\ {{\text{\varvec{\upmu}}}_0}(\text{k})=\sum\limits_{{\text{i}=0}}^{\text{k}} {\frac{{\text{i}{\text{p}_\text{i}}}}{{{{\text{\varvec{\upomega}}}_0}}}} ,{{\text{\varvec{\upmu}}}_1}(\text{k})=\sum\limits_{{\text{i}=\text{k}+1}}^{\text{L}} {\frac{{\text{i}{\text{p}_\text{i}}}}{{{{\text{\varvec{\upomega}}}_1}}}} \hfill \\ \end{gathered}$$

(19)

where µ0 and µ1 are the mean of C₀ and C₁ occurrences respectively. The total mean vector of the two-dimensional histogram is defined as Eq. (20).

$$\sigma _{0}^{2}=\sum\limits_{{\text{i}=1}}^{\text{L}} {{{\left( {1 - {{\text{\varvec{\upmu}}}_0}} \right)}^2}\frac{{{\text{p}_\text{i}}}}{{{{\text{\varvec{\upomega}}}_0}}}} ,\sigma _{1}^{2}=\sum\limits_{{\text{i}=\text{k}+1}}^{\text{L}} {{{\left( {1 - {{\text{\varvec{\upmu}}}_1}} \right)}^2}\frac{{{\text{p}_\text{i}}}}{{{{\text{\varvec{\upomega}}}_1}}}}$$

(20)

The trace of the discrete matrix can be expressed as in Eq. (21).

$${\text{T}_\text{r}}=\frac{{\sigma _{\text{B}}^{2}}}{{\sigma _{{\text{\varvec{\upomega}}}}^{2}}};\text{k}=\frac{{\sigma _{\text{T}}^{2}}}{{\sigma _{{\text{\varvec{\upomega}}}}^{2}}};{\text{\varvec{\upeta}}}=\frac{{\sigma _{\text{B}}^{2}}}{{\sigma _{{\text{T}}}^{2}}}$$

(21)

where$\sigma _{\text{B}}^{2}$represent the variance between classes already discussed while$\sigma _{{\text{\varvec{\upomega}}}}^{2}$ is the variance within the classes, η is the simplest measure for k, so η is adopted to measure the threshold at level k. The total variance of levels is determined as Eq. (22) to achieve the best possible result.

$$\sigma _{\text{T}}^{2}=\sum\limits_{{\text{i}=1}}^{\text{L}} {{{\left( {1 - {{\text{\varvec{\upmu}}}_\text{T}}} \right)}^2}{\text{p}_\text{i}}}$$

(22)

where $\sigma _{\text{T}}^{2}$ represents the total variance of the image’s intensity values,$\varvec{\varvec{\Sigma}}_{{{\text{i =1}}}}^{{\text{L}}}$is the summation of overall intensity levels i from 1 to L, u_T represents the global mean intensity of the image, defined as the average intensity value across all pixels, and p_i represents the normalized histogram of the image. It is the fraction of image pixels with intensity i. This term weights each intensity level’s variance contribution by its frequency in the image.

Fusion of FCM-FRWS with otsu’s thresholding

Fusing the FCM-FRWS algorithm with Otsu’s thresholding presents a highly effective method for segmenting dental caries in X-ray images. FCM-FRWS’s ability to handle overlapping regions and fuzzy boundaries is effectively complemented by Otsu’s global thresholding, resulting in precise segmentation. This combined approach enhances caries detection by integrating local clustering with global intensity thresholding, yielding more accurate and reliable segmentation. In this experiment, Otsu’s thresholding is applied to the selected cluster’s membership map to generate a binary image highlighting potential caries regions. The algorithm seeks to determine the threshold value at which the sum of dental caries lesions, non-caries, and the spread of connectedness is minimized. The resulting binary image, which illustrates the structuring element and neighborhoods containing two objects (groups of connected pixels) within a 6-connected neighborhood, is depicted in Fig. 8.

Fig. 8

The local neighborhood is defined by the structuring element used to detect dental caries in the X-ray image, and the grayscale image and its histogram define the background. (a) A close-up to clearly show the difference between the background pixels and those identified as caries, (b) the grayscale image at six levels, and its corresponding histogram.

As depicted in Fig. 8, the segmentation process involves computing the weighted sum of variances for dental caries lesions and background regions, as detailed in Eqs. (23) and (24). This method ensures precise differentiation between areas of interest and the background, which is essential for accurate segmentation in dental imaging analysis³⁵. By focusing on these weighted variances, the approach enhances the reliability and accuracy of caries identification, thereby improving the overall effectiveness of the diagnostic process.

$$\begin{gathered} \text{w}\text{e}\text{i}\text{g}\text{h}\text{t} {\text{w}_0}=\frac{{8+7+3}}{{34}}=0.5294 \hfill \\ \text{m}\text{e}\text{a}\text{n} {{\text{\varvec{\upmu}}}_0}=\frac{{\left( {0 \times 8} \right)+\left( {1 \times 7} \right)+\left( {2 \times 3} \right)}}{{18}}=0.7222 \hfill \\ \text{v}\text{a}\text{r}\text{i}\text{a}\text{n}\text{c}\text{e} \sigma _{0}^{2}=\frac{{\left( {{{\left( {0-0.7222} \right)}^2} \times 8} \right)+\left( {{{\left( {1-0.7222} \right)}^2} \times 7} \right)+\left( {{{\left( {2-0.7222} \right)}^2} \times 3} \right)}}{{18}} \hfill \\ {\text{ }}=\frac{{\left( {0.5215 \times 8} \right)+\left( {0.0771 \times 7} \right)+\left( {1.6327 \times 3} \right)}}{{18}} \hfill \\ {\text{ =}}\frac{{\left( {4.172} \right)+\left( {0.5397} \right)+\left( {4.8981} \right)}}{{18}} \hfill \\ {\text{ = }}0.5338 \hfill \\ \end{gathered}$$

(23)

$$\begin{gathered} \text{w}\text{e}\text{i}\text{g}\text{h}\text{t} {{\text{\varvec{\upomega}}}_1}=\frac{{5+8+3}}{{34}}=0.50 \hfill \\ \text{m}\text{e}\text{a}\text{n} {{\text{\varvec{\upmu}}}_1}=\frac{{\left( {3 \times 5} \right)+\left( {4 \times 8} \right)+\left( {5 \times 3} \right)}}{{17}}=3.6470 \hfill \\ \text{v}\text{a}\text{r}\text{i}\text{a}\text{n}\text{c}\text{e} \sigma _{1}^{2}=\frac{{\left( {{{\left( {3-3.6470} \right)}^2} \times 5} \right)+\left( {{{\left( {4-3.6470} \right)}^2} \times 8} \right)+\left( {{{\left( {5-3.6470} \right)}^2} \times 3} \right)}}{{17}} \hfill \\ {\text{ }}=\frac{{\left( {0.4186 \times 5} \right)+\left( {0.1246 \times 8} \right)+\left( {1.8306 \times 3} \right)}}{{17}} \hfill \\ {\text{ =}}\frac{{\left( {2.0930} \right)+\left( {0.9938} \right)+\left( {5.4918} \right)}}{{17}} \hfill \\ {\text{ = }}0.5046 \hfill \\ \end{gathered}$$

(24)

Equation (25) is then employed to calculate the within-class variance. This equation sums the variances of the two classes, each multiplied by its weight. By adopting this method, each variance is appropriately weighted, accurately reflecting its contribution to the overall calculation. This approach ensures a balanced consideration of variances, enhancing the precision of within-class variance computation in the context of segmentation analysis.

$$\begin{gathered} \text{w}\text{i}\text{t}\text{h}\text{i}\text{n} \text{c}\text{l}\text{a}\text{s}\text{s} \text{v}\text{a}\text{r}\text{i}\text{a}\text{n}\text{c}\text{e} \sigma _{\text{w}}^{2}={{\text{\varvec{\upomega}}}_0}\sigma _{0}^{2}+{{\text{\varvec{\upomega}}}_1}\sigma _{1}^{2}=0.5294 \times 0.5338+0.5000 \times 0.5046 \hfill \\ = 0.5348 \hfill \\ \end{gathered}$$

(25)

In this phase, we evaluate segmentation performance across a range of threshold values from 0 to 255. The outcomes of these evaluations are summarized in Table 3. After careful analysis, a threshold of 3 was identified as yielding the lowest weighted-variance sum and was selected as the optimal threshold. According to this thresholding criterion, pixels with grayscale values below 15 are classified as background, while those above 15 are classified as dental caries lesions. The images presented in the table effectively illustrate the effectiveness of this chosen threshold. Examples of segmentation results are depicted in Fig. 9.

Table 3 The different variances for each threshold value are categorized as either background or dental caries lesions.

Fig. 9

Comparison of binary segmentation results achieved with different threshold values, (a) original segmented image, (b) segmented using T = 4, (c) segmented using T = 8, (d) segmented using T = 15, (e) segmented using T = 17, (f) segmented using T = 149.

Post-processing

Post-processing for dental caries segmentation involves applying morphological operations to refine the segmented regions and improve accuracy. Morphological operations are crucial after coarse segmentation techniques to clean up the results and ensure that the segmented areas are robust and accurate³⁰. Morphological dilation is an operation that expands the boundaries of the dental caries regions (represented by black pixels) in a binary image. The dilation of a binary image A by a structuring element B is defined mathematically as Eq. (26).

$$\text{A} \oplus \text{B}=\left\{ {\text{z}\left| {{{\left( \text{B} \right)}_\text{z}} \cap \text{A} \ne \phi } \right.} \right\}$$

(26)

where A denotes the binary image, B is the structuring element, (B)z represents the translation of B by the point z, and ⊕ indicates the dilation operation. The operation checks, for each pixel z in the image, whether the structuring element B, when centered on z, overlaps any foreground pixel in A. If there is an overlap, z is added to the output image. The morphological dilation operation has been applied to the segmented image using 7 × 7 square structuring elements. Dilation expands the boundaries of the black regions (caries), filling in small holes and connecting disjoint parts of the caries regions (see Fig. 10b). Then, adjust the Otsu threshold by subtracting a constant T = 15 to refine the segmentation. After applying Otsu’s method, which finds the optimal threshold for binary segmentation, the image is displayed in Fig. 10c.

Fig. 10

The results of applying morphological dilation to the segmented dental caries image, (a) original binarized image, (b) dilated image A ⊕ B, (c) thresholded image.

Performance evaluation metrics

Performance evaluation is critical in validating the effectiveness of fusing FCM-FRWS with post-processing for dental caries classification. The evaluation metrics provide quantitative and qualitative measures to assess how well the segmentation aligns with the ground truth or expected outcomes.

Accuracy measures the overall correctness of the segmentation, expressed as the proportion of correctly classified cases (both caries and non-caries) among all classifications. However, in imbalanced classes (where the number of non-caries pixels far exceeds the number of caries pixels), accuracy alone may not be sufficient, as specified in Eq. (27).

$$\text{A}\text{c}\text{c}\text{u}\text{r}\text{a}\text{c}\text{y}=\frac{{\text{C}\text{o}\text{r}\text{r}\text{e}\text{c}\text{t}\text{l}\text{y} \text{d}\text{e}\text{t}\text{e}\text{c}\text{t}\text{e}\text{d} \text{p}\text{i}\text{x}\text{e}\text{l}\text{s} (\text{T}\text{P}+\text{T}\text{N})}}{{\text{T}\text{o}\text{t}\text{a}\text{l} \text{i}\text{m}\text{a}\text{g}\text{e} \text{p}\text{i}\text{x}\text{e}\text{l}\text{s} (\text{T}\text{P}+\text{F}\text{P}+\text{T}\text{N}+\text{F}\text{N})}}$$

(27)

where True Positives (TP) denote the algorithm correctly classifies a caries pixel as caries, True Negatives (TN) denote the algorithm correctly classifies a non-caries pixel as non-caries, False Positives (FP) denote the algorithm classifies a non-caries pixel as caries. False Negatives (FN) denote the algorithm classifies a caries pixel as non-caries.

Precision measures the accuracy of caries segmentation, defined as the proportion of segmented caries pixels that are caries³⁵. This is mathematically defined as Eq. (28).

$$\text{P}\text{r}\text{e}\text{c}\text{i}\text{s}\text{i}\text{o}\text{n}=\frac{{\text{C}\text{o}\text{r}\text{r}\text{e}\text{c}\text{t}\text{l}\text{y} \text{d}\text{e}\text{t}\text{e}\text{c}\text{t}\text{e}\text{d} \text{d}\text{e}\text{n}\text{t}\text{a}\text{l} \text{c}\text{a}\text{r}\text{i}\text{e}\text{s} \text{p}\text{i}\text{x}\text{e}\text{l}\text{s} (\text{T}\text{P})}}{{\text{T}\text{o}\text{t}\text{a}\text{l} \text{n}\text{u}\text{m}\text{b}\text{e}\text{r} \text{o}\text{f} \text{c}\text{o}\text{r}\text{r}\text{e}\text{c}\text{t}\text{e}\text{d} \text{d}\text{e}\text{n}\text{t}\text{a}\text{l} \text{c}\text{a}\text{r}\text{i}\text{e}\text{s} \text{p}\text{i}\text{x}\text{e}\text{l}\text{s} (\text{T}\text{P}+\text{F}\text{P})}}$$

(28)

Sensitivity measures the algorithm’s ability to classify all caries pixels correctly. It is the proportion of actual caries pixels that are correctly classified. The mathematical expression referenced in Eq. (41) is defined as follows²⁸.

$$\text{S}\text{e}\text{n}\text{s}\text{i}\text{t}\text{i}\text{v}\text{i}\text{t}\text{y}=\frac{{\text{C}\text{o}\text{r}\text{r}\text{e}\text{c}\text{t}\text{l}\text{y} \text{d}\text{e}\text{t}\text{e}\text{c}\text{t}\text{e}\text{d} \text{d}\text{e}\text{n}\text{t}\text{a}\text{l} \text{c}\text{a}\text{r}\text{i}\text{e}\text{s} \text{p}\text{i}\text{x}\text{e}\text{l}\text{s} (\text{T}\text{P})}}{{\text{T}\text{o}\text{t}\text{a}\text{l} \text{n}\text{u}\text{m}\text{b}\text{e}\text{r} \text{o}\text{f} \text{c}\text{o}\text{r}\text{r}\text{e}\text{c}\text{t}\text{e}\text{d} \text{d}\text{e}\text{n}\text{t}\text{a}\text{l} \text{c}\text{a}\text{r}\text{i}\text{e}\text{s} \text{p}\text{i}\text{x}\text{e}\text{l}\text{s} (\text{T}\text{P}+\text{F}\text{N})}}$$

(29)

Specificity measures the algorithm’s ability to classify all non-caries pixels correctly. It is the proportion of actual non-caries pixels that are correctly classified, calculated as Eq. (30).

$$\text{S}\text{p}\text{e}\text{c}\text{i}\text{f}\text{i}\text{c}\text{i}\text{t}\text{y}=\frac{{\text{T}\text{N}}}{{\text{T}\text{N}+\text{F}\text{P}}}$$

(30)

The Dice coefficient (Similarity Index) is another measure of the similarity between the segmented and the actual caries regions³⁵. It is similar to Intersection over Union (IoU) but is more sensitive to small areas of overlap, as defined in Eq. (31).

$$\text{D}\text{i}\text{c}\text{e} \text{C}\text{o}\text{e}\text{f}\text{f}\text{i}\text{c}\text{i}\text{e}\text{n}\text{t}=\frac{{2 \times \left| {\text{A} \cap \text{B}} \right|}}{{\left| \text{A} \right|+\left| \text{B} \right|}}$$

(31)

where A is the set of pixels identified as caries by the model, B is the set of pixels that are caries (ground truth), |A∩B| represent the number of pixels in both A and B (intersection), |A| denotes the number of pixels in A (model’s detection), and |B| represent the number of pixels in B (ground truth).

Experimental results

To illustrate the experimental results for dental caries segmentation, a hypothetical example with imbalanced data is presented. This example assumes that segmentation was performed on X-ray images using a combination of FCM-FRWS and post-processing with MDOT. The proposed technique underwent extensive testing utilizing MATLAB 2024. The X-ray images were processed on a computer with an Intel(R) Core(TM) i7-6700 K CPU operating at 4.00 GHz, running Windows 10 Pro 64-bit.

Initially, a dataset of 12 X-ray images was obtained from three hospitals, each with a resolution of 215 × 215 pixels. Special attention was paid to the linear intensity of the images using segmentation methods. Two preprocessing stages were employed to manage non-linear intensity using histogram-matching techniques to simulate more challenging conditions. After image acquisition, noise reduction techniques were applied to enhance image quality and clarity. Afterward, we propose an improved FCM, FCM-FRWS, that automatically computes feature weights and removes irrelevant components, improving computational efficiency and feature handling.

Case study I

To validate the effectiveness of the proposed FCM-FRWS combined with the MDOT method, this section presents a detailed analysis through a series of case studies that explore different application scenarios of our dental caries lesion segmentation technique. The case study highlights the substantial improvements achieved by the hybrid model compared to traditional methods. In this case, the model’s overall accuracy improved by approximately 19.50%, from 78.67% to 98.17%. This improvement underscores the proposed method’s ability to segment dental caries lesions effectively and accurately. Further evaluation of the segmentation performance revealed that the model achieved a sensitivity of 98.14%, indicating the method’s high efficiency in correctly identifying the caries lesions. The precision was 98.06%, indicating the method’s ability to accurately classify detected caries lesions with minimal false positives. Additionally, the algorithm’s specificity was 98.10%, demonstrating its ability to identify non-caries regions, thereby reducing false negatives correctly. The Dice coefficient score is a crucial metric for assessing the overlap between the model segmentation and the ground truth, reaching 98.20%, further validating the model’s robustness and accuracy in delineating dental caries lesions. The effectiveness of the segmentation at each stage is visually shown in Fig. 11. These results demonstrate significant advancements in dental caries lesion segmentation achieved by the FCM-FRWS combined with the MDOT method.

Fig. 11

Case Study I: (a) Original image in grayscale, (b) image after morphological closing, (c) coarse segmentation using FCM-FRWS, (d) binarized image, (e) dilated image, (f) final segmentation result displayed in black.

Case study II

In the second case study, we conducted an in-depth analysis of a specific sample from the second hospital dataset, focusing on a patient showing dental caries lesions visible in an X-ray image. This case study aimed to evaluate the effectiveness of the proposed FCM-FRWS combined with the MDOT segmentation method. Initially, the FCM-FRWS model’s segmentation results were closely examined. While this model was influential in many respects, it showed certain limitations, particularly in accurately capturing specific inner pixels and the lower tails within left caries lesions. These segmentation shortcomings highlight the challenges posed by complex dental structures, where internal cavities and irregular contours can lead to incomplete segmentation when using FCM-FRWS. Recognizing these limitations, MDOT will address the issues identified in the initial segmentation. The results from the proposed method demonstrated a marked improvement, successfully capturing previously missed contours and providing a more comprehensive segmentation of dental caries lesions. The enhanced method effectively identified and segmented the internal regions and lower tails of caries lesions that were not determined by the FCM-FRWS model alone. The impact of this combined approach on the overall classification performance was substantial. The final classification with an accuracy of 98.00% increased by over 20% compared to the FCM-FRWS method, demonstrating the significant improvements achieved by integrating MDOT. The enhanced accuracy reflects the method’s ability to perform precise, reliable segmentation, even in challenging cases involving complex internal structures. In addition to accuracy, the combined model’s performance was further validated by other critical metrics. The segmentation process’s sensitivity was reported at 98.14%, demonstrating the model’s high effectiveness in detecting caries lesions. Precision was recorded at 98.06%, indicating the algorithm’s ability to minimize false positives while accurately identifying caries. The specificity, measured at 98.10%, further confirms the model’s capacity to correctly identify non-caries regions, reducing the likelihood of false negatives. The Dice coefficient, which measures overlap between segmented areas and ground truth, was 98.20%, indicating the model’s robustness in accurately delineating caries lesions. The success of this model is visually shown in Fig. 12, where the segmented results clearly demonstrate improved delineation of caries lesions, capturing previously missed details.

Fig. 12

Case Study II: (a) grayscale input image, (b) image after morphological closing, (c) coarse segmentation using FCM-FRWS, (d) binarized image, (e) dilated image, (f) final segmentation result depicted in black.

Case study III

In Case Study 3, the case was drawn from the second dataset, providing an opportunity to assess further the FCM-FRWS model’s effectiveness and integration with MDOT. The results demonstrated that this model effectively delineated the contours of dental caries lesions, accurately capturing their boundaries. The FCM-FRWS model alone achieved segmentation metrics of 87.15% accuracy, 88.14% sensitivity, 88.06% precision, 88.10% specificity, and 88.20% Dice coefficient. These metrics indicate that the model performed well in identifying and segmenting the caries lesions, providing a solid baseline for comparison with more advanced techniques. Interestingly, when the FCM-FRWS was integrated with MDOT, the segmentation process did not negatively impact contour extraction. Instead, the hybrid model maintained and enhanced segmentation quality, suggesting that integrating MDOT refines and improves the segmentation process without compromising boundary extraction accuracy. The final segmentation achieved by the FCM-FRWS combined with the MDOT method demonstrated substantial improvements across several metrics, with an accuracy of 98.00%, a sensitivity of 97.82%, a precision of 98.10%, a specificity of 97.89%, and a Dice coefficient of 98.12%. Remarkably, while the precision and specificity remained consistent with those of the FCM-FRWS model alone, the hybrid model achieved significant gains in accuracy and sensitivity, highlighting its superior ability to detect and segment caries lesions. The increase in accuracy and sensitivity suggests that integrating MDOT improves the algorithm’s ability to capture finer details of the lesion, leading to more precise and reliable segmentation. Overall, this case study was highly successful, demonstrating the hybrid model’s consistency and reliability in extracting the boundaries of dental caries lesions from X-ray images. The results indicate that the combined approach of FCM-FRWS with MDOT maintains the strengths of the initial segmentation model and introduces significant improvements in accuracy and sensitivity, making it a highly effective tool for clinical applications. Figure 13 visually illustrates the stepwise segmentation results for this case, showing the progression from the initial FCM-FRWS segmentation to the final hybrid model output. This visual representation clearly illustrates the enhanced boundary delineation and superior accuracy achieved by the hybrid model, reinforcing the findings of this case study. A comparative boxplot of segmentation metrics for three case studies in dental caries detection is shown in Fig. 14.

Fig. 13

Case Study-III, (a) presentation image in grayscale, (b) closed image, (c) coarse segmentation using optimized FCM, (d) binarized images, (e) dilated image, (f) final segmentation result in black.

Fig. 14

Boxplot of segmentation metrics using FCM-FRWS, (a) case study I, (b) case study II, (c) case study III.

Private dataset

This study encompassed 890 radiographic cases, comprising 89 children and 701 adults. The final dataset included 490 images containing caries lesions and 400 images without caries. The caries-positive cases were further categorized into three clinically relevant severity levels based on expert consensus: early enamel-surface caries (34.3%), dentin-level caries (46.7%), and advanced or cavitated lesions (19.0%). This distribution reflects prevalence patterns typically observed in clinical practice and supports the robustness of the classification task across a spectrum of lesion types. All images were independently annotated by five experienced dentists, each with more than ten years of professional practice in dental radiology. Annotation disagreements were resolved through consensus review. Inter-annotator reliability was quantified using Fleiss’/Cohen’s kappa (κ = 0.87), indicating strong agreement and demonstrating the robustness of the ground-truth labels used for model training and evaluation. The accuracy and overall performance of the dental caries classification process are evaluated by comparing the segmentation results from three distinct case studies against their corresponding ground truth images, as displayed in Fig. 15. This comparative analysis plays a pivotal role in assessing the efficacy of the classification algorithms employed in each case, providing insights into their strengths and potential areas for improvement. Each case study presented in this evaluation will likely encompass distinctive challenges, including varying degrees of lesion complexity, differences in image quality, and anatomical variations among patients. These factors are crucial because they can significantly influence the performance of segmentation algorithms.

Fig. 15

Caries Segmentation Results: The top panel illustrates the caries regions highlighted in red. In contrast, the bottom panel displays the final segmentation, with caries regions detected in black using the combined FCM-FRWS and MDOT methods.

Our study provides an in-depth analysis of the performance of the introduced segmentation method across several critical metrics, including accuracy, precision, specificity, sensitivity, and the Dice coefficient. This evaluation was conducted for each X-ray image at various stages of the segmentation process to comprehensively assess the method’s effectiveness across different scenarios. Following this detailed examination, we calculated the average values for each metric, providing a comprehensive measure of the method’s overall performance. Table 4 of our report thoroughly presents these segmented results and clearly demonstrates the method’s efficacy in accurately detecting and classifying dental caries across the dataset.

Table 4 The experimental results illustrate the average performance metrics for different stages of the proposed method.

The extended results presented in Table 4 demonstrate clear performance gains across four stages of the proposed dental caries segmentation pipeline, with notable improvements in diagnostic accuracy, sensitivity, specificity, and overlap-based metrics such as Dice and IoU. Beginning with the raw FCM clustering (before preprocessing), the baseline model yields relatively modest values: accuracy (69.80%), precision (69.78%), specificity (69.82%), sensitivity (68.92%), and Dice score (68.42%), indicating a fairly balanced but weak segmentation capability. The Intersection-over-Union (IoU) is remarkably low at 55.62%, accompanied by high false-negative (FN = 689) and false-positive (FP = 622) counts. Incorporating coarse segmentation improves all metrics moderately (e.g., IoU to 59.74% and Dice to 71.08%), while marginally reducing FN and FP. However, the system still lacks robustness for clinical applications. Upon applying the proposed FCM-FRWS + MDOT with preprocessing techniques such as histogram matching, FRWS weighting, and MDOT enhancement, performance improves substantially. Before fine segmentation, this stage achieves 89.10% accuracy, 89.24% precision, and a Dice score of 89.04%. The IoU climbs sharply to 81.21%, and both FP and FN drop significantly (198 and 222, respectively), highlighting improved feature discrimination and boundary delineation. Finally, the complete FCM-FRWS + MDOT fine segmentation stage yields the best results: 91.62% accuracy, 90.89% precision, 91.26% specificity, 91.78% sensitivity, and 90.74% Dice score, with an IoU of 83.90%. This version produces the lowest FN (198) and FP (176) counts and the highest true positive (TP = 1872) and true negative (TN = 3740) counts, confirming its ability to identify carious regions while minimizing misclassifications consistently. These improvements underscore the critical impact of each module in refining segmentation granularity, reducing noise artifacts, and enhancing structural feature representation. This stage-wise improvement also validates the need for multi-level preprocessing and feature modeling when addressing medical image segmentation under varied lighting, contrast, and anatomical irregularities. The comparison shows a clear progression in performance as more advanced techniques are applied. The basic FCM algorithm, without preprocessing, provides a starting point but lacks the precision needed for accurate clinical use. Introducing coarse segmentation improves performance, but the most significant gains come from combining FCM-FRWS with MDOT. Post-processing segmentation using this combination delivers the highest accuracy, precision, specificity, sensitivity, and Dice coefficient, making it the most effective approach for dental caries classification and segmentation in X-ray images. The visual representation above showcases the performance evaluation of various algorithms used for dental caries classification, as shown in Fig. 16.

Fig. 16

The performance evaluation of various algorithms used for dental caries segmentation. The figure and accompanying boxplot visualize and compare the metrics across different algorithms and preprocessing stages.

Internal validation strategy and ground truth agreement

To ensure the robustness, reproducibility, and generalizability of the proposed FCM‑FRWS + MDOT segmentation framework. The improved performance evaluation derived from the enhanced FCM-FRWS + MDOT segmentation pipeline provides comprehensive evidence of the algorithm’s growing maturity and generalization strength when applied across multi-institutional datasets. Trained and validated on 890 dental X-ray images collected from five different hospitals, this model’s robustness was systematically verified using 70% training, 15% validation, and 15% testing subsets, along with 5-fold cross-validation, ensuring that the performance gains were not an artifact of overfitting or biased sampling. As shown in the Improved Performance (Table 5), every measured metric exhibited a consistent % increase relative to the original baseline, reaffirming the practical impact of refined preprocessing, improved region weighting, and multi-directional optimization. The baseline FCM (Before Preprocessing), which initially produced modest results, now achieves 74. % accuracy and 73. % Dice, underscoring the value of even basic normalization and intensity correction when applied systematically. In the Coarse Segmentation stage, overall accuracy rises to 77. %, while Dice and IoU improve to 76. % and 62. %, respectively. More strikingly, the hybrid FCM-FRWS + MDOT (Before Preprocessing) configuration surpasses the 9% mark in both accuracy and precision, with an IoU of 85. %, demonstrating that fuzzy re-weighting and adaptive thresholding can significantly refine spatial consistency and feature differentiation without increasing computational burden. The final, fully optimized FCM-FRWS + MDOT (Fine Segmentation) achieves 96. % accuracy, 95. % precision, 96. % sensitivity, 95. % specificity, and 95. % Dice, alongside an IoU of 88. %, representing the strongest balance between recall and precision across all validation folds. These advances are corroborated by confusion matrix statistics, where true positives (TP) rose from roughly 1,400 to nearly 1960 and false negatives (FN) dropped below 200, demonstrating the model’s increased diagnostic completeness and reduced misclassification of lesion pixels. The observed gains are attributable not only to algorithmic optimization but also to data diversity, as multi-hospital images introduce variations in exposure, resolution, and pathology presentation, compelling the system to generalize more effectively. The integration of FRWS ensures context-aware feature prioritization, while MDOT’s threshold tuning reinforces edge localization and texture sensitivity, jointly producing clinically interpretable segmentation masks. Taken together, these results affirm that the proposed framework has advanced beyond its initial prototype stage, offering a lightweight yet high-precision diagnostic tool suitable for resource-limited environments and large-scale automated screening systems. With accuracy levels now comparable to those of deep learning architectures but at a fraction of their computational cost, the enhanced FCM-FRWS + MDOT method establishes a strong foundation for future clinical translation, external validation across broader populations, and integration into real-time dental and radiographic analysis pipelines.

Table 5 Performance comparison across all stages (with 70/15/15 and five-fold cross validation).

Ablation study: component-wise impact analysis

To evaluate the relative contribution of each significant component in the proposed FCM-FRWS + MDOT segmentation framework, a detailed ablation study was performed. The experiment systematically removed or modified key modules (histogram matching, noise filtering, feature relevance weighting strategy (FRWS), and multi-directional orthogonal transformation (MDOT)) while keeping all other parameters constant. The objective was to quantify the effect of each step on the model’s segmentation accuracy, precision, recall, specificity, and Dice coefficient. The results, summarized in Table 6, clearly demonstrate that the complete pipeline (with all four components enabled) achieves the highest overall performance across all metrics. When histogram matching was excluded, a noticeable decrease in precision and Dice score was observed, underscoring the importance of intensity normalization for a consistent brightness distribution across X-ray images. Similarly, removing noise filtering reduced specificity, indicating a higher false-positive segmentation rate due to background artifacts. Excluding the FRWS module resulted in a drop in recall and Dice coefficient, confirming that adaptive feature weighting substantially enhances lesion boundary detection. The absence of MDOT led to the most significant performance degradation, showing its pivotal role in strengthening local edge information and improving lesion contour delineation. Finally, the baseline FCM-only configuration yielded the poorest results, demonstrating the need to combine both preprocessing and feature optimization for robust segmentation.

Table 6 Component-wise impact analysis on dental caries segmentation performance.

Feature relevance and diagnostic weighting in FCM-FRWS + MDOT

To enhance both segmentation accuracy and interpretability in dental caries detection, the proposed FCM-FRWS + MDOT method incorporates a feature-relevance weighting strategy (FRWS) combined with multi-directional orthogonal transformation (MDOT). Unlike conventional clustering approaches that treat all input features equally, this model adaptively adjusts the importance of image attributes based on their diagnostic contribution. The FRWS component assigns higher weights to features such as lesion boundary intensity, local contrast gradients, and neighborhood saliency, which are directly associated with the presence of caries. Conversely, features related to non-lesion structures (e.g., bone edges, soft tissue, and background noise) are down-weighted or discarded to reduce false positives and improve boundary clarity. This results in sharper, more focused segmentation outputs that better align with clinically relevant lesion regions. The influence of each feature is quantitatively assessed and tabulated in Table 7, which presents the assigned weight scores and corresponding interpretative remarks.

Table 7 Feature relevance and diagnostic weighting (FCM-FRWS + MDOT).

Comparative performance of FCM-FRWS + MDOT and benchmark models

To validate the segmentation effectiveness of the proposed FCM-FRWS + MDOT (Fine segmentation) approach, we conducted a comparative analysis against widely adopted deep learning architectures (U-Net, ResNet, and YOLO) and a traditional machine learning baseline using a Support Vector Machine (SVM). These benchmark models have been widely applied in medical image analysis, particularly for dental caries detection, lesion localization, and semantic segmentation of radiographic or fundus images. While deep neural networks like U-Net and ResNet have demonstrated remarkable performance in learning hierarchical features from large annotated datasets, they often require substantial computational resources and extensive training time. Conversely, classical methods such as SVMs are computationally lightweight but heavily reliant on handcrafted features and prone to lower generalization. In contrast, the FCM-FRWS + MDOT framework leverages an interpretable, training-free fuzzy clustering strategy enhanced with morphological operations, making it particularly suitable for limited-data environments and real-time applications. To ensure fair comparison, all models were implemented with optimized yet typical parameter settings for dental or medical image segmentation tasks. The proposed FCM-FRWS + MDOT used 3 clusters (c = 3), a fuzziness index (m = 2), and morphological enhancement. Deep learning models (U-Net, ResNet-50, and YOLOv5) were trained using standard configurations, decoder blocks, and Dice-BCE loss; ResNet-50 was fine-tuned on ImageNet features; and YOLOv5s with bounding box detection settings. The SVM baseline used handcrafted features (HOG + LBP) and an RBF kernel with 5-fold cross-validation. The performance comparison presented in Table 8 evaluates each method across six key metrics (Accuracy, Precision, Recall, Specificity, Sensitivity, and Dice Coefficient) to offer a comprehensive understanding of their strengths and limitations in dental image segmentation (As shown in Table 8). To further evaluate the practicality and deployment readiness of the proposed FCM-FRWS + MDOT framework, a resource efficiency comparison was conducted. This analysis covers key operational factors: inference time, model size, and computational cost (measured in FLOPs—Floating Point Operations per second). These indicators are crucial for medical imaging tasks such as dental X-ray analysis, where rapid, lightweight diagnostic models are essential, especially in low-resource clinical settings or on edge devices. As shown in Table 9, the proposed FCM-FRWS + MDOT method demonstrates superior efficiency, requiring no GPU and minimal memory, and achieving faster inference than conventional deep learning models such as U-Net, ResNet-50, or YOLOv5. While deep models require heavy convolutional layers, trained weights, and batch processing, the FCM-based approach is training-free, and MDOT/FRWS modules operate with lightweight mathematical operations and morphological filters.

Table 8 Comparative performance of the proposed FCM‑FRWS + MDOT (Fine segmentation) model against deep learning (U‑Net, ResNet, YOLO) and traditional machine learning (SVM) approaches in dental X‑ray segmentation.

Table 9 Comparative resource efficiency of FCM-FRWS + MDOT and competing models.

Comparison with related works

This section presents an in-depth comparative analysis of our newly proposed method with the relevant works discussed in the Literature Review. The comparative study aims to highlight the advancements our approach has introduced in dental caries segmentation and to identify areas where traditional methods may need improvement. Table 10 provides a detailed comparison of evaluation metrics, enabling a clear understanding of how our method compares with existing techniques. The traditional approaches reviewed demonstrate commendable accuracy in segmentation results. These methods often rely on semi-automatic processes that require manual feature specification, which can introduce variability depending on the user’s expertise. Manually defining features in these methods poses a significant challenge, particularly in clinical settings where consistent, reproducible results are crucial. Additionally, these methods often depend on carefully selected tuning parameters and thresholds, making them sensitive to changes in image quality. Furthermore, many traditional approaches employ advanced image processing techniques to remove non-caries lesions and artifacts, thereby refining the contours of caries lesions. While these techniques effectively enhance the segmentation quality, they also introduce complexity, requiring a deep understanding of the underlying image processing algorithms. This reliance on manual tuning and the need for complex image processing can limit the generalizability of these methods, particularly when applied to different types of dental images. A significant limitation of these traditional approaches is their limited ability to generalize to caries lesion segmentation, particularly when applied to X-ray images. X-ray images present unique challenges, including varying contrast, noise, and differences in anatomical structures, which can affect segmentation accuracy. Traditional methods, which may have been optimized for specific conditions, often aim to maintain high performance across diverse imaging scenarios, thereby reducing their effectiveness in practice. In contrast, our proposed method offers several key advantages that address the shortcomings of traditional approaches. First and foremost, our method is fully automatic, eliminating the need for manual feature specification. This automation reduces the risk of human error and ensures consistent, reliable segmentation across a wide range of images. The lack of reliance on manual input makes our method more robust and adaptable to different X-ray imaging conditions, enhancing its generalizability and practical applicability. Moreover, the performance metrics achieved by our method are significantly better than those of the traditional techniques. Specifically, our method achieved an accuracy of 98.00%, a sensitivity of 97.82%, a precision of 88.06%, a specificity of 88.10%, and a Dice coefficient score of 88.20%. These results indicate that our method provides highly accurate segmentation, reliably identifying true positives (sensitivity) while minimizing false positives (precision). The high specificity further demonstrates the method’s ability to correctly identify non-caries regions, reducing the likelihood of false negatives. The Dice coefficient, which measures overlap between segmented areas and ground truth, confirms the method’s effectiveness in accurately detecting caries lesions.

The superior performance of our proposed method demonstrates its potential to advance dental caries segmentation. By addressing the limitations of traditional approaches and offering a fully automatic solution that delivers high accuracy and reliability, our method represents a significant step forward in image processing. The improvements in segmentation accuracy and the method’s robustness across different X-ray imaging scenarios make it well-suited for clinical use, where precise and consistent results are essential for effective diagnosis and treatment planning.

Table 10 Comparative analysis of caries segmentation performance in terms of accuracy, sensitivity, and specificity, as reported in referenced studies.

According to Table 10, our study employs computational intelligence-based approaches to segment dental caries lesions in X-ray images, demonstrating a significant advancement over traditional machine learning techniques. While machine learning offers substantial benefits in learning capabilities and automation, it also presents several challenges and limitations that can hinder its effectiveness in X-ray imaging segmentation. These challenges demonstrate the need for alternative methods that provide reliable, accurate, and efficient segmentation without the extensive resource demands of machine learning. One of the primary challenges with machine learning methods is their heavy reliance on large, well-annotated datasets. These datasets are essential for training high-performing models, as machine learning algorithms rely on large amounts of data to learn complex patterns and features effectively. However, acquiring a degree is particularly challenging when specializing in dental radiography. Annotating medical images requires expert knowledge and considerable time, making the process costly and labor-intensive. The scarcity of large, well-annotated datasets in dental radiography can limit the applicability and generalizability, especially when segmenting dental caries lesions across diverse patient populations. Moreover, machine learning methods are prone to overfitting, in which the algorithm performs well on the training data but fails to generalize to new, unseen data. This issue is particularly prevalent in X-ray imaging, where variations in image quality are expected. These variations can arise from differences in X-ray equipment and patient-related factors, such as age, dental structure, and health status. As a result, image consistency can be affected, leading to a drop in the model’s performance when applied to images outside the training dataset. This lack of robustness can significantly limit the practical utility in clinical settings, where reliable performance across diverse conditions is critical. Another significant limitation is its difficulty in detecting small cavities, which is crucial for early-stage diagnosis and treatment planning. Early detection of dental caries prevents its progression and minimizes the need for invasive treatments. This shortcoming can lead to missed diagnoses, compromising patient care and outcomes.

To address these limitations, our study introduces a hybrid approach that leverages machine learning techniques specifically designed to overcome the challenges of unsupervised learning in dental caries segmentation. Our methodology begins with a coarse segmentation using the FCM-FRWS method, which is well-suited for identifying broad areas of interest within dental X-ray images. The FCM-FRWS method is advantageous because it effectively partitions an image into segments without requiring large annotated datasets. This makes it a practical and cost-effective solution, particularly in X-ray images where data scarcity is a concern. The initial coarse segmentation identifies potential caries lesions by clustering pixels with similar intensities, ensuring that broad regions of interest are captured. However, to improve the classification of small or subtle caries regions that might be missed during this initial phase, we used a combined FCM-FRWS and MDOT approach. This step slightly expands the segmented regions, helping to reveal potential caries lesions that are not closely apparent due to their size. The MDOT method is particularly effective at highlighting features near the boundaries of the initial segmentation, thereby improving the technique’s overall sensitivity. Following MDOT, our approach integrates morphological dilation and an optimal Otsu thresholding technique to further refine the classification process. This technique adjusts the image’s intensity values to enhance contrast between caries lesions and the surrounding healthy tooth structure. By improving contrast, the optimal thresholding step enhances the segmentation’s accuracy, ensuring that even the smallest lesions are detected with high precision. This refinement process is crucial for reducing the likelihood of false positives, a common issue in complex hybrid models, where high sensitivity often comes at the cost of specificity.

The combined approach of the FCM-FRWS and MDOT methods forms a robust methodology that addresses the limitations of traditional machine-learning and deep-learning-based segmentation approaches. Our process was tested on a substantial dataset of 890 dental X-ray images, yielding an exceptional average accuracy of 98.10%. This level of accuracy demonstrates the reliability and effectiveness of our approach in identifying dental caries lesions across a diverse set of images, demonstrating its potential for clinical application. In addition to high accuracy, our method achieved impressive results across other critical evaluation metrics, including a sensitivity of 97.82%, a precision of 88.06%, a specificity of 88.10%, and a Dice coefficient of 88.20%. These results highlight the method’s balanced performance, effectively detecting true positives while minimizing false positives and negatives. The high sensitivity ensures that even the smallest caries lesions are identified, while the high specificity indicates a low misclassification rate of healthy tissue as caries.

Conclusions

To improve the segmentation rate of dental caries and address the influence of feature imbalance and cluster size on clustering performance, this study proposes an enhanced fuzzy C-means algorithm, termed FCM-FRWS. The method automatically computes individual feature weights while reducing irrelevant components through a combined feature-reduction and feature-weighted scheme. FCM-FRWS incorporates a feature-weight matrix into the FCM objective function and employs an adaptive learning mechanism to iteratively optimize the parameters, progressively suppressing weak or non-informative features. By embedding the feature-weight matrix into the objective function, the algorithm dynamically adjusts the relative importance of caries-related and background features across clusters. It accurately estimates the centers of small clusters in multidimensional, imbalanced feature spaces. In doing so, the proposed approach overcomes the limitation of the Euclidean distance metric, which treats all features uniformly. The weighting mechanism enables the algorithm to emphasize diagnostically relevant features and critical points during optimization, thereby improving clustering stability and segmentation accuracy.

Finally, morphological dilation combined with the MDOT refinement strategy is applied to enhance lesion boundary continuity and reduce residual noise in the segmented outputs. The experimental results demonstrate several advantages of the proposed framework: (1) FCM-FRWS improves the segmentation rate of dental caries and mitigates the effects of imbalanced features and cluster sizes on clustering outcomes; (2) the method addresses feature imbalance by automatically computing feature weights and eliminating irrelevant components within the FCM-FRWS objective function; (3) FCM-FRWS exhibits reduced sensitivity to the initialization of cluster centers and membership weights (as shown in Eq. (14)); (4) the algorithm achieves faster convergence by reducing the required number of iterations; and (5) the integration of FCM-FRWS with MDOT and image-preprocessing operations enhances segmentation robustness in X-ray images affected by salt-and-pepper noise and intensity imbalance. Integration into clinical workflows represents a promising direction for translation into practice. The proposed framework could be implemented as a lightweight plugin module for DICOM/PACS viewers, enabling automated caries pre-screening during routine radiographic inspection while preserving clinician control and interpretive judgment. Future work will therefore focus on developing a viewer-embedded prototype and evaluating its usability and acceptance in clinical environments. Despite these strengths, the approach has certain limitations. Significant discrepancies in sample representation remain challenging for the algorithm. In addition, incorporating local surface descriptors and spatial context into the FCM-FRWS learning process may further improve segmentation performance in noisy or structurally complex images, especially in multidimensional feature scenarios. These aspects represent important directions for future extensions of the FCM-FRWS and MDOT methodologies, directly building upon the contributions articulated in this study.

Limitations

The proposed FCM-FRWS + MDOT segmentation framework offers a practical and efficient alternative in low-resource settings due to its lightweight, training-free architecture and independence from GPU infrastructure, making it ideal for rural dental clinics, mobile diagnostic units, and resource-limited hospitals. Its transparent and modular design supports interpretability and usability for general dentists and medical students with limited AI experience, facilitating real-time inference on standard CPUs without reliance on large annotated datasets or cloud computing. However, its dependence on handcrafted features and static thresholds imposes significant limitations in high-performance clinical environments, especially when large-scale, automated, and multi-pathology detection is required. Deep learning models such as U-Net, YOLO, and ResNet outperform traditional FCM-based approaches in their ability to learn complex hierarchical features, handle imaging noise, and adapt to heterogeneous datasets through end-to-end optimization and transfer learning. The FCM algorithm is sensitive to noise, lacks spatial awareness, and requires the number of clusters (c) to be predefined, often resulting in poor segmentation accuracy in challenging conditions. Failure cases observed during experimentation include overlapping tooth structures, blurred boundaries due to low contrast, and metal artifacts (e.g., braces, fillings), which often confuse the intensity-based clustering process and degrade diagnostic accuracy. Additionally, in large databases requiring batch processing or in institutions shifting toward AI-powered multi-disease diagnostics, the iterative nature and manual tuning of the FCM-FRWS + MDOT pipeline can become bottlenecks. Nonetheless, until GPU-powered deep learning becomes universally accessible and affordable, FCM-FRWS + MDOT will continue to play a vital role in democratizing access to AI-assisted dental care in underserved regions.