Introduction

Odontogenic tumors (OTs) correspond to a group of lesions that originate exclusively from tooth related regions, including maxillary bones and oral mucosa. They derive from epithelial, mesenchymal and/or ectomesenchymal cells that are or were part of the tooth formation process1. OTs are uncommon, representing up to 1% of all oral lesions, with an incidence of less than 0.5 cases per 100,000 inhabitants2. The most common types include odontoma, ameloblastoma (AM) and odontogenic myxoma. OTs can be benign or malignant. Benign tumors are subdivided in three groups based on the type of odontogenic tissue involved, into: (i) tumors composed of odontogenic epithelium with fibrous and mature stroma without odontogenic ectomesenchyme, (ii) mixed epithelial and mesenchymal tumors (mixed odontogenic tumors), and (iii) tumors derived from mesenchyme or ectomesenchyme1.

Among mixed odontogenic tumors, the primordial odontogenic tumor (POT) is a recently described entity, first included in the 2017 World Health Organization (WHO) classification of head and neck tumors and reaffirmed in the 2022 update1,3. Consequently, only a limited number of cases have been reported to date. This tumor shares certain histological features with other mixed odontogenic tumors, which can complicate its differential diagnosis.

Among the group of mixed tumors, POT, ameloblastic fibroma (AF) and developing odontoma (DO) are relevant to this work due to their close histopathological resemblance and relevance in the differential diagnosis. These entities share epithelial and ectomesenchymal components with variable degrees of organization, often leading to diagnostic challenges, particularly in incisional biopsies. The DO represents an early stage of odontogenic tumor development, prior to hard tissue formation, and may histologically resemble POT. The persistent controversy regarding the distinction between POT, AF and DO underscores the importance of a quantitative architectural approach to clarify their morphological features4,5,6.

Digital image processing has become an integral part of modern histopathology, enabling an algorithmic and quantitative analysis of structures of interest. This can be applied to the study of cellular and nuclear morphology with parameters such as area, perimeter, shape, among others, to allow objective data collection and develop statistical classifiers to objectively characterize pathological events and lesions. Several studies have researched the use of machine learning, artificial intelligence (IA) and morphological parameters to differentiate pathological structures and support potential differential diagnoses. Notably, Landini et al. investigated the architecture of the epithelial lining in two odontogenic cyst categories7, while Mahmood et al. and Araújo et al. focused on oral epithelial dysplasia8,9 but such an approach has not been applied to mixed OTs.

The use of algorithmic segmentation could complement diagnostic assessment in incisional biopsies where the complete architecture of the tumor is not represented, by providing reproducible metrics to support the differential diagnosis of mixed odontogenic tumors.

The aim of this study was to morphologically characterize mixed odontogenic tumors and to compare them with normal tooth germs (TG) using computer analysis of digital histopathological images to elucidate some aspects of the structure and spatial organization of the epithelial and ectomesenchymal compartments, to gain a better understanding of these lesions. The goal was to identify both measurable similarities and, importantly, quantitative differences, not just qualitative ones.

Materials and methods

Sample digitalization

In this study, the sample consisted of 15 cases of mixed OTs, including POT (n = 5), AF (n = 5), DO (n = 5) and 5 TGs (cap and bell stages). The DO were cases previously diagnosed as AFO and AFD according to the WHO classifications of head and neck tumors, prior to 2017, which have been re-classified as DO under the new criteria3,10. Formalin-fixed paraffin embedded samples were obtained from Oral Pathology and Histology services in Uruguay, Mexico and Brazil. All samples were routinely stained with HE and all POT samples were also stained with alcian blue.

For slide digitization, a Motic EasyScan© scanner was employed in standard mode with a 40x objective. The PMA.start11 software (Pathomation) was used for the visualization and capture of regions of interest, while ImageJ/Fiji was utilized for image processing12. A standardized dataset was used to validate the procedures, in order to enhance the robustness of the results. Figure 1 summarizes and illustrates the methodology used.

Fig. 1
Fig. 1
Full size image

Resume of used methodology step by step.

Identification of primordial odontogenic tumor subepithelial condensation

The identification of nuclei in epithelial and mesenchymal compartments in POT was conducted to firstly, assess the nuclear density in both compartments and to clarify whether a subepithelial condensation exists in this OT, and secondly, to allow the algorithmic partition of the tissues into virtual cells to investigate the structural makeup of the samples (i.e. layer or strata analysis).

The images of POT stained with Alcian Blue (which provided adequate contrast for visualizing the nuclei) were processed using the ImageJ/Fiji StarDist 2D plugin13 to segment the nuclear profiles (Fig. 2A and B) (details about the parameters used for the Stardist plugin are available from the authors).

To determine nuclear density according to the depth of the ectomesenchyme, the nuclei masks that contacted each other were separated using an ImageJ macro that separated individual nuclear labels14. The images were aligned using the space between the top edge of the image and the first nuclear mask as a reference to allow for a comparable calculation of the depth distance from each nucleus to the superficial epithelial layer. After alignment, a stack was created, where each slice represented a different sample image. The integral of the grayscale (Z-axis projection) was calculated across the stack to compute the likelihood of finding a pixel belonging to a nucleus at a given location tumor sample’s depth.

Fig. 2
Fig. 2
Full size image

Sequence of preliminary steps to determine the epithelial and ectomesenchyme compartments, and nuclear density in the primordial odontogenic tumor. (A) Alcian blue staining. (B) Binary image of nuclear segmentation with StarDist 2D. (C) Segmentation by compartments separated by the epithelial-mesenchymal interface performed with the freehand tool. (D) Combination of B and C.

The alcian blue images were delineated at the epithelial-ectomesenchymal interface using the ImageJ freehand tool, which allowed to separate the epithelial and ectomesenchymal compartments nuclei, so they could be evaluated independently15 (Fig. 2C and D). This required aligning the epithelial-mesenchymal interface with a macro to remove the upper epithelial compartment, leaving only the ectomesenchymal compartment for analysis (and vice versa, for the analysis of the epithelium) by layers or strata16. To analyze the tissue by strata, it is first necessary to partition the ectomesenchymal space into regions (virtual cells, defined later) that are exclusively associated with only one nuclear region. Additionally, the morphological variables those algorithmically defined virtual cells were also extracted: Perimeter: calculated from the center of the pixels of the boundaries, Area: the area defined by the perimeter; Circularity: 4πarea/perimeter2, Shape: perimeter2/area, Rectangularity: area/ArBBox and ArBBox: Feret*Breadth. Details on the software used and morphological parameters have been described elsewhere14. From these values, descriptive statistics were extracted to establish differences between the epithelial and ectomesenchymal nuclei in POT.

Virtual cell analysis

After the identification of POT nuclear density, we proceeded to compare POT morphological differences with those of the mixed OTs and TG.

The analysis of the architecture of epithelial and mesenchymal tissues was done by first segmenting the cell nuclei and subsequently finding their ‘areas of influence’ to generate ‘virtual cells’. These were then grouped into layers according to their adjacency relations17 in relation to a reference region (e.g. adjacent to the epithelial compartment). H&E-stained slides were first processed to detect nuclear profiles using the Stardist 2D plugin as described earlier13, which were then separated (if overlapping or adjacent) using an ImageJ macro to separate such labels (Fig. 3A and B). Subsequently, a binary image was created and the free space (without tissue) was identified as the empty “background” which was used as reference to detect the epithelial compartment (adjacent to the background) and the ectomesenchyme compartment (adjacent to the epithelium).

The partitioning of the binary compartment into virtual cells (each of which contain a single nucleus) was obtained using the watershed transform18, where pixels not belonging to the influence zone of any particular nucleus defined the boundaries (watershed lines) between adjacent virtual cells (Fig. 3C). Layers were generated using the method described in17, considering the epithelium as the layer one (Fig. 3D). Since the images of tissues only cover a section of the specimen, the number of detectable layers is restricted by the size of the image as well as the presence of epithelial islands (if any). This is the reason why the number of layers is not necessarily characteristic of a tumor and may be influenced by the size of the image analyzed (or by the size of the tumor, if it could be imaged in a single image). However, by comparing same-sized images, this type of analysis can still provide a measure of the tissue arrangement within those spatial bounds that could be useful to differentiate between entities.

The resulting segmentation was further analyzed with the Particles8 plugin to extract various morphological descriptors from the different layers of virtual cells16 (i.e. layer-by-layer analysis).

Fig. 3
Fig. 3
Full size image

Sequence of the analysis of virtual cell morphology in a tooth germ. (A) HE staining showing the dental papilla (bulk of tissue and the inner enamel epithelium (top). (B) Nuclei segmentation with StarDist 2D. (C) Epithelial segmentation of virtual cells using watershed transform. (D) Labelled of layers/strata of virtual cells starting from the upper layers of the tissue (epithelium labelled in blue) advancing towards the depth.

Statistical analysis

The epithelial cell area was chosen as the comparison parameter due to its robustness against histological distortions and its lower variability across images, as observed in unpublished pilot studies.

Given that, the variables were determined to be not normally distributed; non-parametric statistics were employed to compare the areas of the virtual cells of the tumors. The mean area values of the epithelial compartment of each odontogenic entity were compared statistically using Student’s t-test (adjusted to Mann-Whitney) and descriptive statistics was also computed in GraphPad (Prism version 8.0.1 for Windows, GraphPad Software, Boston, Massachusetts USA, www.graphpad.com).

Results

Identification of primordial odontogenic tumor subepithelial condensation

The nuclear density analysis of POT showed a higher nuclear concentration in the upper third of the tissue, corresponding to the epithelial and subepithelial zones, with a lower nuclear density in the deep ectomesenchyme (Fig. 4). This analysis shows that POTs have two different presentations, one with subepithelial condensation (84–86% of the tissue samples) and another without subepithelial condensation (POT-NC).

Fig. 4
Fig. 4
Full size image

Analysis of nuclear density in a primordial odontogenic tumor. (A) Alcian blue staining. (B) StarDist 2D segmentation. (C) The integral of all images (n = 71) shows an increase in nuclear density in the upper third of the image. A colorimetric scale of values is shown, where the color white represents the area of highest nuclear density.

The analysis of nuclear morphological parameters revealed that the nuclei in the whole ectomesenchyme compartment were slightly larger than those in the epithelial compartment (in terms of perimeter 4.5% larger, and area 7.6% larger). A student’s t-test (adjusted to Mann-Whitney) of the morphological parameters of epithelium and the ectomesenchyme compartments showed a significant difference (p-value < 0.0001) for shape, circularity, perimeter, and area, and a p-value < 0.0273 for rectangularity. However, the descriptive statistics (Tables 1 and 2) show little differences between nuclei of both compartments. The values of percentiles, minimum, and maximum are presented for informational purposes, but the mean and standard deviation will be used for discussion between the two compartments.

Table 1 Descriptive statistics for epithelial compartment nuclei in primordial odontogenic tumor.
Full size table
Table 2 Descriptive statistics for Ectomesenchymal nuclei in primordial odontogenic tumor.
Full size table

Taking the mode into consideration, for the shape a digital circle is between a value of 13–14 and for the epithelium and the ectomesenchymal tissue the values are 14.87 and 14.94 respectively; the same is true for the circularity where a perfect circle has a value of 1 and values smaller than 1 correspond to non circular shapes as we can observe for both the epithelium and ectomesenchymal tissues, corresponding to 0.85 and 0.83 respectively; for the rectangularity the values close to 0,79 correspond to a circular tendency and in our results both tissue exhibit values of 0,72. According to these results, both tissue exhibit a shape with a circular tendency.

Virtual cell analysis

When the nucleus density in the tissue is high, the influence zones tend to be smaller in size because the virtual cells are a geometrical construct based on the remaining non-nuclear pixels in the image. Consequently, regions like the epithelial compartment and subepithelial region (if featuring condensation) show a higher density compared with deep ectomesenchymal zones, which are characterized, qualitatively, as more lax in composition. Considering layer 1 as the whole epithelial compartment (represented in blue - or in red for OD - in Fig. 5), we noted that virtual cells in layer 2 tend to exhibit a larger area because they include the region corresponding to the basal membrane (odontogenic epithelium features reversed polarity) and the cell -free hyalinization space, observed in some regions.

Virtual cell analysis - Primordial odontogenic tumor

An average of 50 layers were detected in POT with subepithelial cell condensation. The virtual cells corresponding to the epithelium had smaller areas (82.8 ± 58.5) µm2 compared to those of the deep mesenchyme (547.3 ± 58.5) µm2 (Fig. 5A). The segmentation of POT-NC detected an average of 38 cellular layers (Fig. 5B). As the images had the same size and magnification, this means that there were 12 more layers in POT than in POT-NC. This may be due to the fact that subepithelial cells are more condensed and closely packed, allowing the detection of more layers in the same space.

Fig. 5
Fig. 5
Full size image

Analysis of virtual cell morphology in all the entities. (A) Primordial odontogenic tumor. (B) Primordial odontogenic tumor without subepithelial condensation. (C) Developing odontoma. (D) Ameloblastic fibroma. (E) Tooth germ.

Virtual cell analysis - Developing odontoma

The analysis of DO revealed an average of 17 ectomesenchymal layers (Fig. 5C). This was the only case where the cells located on the center of the islands, observed in blue, were not epithelial, being the epithelium represented in color red. Upon studying the H&E image, it was observed that the distribution correlated with the presence of epithelial islands, which limited the number of ectomesenchymal layers between them.

Virtual cell analysis - Ameloblastic fibroma

In the analysis of the AF an average of 22 layers were detected (Fig. 5D). The size of mesenchymal virtual cells clearly depended on the proximity of epithelial islands to each other, sharing more or less mesenchymal space. Besides comparing FA and DO, which had the same magnification and similar image size, for DO the number of layers was smaller than for FA.

Virtual cell analysis - Tooth germ

For TG, an average of 29 layers of virtual cells were identified (Fig. 5E). Even though TG samples have different sizes due to the morphology of the tissue, the amount of layer is considerable compared with FA and DO. In some zones of TG cases a white hyalinization space was present, modifying the virtual mesenchymal cells area close to it.

For all the odontogenic entities, the epithelial compartment was the one with the virtual cells with the smallest area (Table 3) in comparison with the rest of the sample layers, as there is more nuclear density in the epithelium. Because the size of the images limits the quantity of mesenchymal layers that differs from one sample to another and considering the differential histopathological characteristics between the tissues, the statistical analysis was conducted solely on the epithelium (layer 1). Moreover, all morphological analyses of OTs began with this point of identification. Additionally, the area was the chosen parameter for studying the virtual cells. Statistical comparison of the area of epithelial virtual cells of POT, POT-NC, AF, DO and TG was conducted using Student’s t-test (adjusted to Mann-Whitney), revealing a significant p-value: p* < 0.0001 (Fig. 6). P value was 0.0045 when comparing TG and POT-NC, so no significant difference was found between these two entities.

Fig. 6
Fig. 6
Full size image

Comparison of the area of epithelial virtual cells of all the cases included in the study. Note the significance between the cases. P < 0,001 = ****.

Additionally, descriptive statistics were conducted for the analysis of epithelial virtual cells area of all odontogenic entities (Table 3), revealing similarities in the mean values between POT-NC and TG, and DO cases showing higher values compared to the rest.

Table 3 Descriptive statistics of the area of epithelial virtual cells for all entities studied.
Full size table

Discussion

Digital image processing can be a valuable tool for quantitative histomorphology study. This is particularly important because incisional biopsies do not always provide sufficient data for diagnostic confirmation and therefore, supplementary tools may offer a more detailed understanding of tissue architecture. In this context, the aim of this study was to morphologically characterize and compare mixed Ots (POT, AF, DO) as well as TG, in order to elucidate the structure and spatial organization of the epithelial and ectomesenchymal compartments. Our findings showed that samples stained with H&E provide sufficient morphological information for algorithmic image segmentation into distinct cellular compartments and layers, which could assist in the automated analysis of histopathological images and potentially contribute to the differential diagnosis of odontogenic tumors.

Currently, there are few studies that use and integrate morphological and spatial parameters, in tumor characterization and behavior19,20. Histopathologically, POT is composed of loose tissue with variable fibroblast-like cells and minimum collagen production, with areas close to the epithelium rich in ectomesenchymal cells, surrounded entirely by cuboidal-columnar epithelium1. To better characterize this area of high cellular density, in POT an analysis of cellular density was conducted based on tissue depth, from the epithelium towards the center of the mesenchyme.

Our results demonstrate that in POT, the epithelium and subepithelium are where nuclear density (number of nuclei per unit area) is at its maximum, with a gradual decrease observed towards the deeper bulk of the tissue. When studying the morphological characterization of nuclear profiles, differences were observed between the epithelium, subepithelium and deep ectomesenchyme. The results for circularity in the epithelium indicate a tendency towards circularity; similarly, the rectangularity of these nuclei shows a tendency toward circular shape. The same is true for mesenchymal nuclei, which also exhibit parameters leaning toward a circular shape according to the shape, circularity and rectangularity results. On the other hand, nuclear size tends to be slightly larger in the ectomesenchyme in terms of perimeter and area parameters.

The importance of these results lies in the acquisition of quantitative data that confirms what was suggested qualitatively regarding a possible nuclear condensation in the subepithelial area of POT. The evident increase in nuclear density in the studied areas supports the hypothesis of a possible active epithelial-mesenchymal interaction in that zone, reminiscent of early stages of odontogenesis where the mesenchymal tissue close to the inner epithelium starts condensing prior to odontoblastic differentiation and dentine formation21. This area with high nuclear density reflects a biological activity characteristic of the tumor, which defines POT. Moreover, the nuclear density analysis characterization in POT correlates with a change in the composition of acidic mucopolysaccharides and the expression of CD34 reported elsewhere19, suggesting a difference in the composition (and function) of the subepithelial zone compared to the deeper ectomesenchyme.

Nuclear area has been linked to cellular differentiation, where significant increases in size often reflect a high replication rate22. According to Bologna et al., POT exhibits a low proliferation rate (< 5%) in the epithelium and ectomesenchyme, with higher rates observed in the cells within the subepithelial ectomesenchymal condensation23. Additionally, the authors suggested a variable degree of cellular differentiation within the odontogenic epithelium. This observation seems to correlate with our findings of nuclear area and perimeter, which show a tendency towards variability in the epithelium and ectomesenchyme across the entire tissue. Despite this, comparing nuclear area and perimeter between the strata of the same tumor can only aid to comprehend its behavior and structure, but is not a parameter for analyzing and comparing replication rates and cellular differentiation between the entities24.

To have a quantitative estimate of the differences found in the virtual cell analysis, statistical comparison of the area of epithelial virtual cells of the entities was performed, finding a significant p-value: p* < 0.0001 between them, except when comparing TG with POT-NC. In POT, the subepithelial condensation19 is not characteristic of all subepithelial zones in the tumor. POT cases showing subepithelial condensation were characterized by cells with smaller area compared to cells in the deep mesenchyme. Moreover, the morphology of POT was different between regions with or without condensation, suggesting different states of cellular maturation and differentiation. In other words, POT appears to be a non-static tumor whose growth and evolution could potentially change in different zones23. These differences may occur alternately depending on tumor growth and expansion by infiltrating the surrounding bone tissue, where other factors associated with the tumor microenvironment may intervene. Our results support previous studies describing the unique subepithelial condensation in POT, distinguishing it from other odontogenic tumors19,25.

In cases of AF, the epithelium presented the smallest virtual cell area according to the descriptive statistics, and the size of ectomesenchymal virtual cells depended on the proximity between epithelial islands. Furthermore, no subepithelial condensation, or mesenchymal virtual cell condensation, was found in the cases studied. Surrounding some epithelial islands a clear hyalinization zone could be identified, but was not comparable with the histopathological morphology of the POT cases. Comparing virtual cells between POT and AF revealed significantly different areas, with smaller percentiles for AF and the maximum value found in POT deeper zones. This could be because AF features epithelial islands, with less inter-epithelium space resulting in fewer virtual ectomesenchymal cells and without condensation in the mesenchymal. Once again, POT appears as a distinct odontogenic neoplasm, morphologically different from DOs or early stage AFs as previously suggested4,5.

In DO cases, the distribution of ectomesenchymal virtual cells also appears to depend on the presence of epithelial islands, resulting in varying amounts of mesenchymal space between them. In spite of this, descriptive statistics showed larger epithelial virtual cells in comparison with the other entities studied, suggesting a potentially useful morphological parameter. Similar to AF, a hyalinization zone was observed surrounding some epithelial islands but no ectomesenchymal condensation was found in any case. The absence of dense cellularity in the sub-epithelial regions of DOs and AFs emphasize the architectural differences with POT. It should be noted that the limited number of DO cases included in this study may have influenced the absence of areas showing ectomesenchymal condensation. All analyzed cases exhibited similar histological patterns, suggesting that they correspond to early developmental stages, before the deposition of dentin or dentinoid tissue. Although areas of subepithelial condensation would be expected at more advanced stages, the algorithmic analysis did not detect such features in any of the DO samples. This finding reinforces the quantitative observation and may reflect the dynamic histogenesis of these lesions rather than a methodological limitation.

Comparisons between DO and AF, both containing epithelial islands, demonstrated that AF exhibited a higher number of epithelial layers than DO. This could be caused by the larger size of the epithelial islands in DO and the presence of a background area in the samples. Moreover, these entities show less amount of layers than the ones exhibiting epithelium at the top of the image, suggesting that the space between the islands could limit the concentric growth of layers limiting this study. DO virtual epithelial cells area and layer distribution were significantly different from TG samples, suggesting that even though this tumor exhibits tissues that resemble a developing TG, its architecture is structurally different1. It is important to highlight that DO samples were previously diagnosed as AFO and AFD, two entities that remained classified as developing odontomas in the WHO 2022 classification1, although since those lesions may have a neoplastic component and in this study they deviated from the morphological characteristics presented in TG, reconsideration on the classification among molecular studies should require a more in-depth analysis. Moreover, the presence of BRAF p.V600E mutations in these two entities has supported the arguments that at least some of these lesions are in fact neoplastic, particularly those with a locally aggressive biological behavior, large size and recurrence26,27.

TG samples in the cap and bell stages presented a hyalinization zone beneath the epithelium, similar to other entities described. Leaving aside this consideration, the samples were comparable to POT-NC, with no significant differences observed in epithelial virtual cells area. The lack of subepithelial condensation may therefore reflect an early phase of tumor differentiation. This may reflect that several markers, including Syndecan-1, vimentin, CAV-1, Glut-1, PITX2, amelogenin, CK19, and MOC-31, show heterogeneous expression—negative in some regions and focally positive in others—supporting variable stages of odontogenic differentiation23.

This information could be valuable for auxiliary diagnostic purposes, offering key insights into the tumor morphology. While histopathological analysis with clinical correlation remains the gold standard for definitive diagnosis, this study introduces a quantitative approach that could enhance the development of automated analysis. The detection of subepithelial features in POT may assist in differential diagnosis, serving as a distinguishing characteristic when compared to other mixed odontogenic tumors, and may further support its consideration as a separate entity within the WHO classification of odontogenic tumors.

Limitations

This study has several limitations that should be acknowledged. First, the sample size is relatively small, which is largely attributable to the exceptional rarity of primordial odontogenic tumor and related mixed odontogenic tumors. This limitation may affect the generalizability of the findings and precludes robust subgroup analyses. Second, some degree of heterogeneity across cases is inherent to these lesions, which may influence quantitative architectural parameters. Third, the analysis was performed on incisional biopsy material, where inflammatory changes or sampling-related distortions could partially modify the native tissue architecture and act as potential confounders in algorithmic measurements. Finally, although the proposed virtual cell–based approach provides objective and reproducible quantitative descriptors, it should be interpreted as an adjunctive tool to conventional histopathological assessment. Validation in larger, independent, multi-center datasets is required to confirm the diagnostic utility and robustness of this methodology.

Conclusions

Quantitative analysis of tissue architecture revealed distinct morphological patterns among mixed odontogenic tumors. In particular, focal areas of POT showed architectural similarities to TG, while DO and AF exhibited different epithelial virtual cell characteristics. These results support the concept that algorithmic evaluation of epithelial morphology may serve as a complementary quantitative approach in cases where conventional histopathological assessment is inconclusive. However, given the limited sample size and the rarity of these lesions, further validation in larger and independent cohorts is necessary before broader diagnostic application.