Introduction

Artificial intelligence (AI) has demonstrated remarkable performance in dermatology, often surpassing dermatologists in certain controlled conditions1. Those studies, mostly encompassing skin cancer and utilizing clinical photographs and dermoscopy images, have shown that AI can outperform dermatologists when diagnoses are based solely on image data2,3,4,5. The performance of various AI models has been validated using a variety of publicly available dermatology datasets6,7,8,9,10,11,12,13,14.

Unlike in radiology and ophthalmology, vision AI models in dermatology have not been yet clinically successful for several reasons. First, AI performance in dermatological conditions tends to be overestimated in the reader test when conducted with limited information. Unlike radiologists who detect specific findings from X-ray images or ophthalmologists who interpret fundus imaging to determine diabetic retinopathy, dermatologists rely on a variety of contextual and patient-specific information outside of imaging to make a diagnosis. Second, AI performance can decrease due to out-of-distribution (OOD) cases in real-world situations. While images utilized in radiology or ophthalmology are taken with specialized medical equipment or by medical staffs following DICOM standards, resulting in more standardized samples, most images used in dermatology outside research settings are taken with non-medical equipment such as non-professional consumer cameras or smartphone cameras in a non-standardized manner and are often acquired by non-medical personnel, resulting in frequent OOD scenarios15.

Consequently, while AI may outperform dermatologists in purely image-based diagnoses using preprocessed in-distribution datasets, clinicians in real practice tend to be more accurate16. Historical examples, such as the MelaFind algorithm, which received FDA approval but ultimately disappeared from the market due to high false-positive rates, highlight how AI algorithms that perform well under controlled conditions can struggle in real-world applications17. In addition, the results may be distorted because most test sets do not reflect disease prevalence, as the ratio of benign to malignant disease in most test datasets is lower than 10:1. Real-world data is also complicated and influenced by the wide variation in skin color and ethnic skin types which creates unique variations that creates specific challenges to AI algorithms. For these reasons, it is important to measure AI algorithms performance using real-world data, especially when evaluating specificity.

In a prior national validation study performed in Korea, we separately calculated the sensitivity and specificity. In order to precisely assess sensitivity, we created a dataset comparable to the annual number of skin cancer cases in Korea. To evaluate specificity, we analyzed the malignancy predictions made by the algorithm in actual use. Additionally, to assess multi-class performance, we analyzed 152,443 images across 70 diseases. Finally, to understand the implications of worldwide skin variation, we investigated global disease prediction patterns using the algorithm, aiming to get insights into user interests and disease prevalence.

Results

NIA and SNU dataset analysis

The NIA (National Information Society Agency) dataset consists of 152,443 clinical photographs, covering 70 distinct diseases. (Table 1) The SNU (Seoul National University Hospital) test dataset consists of 240 clinical photographs, covering 80 distinct diseases (Supplementary Table 1).

Table 1 Demographics of the NIA Dataset and Top-1 and Top-3 Accuracies of the Algorithm
Full size table

For binary classification of malignancy or not, the algorithm achieved an AUC of 0.946 (95% CI 0.942–0.950) in the NIA dataset, using the data on benign and malignant tumors (n = 51,038 images, 33 diseases). (Fig. 1a) The sensitivity and specificity of the algorithm were calculated through the three differential diagnoses suggested by the algorithm (Top-3), which is an established metric5. The sensitivity and specificity of the algorithm were 78.2% (95% CI 77.0–79.4%) and 93.0% (95% CI 92.8–93.3%) for the NIA dataset. For the SNU test dataset, the AUC was 0.958 (95% CI 0.919–0.997). (Fig. 1b) When calculated from the Top-3, sensitivity and specificity were 87.5% (95% CI 73.2–95.8%) and 91.0% (95% CI 86.1–94.6%) for the SNU test dataset (Supplementary Table 2).

Fig. 1: ROC curve for determining malignancy using the NIA dataset and SNU test dataset.
figure 1

The performance of the algorithm, calculated based on Top-1 or Top-3 predictions, is drawn separately in the figure. a TEST = Benign and malignant tumors in the NIA dataset (51,038 images, 33 diseases). b TEST = SNU Test Dataset (240 images, 80 diseases); The results from six continents are based on 61,066 assessments collected from 138 countries. Country-level results are presented only for the top 10 countries with the highest number of responses. Detailed data by country can be found in Supplementary Table 3. AF Africa, AS Asia, BR Brazil, DE Germany, ES Spain, EU Europe, FR France, HU Hungary, IT Italy, KR South Korea, NA North America, OC Oceania, PL Poland, RU Russia, SA South America, US United States.

Full size image

For multi-class classification of matching exact diagnosis, the algorithm achieved micro-averaged mean Top-1 and Top-3 accuracies of 43.3% and 66.6% on the NIA dataset, respectively. (Table 1) The micro-averaged mean Top-1 and Top-3 accuracies showed 63.3% and 83.2% on the SNU test dataset, which were comparable to the 49.9% and 67.2% performance involving 2 dermatologists and 2 dermatology residents in the previous study4.

In the reader test (SNU test dataset), involving 61,066 assessments from 138 countries for determining malignancy or not, the mean sensitivity and specificity of global users were 55.2% (95% CI 54.2–56.2%) and 84.3% (95% CI 84.0–84.6%), respectively, which were significantly lower than the performance of the algorithm for the same 61,066 cases (86.8% [95% CI 86.2–87.5%] and 91.0% [95% CI 90.8–91.2%]) (both p < 0.0001, McNemar). (Supplementary Table 3) As shown in Fig. 1b, reader sensitivity was significantly higher in North America (64.0%), Oceania (62.6%), and Europe (59.2%) than in Asia (51.7%), South America (50.6%), and Africa (43.8%) (p < 0.0001, Chi-square with Bonferroni correction).

Global user performance

Global usage statistics from 1,691,032 requests showed notable regional variations. The distribution of requests by region was as follows: Europe (47.9%), Asia (31.5%), North America (10.1%), South America (5.7%), Africa (2.9%), and Oceania (1.9%). The top five countries by usage were Korea (22.6%), Italy (9.0%), United States (6.2%), Germany (5.9%), and France (4.0%) (Table 2, Supplementary Table 5).

Table 2 Proportions of Disease Groups Obtained from the Algorithm’s Top-1 Prediction by Continent and Countries
Full size table

The algorithm’s predictions showed distinct regional diagnostic trends. The highest proportion of the algorithm’s predictions were for benign tumors, which accounted for 65.2% in Korea. (Table 2) In terms of benign tumor predictions, the ranking by region was as follows (‘>’ indicates p < 0.0001 by Chi-Square with Bonferroni correction): Asia (55.5%) > Oceania (46.8%) > (p = 1.000) Europe (46.5%) > North America (38.0%) > South America (32.3%) > Africa (22.4%).

Regarding malignancy predictions using Top-1 diagnosis, the rate was 2.1% in Korea. (Table 2) The regional distribution of malignancy predictions was as follows (‘>’ indicates p < 0.0001 by Chi-Square with Bonferroni correction): North America (2.6%) > Europe (1.9%) ≈ (p = 0.5402) Asia (1.8%) ≈ (p = 0.9308) Oceania (1.7%) > (p = 0.0002) South America (1.4%) > Africa (0.9%).

Infectious conditions were predominantly predicted in Africa and South America than other regions, with higher frequencies observed in the Middle East and Northern Africa on the country map (Fig. 2c). The regional distribution was as follows (‘>’ indicates p < 0.0001 by Chi-Square with Bonferroni correction): Africa (17.1%) > South America (14.8%) > North America (12.9%) > Europe (11.5%) > Asia (10.0%) > Oceania (8.6%). Unlike other disease groups, allergic diseases showed relatively consistent trends across regions (Fig. 2d).

Fig. 2: Global distribution of disease predictions - four disease groups.
figure 2

A worldwide map illustrating the differences in the algorithm’s predictions across various disease groups. This reflects both the interest in the disease and its prevalence. Interactive demo is available at https://whria78.github.io/nia/demoa Malignant tumors (countries with fewer than 1000 requests were excluded) b Benign tumors c Infectious diseases d Allergic diseases.

Full size image

The ten most commonly predicted diseases were as follows: melanocytic nevus (20.9%), seborrheic keratosis (4.6%), cherry hemangioma (2.2%), dermatofibroma (2.1%), folliculitis (2.0%), wart (2.0%), lentigo (2.0%), hemangioma (2.0%), urticaria (1.9%), and actinic keratosis (1.9%). (Supplementary Table 6) Folliculitis was more prevalent in the Middle East and Southeast Asia on the map, while actinic keratosis was more common in North America and Australia (Fig. 3).

Fig. 3: Global distribution of 10 common conditions predicted by the algorithm.
figure 3

A worldwide map illustrating how the algorithm’s predictions and usages vary across common diseases. The selection of the 10 common diseases was based on algorithm usage as shown in Supplementary Table 6: a melanocytic nevus, b seborrheic keratosis, c cherry hemangioma, d dermatofibroma, e folliculitis, f wart, g lentigo, h hemangioma, i urticaria, and j actinic keratosis.

Full size image

Regarding malignancy diagnosis using Top-3 predictions, the rate was 12.0% (95% CI 11.8–12.1%) in Korea and 10.0% (95% CI 9.9–10.0%) globally. (Supplementary Table 4) Assuming that all malignancy diagnoses by the algorithm were false positives, the estimated specificity using Top-3 in Korea was 88.0% (95% CI 87.9–88.2%), and 90.0% (95% CI 90.0–90.1%) globally. By continent, the malignancy predictions using Top-3 was as follows: North America (11.7%), Asia (10.9%), Europe (9.3%), Oceania (9.0%), South America (8.0%), and Africa (6.7%).

Discussion

Here, we present the worldwide study evaluating the real-clinical use of an open-access global dermatology AI service with 1.69 million assessments from 228 different countries. These large number of requests reflects strong user interest in dermatology AI apps and confirm the utility apps in daily clinical practice. However, evaluating the real-world performance of AI in dermatology is highly challenging because a continuous dataset linking AI results to biopsy outcomes is required. However, due to lack of population-based reference skin cancer datasets and the current lack of digitalization in clinical practice, obtaining large datasets with serial results is virtually impossible18. Furthermore, evaluating ‘specificity’ is problematic, as curated hospital datasets fail to capture the wide range of out-of-distribution conditions encountered in real-world settings.

A substantial discrepancy exists between the performance of diagnoses based on images alone and diagnosing in actual clinical practice. For the diagnosis of 43 tumor types, attending physicians achieved Top-1 and Top-3 accuracies of 68.1% and 77.3%, whereas physicians in the reader test achieved only 37.7% and 53.4% for the same cases, respectively16. Although AI has demonstrated exceptional performance in controlled settings, it must prove its effectiveness in real-world environments to be meaningful in clinical practice19,20. Despite the large number of retrospective studies, real-world evidence remains insufficient. As of 2024, only 86 medical AI algorithms have reported RCT-level evidence, and among these, 70 achieved successful outcomes, and of these, only 1 exists in the field of dermatology21,22.

We curated a large dataset representing cancer cases in Korea to estimate the sensitivity for cancer diagnosis. To assess specificity, we analyzed usage statistics under the assumption that all AI-determined malignancy predictions were false positives. This separated approach enabled us to estimate the algorithm’s real-world performance in terms of sensitivity and specificity.

First, regarding the sensitivity, the maximum sensitivity was estimated under the assumption that false negatives are minimal if using the national-scale hospital dataset. The NIA dataset is large enough to represent skin cancer cases in Korea. Therefore, the sensitivity of 78.2% (95% CI, 77.0–79.4%), predicted by three differentials, was presented as the ideal maximum value achievable in real-world settings. If users capture only low-quality images, the sensitivity could be lower15,23. Conversely, in practice, repeated testing of a lesion over days may increase sensitivity.

Next, regarding specificity, the minimum specificity was estimated under the assumption that there are no true negatives (= all false positives) in the usage data. The analysis of usage records in Korea showed that the algorithm predicted malignancy at a rate of 12.0% (95% CI, 11.8–12.1%) based on three differentials. Given the relatively low prevalence of cancer, if we assume all malignancy predictions by the algorithm are false positives, the specificity was estimated at 88.0% (95% CI, 87.9–88.2%). (Supplementary Table 4) This figure (88.0%) is lower than the specificity reported for the NIA dataset (93.0%; Table 1). At first glance, benign tumors requiring hospital visits may seem harder for AI to diagnose than various skin conditions seen in daily life. However, real-world use includes out-of-distribution scenarios, likely affecting the degradation of AI performance. On the other hand, actual specificity may be slightly higher than our 88.0% estimate, as users often confirm malignant results through repeated tests before hospital visits.

From the perspective of disease screening, WHO tuberculosis guidelines set minimal requirements of 90% sensitivity and 70% specificity24. Similarly, the Breast Cancer Surveillance Consortium benchmarks highlight 86.9% sensitivity and 88.9% specificity for breast cancer screening25. However, for skin cancer screening, no precise guidelines for the sensitivity and specificity currently exist26. Furthermore, concerns about overdiagnosis in dermatologist-led skin cancer screenings highlight the need to discuss using algorithms with lower real-world performance. As seen in Fig. 2a, the higher malignancy rate on the map may be related to greater algorithm usage for skin cancer, similar to how an increase in dermatology clinics is associated with a higher regional incidence of melanoma27,28. However, unlike human physicians, algorithms can be adjusted to operate with either very high specificity or low by modifying the threshold. For example, in scenarios where concerns about overdiagnosis, the threshold could be raised to identify only cancers with the highest diagnostic certainty, particularly in populations with limited access to healthcare. From this perspective, there should be an effort to find proper settings by each region and demonstrate the improvement of clinical outcomes (mortality, morbidity, and costs).

The usage statistics revealed significant regional differences in the types of diseases predicted by the algorithm. (Figs. 2, 3, Supplementary Table 3) Neoplastic disorders were more commonly predicted in Asia, Europe, and North America. (Fig. 2) Premalignant conditions such as actinic keratosis were observed at higher rates in Australia and North America. (Fig. 3) Interestingly, in line with these findings, users from the EU, North America, and Oceania showed higher sensitivity in the global reader test compared to those from other regions. (Fig. 1b) In contrast, infectious diseases were more frequently observed in regions such as Northern Africa and the Middle East. (Fig. 2c) These regional differences reflect variations in disease prevalence, the age demographics of users, and the diseases users are particularly concerned about in each region. For example, in South Korea, despite the low prevalence of skin cancer, high interest in skin cancer in the media may lead to a higher malignancy ratio as shown in Fig. 2a. These data, collected before patients even visit the clinic, can offer more accurate insights into the prevalence and interest in specific skin conditions. The global burden of disease study has been attempted, but only for certain disease groups, with no individual approach to dermatological diseases29,30,31. This indicates that AI-based big data analytics could significantly contribute to understanding skin disease trends globally.

Our study has several limitations. The first part of the study was limited to patients with skin types III and IV, which account for the vast majority of skin types in Korea. Therefore, we were unable to provide stratified data by race and skin type, especially in the population of dark skin tones32. Additional studies are required to evaluate the sensitivity for skin cancer in white and black populations.

Second, the NIA dataset does not include less common skin cancers beyond the four major types (e.g. Merkel cell carcinoma, Kaposi sarcoma). According to 20 years of incidence statistics in Korea, these other types account for approximately 11.3% of all skin cancers33. In particular, early detection with AI can be beneficial for rare but very poor-prognosis skin cancers such as cutaneous angiosarcoma34. Therefore, rare cancers not represented in the NIA dataset should be collected and analyzed using appropriate statistical sample size considerations to ensure sufficient power and validity.

Third, the sensitivity and specificity calculated in this study needs to be re-evaluated through further digital transformation efforts, separately for each indication. The global study lacked a gold standard in order to calculate sensitivity and specificity. Ideally, users could provide feedback on final diagnosis based on clinical follow-up, laboratory work-up or final histopathological assessment. Further validation should be tried individually in each country, tailored to the healthcare environment35.

Fourth, in terms of multi-class results, the algorithm achieved Top-1 and Top-3 accuracies of 43.3% and 66.6% on the NIA dataset. Although the multi-class performance of the algorithm was validated using the large dataset, its real-world accuracy also needs to be reassessed in further studies.

Fifth, because of this being the study to evaluate real-use of global users of a single algorithm, it was challenging to determine the best approach. Since usage varies over time, it is displayed as the disease ratio within each country.

Sixth, although this was a global study, data from Africa, South America, and Oceania were relatively underrepresented, which may reflect regional differences in population size, access to digital healthcare, and overall engagement with digital health initiatives.

Finally, while the algorithm was designed to prioritize high specificity, it may be perceived as insufficient from the perspective of maximizing sensitivity. Although high sensitivity has been favored to avoid missing malignant cases, this approach often results in a high false-positive rate, as seen in the case of MelaFind17. In the context of AI tools that may be used frequently by laypersons without clinical oversight, insufficient specificity could amplify overdiagnosis27 and unnecessary anxiety. Therefore, our specificity-focused design aligns with concerns about overdiagnosis and false alarms, particularly in low-risk populations. However, it limits the algorithm’s ability to capture all true positives.

In conclusion, using a national-scale curated dataset and real-world usage data, the performance of AI in diagnosing skin cancer in Korea could be estimated at 78.2% (95% CI, 77.0–79.4%) sensitivity and 88.0% (95% CI, 87.9–88.2%) specificity. In multi-class classification, our algorithm achieved Top-1 and Top-3 accuracies of 43.3% and 66.6%, respectively, replicating the results obtained from the past small study4. Furthermore, this study highlights the potential of AI algorithms to provide a global perspective on skin diseases, offering a quantitative reflection of regional variations using AI-based big data analytics. Further research is needed to identify clinical settings where AI can effectively improve clinical outcomes. Additionally, the need for randomized controlled trials (RCTs) and validation in underrepresented regions, such as Africa and South America, must be explicitly emphasized to ensure the algorithm’s effectiveness and generalizability across diverse populations.

Methods

This study adheres to the STARD 2015 guidelines and was approved by the Kangnam Sacred Hospital Institutional Review Board (2021-10-011, 2024-02-028-004) as a retrospective study.

Datasets for validation

  1. 1.

    The NIA dataset consists of 152,443 clinical photographs (2001–2021) taken with professional DSLR cameras by dermatology residents or professional photographers from 92,004 random cases, covering 70 distinct skin diseases36. (Table 1) Among these, 4250 cases (4.6%) are malignancies, including 734 melanomas, 1804 basal cell carcinomas (BCC), 1601 invasive squamous cell carcinomas (SCC), and 111 in situ squamous cell carcinomas (Bowen’s disease) cases. The number of malignancies in the NIA dataset is comparable to Korea’s annual average number of skin cancer cases (4498) based on 20 years of statistics33. Assuming 80% sensitivity and a sample size of 4250, the 95% confidence interval for precision was ±1.18% (Wilson score method). Diagnoses were validated based on clinical diagnoses or pathology reports in cases of malignancy. The NIA dataset was compiled from 9 university hospitals in Korea. From each clinical image, the lesion of interest was manually cropped to generate standardized square images. These cropped images were then reviewed by two dermatologists, and only those deemed consistent with the diagnosis were included. Images that were blurry, poorly focused, diagnostically ambiguous, or not included in the predefined set of 70 diagnoses were excluded. A total of 52 board-certified dermatologists and 35 dermatology residents participated in data processing.

  2. 2.

    The SNU test dataset consists of 240 clinical photographs, covering 80 distinct diseases. (Supplementary Table 1) The dataset includes 40 malignancy cases, comprising 7 Bowen’s disease, 3 keratoacanthoma, 8 BCC, 8 SCC, and 14 melanoma cases. Both malignant and benign tumors had histopathological confirmation, while other benign non-tumoral conditions were diagnosed clinically. This public dataset was compiled in a previous study that includes the reader test results from 21 dermatologists and 23 laypersons4. In this study, a global reader test was conducted using the same SNU test dataset in a mobile environment (https://www.modelderm.com/#quiz; Supplementary Note 1). The global reader test consists of image-based questions, randomly sorted and presented in a binary (yes/no) format, asking whether the condition is cancerous. To prevent cheating (e.g., using multiple devices to find the correct answers), each installation of the app presents a unique, shuffled order of questions.

Global usages study

We analyzed 1,691,032 requests submitted over 2.5 years (2022.7.7–2025.1.18) by 371,757 global users to the mobile app (https://modelderm.com). (Table 2, Supplementary Table 3) At no point were potentially identifiable data such as user registration information, device identifiers, or patient images collected or stored, and all data used for analysis remained at the level of de-identified server logs. As the submission was fully anonymized, informed consent was not required under current regulatory standards (U.S. Common Rule, 45 CFR §46.104[d][4]; European GDPR, Recital 26). All user-submitted images were included in the analysis, including those that received a low-quality warning from the app, which may have introduced variability in image quality and potentially affected diagnostic performance. Algorithm’s predictions were categorized into five groups: benign tumors, malignant tumors, infectious diseases, allergic diseases, and miscellaneous. (Table 1) Wart was counted under both benign tumors and infectious diseases because it should be distinguished from verrucous skin cancer.

Algorithm

The NIA and SNU test dataset were tested using ModelDerm Build2024 (https://modelderm.com, Supplementary Note 1). For global usage statistics, Build2024 was used during 2024–2025, while Build202115 was used during 2022–2023. The architecture of the algorithm is an ensemble of ResNet variants. The algorithm was developed using an R-CNN-based approach, designed to incorporate out-of-distribution data as much as possible. This method involved manually annotating lesions from numerous R-CNN predictions, extracting them from a vast collection of internet and public images.

Statistical analysis

Model performance was evaluated using sensitivity, specificity, and top-n accuracy. Sensitivity and specificity for determining malignancy can be derived from one prediction (Top-1) or three predictions (Top-3) of the algorithm. The results of sensitivity and specificity were calculated using the Top-3. The ROC curve was drawn using the malignancy score which is composed of the output for malignant and premalignant diseases4. A 95% confidence interval (CI) was reported for all results using the binomial distribution method. When reporting the average, the macro-averaged mean is used unless otherwise indicated, and the micro-averaged mean is explicitly specified when applicable. The standard error for a proportion is the square root of the proportion times its complement, divided by the sample size, and the z-score of 1.96 is used to calculate the 95% confidence interval. To account for Type I error when performing Chi-square tests to sequentially compare values across six continents, we applied the Bonferroni correction by multiplying the p-value by 5. All statistical analyses were performed using R (version 4.3.3; https://www.r-project.org/). ROC curves were created using matplotlib (version 3.10.0; https://matplotlib.org/).