Introduction

Spontaneous intracerebral (ICH) and subarachnoid hemorrhage (SAH) account for up to one-third of all strokes1. Both ICH and SAH are associated with high morbidity and mortality rates2,3, and intraventricular hemorrhage (IVH) also commonly occurs with either condition. Due to the high disease burden of ICH and SAH, prompt and accurate diagnosis is critical, along with quickly initiated therapeutic actions4,5.

Since non-contrast head CT scans (NCCTs) have high sensitivity for detecting acute intracranial blood within 12 h from symptom onset, the diagnosis of acute intracranial hemorrhages is based on emergent NCCTs6,7. The number of medical imaging studies is increasing globally8,9,10. At the same time, there are increasing concerns regarding the fatigue of radiologists and its effect on diagnostic accuracy11,12. Therefore, new technological solutions to assist clinicians and radiologists with rapid and accurate interpretation of imaging studies could alleviate this issue. According to a recent review article13, multiple algorithms have been developed for this task. However, the performance metrics of these solutions are often poorly reported, and suboptimal for clinical use.

Recently, we trained a model that identified SAH with high sensitivity, although the false positive rate was also high14. To overcome this false positive problem, the number of images required for training a deep learning model can be exceptionally high. Based on these premises, we aimed to develop a combination of multiple deep learning (DL) algorithms that would detect spontaneous intracranial hemorrhages, namely ICH, SAH and IVH, with a high sensitivity and low false positive rate despite limited amount of training data. Given the inherent diagnostic limitation of modern head CT scanners in identifying subacute blood (accuracy highest in early imaging)6,15, we aimed to train the solution to give optimal results when applied to cases with acute hemorrhages. To evaluate the solution’s performance, we gathered a comprehensive dataset of emergency NCCT scans from different emergency rooms, imaged using various scanner devices.

Materials and methods

The institutional review board of Helsinki University Hospital (HUH) approved the study and granted a waiver for acquiring informed consents (HUS/365/2017; HUS/163/2019; HUS/190/2021). According to the Finnish legislation, no ethics committee approval is needed for retrospective studies using registry or archive data. We conducted study reporting following the Standards for Reporting of Diagnostic Accuracy Study (STARD) and Checklist for Artificial Intelligence in Medical Imaging (CLAIM) guidelines16,17.

Deep learning solution

Our solution included four base U-Nets18 and a metamodel that was trained on top of the base U-Nets. We trained one ICH U-Net, one IVH U-Net and two SAH U-Nets to detect specific types of spontaneous intracranial hemorrhages. The training and performance metrics of one of the SAH U-Nets has been described previously14. In short, this SAH model was developed with 120 NCCTs, of which 98 included SAH, and 22 were negative for SAH. We trained the new ICH, SAH and IVH U-Nets with 180 NCCTs collected from HUH picture archiving and communication systems. For training the three new U-Nets, we segmented only the bleeding type of interest present in the NCCTs. The new SAH U-Net was developed in order improve the detection of focal SAHs.

The training dataset of the three new U-Nets consisted of 63 (ICH), 50 (IVH) and 67 (SAH) head NCCT multiplanar reconstruction (MPR) reformates (with 512 × 512 dimensions, 3 mm slice thickness). All patients were imaged and treated at HUH prior to October 2021. Segmentations were done using a Philips IntelliSpace Discovery (Philips Healthcare, 3000 Minuteman Rd, Andover, MA) and 3D Slicer19. In segmentations, Hounsfield unit (HU) based thresholds were used. Eventually, the segmentation masks were saved in a binary format with each hemorrhage type segmentation with a unique label.

After training the base U-Net models, we trained the metamodel with 55 NCCTs randomly sampled from the 180 previously mentioned NCCTs. For these 55 NCCTs, we segmented all three bleeding types. All NCCTs used for model training were imaged prior to October 2021, and therefore did not overlap with our validation dataset.

During the inference phase, the NCCT slice is first fed to the four base models, which produce output segmentations of ICH, IVH and SAH (one segmentation for ICH, one segmentation for IVH and two segmentations for SAH). The four resulting segmentations together with the original NCCT slice are then directed to the metamodel, which produces outputs as the final segmentations for ICH, IVH and SAH. The DL solution performs semantic segmentation (i.e. predicting pixel-wise probability for the presence of intracranial hemorrhage). Unlike conventional stacked generalization, our approach provides the metamodel with both the base model predictions and the original NCCT slice as input.

After the base U-Net and metamodel development, the solution was implemented with a post-processing pipeline for improving the solution detection accuracy. First, segmentation clusters smaller than 10 pixels were removed. This was followed by a soft-voting step. In the soft-voting step, the segmentations from the base models were summed, averaged and compared against the segmentation mask of the metamodel. If the summed segmentation did not overlap with the metamodel segmentation, the segmentation was directed to test-time augmentation (TTA) phase. Otherwise, the TTA phase was not applied. In the final step of the post processing, the positive segmentation clusters were combined with the segmentations of the base U-Nets. If the combined cluster size exceeded 125 pixels, the segmentation was classified as a positive prediction.

During the inference phase, we measured the total processing times for the NCCTs, which included reading the DICOM files, preprocessing the imaging data, predicting the presence of a hemorrhage, post-processing step, and saving the predictions in a NIfTI file format.

The detailed description of DL solution training and post-processing pipeline is provided in the article supplementary material (Supplementary Materials and Methods, Supplement 1). The code for developing the solution and performing the inference tasks are available at GitHub (https://github.com/Juusotak/Intracranial_hemorrhage_metalearning).

Validation dataset

To simulate a real-world emergency imaging setting, we collected a retrospective dataset from 10 different hospitals in the HUH catchment area, which covers over 1,700,000 inhabitants. The validation dataset consisted of all consecutive emergency head NCCTs imaged between October 1st and December 31st 2021. In more detail, if an NCCT scan of adult patients (18 years or older) was performed either with emergent or immediate priority, it was included in the dataset. We also collected corresponding on-call radiology reports, patient reports of the emergency clinic visit, and ambulance reports, when available. In addition to these, we gathered information about patient demographics (age and sex).

The time of the symptom onset and the etiology of hemorrhages were evaluated based on patient and ambulance reports. The on-call radiology reports were considered as ground truths for the presence of intracranial hemorrhage in NCCTs. If the clinical reports stated that the hemorrhage occurred without a history of head trauma, the hemorrhage was classified as a spontaneous intracranial hemorrhage. If the reports mentioned a previous head trauma, or if the etiology could not be confirmed based on available information, the hemorrhage was classified either traumatic or unclear etiology, and the case was excluded from the validation dataset. Primary spontaneous intracranial hemorrhages included non-traumatic aneurysmal and non-aneurysmal SAHs, non-traumatic deep and lobar ICHs, and non-traumatic IVHs. Secondary spontaneous intracranial hemorrhages included SAHs, ICHs or IVHs related to ischemic strokes and tumors, and were excluded from the analysis. The dataset included also acute subdural hemorrhages (ASDHs) and epidural hemorrhages (EDHs), if the medical history did not include preceding head traumas.

The head NCCT scans in which the on-call radiologist had reported hemorrhage were independently annotated on a slice-level by two study authors (JT; Medical doctor with 3 years of experience in neuroimaging and deep learning research and KV; Radiologist in-training with 2.5 years of experience) for the presence of hemorrhage using 3D Slicer19. Senior study author (MK; Consultant cerebrovascular neurosurgeon with 20 years of experience) solved any conflicts between the annotators either by removing annotations from the slice or accepting the slice annotations.

The final validation dataset included 118 NCCTs in which on-call radiologists reported spontaneous intracranial hemorrhages, and 7679 NCCTs that were reported negative (Fig. 1). Of the 7797 NCCTs, 4078 represented women (median age 74 years old, IQR 29 years) and 3719 men (median age 67 years old, IQR 27 years) (Table 1). The median age of all imaged patients was 71 years (IQR 29 years) (Table 1). The 7797 NCCTs were imaged using 12 different CT scanners from four vendors (Table 1).

Table 1 Validation dataset patient demographics and CT scanner devices used in imaging. IQR = interquartile range.
Full size table

Of the 118 NCCTs in which on-call radiologists had reported hemorrhage, 59 were imaged within 12 h, 19 presumably within 12 h and 17 presumably/certainly within 12 to 24 h from the symptom onset (Table 2). Of the 59 early imaged (within 12 h) spontaneous intracranial hemorrhages, 32 contained only one hemorrhage type, 22 two hemorrhage types and 6 three hemorrhage types. Overall, 49 NCCTs included only one hemorrhage type, 38 included two hemorrhage types, and 10 images included three hemorrhage types (Table 2). The dataset also included one ASDH. The patient presented without a history of head trauma.

Table 2 Hemorrhage types in validation dataset. Counts of different bleed types and combinations of bleed types in dataset. ICH = intracerebral hemorrhage, ivh = intraventricular hemorrhage, sah = subarachnoid hemorrhage, asdh = acute subdural hemorrhage.
Full size table

Statistical analyses

The case- and slice-level metrics were calculated using NumPy (version 1.24.0) and SciPy (version 1.9.3) Python packages. These reported metrics included sensitivity, specificity, false positive rate, negative predictive, positive predictive value, accuracy, and 95% confidence intervals (CI) for these metrics. We used the same Python packages for calculating the validation dataset demographics and interquartile ranges (IQR).

Results

Technical performance

Figure 2 depicts the time range taken for analyzing all slices in one head axial 3 mm NCCT scan (median number of slices 53). The median time of analyzing all slices in a single scan was 6.7 s (range from 4.2 to 33.1 s). Example of DL solution output is presented in Fig. 3.

Performance of metamodel and base U-Nets

Table 3 represents the case-level performance metrics for the four base U-Nets and the metamodel. The sensitivity and specificity of the metamodel was 92.4% (95% CI, 87.6–97.2%) and 53.8% (95% CI, 52.7–54.9%), respectively. The case-level false positive rate of the metamodel was 46.2% (95% CI, 45.1–47.3%). Both SAH1 and SAH2 base U-Nets had 100% sensitivity for hemorrhage detection at the case-level.

Table 3 Case-level performance of the base U-Nets and metamodel for all scans in validation dataset. SAH1 base U-Net is model trained during this study. SAH2 is model which development and validation was previously described by thanellas et al.14. The best achieved metrics are bolded. TP = true positive, tn = true negative, fp = false positive, fn = false negative, fpr = false positive rate, npv = negative predictive value, ppv = positive predictive value, ci = confidence interval.
Full size table

Supplementary Table 1 (Supplement 1) presents slice-level performance metrics for the base U-Nets and metamodel. The sensitivity and specificity of the metamodel was 71.7% (95% CI, 69.7–73.7%) and 98.2% (95% CI, 98.1–98.2%), respectively. The slice-level false positive rate of the metamodel was 1.8% (95% CI, 1.8–1.9%). At the slice-level, the ICH, IVH, SAH1 and SAH2 base U-Nets had false positive rates of 4.3% (95% CI, 4.3–4.4%), 4.3% (95% CI, 4.3–4.4%), 9.9% (95% CI, 9.8–10.0%) and 10.1% (95% CI, 10.0–10.2%), respectively.

Metamodel performance with full post-processing

Table 4 represents the case-level performance metrics of the metamodel with full post-processing. The case-level sensitivity and specificity of the metamodel were 89.8% (95% CI, 84.4–95.3%) and 89.5% (95% CI, 88.8–90.2%), respectively. The case-level false positive rate for the metamodel was 10.5% (95% CI, 9.8–11.2%).

Table 4 Case-level performance metrics of the metamodel with full post-processing. The performance metrics for scans with hemorrhage are also stratified according to delay from symptom onset to imaging. TP = true positive, tn = true negative, fp = false positive, fn = false negative, fpr = false positive rate, npv = negative predictive value, ppv = positive predictive value, n/a = not applicable, ci = confidence interval.
Full size table

The metamodel showed 69.3% (95% CI, 67.3–71.3%) slice-level sensitivity and 99.6% (95% CI, 99.6–99.6%) slice-level specificity. The slice-level false positive rate of the metamodel was 0.4% (95% CI, 0.4–0.4%). In other words, of the 7679 NCCTs reported negative for hemorrhage by the on-call radiologists, the solution predicted falsely positive 1594 slices out of the 408 426 slices. The slice-level performance of the metamodel with full post-processing is presented in Supplementary Table 2 (Supplement 1). Most of the false positive pixel clusters pointed out blood in normal and highly vascularized anatomical structures, e.g. superior sagittal sinus, cerebellar tentorium, straight sinus, and falx cerebri (Supplementary Figs. 1 and 2, Supplement 1). Detailed summaries of case- and slice-level performance metrics for each component of the DL solution are provided in Supplementary Figs. 3 and 4 (Supplement 1).

Metamodel performance by time of symptom onset

The metamodel with full post-processing detected hemorrhages in 59 out of 59 patients (sensitivity 100.0%) who were imaged within 12 h from symptom onset (Table 4). For 19 patients imaged most likely (not 100% certain) within 12 h from symptom onset, the solution’s sensitivity was also 100% (Table 4). An additional 18 patients were imaged 12 to 24 h from the symptom onset. The metamodel sensitivity with full post-processing among this group was 77.8% (95 CI, 56.3–96.6%) (Table 4). The four missed cases included two small ICHs and two focal small SAHs (Supplementary Figs. 5–8, Supplement 1). For patients imaged between 24 h and 7 days, the sensitivity of the metamodel with full post-processing was 71.4% (95% CI, 47.8–95.1%). The missed hemorrhage cases included one ICH and three focal SAHs (Supplementary Figs. 9–12, Supplement 1). The sensitivity for hemorrhages that were imaged after 7 days from symptom onset or had unclear symptom onset time was 55.6% (95% CI, 23.1–88.0%). The missed hemorrhage cases included three ICHs and one focal SAH (Supplementary Figs. 13–16, Supplement 1). The sensitivity of the metamodel with full post-processing for scans in different symptom-onset time points is represented in Fig. 4.

Cases missed in on-call reports but identified by solution

The metamodel with full post-processing identified five spontaneous intracranial hemorrhages that were not reported in the initial on-call reports. Of these five cases, one was ICH, two were SAHs, and one was IVH. All identified hemorrhages are presented in Supplementary Figs. 17–21 (Supplement 1).

Discussion

The developed DL solution detected spontaneous intracranial hemorrhages with 100% sensitivity on head CT scans imaged within 12 h from symptom onset, while maintaining a low processing time per scan. With full post-processing, the solution falsely detected an intracranial hemorrhage in approximately 1 out of every 10 true negative head NCCTs. Most of these false positive findings were present only in a few slices and in the same anatomical locations. Since the DL solution produces an easily visualized segmentation map, these small false positive pixel clusters could be relatively easily assessed as false positive findings by on-call radiologists or clinicians. Structures containing high vascularity or low pressure (slow flowing) blood, such as the choroid plexuses and intracranial sinuses, makes achieving a zero false positive rate a challenging task if a high sensitivity is a priority.

One of the DL solution’s base models has already been validated previously in both internal and external settings14. Despite the model had high sensitivity for detecting SAH, there was a relatively high number of false positive findings. Due to the high false positive rate, model‘s standalone clinical usability was considered suboptimal. Therefore, we developed this new solution that combines the outputs of high-sensitivity but low-specificity base U-Net models using a metamodel and post-processing steps. The lowest case-level base U-Net false positive rate was 61.3% which was reduced to 10.5% by applying the trained metamodel and rule-based post-processing steps. This translates to an improvement of over 80% at the case-level false positive rate.

The training material used in this study included only 300 head NCCTs. The training data was segmented by HU threshold-based method. In comparison to often used consensus-based approaches in image annotation, our approach requires less images and human resources. The development of deep learning models commonly relies on large amounts of training data, leading to a large amount of work and costs related to data preprocessing and model training. Thus, the study also suggests that the proposed segmentation method, ensemble-learning approach and development of rule-based post-processing pipeline can be used for training and optimizing clinically potential deep learning systems with limited amounts of training data. Additionally, the proposed approach enabled us to combine existing deep learning models into a unified framework, which in turn can reduce the total development time and also the need for extensive retraining with task-specific data.

Our study has also acknowledged limitations. First, the final number of true hemorrhage cases in the validation dataset can be considered low. However, the dataset represents a true consecutive patient cohort imaged during a 3-month long period in 10 different hospitals with a catchment area of around 1.7 million inhabitants. Second, the validation dataset was collected from 10 different hospitals, but the validation setting was still internal. Therefore, the solution’s results cannot be generalized outside the study country. Third, we did not have a possibility to assess our solution’s usefulness in the clinical workflow, as the solution is not an officially approved medical device. Fourth, using on-call radiologist reports as a ground truth might be considered a shortcoming, as it is recognized that some degree of error is inherent in the interpretation of medical imaging studies20,21. However, since our aim was to assess if the model could improve real-world diagnostics, using the reports as ground truth was necessary. By using this approach, we were able to evaluate whether the system could identify hemorrhage cases that have been missed in a real-life situation. Fifth, since the sensitivity diminishes by the delay from symptom onset to imaging, the model suits best for acute (12 h) cases. The training data was segmented using threshold-based on HU values, which represented acute blood in ICHs, IVHs, and SAHs. This segmentation method is likely to affect the DL solution’s performance when used for detecting subacute and chronic hemorrhages. This may be considered as a shortcoming, even though the DL solution was designed for emergency use and for acute cases only.

The sensitivity of acute intracranial hemorrhage detection of our solution is essentially similar to the reported performance metrics of commercially available and clinically used solutions22,23,24,25. The performance of the proposed DL solution is also comparable to other algorithms presented in the literature26,27. However, reliable comparisons between different solutions are difficult to conduct due to the lack of standardized comparison protocols and datasets.

Although emergency head NCCTs interpreted by radiologists have a high sensitivity for acute blood6,7, misidentification of acute intracranial hemorrhage still occurs in clinical settings. Our solution successfully detected five cases of acute spontaneous intracranial hemorrhages that were not reported in on-call reports, highlighting the potential impact the solution could have in diminishing misdiagnoses of intracranial hemorrhages in clinical practice. Due to relatively low computational costs of the 2-dimensional U-Net architecture, the solution could also be run in a local set up without requiring costly high-end graphics processing units or central processing units.

In conclusion, the developed DL solution with the post-processing pipeline had high sensitivity of detecting spontaneous intracranial hemorrhages in the acute period. This kind of solution could help to rule out spontaneous intracranial hemorrhages in a clinical setting. Compared to similar applications, to our knowledge, the presented solution is the first primarily intended to exclude acute spontaneous hemorrhages. This way, the solution does not burden or interfere with diagnostics in most cases. Even though the solution is not an officially approved medical device, it could be evaluated for quality assessments and research purposes.

Fig. 1
figure 1

Dataset Selection. We Reviewed all 8494 on-call report for the NCCTs. The final dataset included 118 NCCTs with spontaneous hemorrhage and 7679 NCCTs which were reported negative for hemorrhage in the on-call report. The dataset did not include head NCCTs from patients who were already admitted to a hospital ward before imaging. Also, if a single patient was imaged multiple times during different emergency clinic visits during the 3 months period, all first scans (excluding follow up scans) were included in the dataset.

Full size image
Fig. 2
figure 2

Deep Learning Solution’s Run Times. Histogram of inference times for scans using Nvidia Tesla V100 graphics processing unit (GPU) for running the inference task in Microsoft Azure Machine Learning Studio. The median time for analyzing a single scan was 6.7 s and 95% of the scans could be analyzed within 10.6 s. The run time included processing of all slices in 3 mm axial reformat scan and saving the prediction.

Full size image
Fig. 3
figure 3

True Positive Detections of Hemorrhage and Segmentations. (a) intracerebral hemorrhage, (b) intraventricular hemorrhage and (c) subarachnoid hemorrhage in axial non-contrast head CT 3 mm reformat slice. The left sided image in each panel shows the original CT scan slice, the middle image shows the ground truth segmentation with red color using the same HU ranges as those used for segmenting the training data, and the right sided image displays the DL solution’s detection of hemorrhage in the slice with purple color.

Full size image
Fig. 4
figure 4

Deep Learning Solution’s Case-Level Sensitivity with Full Post-processing. Sensitivity of the deep learning solution for detecting hemorrhages at different time point groups. The shaded area in the plot represents the 95% confidence interval.

Full size image