Introduction

Idiopathic pulmonary fibrosis (IPF) is a progressive and ultimately fatal disorder of unknown etiology, characterized by interstitial fibrosis of the lungs1,2. Repetitive alveolar epithelial injury triggers the early development of fibrosis and leads to tissue remodeling and the fibrosis seen in end-stage pulmonary fibrosis. The Ashcroft fibrosis score is a metric used to assess fibrotic changes in clinical samples under microscopic evaluation3. This assessment is traditionally performed by pathologists with a 10x objective in multiple fields of view, with scores ranging from 0 (normal lung) to 8 (total fibrous obliteration). The final score averages those of individual microscope fields.

To develop therapeutics for IPF, a widely used non-clinical model for IPF is bleomycin-induced pulmonary fibrosis mouse model (BLM model) of lung injury. Unfortunately, while no animal model fully recapitulates the histologic pattern of IPF or exhibits features of progressive disease4, BLM model, unlike human IPF, has characteristics such as rapidity of its development, inflammation preceding fibrosis, and self-resolution nature5. However, the model is currently considered “the best-characterized animal model available for preclinical testing” at this moment. Ashcroft fibrosis score, originally designed for human evaluation, includes late-phase IPF features such as “honeycomb lung”. Honeycomb lung appears rare in mouse models, but has been forcibly induced for the evaluation of BLM model6,7. Furthermore, one intractable problem with Ashcroft fibrosis score is that it is time-consuming and requires highly trained pathologists who are nevertheless prone to intra- and inter-observer variations6,8. Therefore, a new high-throughput and reproducible scoring system for assessing lung fibrosis in BLM model would be greatly aid the development of therapeutics9,10,11.

Recently, several reports have been published on image analyses for quantifying fibrosis areas in lung8,12,13. However, IPF grading requires highly nuanced interpretation of pathological images, including the accumulation of myofibroblasts in clusters called fibroblastic foci, and extensive extracellular matrix (ECM) deposition within the interstitium that results in the destruction of alveolar architecture14. Moreover, several reports have shown that tissue density affects fibrosis development12,15. As such, simple measurements of the fibrosis area may not be enough to capture the severity of IPF. Therefore, we developed a workflow in BLM model and generated deep learning models for grading lung fibrosis. The models function with minimal variability and, furthermore, provide a continuous score, offering opportunities for research with higher granularity.

Materials and methods

Dataset

We collected 173 whole slide images (WSI) (ndpi, x20) for C57BL6/J mouse lung fibrosis slides stained with Sirius red. Although the slides of left lung were prepared in two different facilities, the Sirius red staining was performed following a standard procedure. These slides were obtained from five independent archived animal studies with mice treated with bleomycin including control mice (Table 1). In all studies, animal procedures were conducted in accordance with the Institute’s Guide for the Care and Use of Laboratory Animals, and all experimental protocols were approved by the Institutional Animal Care and Use Committee at Chugai Pharmaceutical Co., Ltd. All C57BL/6J male mice were purchased from The Jackson Laboratory Japan, Inc. (Kanagawa, Japan) and were acclimated for 1 week before the start of treatments. Lung fibrosis was induced by the following three methods. Study 1 and 2 were established via intrathecal administration using 6-week-old mice. Bleomycin (Nihonkayaku, Tokyo, Japan) was administered intratracheally once at 0.75 mg/kg. Lung fibrosis was evaluated at day 21 from the administration of the bleomycin. Study 3 and 4 were established via osmotic pump using 8-week-old mice. 60 mg/kg Bleomycin was administered through subcutaneously by implanting osmotic infusion pump (ALTEZ 1007D) for 10 days. Lung fibrosis was evaluated at day 28 after pump implantation. Study 5 were established via subcutaneous administration using 8-week-old mice. Bleomycin was administered 200 µg/100 µL subcutaneously in total 10 shots (5 days consecutively administration, 2 days of withdrawal, and following 5 days consecutively administration). Lung fibrosis was evaluated at day 28 from 1st administration of the bleomycin. Lung sampling was performed under isoflurane anesthesia after exsanguination euthanasia in all studies. The study is reported in accordance with ARRIVE guidelines.

Table 1 Summary of the dataset.
Full size table

Histopathological evaluation

Pathologists assessed fibrosis grade for each animal study using light microscopy or WSI to establish ground truth. Fibrosis grades were categorized into four levels: none; 0% (Grade0), weak; less than 10% (Grade1), moderate; between 10% and 20% (Grade2), severe; more than 20% (Grade3) (Fig. 1a; Table 1). The grade was defined based on the area of fibrosis spreading within the lung tissue in each WSI. These pathologists provided the label for generating our deep learning models; however, to fully evaluate the performance of our models in comparison to pathologists, two additional pathologists (P#3 and P#4) also graded fibrosis using WSIs independently (Study 1 and 2 in Table 1).

Fig. 1
figure 1

Prediction of the grade of lung fibrosis. (a) Lung fibrosis was defined as having four grades: none, Grade0; weak, Grade1; moderate, Grade2; severe, Grade3. The grade was defined based on the area of fibrosis spreading. The squares in the top figures indicate the areas of high magnification in the lower section. Bar, 2.5 mm. (b) Overview of workflow for predicting the lung fibrosis grade. Whole slide images (WSIs) were cropped into rectangular images encapsulating the alveolar area and were divided into 1024 × 1024-pixel patches. Each patch was scored according to the area of fibrosis region detected by HALO AI. Next, we generated 5 × 5 tiled images using top 25 scored patches from each WSI. Tiled images were used as inputs for our models to test and evaluate.

Full size image

Workflow

First, WSIs were preprocessed with HALO AI image analysis software (Indica Labs, NM, USA, v3.3 or v3.4) to distinguish the alveolar area for analysis. Using the “classifier” function, a pathologist annotated “alveolar” and “bronchus”. The software then iteratively trained and output the classified results, and the “alveolar” area was defined as the object of analysis. Next, the fibrosis area within “alveolar” area was extracted using the “Area quantification v2.1.11” module. These results, including images of the detected fibrosis area, annotation data, and the original WSI file, were exported.

The resolution of unmodified WSIs was too high for use as inputs for our models; however, reducing image size could result in the loss of information needed for fibrosis analysis. Therefore, we used a pre-processing strategy to preserve necessary information while focusing on areas critical for determining the fibrosis grade. The WSIs were cropped into rectangular images of the alveolar areas and were partitioned into 1024 × 1024-pixel patches. The background regions within these patches were then removed. Each patch was scored according to the area of fibrosis region detected by HALO AI. Next, we generated 5 × 5 tiled images using the top 25 scored patches from each WSI as input images to test and evaluate our models. For training the models, we also generated tiled images using 25 patches randomly selected from the top 50 scored patches, and each patch was randomly rotated and flipped. (Fig. 1b). We generated 50 tiled images corresponding to a single WSI, and during each training epoch, a random tiled image was selected.

To evaluate the effectiveness of patch pre-selection, we trained and evaluated another model without using patch pre-selection. As with the patch pre-selection approach, the WSIs were divided into 1024 × 1024-pixel patches. Instead of selecting top-scored patches, all patches were used to generate 5 × 5 tiled images. Consequently, multiple tiled images were generated from a single WSI. For evaluation, the outputs from multiple tiled images of a single WSI were averaged to determine the predicted class.

Our models are convolutional neural networks (CNNs) with an EfficientNet-B2 backbone, pre-trained on ImageNet-1 K, and two fully connected layers as the classifier. We trained the model using a stochastic gradient descent (SGD) optimizer with 300 epochs, learning rate of 10− 3, and batch size of 64. We evaluated the models using 5-fold cross-validation. Each fold comprised 143 training data samples and 30 validation data samples. Given the limited number of data samples in this study, particularly with only six samples for Grade3, we did not simply divide the dataset into five parts and assign them to training and validation. Instead, we chose the validation data to ensure sufficient samples for each class per fold. The model training and validation were performed using Python v3.8.10 with Pytorch v1.10.2, Torchvision v0.11.3, Numpy v1.22.2 and Albumentations v1.3.0.

Statistical analysis and evaluation metrics

Statistical analysis and evaluation metrics were performed using Python v3.9.16 with Numpy v1.23.5 and Scipy v1.12.0. The effectiveness of each model compared to the pathologist evaluations was verified based on accuracy. The level of agreement between pathologist-pathologist pairs and machine-pathologist pairs was determined using both agreement coefficient and Cohen’s kappa coefficient16 (Kcoef). The means of the two groups was compared using Welch’s t-test.

Result

Validation of the models for grading fibrosis

To validate the quality of our models, we compared the fibrosis grades assigned by pathologists with the scores predicted by our deep learning models using 5-fold cross-validation (Fig. 2). The accuracy of all analyses was high, with the confusion matrix showing no estimation errors more than two classes apart. The models’ scores were calculated by multiplying SoftMax values with the grade numbers. The figure shows a correlation between the pathologists’ grades and the scores predicted by our models. The average accuracy of the five analyses for the model with patch pre-selection was 83%. In contrast, the average accuracy without patch pre-selection was 75%, demonstrating higher overall accuracy for the former. The model without pre-selection was more likely to underestimate grade, particularly predicting Grade 2 for data that pathologists evaluated as Grade 3 than the model with pre-selection. The model with patch pre-selection requires one input image for a single WSI, reducing computational cost.

Fig. 2
figure 2

Validation of the model for grading fibrosis. (a) (top) We calculated the accuracy by comparing the scores of the actual pathologists and those of predicted by our models using 5-fold cross-validation. (a) (bottom) The comparison of the pathologist evaluation and the grade score estimated by the models. The models’ grade score was calculated by multiplying SoftMax values and grade numbers. The dashed line was fitted by the least-squares method and r value represents the correlation coefficient between the pathologist evaluation and the calculated grade score. (b) (top) To compare with (a), we calculated the accuracy of the models without patch pre-selection. (bottom) Similarly, we compared the pathologist evaluation and the grade score estimated by the models without patch pre-selection.

Full size image

Correlation prediction from the models and pathologists

To evaluate the performance of our models in comparison to pathologists, we examined the level of agreement coefficient and Kcoef (Fig. 3a and b). We further compared the variations in both agreement coefficient and Kcoef between pathologist–pathologist pairs and machine–pathologist pairs (Fig. 3c and d). No significant difference was observed, indicating that our models’ performance in grading lung fibrosis is on par with that of pathologists.

Fig. 3
figure 3

Model predictions compared with pathologist evaluations. (a) Agreement coefficient and (b) Cohen’s kappa coefficient (Kcoef) for every pair. (c) Variations in agreement coefficient and (d) variations in Kcoef between pathologist–pathologist pairs (P-P) and machine–pathologist pairs (M-P) are shown. The Kcoef is based on the difference between the observed agreement (po) and the probability of chance (pe) and is calculated as po − pe / 1 − pe. P values were calculated using Welch’s t-test (n = 3). G, P#3, P#4 and M indicate ground truth, pathologist #3, pathologist #4 and machine.

Full size image

Discussion

BLM model is a widely used animal model to evaluate therapeutic targets for IPF5,17,18,19. Histopathological analysis of lung fibrosis is an important method for evaluating BLM model. However, challenges such as inter- and intra-observer variations and disappointing throughput complicate this approach6,8. We believe that digital technologies will be able to streamline and improve the evaluation process.

In this study, we successfully developed deep learning models that mimic evaluation by a pathologist. These models achieved an accuracy comparable to the variations among pathologists. For this dataset, WSIs from Sirius Red stained specimens prepared at multiple centers (Table 1) were used. Although there was a concern that this might affect the accuracy of the models, the verification results showed that the data from the facilities used in this study was consistent and acceptable for this study. However, our workflow should be validated further in order to be applied automatically to any stain of any center due to the limitation of n = 2 centers in this study.

We carried out patch pre-selection before delving into deep learning. The process of human annotation presents several challenges, as it can be time-consuming and subject to variability20. Meanwhile, self-supervised learning approaches, capable of capturing image features without the necessity for annotation, demand substantial data and computational resources21. To surmount these hurdles, we developed models capable of interpreting complex image patterns, akin to the abilities of pathologists. This was accomplished beyond the simple area extraction that the HALO AI image analysis software usually performs. This was motivated by our preliminary study showing that simple area extraction and pathologist evaluations did not sufficiently match. This discrepancy was attributed to the fact that pathologists did not grade solely based on the simple Sirius red-positive area. They also considered qualitative factors such as collagen texture and distribution. The pathologist’ perspective has the potential to recognize drug possibilities that might be overlooked in efficacy evaluations based only on the size of the area. We think this qualitative capacity is more important than the operational efficiency and reproducibility provided by our simpler deep learning models.

In conclusion, we developed a deep learning workflow for assessing fibrosis grades in BLM model. This workflow emulates the pathologist’s grading process, including qualitative assessment. The use of this workflow is expected to facilitate the development of drugs for pulmonary fibrosis.