Introduction

Psidium guajava L.(common guava) is an important commercial fruit crop in many tropical countries. It is a highly nutritional fruit rich in vitamins, fiber, and minerals. Existing literature has shown that it has a high antioxidant capacity and other biologically active secondary compounds1, such as vitamins, phenols, carotenoids, and lycopene, and other chemical substances that help to prevent cancer and other chronic and degenerative diseases2. It has a long culinary history all around the world, and research on its medicinal properties, which include being antidiarrheal, antibacterial, antioxidant, antigenotoxic, anti-inflammatory, and hypotensive, can be traced back to the 1940s3. Currently, guavas are commercially grown in a number of tropical and subtropical countries, and selective breeding based on regional needs has greatly increased yield in the past decade. Although international trade in fresh guava is currently limited, both guava and its processed products are gaining popularity in markets such as Europe and North America4.

Due to the short shelf life of guava following harvesting, low-temperature storage is usually employed to delay ripening and fungal deterioration. However, most tropical fruits are highly sensitive to cold, so the low-temperature storage process may induce cold damage such as abnormal ripening and browning and increase the incidence of rot5. The speed at which guavas ripen depends on the variety. Suitable storage and handling methods can prevent supply chain losses due to bruised and overripe fruit, lengthen the storage period, and maintain quality in the environmental conditions of the shipping process6. Disease detection can promote the early discovery of diseases, increase fruit volume7, and prevent the damaged portion from breeding other bacteria and infecting more fruit, which would result in even more losses. If the diseased fruit enters the market, it could also affect human health and cause food safety issues8.

Early measurement methods to detect diseases tended to be invasive. More recently, non-destructive and non-invasive testing methods have been developed. One example is non-invasive hyperspectral imaging (HSI), which measures the spectral ranges of normal and damaged fruit9. With advances in image recognition and deep learning such as structured-illumination reflectance imaging (SIRI) systems and convolutional neural networks (CNNs), image classification methods can now clearly identify diseased regions while reducing other surface characteristics, thereby effectively facilitating early disease detection10and increasing the detection rate of new bruises below the surface11. However, HSI methods and visible/near infrared spectroscopy cannot detect damage in unknown locations12. Thus proposed combining a thermal imager with imaging methods to test degrees of ripeness and test for disease. They also suggested applying CNN models for training and their modified CNN models reached accuracy rates of more than 99% with custom data. However, the models developed using this approach are subject to limitations because the fruits that are deemed to abnormal are all put into one category; that is, the type of the abnormality cannot be differentiated. Deep learning is another approach to defect detection with accuracy rates of over 99%13.

In addition to advances in detection methods, developments in statistical analysis and modeling of obtained data also promote quality detection and fruit grading. Shao et al.14 employed visible-near infrared (Vis-NIR) HSI and factor analysis to create a novel comprehensive quality index (CQI), overcoming the issue of correlation among the internal components of single indices. They established comprehensive quality prediction models using partial least squares regression (PLSR), principal component regression (PCR), and multiple linear regression (MLR) and combined a spectral index and traditional imaging to obtain visualized tomato quality information. Based on various fruit quality parameters, Funsueb et al.15 proposed fruit quality indices FQI1 and FQI2, which are respectively based on simple geometric and weighted mean values. Using near-infrared spectroscopy data, they constructed a partial least squares (PLS) model which can be used to evaluate the overall quality of fruit samples. Wang et al.16compared the Vis-NIR point spectra and Vis-NIR-HIS prediction results of three varieties of apples, proposed a Vis-NIR method to visualize the changes in hardness in the three apples, and discussed the influences of interval variances among the fruits on the prediction results. They pointed out that in addition to gradual improvements made in HIS index statistical methods, deep learning prediction models such as CNNs outperform conventional machine learning methods. However, although deep learning enables effective feature learning, it is difficult to train models accurately small amounts of data.

Existing deep learning detection methods have been applied in a number of studies on topics such as fruit detection, damage assessment, and fruit quality grading. The CNN frameworks most frequently employed in fruit damage detection at present are ResNet-50 and VGG-16, and deep learning methods are at a greater advantage when processing large datasets17. Increasingly, deep learning has been applied to agricultural fields other than those associated with fruit quality, such as disease detection and leaf disease detection in crops18. For the most part, computer vision and data analysis techniques are still applied for seed identification, water stress detection, water erosion assessment, pest detection, and greenhouse monitoring. In recent years, more and more studies have used drone-captured images and deep learning to estimate yield and although these techniques remain prone to issues associated with image capture and accuracy, they promise more effective yield assessment, increased agricultural productivity, and reductions in food waste19.

To resolve difficulties in costs and instant computation, we employed a commercially-available 3840 × 2160 resolution color camera and lens with an adjustable field of view to remove installation distance limitations and enlarge the effective pixels of the targets for better recognition rates. We divided appearance defects into 13 classes and applied the Yolov420 pretrained model framework. After stacking our samples and retraining, we achieved effective fruit defect classification.

Results

Table 1 presents the training results. In the overall image database, a total of 1701 images were marked using thirteen labels (as shown in Table 2). A total of 4044 labels were applied, and the overall false negative rate and false positive rate were 5.03% and 6.62%, respectively.

Table 1 False positive and false negative rates for each class.
Full size table
Table 2 Names and abbreviations/codes of defect classes.
Full size table

Deep learning

During training, assessments were made using box loss, object loss, and class loss, with values closer to 0 indicating higher accuracy. We randomly retrieved 70% of the images in the database as training data (1190 images), and the remaining 30% served as test data (511 images). The appearance defects were divided into 13 classes, as shown in Table 2. We utilized the Yolo v4-coco pretrained network architecture and added the above database to retrain the network weights. It was executed on a computing host with an i7 processor with a processing speed of 2.1 GHz, 64G RAM, and an RTX 3080 graphics card. The training took a total of two days, two hours, and 47 min. With the training set, the total loss, box loss, object loss, and class loss were 0.2576, 0.19043, 0.026229, and 0.040938, respectively.

Figure 1 exhibits some of the image recognition results of the training and test sets. Figure 1(a) ~ 1(d) present some of the image recognition results of the training set. Figure 1(a) shows an example of Colletotrichum gloeosporoides in the training set. As shown in Fig. 1(b), a single image may contain more than one type of defect; this figure shows damage from thrips and browning. Figure 1(c) shows an example of Pestalotiopsis psidii and Fig. 1(d) displays a healthy fruit with no defects. Figure 1(e) ~ 1(h) present some of the image recognition results of the test set. Figure 1(a) shows an example of Pestalotiopsis psidii in the test set. Figure 1(f) displays a healthy fruit with no defects. Figure 1(g) shows an example of damage from Bactrocera dorsalis Hendel. Figure 1(h) shows an example of Colletotrichum gloeosporoides.

Fig. 1
figure 1

Experiment results: (a)~(d) training images; (e)~(h) test images.

Full size image

The term false positive (FP) refers to non-defective cases that are misidentified as defects. They are identified by comparing the results of automatic detection using the trained network with manually-marked samples using the formula in Eq. 1. The term false negative (FN) refers to defective cases that are missed in the detection process. They are determined using Eq. 2. True positives (TP) are calculated as the total number of samples minus the number of missed detections. Table 1 exhibits the training and test set data for each class. In the random training set, the false negative rates in each class were less than 1%, and the false positive rates were less than 13%. The two classes with the highest false positive rates were Planococcus minor (P.M.=13%) and damage from thrips (Thrips = 11%). In the manually-marked samples, the false negative rate was 0.17% and the false positive rate was 5.33%.

$$\:False\:positive\:rate=\:\frac{Number\:of\:false\:positives}{Number\:of\:false\:positives\:+\:Total\:EquationNumber}$$
(1)
$$\:False\:negative\:rate=\:\frac{Number\:of\:false\:negatives}{Number\:of\:false\:negatives\:+\:Total\:EquationNumber}$$
(2)

To prevent overtraining, which would increase the number of accurately recognized results in the training set but lead to abnormal situations with images not in the training set, we used the remaining images as the test set and employed the training set with manually-marked samples to test whether overfitting existed in the trained network weight. In the test set, the class with the highest false negative rate was manmade damage (M.D.), which had a false negative rate of 38%. This was followed by 32% in the physical damage from branches (P.D.) class (both classes had few samples in the database, and there were wide variations between samples). These were followed by damage from Lepidoptera larvae (Le.=29%) and damage from Bactrocera dorsalis Hendel (B.D.H = 29%). The average false negative rate was 16%. The false positive rates of each class were less than 36%. The two classes with the highest false positive rates were browning (Brown = 36%) and physical damage from branches (P.D.=30%). In the manually-marked samples of the test set, the false negative rate was 15.82% and the false positive rate was 11.29%.

The total number of the training and test sets was 4404. Colletotrichum gloeosporoides had the highest number with 1120; the total number of false negatives of Colletotrichum gloeosporoides in the training and test sets was 27, and the false negative rate was 2.35%. The total number of false positives in the training and test sets was 62, and the false positive rate was 5.25%. There were 6 instances of sunburn, with 1 false negative for a false negative rate of 14.29%. There were 20 instances of manmade damage (M.D), with 5 false negatives for a false negative rate of 20%. There were 110 instances of Planococcus minor (P.M.), with 19 false positives for a false positive rate of 14.73%. There were 129 instances of physical damage from branches (P.D.), with 27 false positives for a false positive rate of 17.31%. Overall, the false negative rate was 5.03% and the false positive rate was 6.62%.

Fig. 2
figure 2

Recognition results: (a) smallest recognition result: 13 × 14 pixels; (b) largest recognition result: 266 × 312 pixels.

Full size image

In the experiment results, the smallest and largest objects detected were a chilling injury of 13 × 14 pixels and a healthy fruit of 266 × 312 pixels, respectively, as shown in Fig. 2.

Further analysis was conducted to assess the robustness of the model against FNs and FPs. This was achieved by determining the number of TPs, FPs, and FNs in the training and test datasets, as summarized in Table 1. The initial analysis revealed no FPs or FNs in the training set, indicating that the training results were based on reliable data.

To further assess model performance21, 200 images without the object of interest were added to the test dataset. The model was then assessed in terms of accuracy, precision, recall, and F1-score. Accuracy was defined as the proportion of correct predictions across the entire sample. Precision was defined as the proportion of samples identified as positive that are actually positive (TPs), while recall was defined as the proportion of actual positive samples that were correctly identified by the model. The F1-score indicates the harmonic mean of precision and recall, providing a single measure that balances both aspects.

The data in Table 1 was subsequently used to compare the predictions versus the ground truth based on the numbers of TPs, FPs, FNs in the training and test sets, irrespective of specific categories. An additional 200 images without the object of interest were added to the test dataset. These images were correctly classified in the group without objects from any category, resulting in 200 TNs (see Table 3).

Table 4 Lists the accuracy, precision, recall, and F1-scores calculated based on the values ​​in table 3. All values for the training set exceeded 94.5%, indicating good model stability. The values for the test set exceeded 75%, which is considered satisfactory. The overall mean value across datasets was above 88%. Taken together, these results highlight the model’s robustness and stability without evidence of overfitting.

Table 3 Confusion matrix of basic data and prediction results.
Full size table
Table 4 Four numerical analysis identification results.
Full size table

This study performed real-time detection using a pretrained YOLO v4-COCO object detection model running on a system with an Intel i7 2.1 GHz processor and an RTX 4090 graphics card. The resolution of each image in the stacked test dataset was 3840 × 2160 pixels; however, none of the detected objects exceeded 266 × 312 pixels in size (i.e., approximately one-tenth of the image resolution), as shown in Fig. 2(a). Setting the region of interest (ROI) to 1000 × 1000 improved the detection speed to 12 FPS. When applied to the image dataset, the system achieved a false negative rate (FN) of 5.03%, a false positive rate (FP) of 6.62%, and accuracy of 88.15%.

Discussion

This paper lays the foundation for the development of an automatic quality grading system for Psidium guajava L (common guava). We collected 189 fruits from different farms and captured 1701 images, each with a resolution of 3840 × 2160 pixels. This resulted in a total of 4404 defect samples for offline training. Defects were divided into thirteen classes, including damage from pests, diseases, and humans. Among the thirteen classes, it is more difficult to manually identify Colletotrichum gloeosporoides, Pestalotiopsis psidii, and Phyllosticta psidiicola. Traditionally, these three classes require cutting the fruit open or conducting fungal cultures for accurate classification. However, we were able to achieve false positive and false negative rates of less than 9% in these three categories using non-invasive detection.

Figure 3 illustrates the design of the proposed guava grading system. The front-end comprises multiple platforms where the fruits are stacked (represented as yellow discs in the picture). After manually placing the fruit samples, the platforms are extended by an internal mechanism, positioning the samples within the viewing range of cameras located on each side of the platform. The detection environment is designed with a white or solid-color background (highlighted by the red dot in the picture) to enhance recognition performance. The front-end platform evaluates the quality of the moving fruit samples, and the quality results are further refined through back-end weight classification to quantify the overall fruit quality.

Fig. 3
figure 3

Computer rendering of fruit grading system.

Full size image

As shown in Table 1, the number of samples was greater than 500 and that the overall false positive and false negative rates were less than 8%. In future work, we plan to increase the number of samples in the database so that the numbers of samples in each class are greater than 500.

We applied the weights of the trained network to the stitched panoramic image shown in Fig. 4. Figure 4(a) shows the original stitched image, and Fig. 4(b) presents the recognition results. As we used stitching, the image is not overly deformed; thus, the same training results can be shared for recognition. In future work, we plan to increase the number of panoramic images to more accurately obtain the distributions and proportions of defects on entire fruits.

Fig. 4
figure 4

Stitched panoramic image of a fruit: (a) original image; (b) recognition results.

Full size image

Methods

Offline training was performed using Matlab 2023a run on a PC with an Intel i7-12700 2.1 GHz processor and an RTX-3080Ti graphics card. We next describe how our database was obtained and briefly motivate our selection of the network architecture.

Database

In deep learning, establishing the database and standards is crucial. Thus, planning sample collection is of upmost importance. Psidium guajava L. can be divided into climacteric and non-climacteric varieties. The former produce softer and more fragrant fruits, which are usually processed, whereas the latter yield crisper fruits that usually serve as fresh produce. Non-climacteric varieties include Tai-Guo Ba, Jen-Ju Ba, Shui-Jing Ba, Di-Wang Ba, and Tsai-Hong Ba. Jen-Ju Ba is the main cultivated variety in Taiwan, accounting for over 95%. We therefore chose to collect Psidium guajava L. ‘Jenju Bar’. We used a variable focus color camera and an image resolution of 3840 × 2160 pixels to capture our images. The distance between the camera and the fruit was 30 cm with an LED light set up next to the camera tripod. This set-up is shown in Fig. 5. The experimental fruits were obtained from the twelfth class of the fruit tree production and marketing class of the Yanchao District Farmers Association in Kaohsiung City. This production and marketing class is composed of many different professional farmers. We took images of 189 fruits from different farms. The fruit was placed on a white rotating risk, and an image was taken every 40 degrees, resulting in a total of nine anchor points for each fruit and 1701 images in total.

Fig. 5
figure 5

Set-up for obtaining image samples.

Full size image

The most important aspects of any deep learning implementation are data collection and the accuracy of the ground truths. Thus, after the images were collected, Po-Sung Chen aided in pathological analysis to ensure the accuracy of the ground truths and the importance and criticality of the classes. Phyllosticta psidiicola, Colletotrichum gloeosporoides, and Pestalotiopsis psidii can only be accurately identified by experienced experts and the results must be confirmed by cutting open the fruit and making cell cultures. We used both methods to check the ground truths in our database. The other appearance defects shown in Fig. 6 include chilling injuries, which are associated with season and weather, sunburns, which are more associated with the way orchard branches are pruned, browning, which can be caused by a number of reasons and are therefore more irregular in shape, damage from Planococcus minor, which can occur on the peel throughout the year, damage from Lepidoptera larvae, which gnaw on the peel and create irregular rough spots on the peel, damage from Bactrocera dorsalis Hendel adults, which lay eggs within the peel, creating sunken black spots on the peel, manmade damage, which is mostly caused by pressure and scratches during manual handling, damage from thrips, which are rough and reddish brown marks on the fruit, and physical damage from branches, which is the result of poor branch pruning that affects the shape of the fruits as they grow.

Fig. 6
figure 6

Examples of images in each class.

Full size image

The 1701 images were manually labelled. Except in the healthy fruit images, the other twelve classes could be present simultaneously. In total, there were 281 healthy fruit labels, 811 chilling injury labels, 6 sunburn labels, 101 browning labels, 1120 Colletotrichum gloeosporoides labels, 847 Pestalotiopsis psidii labels, 129 Phyllosticta psidiicola labels, 110 damage from Planococcus minor labels, 116 damage from Lepidoptera larvae labels, 38 damage from Bactrocera dorsalis Hendel labels, 20 manmade damage labels, 336 damage from thrips labels, and 129 physical damage from branches labels. This resulted in a total of 4404 labels.

Deep learning

The most well-known network architecture for real-time detection is Yolo. The first through third generations successfully solved computing problems from 2015 to 201822,23,24. Consequently, many real-time deep learning network architectures were developed. With the approval of the author of the first generation, various teams developed the fourth through eighth generations. The two-stage solution (class and bounding box) under the first-generation network, the CNN architecture, was unified into a regression problem, with a computing speed of 45 FPS. The second generation featured better precision, convergence, and computation speed. The third generation presented improvements in its small object detection capabilities, but at the expense of computation speed. The fourth generation20improved the balance between resolution and number of convolutional layers, and unlike the first three generations, the detected objects could overlap, thereby achieving high recognition rates and high computation efficiency. The fifth25generation optimized the computation of the fourth generation, which doubled the computation speed. The sixth and seventh generations26,27 improved on precision and speed.

In this study, wide variations in size and shape exist in each class of the 4404 labels. While the precise positioning of objects is not required for linguistic segmentation, objects must overlap and remain unfiltered to achieve real-time detection. Our maximum and minimum selection boxes were 300 × 300 pixels and 32 × 32 pixels. We thus chose the YOLO V4 pretrained network architecture. The parameter settings were as follows: Anchor boxes were (7,6), (22,22), (55,38), (51,86), (104,78), (88,167), (170,125), (149,246), and (335,285). We employed a mini-batch-size of 8, max-epochs of 100, a learning rate of 0.001, a warmup period of 1000, l2Regularization of 0.001, and a penalty threshold of 0.5. To avoid overfitting, we randomly selected the training set from the database and used the test images that were not in the training set to detect overfitting.