Detection of commercial crop weeds using machine learning algorithms

Abstract

This work investigates the YOLOv5 object detection algorithms for classifying commercial crops such as tomatoes, chili, and cotton. The data sets comprise 707 images of green chillies, 200 images of tomato crops and 130 images of weeds from Ponnandagoundanoor farms in western agro climatic Zones (WAZ) of Tamil Nadu. The objective of this research is to explore the determination of weed present in the crops and further the machine learning (ML) algorithms that have deployed for computing the F1 score, detection time, and mAP of each machine learning algorithms. As a result, a tomato dataset contains an F1 score of 98%, a mAP of 0.995, and a detection time of 190 ms; a cotton dataset with an F1 score of 91% and a mAP of 0.947; and a chilly dataset with an F1 score of 78% and a mAP of 0.811. A Further investigation has been carried out for the same crops; improving YOLOv5 accuracy includes adaptively spatial feature fusion (ASSF) blocks to its architecture head. An enhanced YOLOv5 algorithm using ASFF modules on the same datasets achieved an F1 score of 99.7% in the tomato dataset and 79.4% in the chilly dataset, resulting in a 1.14% improvement in the F1 score. With a 93.53% F1 score, were able to obtain a 2% enhancement over the cotton dataset. The extended YOLOv5 increased the mAP by about 0.5% and resulted in an insignificant drop in the number of computations carried out, rendering the model more compact.

Introduction

The horticulture and plantation sectors play a significant role in the state of Tamil Nadu, contributing to its agricultural economy and providing employment opportunities to a large number of people¹. The cultivation of fruits, vegetables, and ornamental plants is known as horticulture, while the large-scale cultivation of crops such as tea, coffee, spices, etc. is known as plantation. The diversified climate, fertile soil, and ample water supplies of Tamil Nadu² make it ideal for horticulture practices. The state contains a diverse range of agro-climatic zones, from coastal regions to hill stations that allow for the production of a variety of horticulture crops³.

Weeds can have a considerable influence on horticulture crop output. These unwanted plants that grow along with the crops compete with them for nutrients, reduce sunlight penetration, and serve as hosts for pests, insects, and, in some cases, diseases. They also make harvesting difficult by getting tangled with the crops⁴. Some weed species exhibit the phenomenon of allelopathy⁵, where they release chemicals into the soil that inhibit the growth of nearby plants. To mitigate the negative effects of weeds on horticulture crop yield, weed management practices are essential. These can include cultural practices like crop rotation, mulching, and regular cultivation to suppress weed growth⁶.

In horticulture, traditional weed control methods often utilize a combination of human, mechanical, and chemical procedures. Some commonly used methods include hand plucking, hoeing (using a hoe or similar tool to cut and uproot weeds), mulching, and spraying weedicides⁷. However, these methods are labor-intensive, lack precision and accuracy, and have high manual costs. Automating the weed identification process is necessary to expedite the weed removal process, thereby reducing manpower and expenditure and significantly enhancing production⁸.

Information and computing technologies play a crucial role in modern weed control practices. ICT makes use of data-driven decision-making, which enables the collection, storage, and analysis of large amounts of data related to weed populations, crop growth, environmental conditions, and management practices. Computing technologies, namely satellite imagery, remote sensing, and geographic information systems (GIS), allow for precise mapping and monitoring of weed distribution and dynamics⁹. Machine learning and artificial intelligence algorithms can assist in automated weed detection and identification¹⁰. We can integrate these technologies to create an automatic weeding robot for variable site-specific weedicide spraying.

Image normalization, plant segmentation, annotations, etc. are common procedures carried out in weed detection systems. Machine learning-based algorithms increase the accuracy of weed identifications, but they reduce the robustness. Deep learning approaches showed that they could handle complicated issues with ease¹¹.

When weeds grow in the same area as crops, they deprive the crops of the necessary nutrients, leading to lower yields, increased production costs, difficulty in harvesting, and a reduction in the overall quality of the product¹². Weeds, many of which act as hosts for various plant diseases, also hinder the flow of irrigation water and make it more difficult to water crops. The cost and amount of herbicides applied to a field also harm crop production and soil quality. As a result, it is vital to automate the weed removal process and limit the use of herbicides¹³.

This research work aims to develop a crop-weed detection mechanism that enhances accuracy and speed of detection, with the potential to upgrade to an autonomous weeding system that sprays a limited amount of herbicide on weeds and removes them early. This work develops a CNN-based classification algorithm for crops and weeds.

The structure of this research paper is presented below. Section II offers a thorough overview of the studies conducted in this field. Section III delves into the description of the dataset, the methods that were used, and the proposed technique along with its block diagram. Section IV delves into the experimental data and its interpretation. Section V concludes and discusses future work.

Literature survey

In¹⁴, the authors proposed convolutional neural networks to segment vegetables and classify remaining green objects as weeds to enhance weed identification. A deep learning algorithm is under development for robotic weed removal to detect vegetable-bounding boxes and perform weed segmentation based on color features outside the boxes. They employed the CenterNet al.gorithm¹³ for bok choy detection, which generates bounding boxes based on class probability. Color index-based segmentation is then used to identify weeds. Images of bok choy from Nanjing, China, were augmented to 11,500 and manually annotated using LabelImg software. The trained CenterNet achieved impressive precision (95.6%), recall (95.0%), and F1 score (0.953) by treating the remaining green objects as weeds. Future work involves weed identification in in-situ videos, optimizing the deep learning model, and evaluating accuracy in vegetable detection.

In¹⁵, the authors developed a decision-based predictive algorithm for early weed detection in a green-on-green setting, validated on testing data. A DJI Phantom 3 professional drone extracted two datasets from a weed-infested sugarcane field. The datasets consisted of high-resolution images captured at a fixed altitude of 70 m, resulting in 261 images with a pixel size of 2.13 cm. The authors performed UAV data preprocessing steps using Pix4D software, which included mosaicking, RGB band stacking, and spatial subset segmentation and they marked and masked polygon-shaped regions of interest to effectively encapsulate weed patch structures. Colour-based features were extracted from the training imagery for classification, achieving an 89% accuracy in detecting “early weed” patches in red and “crop” patches in green. The algorithm demonstrated the potential for localized and accurate early-stage weed detection.

The authors in¹⁶ investigated trade-offs between semantic segmentation quality and imaging sensor complexity. The authors used a generative adversarial neural network to generate a fake infrared channel for alignment. OpenDroneMap software aligned unsynchronized frame sequences, and a greedy-region-based procedure constructed a paired dataset for training. They manually label using RGB orthophotos and propose a single, upgraded RGB camera for cost-effectiveness. The evaluation based on mIoU scores shows 95% and 90% for old and young crops, respectively. Inconsistencies in manual labelling are the main source of error. Other agricultural industry-related applications, such as disease and insect detection, can apply the proposed approach.

In¹⁷, the authors classified weeds and maize using a snapshot mosaic hyperspectral imaging sensor, processing images to obtain the spectral reflectance of regions of interest. The objective of the work was to explore the feasibility of a NIR snapshot mosaic hyperspectral camera for classification, determine relevant wavelengths and features, and provide optimal parameters for a RF model. They sowed seeds of C. arvensis, Z. mays, Rumex, and C. arvense in sandy soil-filled pots and recorded them using the snapshot mosaic hyperspectral camera in the plant laboratory of ILVO in Belgium. They assessed and compared the performance of RF and KNN models using the McNemar test and tested RF classifiers with different spectral feature combinations, and selected 30 significant features through a feature reduction procedure based on importance scores. The crop (Z. mays) exhibited exceptional precision (94%) and recall (100%), while precision values for C. arvensis, Rumex, and C. arvense were 95.9%, 70.3%, and 65.9%, respectively. These findings support the potential application of the camera in implementing site-specific weed management (SSWM) for targeted weed management strategies.

In¹⁸, the authors proposed YOLOv4-tiny for detecting sesame crops and weeds, using a labelled dataset of 1300 photos obtained from Kaggle. The dataset includes 512 × 512 colour images of sesame crops and various weed types. We trained the model using 1000 images for training and 300 images for testing, with an input size of 416 × 416. We evaluated our object identification model using Mean Average Precision (mAP), taking into account bounding box coordinates, class probabilities, probability score threshold, and intersection over union (IoU). We trained the model on a Tesla K80 GPU, converted the weights to TensorFlow Lite (TFlite) format, and deployed it on a Raspberry Pi 4 model B, achieving 4 frames per second (FPS). Future work includes performance testing on Nvidia Jetson Nano and Jetson TX2, leveraging the TensorRT library for accelerated inference speed.

The authors in¹⁹ introduced a promising technique for crop and weed detection that addresses the challenges posed by morphological similarities. The authors used a combination of contour masks and filtered LBP operators²⁰. The filtered LBP operators allow for better feature extraction by capturing local texture information, while the contour masks aid in separating plants based on their shape and structure. The experimental results demonstrated a classification accuracy of 98.63% with four classes in the “bccr-segset.” We could conduct further research by integrating the contour masks and filtered LBP operators with other advanced technologies like machine learning algorithms, deep learning models, or computer vision techniques.

In²¹, the authors focused on enhancing the classification of sugar beet and weeds using the VGG-Beet CNN, a modified VGG16 model²² with 11 weight layers. They utilize three diverse datasets: a local dataset captured with the Agrocam sensor, a public dataset captured with the Parrot Sequoia camera, and another publicly available dataset captured with the Red Edge camera sensor. These datasets vary in lighting conditions, geographical locations, and soil conditions. We manually separate crop and weed patches during training to exclude mixed-class patches. We train the VGG-Beet CNN using cross-validation data, stopping at optimal weights. The maximum overall testing accuracy for the Agrocam and Red Edge datasets is 86.3% and 92.4%, respectively. Further, this work will develop methods for classifying mixed-class patch images, reducing mixed-class patches, improving weed detection accuracy, and minimizing the misclassification of crops as weeds.

In²³, the authors investigated the use of convolutional neural networks (CNN) with hyperspectral imaging, patch-based classification methods, and RGB images for weed mapping. The authors employed data augmentation methods such as rotation and random cropping to enhance training data for CNN models, thereby improving recognition accuracy. The study includes uniform band selection and analyses the impact of resolution on weed classification, emphasizing the relationship between patch size and sensitivity. The results showed CNN’s superiority over traditional feature extraction methods and highlighted the advantages of hyperspectral data over RGB images. The sensitivity of CNN performance to patch size underscores its correlation with spatial resolution in image processing.

The authors of²⁴ created a machine learning-based crop-weed identification system capable of performing site-specific detection. The authors utilized an SVM-based classification and detection system in real-time to extract features from crops and weeds. The SVM-based classifier reached a real-time detection speed of 5–6 frames per second and a 96% accuracy. Future work will also include determining the spray amount for each plant based on its size and adjusting the spray flow rate.

The authors in²⁵ focused on developing a real-time monitoring system for detecting broken corn during harvesting. The system includes a camera, a hardware platform, and a designed corn detection method. The camera captured images of both broken and non-broken corns before sending them to the peeling device. We generated synthetic images of broken corn with different backgrounds to train a corn detection network. We fine-tuned the network using real images of broken corn. The authors resized the input image to 416 × 416 pixels and trained the network using YOLOv3 Tiny, which predicts bounding boxes, objectless scores, and class probabilities for corn and non-corn objects at multiple scales. We implemented the proposed corn detection method on NVIDIA TX2, achieving a speed of up to 10 fps, enabling nearly real-time detection.

In²⁶, the authors proposed YOLOv4-tiny for detecting sesame crops and weeds, using a labelled dataset of 1300 photos obtained from Kaggle. The dataset includes 512 × 512 colour images of sesame crops and various weed types. The model was trained using 1000 images for training and 300 images for testing, with an input size of 416 × 416. We evaluated our object identification model using Mean Average Precision (mAP), taking into account bounding box coordinates, class probabilities, probability score threshold, and intersection over union (IoU). They trained the model on a Tesla K80 GPU, converted the weights to TensorFlow Lite (TFlite) format, and deployed it on a Raspberry Pi 4 model B, achieving 4 frames per second (FPS). Future work includes performance testing on Nvidia Jetson Nano and Jetson TX2, leveraging the TensorRT library for accelerated inference speed.

Table 1 shows the comparison of the existing literature with the proposed methodology.

Table 1 Comparison of the existing literatures with Proposed.

Proposed methodology

Consider a crop weed identification system that has Ct, Cc, and Ch numbers for tomato, cotton, and chili crop images, and Wt, Wc, and Wh for the corresponding numbers of weed images. Let N x N be the dimension of the input image, and n x n is that of the pre-processed image. Let B represent the backbone network and N represent the neck network of YOLOv5. The dimensions of the feature map obtained from the BottleNeck CSP architecture are (Fh The architecture of YOLOv5 is represented by B. while the output dimensions of the pooled feature maps are (Fho, Fwo). We denote the pooling w Let k be the number of feature maps. w size as k and the stride as s. Let Fbi stand for the feature maps with dimensions (Hf, Wf) derived from the backbone network, where i ranges from 1 to 3. Let (hin, win) represent a position in the Fbi feature map, and (hout, wout) represent the upsampled feature map. Let w1, w2, w3, and w4 represent their weights.

The variables boxx, boxy, boxw, and boxh are used to denote the x and y coordinates, width, and height of the bounding box. The variables tbx, tby, tbw, and tbh represent the X and Y coordinates, width, and height of the bounding box. The variables “abw” and “abh” represent the dimensions of the anchor box. Let lx, ly denote the Cartesian coordinates of the upper-left corner of the grid that contains the object. The symbol represents the logistic activation function.

This research work is analyzed in terms of parameters, namely precision (λ), recall (µ), F1 score (Ψ), and confidence score^27,28,29. Let tp stand for true positive, indicating the correct classification of crop images as such, and tn for true negative, indicating the incorrect classification of weed as crop. When we incorrectly classify a crop as weed, let fn stand for false negative, and fp for false positive.

System architecture

Fig. 1

System architecture.

Figure 1 shows the system architecture of the proposed YOLOv5. A custom dataset of green chili and an open access dataset of tomato and cotton to train and test the CNN model. Then preprocess the raw data to improve its quality and reliability. We fed the pre-processed images into two detection and classification mechanisms, YOLOv5³⁰ and Adaptable YOLOv5, to make accurate predictions and classify crops and weeds. Performance validation of both mechanisms evaluated the robustness of the model.

Data acquisition

This study examined three different crop types: tomato, chilli, and cotton. The chilli dataset was generated using unique data gathering equipment with high-resolution sensors. These photographs were taken in January 2023 on fields near Ponnandagoundanoor, Coimbatore. Seven hundred and seventy pictures show the plants and weeds of green chilli. We took these pictures two weeks after the first seedling planting. We captured the images from a height of approximately 2 feet, with a temperature of approximately 29 degrees Celsius. There was not a single image in the collection that featured an overcast sky or dim lighting. Table 2 illustrates the description of Chilli datasets.

Table 2 Chilli dataset description.

Conventional dataset

The “Early crop weed dataset” provided by the Agricultural University of Athens contains images of early-stage tomato, cotton, velvetleaf, and black nightshade in the field³¹. The photographs were taken with a Nikon D700 (2272 × 1704) from three separate farms in Greece between May and June 2019 under identical lighting conditions. Table 3 Summarizes the Early Crop Weed dataset.

Table 3 Early Crop-Weed dataset Description.

Pre-processing

Figure 2 shows the steps involved in the proposed algorithm. Image preprocessing prior to training a model is essential for increasing data quality, enhancing model performance, and ensuring training efficiency. The input image of size N x N is first resized to Nr x Nr to match the network’s input size. To standardize the input, normalize the pixel values of the image to a specific range, such as by dividing them by 255. We then performed data augmentation techniques to enhance the model generalization capability, employing image transforms such as flip, rotate, brightness, and blur. Create anchor boxes in various sizes to serve as reference points for predicting bounding boxes. Convert the ground truth bounding box coordinates to YOLO format to obtain the pre-processed image n x n.

Fig. 2

Steps involved in image preprocessing.

Detection and classification algorithm

YOLOv5³²

Figure 3 illustrates the architecture of YOLOv5 algorithm.

Fig. 3

Architecture of YOLOv5.

The backbone network B of YOLOv5 receives a nxn input image. The image first passes through the BottleneckCSP²⁷ architecture, which extracts the deep semantic information from the input images. The Spatial Pyramid Pooling (SPP) layer³³ extracts the feature maps of dimension (Fhin, Fwin) and passes them on, assisting in removing the fixed-size constraint of the neural network. The SPP layer divides the input feature map into discrete spatial bins and performs pooling from each bin. Every spatial bin undergoes an independent pooling operation, usually max pooling, which is a mathematical calculation of the highest value in each bin.

The output dimensions of the pooled feature map is given by

$${\text{Fho}}\,{\text{=}}\,{\text{floor }}\left( {\left( {{\text{Fhin - k}}} \right){\text{ / s}}} \right)\,{\text{+}}\,{\text{1}}$$

(1)

$${\text{Fwo}}\,{\text{=}}\,{\text{floor }}\left( {\left( {{\text{Fwin - k}}} \right){\text{ / s}}} \right)\,{\text{+}}\,{\text{1}}$$

(2)

where floor is a mathematical operation that rounds a given number down to the nearest integer. The three feature maps, Fb1, Fb2 and Fb3 obtained from the backbone network are 80 × 80 × 256, 40 × 40 × 512, and 20 × 20 × 1024.

The feature maps from B are transferred to the PANet³⁴ neck N, where 1 × 1 convolutions are performed to reduce computations and reduce the number of feature channels. Up sampling is then performed to convert the low-resolution features to higher resolutions and the output is fused with the output of the bottleneck layer of B.

As mentioned, Hfi and Wfi represent the input height and width and Hfo and Wfo represent the output height and width of the feature map from B respectively. In the up sampled feature map, the position (hout, wout) corresponds to the position (hin, win) in the input feature map.

The corresponding position in the input feature map is computed by

$${\text{hin}}\,{\text{=}}\,{\text{hout / SFw}}$$

(3)

$${\text{win}}\,{\text{=}}\,{\text{wout / SFh}}$$

(4)

where SFw and SFh are the scaling factors in both dimensions which is given by

$${\text{SFw}}\,{\text{=}}\,{\text{Hfo / Hfi}}$$

(5)

$${\text{SFh}}\,{\text{=}}\,{\text{Wfo / Wfi}}$$

(6)

The four nearest neighbours surrounding the position hin, win (from Eqs. 3 and 4) are (a1, b1), (a1, b2), (a2, b1), and (a2, b2), were

$${\text{a1}}\,{\text{=}}\,{\text{floor}}\left( {{\text{hin}}} \right)$$

(7)

$${\text{a2}}\,{\text{=}}\,{\text{a1}}\,{\text{+}}\,{\text{1}}$$

(8)

$${\text{b1}}\,{\text{=}}\,{\text{floor}}\left( {{\text{win}}} \right)$$

(9)

$${\text{b2}}\,{\text{=}}\,{\text{b1}}\,{\text{+}}\,{\text{1}}$$

(10)

The weights are calculated based on the distance between hin, win and its neighbours.

$$\text{d}\_\text{a}\,{\text{=}}\,{\text{hin - a1}}$$

(11)

$${\text{d}\_\text{b}}\,{\text{=}}\,{\text{win - b1}}$$

(12)

$${\text{w1}}\,{\text{=}}\,\left( {{\text{1 - d}\_\text{a}}} \right){\text{ * }}\left( {{\text{1 - d}\_\text{b}}} \right)$$

(13)

$${\text{w2}}\,{\text{=}}\,{\text{d}\_\text{a * }}\left( {{\text{1 - d}\_\text{b}}} \right)$$

(14)

$${\text{w3}}\,{\text{=}}\,\left( {{\text{1 - d}\_\text{a}}} \right){\text{ * d}\_\text{b}}$$

(15)

$${\text{w4}}\,{\text{=}}\,{\text{d}\_\text{a * d}\_\text{b}}$$

(16)

The pixel value is interpolated using the four nearest neighbour values and their corresponding weights:

$$\left( {{\text{hout,wout}}} \right)\,{\text{=}}\,{\text{w1*}}\left( {{\text{a1, b1}}} \right)\,{\text{+}}\,{\text{w2*}}\left( {{\text{a1, b2}}} \right)\,{\text{+}}\,{\text{w3*}}\left( {{\text{a2, b1}}} \right)\,{\text{+}}\,{\text{w4*}}\left( {{\text{a2, b2}}} \right)$$

(17)

The up-sampling process is achieved by repeating this process for each pixel in the feature map. The concatenated feature Z from Eq. 17 is represented as

$${\text{Z}}\,{\text{=}}\,{\text{concatenate }}\left( {\left[ {{\text{hout, wout}}} \right]{\text{, axis}}} \right)$$

(18)

The axis parameter indicates the specific axis along which the concatenation operation is executed.

This output undergoes a similar process before being fused with the output of the first bottleneck layer of B. The output here is then passed through similar layers but while performing 3 × 3 convolutions in between to extract the spatial information. The fused feature maps Fn1, Fn2, and Fn3 are then passed on to the head network, H, where 1 × 1 convolutions are done to reduce the number of channels and H predicts the bounding box coordinates bx, by, bh, bw and the class of the object.

$${\text{boxx = (2 *}}\upsigma \left( {{\text{tbx}}} \right){\text{-}}\,{\text{0}}{\text{.5)}}\,{\text{+}}\,{\text{l x}}$$

(19)

$${\text{boxy = (2 *}}\upsigma \left( {{\text{tby}}} \right){\text{-}}\,{\text{0}}{\text{.5)}}\,{\text{+}}\,{\text{ly}}$$

(20)

$${\text{boxw}}\,{\text{=}}\,{\text{abw * (2 *}}\upsigma \left( {{\text{tbw}}} \right){\text{)}}$$

(21)

$${\text{boxh}}\,{\text{=}}\,{\text{abh * (2 *}}\upsigma \left( {{\text{tbh}}} \right){\text{)}}$$

(22)

The logistic activation function, denoted by σ, restricts the output values to the range of 0 to 1. The application of this procedure involves the utilization of tbx and tby to ensure that the centroid of the bounding box is located within the grid. As per the equations provided above, the bounding boxes that correspond to anchors falling within the range of 0.5 to 2 times the anchor size are taken into account. The final output image is a no x no image with bounding box, class label and confidence score.

Adaptable YOLOv5

The adaptable YOLOv5 model introduces adaptively spatial feature fusion (ASFF) blocks into the head layer as shown in the Fig. 4. This allows the head network to learn in an adaptable manner the spatial weights necessary for feature fusion at various scales, thereby increasing the detection accuracy of objects. ASFF balances the conflicts between different level features and improves the effectiveness of the feature pyramid to increase the model’s average detection accuracy³⁵.

The feature maps Fb1, Fb2, and Fb3 are derived from the backbone network, while the feature maps Fn1, Fn2, and Fn3 are obtained from the neck network. To generate x^1→2, Fn1 undergoes convolution to match the number of channels in the Fn2. Subsequently, the feature map with the same dimensions as Fn2 is up sampled. Fn3 undergoes convolution and down sampling operations in order to modify its channel count and dimensions. As a result, x^3→2 is obtained. The adjustment of the Fn2 feature map is achieved by modifying the number of channels subsequent to the convolution operation, resulting in the acquisition of X^2→2. The three feature maps are subjected to the SoftMax function for processing. This yields the weight coefficients α, β, and γ for x^1→2, x^2→2, and x^3→2, respectively. The ASFF fusion calculation is performed utilizing the following formula.

$$\:{y}_{ij}^{l}={\alpha\:}_{ij}^{l}\cdot{x}_{ij}^{1\to\:l}+{\beta\:}_{ij}^{l}\cdot{x}_{ij}^{2\to\:l}+{\gamma\:}_{ij}^{l}\cdot{x}_{ij}^{3\to\:l}$$

(23)

where $\:{y}_{ij}^{l}$ is the feature map obtained using ASFF module.

Fig. 4

Architecture of adaptable YOLOv5.

Results and discussion

This section uses graphs to analyse the obtained simulation results, elucidating the evaluation metrics to provide precise details and debating the conclusions drawn from the outcomes of both models.

Simulation parameters

Table 4 shows the simulation parameters for the proposed research.

Table 4 Simulation Parameters.

Performance metrics

The variable tp stands for true positive, signifying cases where crop images accurately identify as a crop. Similarly, tn stands for true negative, referring to instances where weed images correctly identify as weeds. The notation “fn” signifies a false negative, indicating an incorrect classification of a crop as a weed. Simply put, “fp” signifies a false positive when an incorrect classification of a weed as a crop occurs³⁶.

Precision: Out of the total number of crop images, precision refers to the number of correctly labelled crop images.

$$\lambda \,{\text{=}}\,{\text{tp/ (tp}}\,{\text{+}}\,{\text{fp)}}$$

Recall: The recall measures the proportion of correctly labelled crop images to the total number of crop images.

$$\upmu \,=\,{\text{tp / Total number of Crop Images}}$$

The F1 score is a statistical measure that represents the harmonic mean of a model’s precision and recall values. The values of the variable fall within the interval of 0 to 1.

$$\Psi \,=\,2{\text{ }}[(\lambda x\upmu )/{\text{ }}(\lambda \,+\,\upmu )]$$

The confidence score represents the probability of the algorithm detecting an image accurately. We compute the mean average precision at various intersection over union (IoU) thresholds to determine the scores.

Crop–weed identification

Outcome of crop–weed identification using YOLOv5

We obtain the resulting confidence score by multiplying the conditional class probabilities and confidence score per bounding box. Figures 5 and 6, and Fig. 7 display scores of 0.9, 0.78, and 0.48. These scores represent the probability of the occurrence of a crop and weed in the box.

Fig. 5

Crop–weed identification of tomato using YOLOv5.

Fig. 6

Crop–weed identification of chili using YOLOv5.

Fig. 7

Crop-weed identification of cotton using YOLOv5.

Outcome of crop–weed identification using adaptable YOLOv5

During the testing process, one of the outputs provided is the confidence score per bounding box. The confidence score multiplies the probability of classifying an image as a crop or weed given the presence of an object within the bounding box. The resulting numerical value, as shown in Figs. 8, 9 and 10, determines the final confidence score. These scores are 0.9, 0.8, 0.7, and 0.4.

Outcome of crop–weed identification using adaptable YOLOv5

Fig. 8

Crop–weed identification of tomato using adaptable YOLOv5.

Fig. 9

Crop–weed identification of chilli using adaptable YOLOv5.

Fig. 10

Crop–weed identification of cotton using adaptable YOLOv5.

Performance analysis

From Table 5, it is evident that the developed crop-weed identification mechanism has enhanced the model’s accuracy and PR values around 1.5–3.5%. Adaptable YOLOv5 has outperformed the existing YOLOv5 model in terms of metrics such as F1 score, mAP, precision, and recall, which denote the quality of the classifier.

Table 5 Comparison of YOLOv5 and adaptable YOLOv5 for tomato, chilli and cotton datasets.

Simulation results

F1 score vs confidence

Tomato dataset – YOLOv5

Fig. 11

F1 score vs. confidence score for tomato using YOLOv5.

Tomato dataset – Adaptable YOLOv5

Fig. 12

F1 score vs. confidence score for tomato using adaptable YOLOv5.

The confidence score that optimizes precision and recall is 0.782, which corresponds to the maximum F1 value of 1.00 for Improved YOLOv5. Since higher confidence and F1 values are desirable, it is evident from Figs. 11 and 12 that the improved YOLOv5 has better accuracy.

Chilli dataset – YOLOv5

The confidence score that optimizes precision and recall is 0.393, which corresponds to the maximum F1 value of 0.79 for Improved YOLOv5. Since higher confidence and F1 values are desirable, it is evident from Figs. 13 and 14 that the improved YOLOv5 has better accuracy.

Fig. 13

F1 score vs. confidence score for chilli using YOLOv5.

Chilli dataset – Adaptable YOLOv5

Fig. 14

F1 score vs. confidence score for chilli using adaptable YOLOv5.

Cotton dataset – YOLOv5

The confidence score that optimizes precision and recall is 0.561, which corresponds to the maximum F1 value of 0.92 for Improved YOLOv5. Since higher confidence and F1 values are desirable, it is evident from Figs. 15 and 16 that the improved YOLOv5 has better accuracy.

Fig. 15

F1 score vs. confidence score for cotton using YOLOv5.

Cotton dataset – Adaptable YOLOv5

Fig. 16

F1 score vs. confidence score for cotton using adaptable YOLOv5.

Figures 17 and 18, and 19 indicate a decrease in the percentage of false predictions when transitioning from YOLOv5 to Adaptable YOLOv5. For the tomato dataset, the percentage of false predictions fell from 47% to 33%; for the Chilli dataset, the values dropped from 34.67% to 26%; and for the Cotton dataset, the percentage of false predictions fell from 33.56% to 28%. Thus, the adaptable YOLOv5 model has yielded results of approximately 5–10%.

Fig. 17

Percentage of false predictions for tomato crop.

Fig. 18

Percentage of false predictions for chilli.

Fig. 19

Percentage of false predictions for cotton.

Conclusion and future work

This research explored the YOLOv5 object detection algorithm for classifying crops and weeds in tomato and chili-cultivated farms. Three sets of data were used to train and test the model: a custom dataset with 707 images of green chili crops and weeds from Ponnandagoundanoor farms in Coimbatore district, Tamil Nadu, India; an open access dataset called “The Early Crop Weed set” from Borja Espejo Garcia et al.‘s “Towards weed identification assistance through transfer learning”; and a dataset of images of the tomato crop and its weed, Black nightshade; and the cotton plant and its weed, Velvetleaf. There were 200 images of tomato crops and 130 images of weeds in the dataset. After augmentation, the dataset’s size increased to 736. Similarly, we achieved a total of 707 images for the Chilli dataset. PyTorch serves as the basis for PyTorch serves as the foundation for the algorithm, and we set the training epoch at 200. We computed the F1 score, detection time, and mAP of each machine learning algorithm, resulting in a tomato dataset with an F1 score of 98%, a mAP of 0.995, and a detection time of 190 ms, a cotton dataset with an F1 score of 91% and a mAP of 0.947, and a chilli dataset with an F1 score of 78% and an mAP of 0.811. We improved YOLOv5’s accuracy by adding adaptively spatial feature fusion blocks to its architecture’s head.

We added three ASFFv5 blocks to YOLOv5. This learns to get rid of contradictions across a number of feature scales by spatially filtering input that is at odds with the scales. This makes the scale-feature invariance higher. We tested the improved YOLOv5 algorithm with ASFF modules on the same datasets and found that it achieved an F1 score of 99.7% in the tomato dataset and 79.4% in the chilli dataset, resulting in a 1.14% increase in the F1 score. We achieved an F1 score of 93.53% in the cotton dataset, indicating a 2% increase. The improved YOLOv5 also improved the mAP by about 0.5% and led to a slight decrease in the number of computations performed, thereby making the model more lightweight. After thinking about what has been said, we can say that a new method was created that combines adaptively spatial feature fusion with the YOLOv5 algorithm. The new YOLOv5 is better for real-time detection on an embedded platform and can be built into a robot that pulls weeds in farms. We can further develop the crop-weed identification system into a weed-removal robot with image sensors, which can monitor and perform site-specific weedicides spraying, thereby reducing the need for human labour. We can further enhance the bot to achieve better results.

Future research may employ adaptive tokenisation³⁷ transformers to enable dynamic representation learning for accurate weed-crop differentiation, reducing computational costs while improving precision in complex field environments. Simultaneously, enhancing reinforcement learning via hierarchical game playing with state relay presents a promising framework for autonomous weeding robots³⁸. It enables decision making across various levels and facilitates adaptation to diverse farming systems. The integration of these methods can establish a perception action loop, facilitating the advancement of fully automated and sustainable weed management systems.