Path planning of intelligent tennis ball picking robot integrating twin network target tracking algorithm

Abstract

Intelligent tennis picking robots can effectively improve the efficiency of tennis training and competition, and reduce manual labor intensity. However, the real-time tracking of targets in existing intelligent robot path planning is poor and susceptible to becoming entrenched in local optimal solutions. Therefore, this study proposes an intelligent tennis ball picking robot path planning method that integrates a twin network object tracking algorithm. In terms of target tracking, a hybrid attention mechanism is introduced, which utilizes a transformer structure to achieve hierarchical feature fusion. In terms of path planning, this study combines an improved rapidly-exploring random trees with an artificial potential field ant colony algorithm to enhance the obstacle avoidance capability of robot path planning. Among them, the hybrid attention mechanism enhances local feature extraction and reduces the influence of occlusion by combining grouped convolution transformation and spatially gated embedding. Additionally, the Transformer structure improves tracking accuracy by capturing the global context relationship. In path planning, the improved bidirectional rapidly-exploring random tree algorithm is enhanced through sector constraint sampling to improve search efficiency. The artificial potential field ant colony algorithm optimizes the obstacle avoidance ability and path smoothness. The results showed that in the training dataset, the accuracy of the proposed target tracking algorithm was as high as 0.981, which was 5.40–25.56% higher than existing algorithms such as SiamFC, MORT, SiamRPN, MeMOT, and FROG MOT. In both test datasets, the expected average overlap values were 0.405 and 0.437. The path planning length and time of the proposed method were 42.07 m and 56.12 s, significantly lower than other methods. This indicates that the research method can provide accurate target position information for robots, optimize path planning, and improve the efficiency of picking up tennis balls. This method provides an effective solution for target tracking and path planning of intelligent tennis ball picking robots in complex environments and has important practical application value.

Introduction

The advancement of artificial intelligence and robotics technology has led to the increasingly widespread application of intelligent robots in daily life and industrial fields¹. As a global sports activity, the research and development of automation equipment related to tennis has also attracted much attention. Intelligent Tennis Picking Robot (ITPR), as an automated device with practical application value, can effectively improve the efficiency of tennis training and competition, and reduce manual labor intensity². However, the development of existing ITPR still faces technological challenges in target tracking and Path Planning (PP). The Target Tracking Algorithm (TTA) has shortcomings such as low recognition efficiency and poor tracking accuracy in dealing with complex scene interference and target state estimation³. In addition, in the PP of intelligent robots in complex environments, traditional Path Planning Algorithms (PPAs) have good theoretical performance, but in practical applications, they have shortcomings such as easily getting stuck in locally optimal solutions. For example, F. Kiani et al. pointed out that traditional meta-heuristic algorithms, such as genetic algorithms and Particle Swarm Optimization (PSO), are also prone to getting stuck in local optima when dealing with complex terrain and dynamic environments. These algorithms required more complex mechanisms to overcome local optima and effectively explore the global search space⁴. S. Khanna pointed out that gradient methods often get stuck in local minima, especially in environments with multiple obstacles⁵. Y. Tao et al.‘s research also showed that even advanced algorithms such as Ant Colony Optimization (ACO) are prone to getting stuck in local optima when the pheromone distribution is uneven, leading to premature convergence⁶. Overall, traditional PPAs tend to converge to local optima rather than global optima. In addition, there are multiple local optimal solutions in the search space, which further complicates the optimization process and leads to solutions that are not optimal. Therefore, optimizing TTA, improving its computational efficiency and real-time performance, and combining efficient PPA to enhance the performance of ITPR have become a key focus of current research.

Twin networks, as an efficient TTA, are widely used in various research fields due to their fast speed and high accuracy, which can significantly improve the accuracy of target tracking⁷. N. An et al. raised a multi-TTA based on Siamese neural networks. The proposed methodology employed deep neural networks and object detection to facilitate the multi-target tracking process. A combination of the object detection network with the single object tracking network has been utilized to enhance the tracking performance of the target. This method could effectively reduce the impact of target occlusion and other issues in target tracking⁸. Z. Xia et al. developed an end-to-end collaborative multi-agent reinforcement learning method to enhance the efficiency and accuracy of target tracking. This method addressed the problem of existing TTA relying on motion frames and low success rates in target tracking. This method introduced the power consumption model and energy-saving strategy of the propulsion system and added spatial information entropy to the tracking algorithm. This method had excellent performance in tracking success rate and energy-saving efficiency⁹. H. Bao et al. developed a Siamese attention network for visual tracking based on concatenation, aiming to address occlusion and scale variation in existing TTA. The efficacy of this method was evidenced by its capacity to promote the representation ability of the target by fusing low-level features with high-level features extracted from disparate convolutional layers. The proposed method had good performance in target tracking¹⁰. L. Zhou et al. developed a distributed tracking algorithm based on twin networks. This algorithm integrated sensing and communication resources to achieve distributed tracking and collaboration over short and long distances while reducing communication energy consumption. This method significantly saved 65.7% of communication energy consumption and effectively improved perception performance by 20.0%¹¹. Z. Ji et al. proposed a multi-var trajectory tracking approach for digital twin intersections, aiming to reduce the error of target tracking. This method utilized long short-term machine networks and graph attention networks to extract spatiotemporal features and predict their states. Then, by calculating the distance matrix, the tracking error was fed back to the tracker. The effectiveness of the method was validated through four datasets in the experiment¹². C. Fu et al. explored twin network target tracking methods in the field of Unmanned Aerial Vehicles (UAVs). This study analyzed the adaptability of key technologies such as network structure, feature extraction, and similarity measurement of existing algorithms in complex scenarios. The results showed that the twin network-based method performed well in real-time and accuracy¹³. S. Javed et al. studied a visual object tracking method based on discriminative line filters and Siamese networks. This study analyzed the online update strategies and feature representation techniques and the performance of related filtering and deep learning. The results showed that the twin network combined with discriminative filtering performed superior in real-time tracking, but faced challenges of occlusion and deformation¹⁴.

Robot PPA is the core technology for achieving autonomous navigation of robots. This method has received a lot of research in recent years because it can help robots efficiently and safely complete tasks and adapt to complex and changing environments. B. Li et al. proposed an online PPA for UAVs based on deep reinforcement learning for tracking maneuvering targets and obstacle avoidance control. This method provided decision-making capability for drones through deep deterministic policy gradients and introduced a mixed noise to assist drones in exploring stochastic strategies for online optimal planning. This method significantly improved the generalization ability and training efficiency of UAV tracking controllers in uncertain environments¹⁵. H. Zhang et al. raised a robot PPA based on an indoor spatiotemporal network model and designed a dynamic PPA for robots in intricate and changeable indoor conditions. This method modeled the indoor spatiotemporal network using actual data to verify the feasibility of the algorithm, which had better performance in indoor spatiotemporal grids¹⁶. W. Chi et al. raised an efficient heuristic robot PPA based on the fast exploration of random numbers. This method first extracted lightweight features and used corresponding feature node fusion techniques to remove redundant nodes. Then, it implemented PP through a heuristic path-guided sampling process. This method had good performance in heuristic PP and real-time motion planning¹⁷. Y. Ou et al. raised an improved A* PPA based on grid maps to address slow search speed and excessive turning points in existing robot PPA. This method was predicated on the implementation of a path-smoothing strategy and a security protection mechanism. The purpose of these mechanisms was to determine whether there were any obstacles at the connection between two path nodes. Then, by utilizing a guided cost way and an adaptive cost function, the searching efficiency was improved. The mean path search duration for this method was minimized by 13% and the search for extended nodes was decreased by 11%¹⁸. J. Wang et al. proposed a hybrid PP method that combines bidirectional Rapidly-exploring Random Trees (RRT) and reinforcement learning. This method adopted the target gravity strategy and collision weight strategy and optimized the random exploration process through Q-learning in reinforcement learning. The experimental results showed that the proposed algorithm was significantly superior to traditional methods in terms of PP time and path length¹⁹. D. Li et al. proposed a multi-step ACO algorithm based on the terminal distance index for the PP of mobile robots. This method optimized the heuristic information of the ACO algorithm by introducing the terminal distance index, enhancing its global search capability and PP efficiency. It was found that the proposed algorithm outperformed traditional ACO algorithms in terms of path length, planning time, and path smoothness²⁰. J. Doe et al. proposed an Omnidirectional (OMNI) robot navigation method based on Simultaneous Localization and Mapping (SLAM). This method simulated non-uniform paths and vascular tumor scenes, utilized visual sensors and SLAM to construct and locate maps, and optimized PP strategies. The results showed that robots could effectively avoid collisions in complex environments, significantly improving navigation efficiency and adaptability²¹. A. J. Moshayedi et al. proposed a robot development method from simulation to design. This method optimized the structure and function of robots through preliminary design and testing in a simulation environment and then applied the simulation results to actual robot manufacturing. The results showed that the method performed well in both simulation and actual operation, with efficient navigation and task execution capabilities, verifying the feasibility and efficiency from simulation to design²².

In summary, some breakthroughs have been made in the research of dual network TTA and robot PPA. For example, in the field of target tracking, TTAs have significantly improved the accuracy and real-time performance of target recognition by introducing attention mechanisms and feature fusion techniques. In terms of PP, improved methods such as artificial potential field and ACO algorithm can effectively improve the efficiency and obstacle avoidance ability of PP. However, there are still issues that need to be further addressed, such as high complexity, insufficient real-time performance, poor adaptability to complex environments, and PPAs that are prone to get stuck in local optima and inefficient planning. Therefore, this study combines a dual network object tracking algorithm with Hierarchical Feature Fusion (HFF) with an improved PPA that combines RRT and Artificial Potential Field Ant Colony Optimization (APF-ACO), which can solve the above problems. This study first develops an advanced TTA that improves the accuracy and efficiency of tennis tracking in complex environments by integrating Lightweight Attention (LA) and HFF. A path optimization planning method for ITPR is proposed, which combines the improved Bidirectional RRT (Bi-RRT) algorithm with APF-ACO to improve the obstacle avoidance ability and path efficiency of the ITPR.

The research objectives are as follows: (1) Develop an advanced object tracking algorithm that improves the accuracy and robustness of tennis tracking in complex environments by integrating (HAM and HFF. (2) Design an efficient path planning method for an intelligent tennis ball picking robot, combining an improved Bi RRT algorithm and an APF-ACO algorithm to optimize path efficiency and obstacle avoidance ability. (3) The effectiveness of the proposed method is verified through comprehensive experiments and simulations, demonstrating its superiority in tracking accuracy, path planning efficiency, and overall robot performance.

The novel contribution of this study lies in threefold. Firstly, a new Hybrid Attention Mechanism (HAM) is introduced, which replaces the traditional SE attention mechanism by combining Group Convolutional Transform (GCT) and Spatially Gated Embedding (SGE), significantly improving real-time performance while maintaining high tracking accuracy. Secondly, an HFF strategy based on a Transformer structure has been developed, which can achieve comprehensive feature extraction and global context modeling, effectively addressing the limitations of existing methods in preserving spatial and semantic information. Thirdly, by combining Bi-RRT and APF-ACO, the PPA has been improved, enhancing obstacle avoidance capabilities, reducing path redundancy, and shortening computation time, making it highly suitable for dynamic environment applications.

The innovation of this study lies in the introduction of an adaptive environment perception module, which can monitor changes in the external environment in real time, such as light intensity, surface conditions of the court, and distribution density of tennis balls. Based on these perceptual information, the system dynamically adjusts the parameters and path planning strategies of the target tracking algorithm. (2) Through the fusion of multimodal data, the system can more comprehensively perceive the state of the target, and accurately locate and track the tennis ball even in situations where the target is partially occluded or the background is complex. (3) Introduced reinforcement learning algorithms to optimize path planning. By constructing a reward function, factors such as path length, obstacle avoidance ability, path smoothness, and task completion time are taken into consideration.

Methods and materials

This section first develops a twin network object tracking optimization (TNOTO) algorithm that combines LA and HFF, aiming to improve the algorithm’s computational efficiency. Then, an ITPR-PPA method based on improved RRT and APF-ACO is proposed to find the optimal path for intelligent robots to pick up tennis balls.

Twin network tracking optimization algorithm combining la and HFF

Twin network TTA represents a meaningful research direction within the domain of computer vision, with extensive applications in video surveillance, robotics, and UAV driving²³. However, the existing twin network TTA has problems such as high computational complexity, inaccurate target scale estimation, poor real-time performance, and susceptibility to interference from semantically similar objects²⁴. Therefore, this study proposes a TNOTO algorithm that combines LA and HFF. MobileNetv3-Large can minimize the complexity of the model and help improve computational efficiency. Therefore, this study selects the lightweight network MobileNetv3-Large as the backbone network and improves it to enhance the running speed and parallel processing capability of TTA. It reduces the amount of parameters and improves the efficiency of target recognition by introducing a HAM. Then, the HFF strategy is utilized to preserve the important and spatial information of the algorithm, thereby improving its target tracking and recognition capabilities.

Due to the high real-time requirements of TTA, the SE attention mechanism in MobileNetv3 Large network can slow down the inference speed of the network, affecting the real-time response capability of the algorithm²⁵. Therefore, this study introduces a HAM. This mechanism integrates GCT and SGE, replacing the original SE attention mechanism, to achieve accurate target recognition²⁶. The HAM based on GCT and SGE is shown in Fig. 1.

Fig. 1

Schematic diagram of HAM based on GCT and SGE.

In Fig. 1, the input feature map is first processed through GCT. This technology involves dividing the input feature map into multiple groups and applying convolutional transformations within each group. This method preserves the necessary features of the input while reducing computational complexity. GCT helps to capture local features more efficiently by reducing the number of parameters and required computational complexity. After GCT, the feature map is further processed through SGE. Space gating helps reduce noise and focus on the most relevant parts of the feature map, thereby improving the accuracy of object recognition. The outputs of GCT and SGE are combined to form the final feature representation. This combination utilizes the advantages of two techniques, with GCT efficiently capturing local features and SGE enhancing the spatial correlation of these features.

After introducing a HAM in lightweight networks, the backbone network of the network is improved to reduce computational complexity and efficiently capture the local features of the target. This method first adjusts the step size of the backbone network to lower the quantity of network parameters. To maximize the lightweight effect of the network, this study uses deep convolution in depthwise separable convolution to efficiently process the convolution kernels and combines the output convolution kernels using point convolution²⁷. Among them, the output feature map and the computational expression of a single convolution kernel are shown in Eq. (1).

$$\left\{ \begin{gathered} {I_F}=W \cdot H \cdot n \hfill \\ C=I_{u}^{2} \cdot m \cdot n \cdot I_{F}^{2} \hfill \\ \end{gathered} \right.$$

(1)

.

In Eq. (1), ${I_F}$ is the output feature map. C and ${I_u}$ are the computational complexity and size of the convolution kernel. W and H are the width and height of the feature map. n means the amount of output channels. m means the amount of input channels. The computational complexity of its deep convolution is shown in Eq. (2).

$${D_c}=I_{u}^{2} \cdot m \cdot I_{F}^{2}+I_{F}^{2} \cdot m \cdot n$$

(2)

In Eq. (2), ${D_c}$ represents the computational complexity of deep convolution. The computational complexity of convolution kernels and deep convolutions ${D_c}/C$ is shown in Eq. (3).

$${D_c}/C=\left\{ \begin{gathered} (I_{u}^{2} \cdot m \cdot I_{F}^{2}+I_{F}^{2} \cdot m \cdot n)/(I_{u}^{2} \cdot m \cdot n \cdot I_{F}^{2}) \hfill \\ \frac{1}{{I_{u}^{2}}}+\frac{1}{n} \hfill \\ \end{gathered} \right.$$

(3)

.

In Eq. (3), ${D_c}/C$ is inversely proportional to the size of the convolution kernel. To make the network’s computations less complicated, this paper uses a method called point convolution. This method allows the network to perform convolution operations on groups of feature maps. Then, it uses a method called channel rearrangement to combine the feature map subgroups as new inputs. This improves the network’s ability to extract features²⁸. In the improvement of the backbone network, the framework of depthwise separable convolution and channel rearrangement is denoted in (Fig. 2).

Fig. 2

Depthwise separable convolution and channel rearrangement.

The twin network TTA cannot effectively preserve semantic and spatial information, resulting in a decrease in the ability to track targets. Therefore, this study proposes an HFF method based on Transformer structure. It comprehensively and deeply extracts the features of tennis balls through the Multi-Head Attention Mechanism (MHAM) in this structure, capturing the global contextual relationship between tennis balls and the surrounding environment, thereby generating rich feature representations and improving the accuracy of tennis ball recognition. In twin networks, this method uses MHAM and Multi-Cross Attention Mechanism (MCAM) to accurately match the tennis features in the template and search area, reducing false positives and false negatives²⁹. The calculation for obtaining the comprehensive feature vector of this method is shown in Eq. (4).

$$\left\{ \begin{gathered} {M_c}=\left[ {{A_{{{M^{\prime}}_2}}}({{M^{\prime}}_2}),{A_{{{M^{\prime}}_1}}}({{M^{\prime}}_2}),{A_{{{M^{\prime}}_3}}},({{M^{\prime}}_2}))} \right] \hfill \\ {{M^{\prime}}_c}=Linear\left[ {{M_c}} \right] \hfill \\ \end{gathered} \right.$$

(4)

.

In Eq. (4), ${M_c}$ denotes the comprehensive feature vector. ${M^{\prime}_2}$ is the target feature. ${M^{\prime}_1}$ and ${M^{\prime}_3}$ are source features. A is a MHAM operation. ${M^{\prime}_c}$ is the eigenvector after linear transformation. The final output feature calculation is shown in Eq. (5).

$$\left\{ \begin{gathered} {M_o}=LN({{M^{\prime}}_c}+{{M^{\prime}}_2}) \hfill \\ {X_o}=LN({M_o}+MLP({M_o})) \hfill \\ \end{gathered} \right.$$

(5)

In Eq. (5), ${M_o}$ denotes the residual connection between the linearly transformed feature vector and the target feature. ${X_o}$ is the final output feature. $LN$ denotes layer normalization. $MLP$ mans a multi-layer perceptron. The framework diagram of HFF method based on Transformer structure is shown in Fig. 3.

Fig. 3

Framework diagram of HFF method based on Transformer structure.

During the training of the twin network TTA model, the contribution of its loss function is easily masked due to the small number of positive samples. Therefore, to balance the number of positive and negative samples, this study uses the binary cross entropy method to calculate the loss function, and its expression is shown in Eq. (6)³⁰.

$$C(p,z)=\left\{ \begin{gathered} - \log (p)\quad \quad \,z=1 \hfill \\ - \log (1 - p)\quad others \hfill \\ \end{gathered} \right.$$

(6)

.

In Eq. (6), $C(p,z)$ means the value of the cross entropy loss function. p means the probability of prediction. z is the actual label. To reduce the classification loss of negative samples, this study adds hyperparameters to the loss function, as shown in Eq. (7).

$$FL({p_t})= - \alpha {(1 - {p_t})^\beta }\log ({p_t})$$

(7)

In Eq. (7), $FL({p_t})$ is the value of the entropy loss function. $\alpha$ means a hyperparameter. $\beta$ is the modulation factor. The calculation of the bounding box regression loss for this TTA is shown in Eq. (8).

$${L_r}=\left\{ \begin{gathered} \frac{1}{2}{(t - t^{\prime})^2}\quad \left| {t - t^{\prime}} \right|<1 \hfill \\ \left| {t - t^{\prime}} \right| - \frac{1}{2}\quad \,\left| {t - t^{\prime}} \right| \geqslant 1 \hfill \\ \end{gathered} \right.$$

(8)

.

In Eq. (8), ${L_r}$ is the bounding box regression loss. t and $t^{\prime}$ are the positional offsets between the predicted box, the real box, and the corresponding anchor box. To further evaluate the robustness of the proposed TNOTO algorithm, supplementary experiments are conducted under complex conditions such as target occlusion. This study selects video sequences containing target occlusion from different datasets to test the tracking accuracy and robustness of the algorithm under occlusion conditions. How the HAM module reduces interference through local feature extraction and spatial gating under occlusion conditions is analyzed. To test the algorithm’s ability to re-capture targets, dynamic occlusion experiments are conducted to simulate dynamic occlusion scenes and verify the target capture performance. In addition, the study tests the algorithm’s tracking performance on multiple occluded targets in complex scenes to verify the HFF module’s ability to maintain target semantic information through global contextual relationships.

PPA based on improved RRT algorithm and APF-ACO algorithm

After identifying and locating tennis balls through the twin network TTA, this study needs to further plan the path of ITPR to find the optimal path. However, the existing robot PPA exhibits slow convergence speed and is vulnerable to getting stuck in local optima problems³¹. Therefore, the paper designs an ITPR algorithm built on improved RRT and APF-ACO algorithms. This method first utilizes an improved RRT to improve the search efficiency of tennis balls and then adopts a sector-constrained sampling method to enhance the pathfinding purposefulness of intelligent robots. Finally, it uses the APF-ACO to find the best path for the intelligent robot to pick up tennis balls while avoiding obstacles.

In the process of PP, traditional RRT algorithms are computationally complex, inefficient, and prone to redundancy. Therefore, this study improves the algorithm and proposes a Bi-RRT algorithm to enhance the guidance of intelligent robots from start to finish³². The generated tree and generated path effects of this algorithm are shown in (Fig. 4).

Fig. 4

The generation tree and generation path effects of the Bi-RRT.

To further improve the efficiency of Bi-RRT, this study adopts sector-constrained sampling method to reduce computation time. This method limits the sampling area within an ellipse, shortens the search time of the algorithm, and enhances the pertinence of path finding. Then, it uses the greedy pruning optimization method to smooth the path and improve the stability of the intelligent robot’s behavior, as shown in Eq. (9)³³.

$$R(\alpha )=\sum\limits_{{i=0}}^{n} {{R_i}{W_{i,k}}(\alpha )}$$

(9)

.

In Eq. (9), $R(\alpha )$ is the point of the spline curve at parameter $\alpha$. ${R_i}$ is the control point. k is the order of the spline. ${W_{i,k}}$ is an k-order spline basis function. $\alpha$ is the parameter. The schematic diagram of the constraint sampling method and greedy pruning optimization is shown in Fig. 5.

Fig. 5

Sector constrained sampling method and greedy pruning optimization.

In Fig. 5, the starting point of the robot is connected to the target point through a fan-shaped area. This method limits the sampling space to reduce the search time of the path algorithm. The size of sectors in the sector area constraint method is determined by a combination of initial settings and dynamic adjustments based on real-time data. The initial radius and angle of the sector are set to ensure effective coverage and computational efficiency³⁴. The radius is usually set to a distance that allows the robot to reach potential target points within a reasonable number of steps, while the angle is chosen to balance the coverage of the search space and computational load. The size of the sector will be dynamically adjusted based on the distance to the target point and the presence of obstacles. In areas with fewer obstacles, wider angles can be used to explore more potential paths, while in cluttered areas, narrower angles may be more suitable for avoiding obstacles and finding clear paths³⁵. In the greedy pruning optimization method, intelligent robots start from the starting point and connect the nodes of the path. When there are obstacles, they perform a greedy detection on the previously connected redundant nodes. The PP of ITPR has poor real-time performance and may encounter local optimal solution problems. Therefore, this study proposes an APF-ACO aimed at improving the obstacle avoidance ability and PP accuracy of robots. The heuristic information function expressions of the traditional ACO algorithm and APF-ACO algorithm are shown in Eq. (10).

$$\left\{ \begin{gathered} {E_{ij}}(t)=1/{D_{(i,j)}} \hfill \\ {E_f}(t)={\varepsilon ^{f\cos \theta }} \hfill \\ \end{gathered} \right.$$

(10)

.

In Eq. (10), ${E_{ij}}(t)$ is the heuristic information function value for positions i to j at time t. ${D_{(i,j)}}$ is the heuristic information function value considering the influence of the potential field. $f$ represents the resultant force of the potential field. $\varepsilon$ is a coefficient greater than 1. $\theta$ means the angle between the current tennis target point’s resultant force and the next tennis target point. The heuristic information coefficient that affects the APF is shown in Eq. (11).

$$e=\frac{{{U_{\hbox{max} }} - {U_m}}}{{{U_{\hbox{max} }}}}$$

(11)

In Eq. (11), e is the coefficient that affects the potential field heuristic information. ${U_{\hbox{max} }}$ means the max amount of iterations. ${U_m}$ means the current iteration count. From the above equation, the formula for the heuristic information function of the ACO with the addition of an APF ${E^{\prime}_{ij}}(t)$ is shown in Eq. (12)³⁶.

$${E^{\prime}_{ij}}(t)={E_{ij}}(t) \cdot {E_{if}}(t) \cdot e=\frac{{{\varepsilon ^{f\cos \theta }}}}{{{D_{(i,j)}}}} \cdot \frac{{{U_{{\text{max}}}} - {U_m}}}{{{U_{{\text{max}}}}}}$$

(12)

During the training process, as the amount of iterations grows, the size of the ant colony gradually decreases, as shown in Eq. (13).

$$G(t)=\left\{ \begin{gathered} G\frac{{{U_{\hbox{max} }} - {U_m}}}{{{U_{\hbox{max} }}}}\quad \;\;\,G(t)>{G_{\hbox{min} }} \hfill \\ {G_{\hbox{min} }}\quad \quad \quad \quad \quad others \hfill \\ \end{gathered} \right.$$

(13)

.

In Eq. (13), $G(t)$ is the initial ant colony size. ${G_{\hbox{min} }}$ is the smallest scale. In the actual PP of intelligent robots, the lack of information concentration can prevent the robot from finding the optimal path, reducing efficiency. Therefore, this study increases the concentration of pheromones in the diameter path between the starting and target points to improve the efficiency of finding the optimal path. The allocation formula of this method is denoted in Eq. (14)³⁷.

$${\lambda _i}=\left\{ \begin{gathered} B\quad i \in F \hfill \\ 1\quad \;\,others \hfill \\ \end{gathered} \right.$$

(14)

.

In Eq. (14), ${\lambda _i}$ denotes the constructed pheromone concentration matrix. B denotes a constant greater than 1. F is the set of points between the starting point and the destination of the path. The management of pheromone concentration is crucial in the APF-ACO. As the number of iterations rises, the positive feedback mechanism can lead to a differentiation in pheromone concentration, causing the algorithm to fall into local optima. To address this issue, this study sets minimum and maximum limits on pheromone concentration to enhance the accuracy of the algorithm’s global optimal solution³⁸. The threshold adjustment of pheromone concentration in this method is shown in Eq. (15).

$${\lambda _{ab}}(t+n)=\left\{ \begin{gathered} {\lambda _{\hbox{min} }}\quad \quad {\lambda _{ab}}(t+n)<{\lambda _{\hbox{min} }} \hfill \\ {\lambda _{ab}}(t)\quad \;\;{\lambda _{ab}}(t+n)<{\lambda _{\hbox{max} }} \hfill \\ {\lambda _{\hbox{max} }}\quad \quad {\lambda _{ab}}(t+n)>{\lambda _{\hbox{max} }} \hfill \\ \end{gathered} \right.$$

(15)

In Eq. (15), ${\lambda _{ij}}(t+n)$ is the concentration of pheromones from the starting point a to the target point b at time $t+n$. ${\lambda _{\hbox{max} }}$ and ${\lambda _{\hbox{min} }}$ denote the max and mini thresholds for pheromone concentration. The APF algorithm and local optimal problem are shown in Fig. 6.

Fig. 6

APF algorithm and local optimal problem.

The implementation steps of ITPR-PPA: Step 1 is to model using the grid method and initialize the parameters. Step 2 is to build a comfort pheromone concentration matrix. When the maximum amount of iterations is reached, the optimal path is output. Otherwise, the new state is transferred to the next point and updated using the APF method. When the maximum amount of ants is reached, the pheromone concentration is updated to ensure that it does not exceed the threshold and to determine whether the maximum amount of iterations has been reached. Otherwise, the above steps are repeated.

The structure of an ITPR mainly includes mechanical structure, sensing and control system, and PP integration. Among them, the body of the robot is designed to be lightweight and sturdy, capable of withstanding the dynamic environment of a tennis court. Attached to the mobile base is a robotic arm specifically designed for picking up tennis balls. The arm is equipped with sensors for detecting the presence and position of tennis balls, ensuring accurate and efficient ball retrieval. The sensing and control system includes cameras for visual inspection and distance sensors for obstacle avoidance. The camera is used to capture images of the tennis court, which are then processed by TTAs to locate the tennis ball. The control system integrates TTA with PPA, enabling the robot to navigate to the tennis ball position and perform ball picking actions.

Results

This section first validates the effectiveness of the TNOTO algorithm, and then conducts simulation experiments on ITPR-PPA to explore the application effect of the algorithm.

Performance verification of TNOTO algorithm

To prove the effectiveness of the TNOTO, this study conducts experiments using the TrackingNet and GOT-10k datasets as training sets and the LaSOT and OBT100 datasets as test sets. The experiment uses stochastic gradient descent to train the proposed algorithm. The selection of key parameters, including learning rate, decay rate, and iteration count, is based on the system’s adjustment process. The learning rate determines the step size of each iteration in the process of minimizing the loss function and is initially set to 0.001 after preliminary grid search. The decay rate is used to gradually reduce the learning rate, allowing for more detailed adjustments in the later stages of training, set to 0.0001. The number of iterations is determined through a series of experiments, and it is observed that increasing the number of iterations beyond 100 does not significantly improve performance, while greatly increasing computational costs. Therefore, 100 iterations are chosen as a compromise between performance and efficiency. Table 1 shows the relevant experimental parameters and environment.

Table 1 Experimental environment and parameter settings.

To verify the rationality of parameter selection, a series of experiments are conducted to evaluate the impact of different parameter settings on algorithm performance. These experiments involve varying the learning rate from 0.0001 to 0.01, the decay rate from 0.00001 to 0.001, and the number of iterations from 50 to 200. The results indicate that the selected parameters have indeed achieved optimal performance, as shown in Table 2. In Table 2, the combination of a learning rate of 0.001, an attenuation rate of 0.0001, and 100 iterations produces the highest mean Average Precision (mAP) and precision while maintaining a high Frames Per Second (FPS), verifying the rationality of the selected parameters.

Table 2 Impact of different parameter settings on algorithm performance.

This study first compares the training losses of different TTAs in TrackingNet and GOT-10k. TTA includes Siamese Fully-Convolutional (SiamFC), Multiple-Object Tracking with Transformer (MORT), Siamese Region Proposal Network (SiamRPN), Multi-Object Tracking with Memory (MeMOT), and Fast and Robust Generic Multiple-Object Tracking (FRoG-MOT), as shown in Fig. 7. In the TrackingNet dataset of Fig. 7 (a), when the iteration period is 10, the loss values of SiamFC, MORT, and SiamRPN are 0.692, 0.583, and 0.496, while the loss values of MeMOT, FRoG-MOT, and the proposed algorithm are 0.398, 0.197, and 0.152. When the iteration cycle reaches 40, the loss values of the five comparison algorithms decrease to 0.327, 0.305, 0.236, 0.201, and 0.167, while the loss value of the research algorithm is 0.112. In the GOT-10k dataset of Fig. 7 (b), when the iteration cycle is 10, the loss values of the five compared algorithms are 0.712, 0.594, 0.513, 0.408, and 0.185. The loss value of the proposed algorithm is 0.008 and tends to stabilize. This indicates that as the iteration period increases, the loss of TTA decreases, and the performance of the research algorithm is better. This indicates that the proposed algorithm can converge more effectively during the training process, avoiding getting stuck in local optima and achieving better global performance.

Fig. 7

Training loss for different TTAs in TrackingNet and GOT-10k datasets.

To verify the feasibility of TTA, this study analyzes the training accuracy of different algorithms on two different training datasets, as denoted in (Fig. 8). In the TrackingNet dataset of Fig. 8a, with an iteration period of 10, the accuracies of SiamFC and MORT algorithms are 0.306 and 0.345, SiamRPN and MeMOT are 0.398 and 0.462, and FRoG-MOT and the proposed algorithm are 0.515 and 0.687. When the iteration cycle increases to 50, the accuracy of the five algorithms is 0.725, 0.784, 0.806, 0.872, and 0.916, and the raised algorithm is 0.974. Compared with other algorithms, the accuracy of the research algorithm has enhanced by 25.56, 19.51, 18.25, 10.47, and 5.95%. In Fig. 8b, it is found that under the GOT-10k dataset, when the iteration period is increased to 50, the accuracy of the above algorithm is 0.782, 0.815, 0.837, 0.896, and 0.928. The accuracy of the research algorithm is 0.981, which increases by 20.09, 16.92, 14.68, 8.66, and 5.40% compared to other algorithms. This indicates that the proposed TTA has high training accuracy. This high-accuracy training result indicates that the proposed algorithm can better learn the features of the target and avoid getting stuck in local optima in complex scenes, thereby improving the tracking accuracy of the target.

Fig. 8

Training precision for different TTAs in TrackingNet and GOT-10k datasets.

This study tests the raised algorithm on the LaSOT and OBT100 datasets and compares it with the robustness values and Expected Average Overlap (EAO) of state-of-the-art TTA. The purpose is to explore the stability of the algorithm and measure the degree of overlap between the target position of the proposed algorithm and the true target position. In Fig. 9a, under the LaSOT dataset, the robustness values of SiamFC, MORT, SiamRPN, MeMOT, FRoG-MOT, and the research algorithm are 0.673, 0.592, 0.518, 0.496, 0.447, and 0.295. In the OBT dataset, its robustness values are 0.636, 0.592, 0.508, 0.479, 0.416, and 0.274. Compared with SiamFC and FRoG-MOT, the robustness values of the research algorithm decrease by 56.17% and 34.00% on the LaSOT dataset, and 56.92% and 34.13% on the OBT dataset. In Fig. 9b, the EAOs of the five comparative algorithms under LaSOT are 0.183, 0.210, 0.281, 0.328, and 0.379, while they are 0.215, 0.226, 0.317, 0.349, and 0.396 on the OBT100 dataset. The proposed algorithm has EAOs of 0.405 and 0.437 on two datasets, indicating superior robustness and high tracking accuracy. This robustness and high tracking accuracy indicate that the proposed algorithm can effectively avoid the influence of local optimal solutions in complex scenarios, thereby better adapting to the dynamic changes of the target and improving the stability and accuracy of tracking.

Fig. 9

Robustness values and EAO of various algorithms in two test datasets.

The fast convergence and low stable loss value indicate that the TNOTO algorithm can effectively avoid getting stuck in local optima during the training process. This is attributed to the HAM and HFF strategies, which enable the algorithm to efficiently capture local and global features. By focusing on the relevant parts of the feature map and reducing noise, HAM helps the algorithm avoid being misled by irrelevant or occluded regions. The HFF strategy further enhances the algorithm’s ability to effectively navigate the search space, thereby achieving better global performance. High precision indicates that the TNOTO algorithm can learn target features more effectively and avoid local optima. The HAM combines GCT and SGE to enhance the algorithm’s ability to capture local features and reduce the impact of occlusion. The HFF strategy based on Transformer structure further improves the algorithm’s global understanding ability of the target and its environment, thereby achieving more accurate tracking. By capturing local and global features, this algorithm can better distinguish targets from similar objects and background noise, thereby improving tracking accuracy and stability. The robustness and high tracking accuracy indicate that the TNOTO algorithm can effectively avoid the influence of locally optimal solutions in complex scenes. The HAM and HFF strategies enable the algorithm to maintain high accuracy even when the target is partially or completely occluded. By capturing local and global features, the TNOTO algorithm can better adapt to the dynamic changes in the appearance and position of the target, thereby improving the stability and accuracy of tracking.

To prove the effect of the TTA, this study compares the Frames Per Second (FPS) of different algorithms on two test datasets. In Fig. 10a, under the LaSOT dataset, the average FPS of SiamFC, MORT, and SiamRPN algorithms are 36 frames/s, 52 frames/s, and 73 frames/s, while the average FPS of MeMOT, FRoG-MOT, and the proposed algorithm are 98 frames/s, 104 frames/s, and 158 frames/s. In contrasted with the other five algorithms, the FPS of the research algorithm increases by 77.22, 67.09, 53.80, 37.97, and 34.18%. In Fig. 10b, the average FPS of the five algorithms is 39 frames/s, 62 frames/s, 79 frames/s, 96 frames/s, and 125 frames/s. The average FPS of the research algorithm is 176 frames/s, which is 77.84, 64.77, 55.11, 45.45, and 28.98% higher than other algorithms. This indicates that the proposed algorithm has strong real-time performance and can respond faster to the dynamic changes of the target.

Fig. 10

FPS comparison of various algorithms in two test datasets.

To further evaluate the generalization ability of the proposed algorithm, experiments are conducted on two new datasets: the VOT dataset and the UAV123 dataset. The VOT dataset contains various complex scenarios, while the UAV123 dataset contains significant changes in target morphology, providing a more comprehensive evaluation of algorithm performance. The results are shown in (Table 3). In Table 3, the robustness, EAO, and FPS of the proposed method in the VOT dataset are all excellent, with values of 0.387, 0.459, and 165.8, respectively. In the UAV123 dataset, the training loss and training accuracy of the proposed method are superior to other methods, with values of 0.137 and 0.925, respectively. The results show that the proposed method has excellent accuracy, stability, and real-time performance in target tracking tasks, and has strong generalization ability.

Table 3 Comparison of experimental results between VOT dataset and UAV123 dataset.

To assess the reliability of the proposed algorithm, this study organizes ablation experiments on the LaSOT, OBT100, VOT, and UAV123 datasets, and the results are shown in (Table 4). The parameter sizes of the proposed algorithm in the LaSOT and OBT100 datasets in Table 4 are 3.55 and 3.52, with mAPs of 89.67 and 92.26. This algorithm can accurately locate and classify targets. FPS values of 158 and 176 indicate that it has superior response speed. In the VOT dataset and UAV123 dataset, the parameter sizes of the proposed algorithm are 3.58 and 3.54, respectively, with average precision of 88.79% and 90.15%, respectively. The research algorithm not only reduces computational complexity but also has better real-time performance and can better adapt to dynamic scenes.

Table 4 Ablation results under two test datasets.

ITPR path planning simulation results

This study conducts simulation experiments on ITPR-PPA using the MATLAB platform. The size of the ant colony is 25, the initial pheromone concentration is 1, its volatility coefficient is 0.3, and the information elicitation and expected elicitation factors are 1 and 5. The maximum range of influence of obstacles is 5, and the gain factors of attraction and repulsion are set to 10 and 15.

To prove the superiority of the designed PPA, a comparative analysis is carried out on the PP effects of different algorithms. Algorithms include Bi-RRT, APF-ACO, and Bi-RRT + APF-ACO algorithm. In Fig. 11, the PP length using Bi-RRT is significantly higher than that of APF-ACO and the research algorithm. The Bi-RRT has a total of 23 turns in PP, while APF-ACO and the proposed PPA have 10 and 3 turns. The Bi-RRT + APF-ACO algorithm has many inflection points and unnecessary turns. The research algorithm can reduce the turning operation of robots and enhance the efficiency of intelligent robots in picking up tennis balls.

Fig. 11

Path planning effect of ITBPR with different algorithms.

This study compares the smoothness of paths and the number of turns using different algorithms. In Fig. 12a, the average smoothness values of RRT and Bi-RRT are 0.276 and 0.432, while the average smoothness values of APF-ACO and the research algorithm are 0.710 and 0.868. Compared with RRT, Bi-RRT, and APF-ACO, the smoothness values of the research algorithms increase by 68.20, 50.23, and 18.20%. In Fig. 12b, the average number of path turns for RRT, Bi-RRT, and APF-ACO algorithms is 34, 23, and 12, with a research algorithm of 4. This indicates that the PPA can reduce the number of path turns and improve the ball picking efficiency of intelligent robots.

Fig. 12

Comparison of smoothness and turn times of different path planning algorithms.

In the simulation, the paper further analyzes the PP length and planning time of the proposed intelligent robot PPA, and compares it with the algorithm before improvement. In Fig. 13a, the average PP lengths of RRT and Bi-RRT are 49.24 and 47.82 m, while APF-ACO and the proposed algorithm are 44.96 m and 42.07 m. Compared with other algorithms, the average PP length of the research algorithm decreases by 14.56, 12.02, and 6.43%. In Fig. 13b, the average PP time for RRT, Bi-RRT, and APF-ACO is 68.42, 65.58, and 61.47 s. The average PP time of the research algorithm is 56.12 s, which is 17.98, 14.43, and 8.70% shorter than RRT, Bi-RRT, and APF-ACO. Research algorithms can effectively reduce the length and time of robot PP.

Fig. 13

Path planning length and time of different path planning algorithms.

Discussion

The experimental results showed that the ITPR-PPA integrated with dual network TTA significantly improved both target tracking and PP performance. The proposed TNOTO that combines LA and HFF achieved high accuracy and robustness in object tracking. In the training datasets (TrackingNet and GOT-10k), this algorithm demonstrated excellent performance compared to existing methods such as SiamFC, MORT, SiamRPN, MeMOT, and FRoG MOT. The training accuracy reached 0.981, which was 5.40-25.56% higher than the comparison algorithms. In the test datasets (LaSOT and OBT100), the EAO values of the proposed algorithm were 0.405 and 0.437, respectively, significantly higher than the comparison algorithms. This indicated that the proposed method could maintain accurate tracking even in complex scenes with occlusion and dynamic target changes. In addition, the proposed PPA that combines improved Bi-RRT and APF-ACO showed significant improvements in path efficiency and smoothness. The average path length of the proposed method was 42.07 m, which was 14.56%, 12.02%, and 6.43% less than the RRT, Bi-RRT, and APF-ACO algorithms, respectively. The average PP time of the proposed method was 56.12 s, which was 17.98%, 14.43%, and 8.70% shorter than RRT, Bi-RRT, and APF-ACO algorithms, respectively. This indicated that the proposed PPA could effectively reduce computational complexity and improve the efficiency of PP.

Compared with the literature, the proposed method exhibits multiple key advantages in unseen environments. For example, reference¹⁵ proposes a UAV target tracking method based on MN-DDPG and transfer learning, which exhibits good performance in dynamic environments. However, it mainly focuses on UAV applications and does not address the specific challenges faced by tennis picking robots in complex and cluttered environments. In contrast, the proposed method is specifically designed for ITPR and advanced technologies such as HAM and HFF are introduced to enhance tracking accuracy and robustness in unseen scenarios. Reference²⁰ proposes a multi-step ACO algorithm for the PP of mobile robots. Although this method performs well in obstacle avoidance and PP, it does not integrate advanced TTA like the proposed TNOTO. The proposed method combines target tracking and PP, providing a more comprehensive solution for intelligent robots operating in dynamic environments, as demonstrated by experimental results in unseen datasets (VOT and UAV123).

Conclusion

In summary, the proposed method has demonstrated significant advancements in the field of ITPR. The innovation of this study includes the combination of HAM, HFF, Bi-RRT, and APF-ACO, effectively addressing the challenges of target tracking and PP in complex environments. The experimental results validate the superior performance of the proposed method in terms of tracking accuracy, real-time performance, path length, planning time, and path smoothness. The proposed methods have the following benefits for related fields: (1) By integrating LA and HFF into the TNOTO algorithm, the accuracy of target tracking in complex environments has been significantly improved. This progress can be applied to various robot applications that require precise target tracking, such as monitoring, search and rescue, and autonomous navigation. (2) By combining the improved Bi-RRT and APF-ACO algorithms, the efficiency and smoothness of PP have been significantly improved. This method reduces computational complexity and improves the ability of robots to avoid obstacles in dynamic and complex environments, which is very beneficial for autonomous robots operating in dynamic and complex environments. This study provides an effective solution for target tracking and PP of ITPRs in complex environments. The proposed method not only improves the accuracy and efficiency of target tracking but also enhances the navigation and obstacle avoidance capabilities of robots in dynamic and complex environments.

The limitation of this study lies in the limited consideration of multi robot collaboration scenarios. In practical applications, multiple robots working together may face issues such as communication delay, task allocation, and path coordination. Therefore, in future research, further exploration of PP and task allocation strategies for multi-robot collaboration is needed to achieve efficient collaboration among multiple ITPRs through distributed computing and communication technologies. The innovative methods that can be explored in future research include: (1) Distributed reinforcement learning algorithm, which aims to achieve real-time decision-making and adaptive PP in dynamic environments. (2) Swarm intelligence algorithm (such as PSO or ACO), which is utilized to optimize multi-robot coordination and task allocation. (3) Advanced sensing technology. By introducing advanced sensing technologies such as LiDAR and high-resolution cameras, the accuracy of environmental perception and target detection can be improved.