Scheme evaluation method of coal gangue sorting robot system with time-varying multi-scenario based on deep learning

Abstract

The rapid advancement of artificial intelligence and robot technology has spurred the proposal and innovation of a coal gangue sorting robot system (CGSRS) paradigm. The time-varying raw coal flow (TVRCF) with multi-scene and full working conditions affects the gangue queue. Configuring the CGSRS scheme correctly is combative. The field environment puts forward higher requirements for the time complexity of the CGSRS multi-task allocation strategy. Therefore, this paper proposes a scheme evaluation method of the CGSRS with time-varying multi-scenario based on deep learning. Firstly, the gangue queue data set of multi-scene and full-condition TVRCF was obtained according to the belt speed, the maximum coal flow, and the uncorrelated nonlinear changes of coal flow and gangue content. The CGSRS scheme is established based on robot number and rule combination, and the multi-task allocation strategy is adjusted to generate the labels of the gangue queue. Then, the RGB sample set is established based on the labels of the gangue queue. The CGSRS scheme evaluation model is trained based on DenseNet. Finally, the CGSRS scheme evaluation method was designed to realize the prediction of a random gangue queue. In this paper, the CGSRS scheme evaluation model, the stability of the solution, and the comparison of methods are carried out. Experimental results show that the solution of the CGSRS scheme evaluation model is accurate and stable. The time complexity is significantly reduced and very stable. The CGSRS scheme evaluation method is applied to the CGSRS multi-task allocation problem, and the stability of the solution is not affected by the data. It is significantly better than the multi-task allocation strategy. The proposed method is the first attempt to apply deep learning to a multi-task allocation problem in CGSRS.

Introduction

The intelligent development of coal mines has injected new vitality into improving coal separation technology¹. Robotics have been extensively employed in the coal preparation process. The combination of time-varying raw coal flow (TVRCF), belt conveyor, and multi-robot system describes a coal gangue sorting robot system (CGSRS) paradigm². The coal gangue sorting robot consists of an identification and positioning module, a control module, and a multi-manipulator module³. The key technologies encompass coal gangue identification technology, machine vision technology⁴, robot technology⁵, and artificial intelligence technology⁶, intending to sort coal or gangue efficiently. In the coal preparation process, the scenes of the coal preparation plant and the working conditions of TVRCF are diverse. Therefore, the research on coal gangue identification and positioning technology, dynamic target tracking technology and multi-task allocation strategy of CGSRS have been the focus of scholars in recent years.

Identification and positioning technology and dynamic target tracking technology of coal gangue have developed rapidly. However, the multi-task allocation strategy can not efficiently pick and place the identified and positioned gangue queue in multi-scene and all working conditions. This also makes dynamic target-tracking technology challenging to perform. Regarding coal gangue identification and positioning, scholars are committed to improving the accuracy and efficiency of coal gangue identification, reducing positioning errors, and obtaining the volume information of gangue in multiple scenarios. Zhou et al. proposed the identification of coal and gangue based on the R-value method for dual-energy X-ray of Geant4 simulation⁷. The identification accuracy of this method for coal and gangue with different shapes and thicknesses reaches 95%, but the mechanism of influence of R-value on thickness, shape and composition needs to be studied. Zhang et al. studied the effect of liquid intervention on coal and gangue identification⁸. The experimental results show that the temperature difference between coal gangue can reach 1.5℃ after only 3 s of liquid intervention, which is conducive to the infrared detection efficiency of coal and gangue. Yan et al. proposed a coal-gangue identification method based on multi-spectral imaging technology and target detection⁹. Huang et al. proposed a coal gangue identification method based on volume detection and dynamic weighing, considering gangue queuing and binocular vision¹⁰. The YOLO series algorithms have been extensively applied to address coal gangue detection challenges. Notably, enhancing tiny YOLO v3 and incorporating a resnet18 backbone feature network have significantly bolstered detection efficiency and recognition accuracy^11,12. In terms of dynamic target tracking technology, combined with global planning and local vision, the coal gangue sorting robot can quickly pursue and follow up the dynamic gangue target and reach the purpose of quasi-grasping¹³. The impact load arises from the velocity differential between the manipulator’s end and the target during grasping, potentially resulting in grasp failure and manipulator damage. Wang et al. propose a dynamic gangue trajectory planning method for the manipulator synchronous tracking under multi-constraint conditions¹⁴. Finally, the collaborative control of multiple manipulators will be used to obtain identification and positioning information of coal and gangue, generating trajectory planning information for the manipulators.

In summary, coal gangue identification and positioning technology and dynamic target tracking technology have been realized as applications in the coal preparation process, whether multi-scene or full working conditions. In terms of the multi-task allocation strategy, Cao et al. Proposed a multi-manipulator sorting strategy with a feedback mechanism to ensure the sustainability of gangue picking and placing by the manipulator when the position coordinate of manipulators in the fixed workspace is random¹⁵. When the position coordinate of manipulators in the fixed workspace is determined, Ma et al. proposed a multi-arm global cooperative coal gangue sorting method based on an improved Hungarian algorithm, which considers the influence of different belt speeds but does not discuss the relationship between robot number and sorting efficiency¹⁶. In addition, Wang et al. proposed a cooperative strategy of multi-arm coal gangue sorting robot based on immune dynamic workspace, which improved sorting efficiency and solved the deadlock problem in the collaborative process of multi-manipulator¹⁷. Wu et al. proposed a multi-task allocation framework of a coal gangue sorting robot system for the time-varying raw coal flow and proposed task decomposition, task distribution, and executable collaborative strategy². The coal gangue sorting robot system is applied to the coal preparation process. Among them, the belt speed of multiple scenes is different. The maximum coal flow and TVRCF of full-working conditions are varied. In any scene and working condition, if CGSRS is configured with the best robot number and cooperative strategy, the time complexity of the existing multi-task allocation strategy is enormous. Even in a scene with high belt speed, the time complexity of an online multi-task allocation strategy to select the best cooperative strategy is challenging in meeting real-time requirements. However, there is still a lack of research on the coal gangue sorting robot system for multi-scene and all-condition time-varying raw coal flow.

With the development of artificial intelligence, neural networks are widely used in various fields^18,19. In a multi-robot system. Bhadra et al. developed an end-to-end 3D CNN model to predict soybean yield using multi-temporal RGB images acquired by drones²⁰. This is the earliest study of neural networks for soybean yield prediction. Experimental results show that DenseNet optimizes the accuracy of yield prediction and model complexity. Cao et al. discussed an end-to-end detection model based on DenseNet for real-time and high-precision robot grasping detection of fragile fruits²¹. This method considers the diversity of fruit shapes and sizes and designs four end-to-end detection models based on three convolutional neural network architectures to obtain the fruit grasp position. In the experimental results, the accuracy of the four models ranged from 83.86 to 93.64%. All the models were capable of rapid real-time detection. In the CGSRS paradigm, a fundamental challenge faced by the multi-manipulator module is optimizing the robot number and cooperation strategy for the gangue queue with multiple scenes and full working conditions²². On this basis, the combination of scheduling rules is selected online for the gangue queue of determined scenes and conditions, and this multi-task allocation strategy is robust to TVRCF²³. Therefore, neural networks are applied to the CGSRS paradigm. DenseNet establishes the CGSRS scheme evaluation model of multi-task allocation strategy to reduce the time complexity. It is beneficial for optimizing robot number and cooperative strategy and selecting online scheduling rule combinations.

Aiming at the gangue queue with multi-scene and full working condition, this study constructs the RGB image sample set and uses DenseNet to establish the CGSRS scheme evaluation model based on a multi-task allocation strategy. The CGSRS scheme describes robot number and cooperative strategy. The task completion success rate, sorting rate and robot utilization rate are used as evaluation indexes. This method calculates the approximate solution of each CGSRS scheme evaluation model, selects the CGSRS scheme with the optimal solution, and calculates the deterministic solution of the gangue queue through the multi-task allocation strategy. The neural network realizes the end-to-end solution of the multi-task allocation strategy, which effectively reduces the selection time of the CGSRS scheme and the optimization time of the online cooperative strategy.

Aiming at the gangue queue of multi-scene and full working condition in the coal preparation process, this paper proposed a scheme evaluation method for CGSRS with time-varying multi-scenario based on deep learning. Specific contributions are as follows.

According to the time, space and feature of the gangue queue, the RGB image sample set is constructed. DenseNet is used to establish the CGSRS scheme evaluation model, and the optimal CGSRS scheme is efficiently selected for gangue queues in multi-scene and full working condition. This method has good accuracy, and the required time is much lower than the time complexity of using multi-task allocation strategy.

During the operation of CGSRS, the CGSRS scheme evaluation model is used to calculate and select the online cooperative strategy of multi-manipulator. The time of the optimization process is much lower than the time complexity of using a multi-task allocation strategy.

The subsequent sections of this article are organized as follows: Section “Problem description” delineates the problem, while Section “The CGSRS scheme evaluation method for all-condition TVRCF is constructed” expounds upon the proposed methodology. Section “Simulation experiments” presents simulations, results, and discussions. Section “Conclusion” provides conclusions.

Problem description

TVRCF in the coal preparation process

The dimensional information such as time, space and feature of gangue queue is affected by the coal preparation process of multi-scene and full working condition. Among them, multi-scene describes the belt conveyor speed and the maximum value of coal flow in the coal preparation plant, which affects the arrival rate of coal and gangue. The full working condition describes the coal flow and gangue content of fully mechanized mining and caving coal, the coal and gangue granularity of the crusher^24,25, and the coal and gangue position of the vibrating screen. Following the objective reality, the arrival rate of coal and gangue must not lead to the accumulation and blockage of belt conveyor. The granularity of coal and gangue can not be greater than the width of the belt conveyor.

Coal flow and gangue content of raw coal flow are uncorrelated and change nonlinearly with time, defined as TVRCF. In the coal preparation process, the coal flow of TVRCF varies within a specific range and is always smaller than the maximum value of coal flow in this scene. On this basis, the gangue content of TVRCF affects the quantity and granularity of gangue. According to the conservation of mass, the larger of gangue quantity, the smaller of gangue granularity. And vice versa. Wu et al. described the mapping between TVRCF and gangue queue by task decomposition². This contributes to the research work in this paper.

In summary, the CGSRS paradigm is deployed in a multi-scene, full-condition coal preparation process. The gangue queue of TVRCF processed by CGSRS is quite complex. This leads to the problem that the multi-task allocation result of CGSRS is not ideal, and the CGSRS scheme is not universal. The cost of replacing hand-picked gangue with the CGSRS paradigm is that the CGSRS scheme must conform to human adaptability.

CGSRS in the coal preparation process

In order to enhance the ability of CGSRS to pick and place gangue queue, it can be achieved by adjusting the gangue queue, increasing the manipulator number, and optimizing the cooperative strategy. Consider increasing the manipulator number. It can be expected that the scene of the hand-picking gangue does not have infinite space. Moreover, the layout of other devices also limits the manipulator number in CGSRS. Although the space of the transportation scene is abundant, the capacity of a coal mine requires that the belt conveyor speed is generally above 2m/s. The robot specification makes it difficult to pick and place gangue queue at this speed. Consider adjusting the gangue queue. As seen from “TVRCF in the coal preparation process”, in the coal preparation process with multi-scene and full-condition, the gangue queue is directly or indirectly affected by fully mechanized mining, top-caving, crusher, vibrating screen and belt conveyor. It is not easy. Consider the optimization of cooperative strategy. According to the scheduling rules executed by the manipulator, the combination of the scheduling rule with the best task allocation result is selected. In summary, the decision variables of the CGSRS scheme are manipulator number and scheduling rule combination.

According to the problem description, selecting the CGSRS scheme suitable for the gangue queue in the coal preparation process with multi-scene and full working condition is challenging. This study considers dimensional information such as time, space and feature of gangue queue. Firstly, the RGB image dataset of the gangue queue is constructed according to the arrival time interval, position and granularity. Then, the multi-task allocation model is used to generate the label of the RGB image. Finally, DenseNet established the CGSRS scheme evaluation model. It is an end-to-end evaluation of a multi-task allocation problem. For the CGSRS scheme selection of gangue queue and the cooperative strategy optimization of CGSRS, the CGSRS scheme evaluation model can be used to obtain the near-optimal solution, and then the multi-task allocation strategy can be used to calculate the deterministic solution.

Mathematical modeling of the CGSRS scheme

Assumptions

Without loss of generality, the following assumptions are given in this paper:

• The variation of gangue content is known and will not be discussed in this paper.

• The granularity distribution of coal and gangue is known, and the range of granularity distribution is constrained in this paper.

• The conveyor uses a flat belt. The speed is adjustable. There is no slack, skid, or deviation.

• In the CGSRS scheme, the density value of gangue is known.

• In the CGSRS scheme, the robot’s initial position and placing position are determined, and the gangue coordinate affects the picking position.

• The gangue arrival rate follows a Poisson distribution, and the position coordinates are expressed as (x, y). The ratio of x interval value to belt speed defines the gangue arrival time interval. The coordinate value y in the vertical belt speed direction is uniformly distributed.

Evaluation indexes

Generally, the evaluation metrics for the multi-task allocation problem of CGSRS are sorting rate (SR), task completion success rate (TCSR), and robot utilization rate (RUR)². Among them, The SR is the ratio of the total mass of gangue picked up and placed by the manipulator to the total mass of gangue entering the CGSRS. The expression is as follows.

$${\varepsilon }_{\text{Q}}=\frac{\sum_{j}\sum_{k}{m}_{{\text{G}}_{k}}}{\sum_{i}{m}_{{\text{G}}_{i}}} , 0<{\varepsilon }_{\text{Q}}<1$$

(1)

where ε_Q is the symbol of the sorting rate. m_Gk is the mass of the k-th gangue picked up and placed by the j-th manipulator, in kg. m_Gi is the mass of gangue i-th in the gangue queue, in kg.

The ratio of the total amount of gangue picked up and placed by the manipulator to the total amount of gangue entering CGSRS is defined as the TCSR, which is calculated as follows:

$${\varepsilon }_{\text{N}}=\frac{\sum_{j}{n}_{j}}{N} 0<{\varepsilon }_{\text{N}}<1$$

(2)

where ε_N is the symbol of the task completion success rate. n_j is the statistical value of the amount of gangue picked up and placed by the j-th manipulator. N is the total amount of gangue in TVRCF.

The ratio of the working time of the manipulator to the operating time of the system is defined as the RUR. The expression is as follows:

$${\varepsilon }_{{\text{M}}_{j}}=\frac{\sum_{k}{t}_{j}^{k}}{{t}_{{\text{G}}_{N}}} 0<{\varepsilon }_{{\text{M}}_{j}}<1$$

(3)

where ε_Mj is the symbol of the robot utilization rate. t_j^k is the sorting time of the j-th manipulator picking up and placing the k-th gangue, unit s. t_GN is the arrival time of the last gangue, unit s.

This paper uses the accuracy (Acc) to describe the prediction accuracy of the CGSRS scheme evaluation model, including SR, TCSR, and RUR. The specific calculation process is as follows.

$$\text{Acc}=1-\frac{1}{n}\sum \frac{\left|{Value}_{\text{true}}-{Value}_{\text{pred}}\right|}{{Value}_{\text{true}}}$$

(4)

Value_true is the evaluation index predicted by the multi-task allocation strategy. Value_pred is the evaluation index predicted by the CGSRS scheme evaluation model.

The root mean square error (RMSE) describes the prediction stability of the CGSRS scheme evaluation model, including SR, TCSR, and RUR. The specific calculation process is as follows.

$$\text{RMSE}=\sqrt{\frac{1}{n}\sum {\left[\frac{\sum \left({Value}_{\text{true}}-{Value}_{\text{pred}}\right)}{n}-\left({Value}_{\text{true}}-{Value}_{\text{pred}}\right)\right]}^{2}}$$

(5)

Constraints

In the coal preparation process, the crusher processes the granularity of coal and gangue. This paper discusses coal and gangue ranging from 50 to 300 mm. The specific constraints are as follows:

$$50\le {s}_{{\text{G}}_{i}}\le 300$$

(6)

where, s_Gi is the granularity of the i-th gangue in the gangue queue, in mm.

In this paper, the coal flow in the time window is discussed, and the symbol is Q(τ), and the unit is kg/s. The length of the time window is set to one second. The sequence number of the time window is described as τ. The belt speed and maximum coal flow are known in a scene. The change of the coal flow in continuous time windows describes a working condition. The constraint of coal flow is as follows.

$$0< Q\left(\tau \right)\le {Q}_{\text{max}}$$

(7)

The gangue content changed with time. The symbol is μ(τ), and the constraint is as follows.

$$0\le \mu \left(\tau \right) \le 1$$

(8)

The CGSRS scheme evaluation method for all-condition TVRCF is constructed

The CGSRS evaluation method with time-varying multiple scenarios, based on deep learning, integrates the strengths of deep learning and a multi-task allocation model. As shown in Fig. 1, it is the construction flow chart of the CGSRS scheme evaluation method for multi-scenario and full-condition TVRCF. The RGB image generation process, CGSRS scheme evaluation model and CGSRS scheme evaluation method are described in turn.

Fig. 1

The construction flow chart of the CGSRS scheme evaluation method for multi-scene and full-condition TVRCF.

The detailed implementation of RGB image generation, CGSRS scheme evaluation model, and CGSRS scheme evaluation method is as follows.

• Step 1 An RGB image generation process to determine the scene and working condition. First, the TVRCF is mapped to the gangue queue according to a scene and working condition. Then, the granularity of the feature dimension, the time dimension’s arrival time interval, and the space dimension’s position coordinate were normalized and mapped to the Red, Green, and Blue channels, respectively. Finally, the three channels fuse an RGB image, which contains information on the gangue queue’s feature, time and space dimensions. The TVRCF of each working condition can obtain many gangue queues. Each gangue queue generates an RGB image. Therefore, the gangue queue of full-condition TVRCF generates the RGB image data set in this scene.

• Step 2 A CGSRS scheme evaluation model based on DenseNet in the determined scene. In general, the belt speed of the scene is stable. The essence of the CGSRS scheme is the difference in robot number and rule combination, which is the key to adjusting the multi-task allocation strategy. The evaluation indexes such as SR, TCSR, and RUR of the gangue queue are calculated through a multi-task allocation strategy. The evaluation index of encoding is the label of an RGB image, generating an RGB image sample. Finally, the multi-task allocation strategy of this CGSRS scheme needs to calculate the gangue queue data set of TVRCF in the full working condition to establish an RGB image sample set. The CGSRS scheme evaluation model is obtained by training the DenseNet network with the RGB image sample set.

• Step 3 CGSRS scheme evaluation method with time-varying multi-scenario based on deep learning. In the determined scene, the RGB image data set is generated according to the gangue queue data set of full-condition TVRCF. The existing CGSRS schemes are exhaustively enumerated according to the belt speed in this scene. The multi-task allocation strategy of step 2 calculates the evaluation index of the gangue queue, which is the label of RGB image, and will change with the CGSRS scheme. To summarize, a scene’s RGB image data sets are the same. However, different CGSRS schemes build different RGB image sample sets. The CGSRS scheme evaluation model obtained by training the DenseNet network is not unique and is not independent of each other. It is the key to implementing the CGSRS scheme evaluation method.

• Step 4 CGSRS scheme evaluation process. A scene is randomly selected, and all CGSRS scheme evaluation models conforming to this scene are selected. The gangue queue of full-condition TVRCF was collected to generate an RGB image data set. The selected CGSRS scheme evaluation model evaluated the SR, TCSR, and RUR of the RGB image data set. The CGSRS scheme of near-optimal solution is selected, and the multi-task allocation strategy calculates the determined solution. Because the CGSRS scheme evaluation model is end-to-end, the time complexity is much less than that of the multi-task allocation strategy. It allows us to do many evaluations quickly using the CGSRS scheme evaluation model. When selecting the near-optimal solution, the multi-task allocation strategy must only be calculated once to obtain the determined solution for the gangue queue. It is the key to achieving online optimization of the multi-task allocation strategy.

Improved multi-task allocation strategy

Mathematical description of multi-task allocation

This paper aims to discuss the application of deep learning to the problem of multi-task allocation in CGSRS. Some contents are improved to ensure the stability of the multi-task allocation process².

The Gangue queue is expressed as follows.

$${\text{G}}_{i}=\left\{{x}_{{\text{G}}_{i}},{y}_{{\text{G}}_{i}},{z}_{{\text{G}}_{i}},{t}_{{\text{G}}_{i}},{m}_{{\text{G}}_{i}}\right\} i=\text{1,2},\dots , n$$

(9)

where, i is the sequence number of gangue in gangue queue. x_Gi is the X-axis coordinated of the i-th gangue, unit is mm. y_Gi is the Y-axis coordinate of the i-th gangue, unit is mm. z_Gi is the Z-axis coordinate of the i-th gangue, unit is mm. z_Gi is set as 0 mm. m_Gi is the mass of i-th gangue, unit is kg. The gangue density is determined, so the mass is related to the granularity.

The robot is expressed as follows.

$${\text{M}}_{i}=\left\{{x}_{{\text{M}}_{j}},{y}_{{\text{M}}_{j}}, A, {v}_{\text{max}}\right\} j=\text{1,2},\dots , m$$

(10)

According to the state, moving ability and position coordinates of the robot, and the moving ability and position coordinates of the gangue, the possibility of picking and placing gangue by the robot is judged. The robot state is set to P_j. When the value is 1, the robot is idle. Then, the judgment result can be expressed as follows.

$$s_{j,i} = \left\{ {\begin{array}{*{20}l} 1 \hfill & {t_{i} \ge t_{j}^{i} \quad P_{j} = 1} \hfill \\ 0 \hfill & {t_{i} < t_{j}^{i} \quad P_{j} = 1} \hfill \\ 0 \hfill & {P_{j} = 0} \hfill \\ \end{array} } \right.$$

(11)

where, t_i is the waiting time for i-th gangue to leave the picking and placing area, according to the moving ability and position coordinates of gangue. According to the moving ability and position coordinates of the robot, tⁱ_j is the time for the robot to pick and place gangue.

The waiting time is calculated as follows.

$${\text{t}}_{i} = \frac{{l}_{\text{max}}^{j}-{x}_{{\text{G}}_{i}}}{{v}_{\text{b}}}$$

(12)

where, l^j_max is the boundary value of the picking and placing area, unit is mm. The v_b is the belt speed, unit is m/s. This variable also describes the moving ability of gangue.

The picking and placing time is calculated as follows.

$${t}_{j}^{i}={f}_{\text{tra}}\left(\left({\text{x}}_{{M}_{j}}\text{, }{\text{y}}_{{M}_{j}}\right),\left({\text{x}}_{{G}_{i}}\text{, }{\text{y}}_{{G}_{i}}\right), {\text{A}}\text{,}{\text{ v}}_{\text{max}}\right)$$

(13)

The iteration of the gangue queue in the picking and placing area can be regarded as a Markov process. Taking the picking and placing time of the robot or the arrival time interval of the gangue as the iteration time will cause instability in the solution process. Therefore, a fixed step size is chosen as the iteration time in this paper, described as T_s.

$$\left\{{{\varvec{E}}}_{s} | \, {{\varvec{E}}}_{s-1}\text{, }{T}_{s}\right\}$$

(14)

where s is the number of iterations. E_s is the environment state, which describes the picking and placing time, gangue mass, serial number, and the judgment result. It is described as follows.

$${{\varvec{E}}}_{s}=\left[\begin{array}{cc}\begin{array}{ccc}\left(\begin{array}{cc}\begin{array}{cc}{\text{t}}_{1}^{1}& {m}_{{\text{G}}_{1}}\end{array}& \begin{array}{cc}1& {s}_{{1}\text{,}{1}}\end{array}\end{array}\right)& \cdots & \left(\begin{array}{cc}\begin{array}{cc}{\text{t}}_{1}^{i}& {m}_{{\text{G}}_{i}}\end{array}& \begin{array}{cc}i& {s}_{1,i}\end{array}\end{array}\right)\\ \vdots & \ddots & \vdots \end{array}& \begin{array}{cc}\cdots & \left(\begin{array}{cc}\begin{array}{cc}{\text{t}}_{1}^{n}& {m}_{{\text{G}}_{n}}\end{array}& \begin{array}{cc}n& {s}_{1,n}\end{array}\end{array}\right)\\ \ddots & \vdots \end{array}\\ \begin{array}{ccc}\left(\begin{array}{cc}\begin{array}{cc}{\text{t}}_{j}^{1}& {m}_{{\text{G}}_{1}}\end{array}& \begin{array}{cc}1& {s}_{j,1}\end{array}\end{array}\right)& \cdots & \left(\begin{array}{cc}\begin{array}{cc}{\text{t}}_{j}^{i}& {m}_{{\text{G}}_{i}}\end{array}& \begin{array}{cc}i& {s}_{j,i}\end{array}\end{array}\right)\\ \vdots & \ddots & \vdots \\ \left(\begin{array}{cc}\begin{array}{cc}{\text{t}}_{m}^{1}& {m}_{{\text{G}}_{1}}\end{array}& \begin{array}{cc}1& {s}_{m,1}\end{array}\end{array}\right)& \cdots & \left(\begin{array}{cc}\begin{array}{cc}{\text{t}}_{m}^{i}& {m}_{{\text{G}}_{i}}\end{array}& \begin{array}{cc}i& {s}_{m,i}\end{array}\end{array}\right)\end{array}& \begin{array}{cc}\cdots & \left(\begin{array}{cc}\begin{array}{cc}{\text{t}}_{j}^{n}& {m}_{{\text{G}}_{n}}\end{array}& \begin{array}{cc}n& {s}_{j,n}\end{array}\end{array}\right)\\ \ddots & \vdots \\ \cdots & \left(\begin{array}{cc}\begin{array}{cc}{\text{t}}_{m}^{n}& {m}_{{\text{G}}_{n}}\end{array}& \begin{array}{cc}n& {s}_{m,n}\end{array}\end{array}\right)\end{array}\end{array}\right]$$

(15)

Scheduling rules

This paper discusses the application of three scheduling rules in the multi-task allocation strategy of CGSRS, namely first-in-first-out (FIFO)²⁶, shortest processing time (SPT)²², and mass first (MF)². Among them, FIFO prioritizes the gangue with the smallest sequence number. MF gives priority to the gangue with the largest mass. SPT gives priority to the gangue with the smallest picking and placing time. The details are described as follows.

$$\left\{\begin{array}{c}min\left\{i\right\} , FIFO\\ max\left\{{m}_{{\text{G}}_{i}}\right\}, MF\\ min\left\{{t}_{j}^{i}\right\} , SPT \end{array}\right.$$

(16)

CGSRS scheme evaluation method based on DenseNet

Sample set construction based on decomposition method

This paper uses the decomposition method of TVRCF to simulate the gangue queue data set with multi-scene and full working condition. It contains the position coordinates of the X-axis, Y-axis, and Z-axis of the gangue queue and the gangue mass. It is the basis for obtaining an RGB image data set.

The gangue mass is related to the granularity. Because the gangue density is determined, the symbol is ρ_G, set as 2100 kg/m³. The gangue shape is similar to a cube, and the compensation factor similar to the cube is set as α, a value of 0.85. The calculation process for gangue mass is as follows.

$${m}_{{\text{G}}_{k}}={10}^{-9}\cdot \alpha {\rho }_{\text{G}}{\left({s}_{{\text{G}}_{k}}\right)}^{3}$$

(17)

The specifications and parameters of the crusher affect the granularity distribution of gangue, and the specific constraints are as follows:

$${s}_{{\text{G}}_{k}}=\left\{\begin{array}{c}300\times \frac{{n}^{\left(v-2\right)/2}{e}^{-n/2}}{{2}^\frac{v}{2}\Gamma \left(v/2\right)} , {s}_{{\text{G}}_{k}}\ge 50 \\ 50, {s}_{{\text{G}}_{k}}<50\end{array}\right.$$

(18)

where n is set to 1. v is set to 130. Г is the gamma function.

The arrival time interval of gangue along the X-axis follows an exponential distribution with mean μ^-1. The calculation of x_Gk is as follows:

$$\left\{\begin{array}{c}\overline{{t }_{\tau }}=\frac{{T}_{\tau }}{{n}_{\tau }}, {n}_{\tau }>0\\ {t}_{{\text{G}}_{k}}= \left(\frac{1}{\mu }\right){e}^{\frac{-{n}_{\tau }}{\mu }}\cdot \overline{{t }_{\tau }}\\ {x}_{{\text{G}}_{k}}= {x}_{{\text{G}}_{k}}- {v}_{\text{b}}\cdot {t}_{{\text{G}}_{k}} k=\text{2,3},\dots ,{n}_{\tau }\\ {x}_{{\text{G}}_{k}}= -{v}_{\text{b}}\cdot {t}_{{\text{G}}_{k}} k=1\end{array}\right.$$

(19)

In a time window, according to the law of mass conservation, gangue content, coal flow, gangue quantity, and gangue mass have the following relationship.

$$Q\left(\tau \right)\cdot \mu \left(\tau \right)=\sum_{k\in {n}_{\tau }}{m}_{{\text{G}}_{k}} , k=\text{1,2},\dots ,{n}_{\tau }$$

(20)

The belt width is 800 mm, and the distance between the two sides is reserved to avoid the fall of large-grain gangue. The coordinates of gangue perpendicular to the direction of belt movement are uniformly distributed.

$${y}_{{G}_{k}}\sim {\varvec{U}}\left[150, 650\right]$$

(21)

The sample set construction is discussed based on the decomposition method of TVRCF. It includes TVRCF of multi-scene and full-working condition, Decomposition method of TVRCF, CGSRS scheme set, Scene consistency, and RGB sample set of full-condition TVRCF.

TVRCF of multi-scene and full working condition. The scene affects the belt speed. The working condition affects the maximum coal flow of TVRCF. To this end, a working condition is taken as an example. The function of coal flow is unchanged. The function of gangue content undergoes eight phase shifts in one cycle. The sign of phase shift is δ, and this fixed value is set to π/4.

Decomposition method of TVRCF. Select a scene where the belt speed is 0.5m/s. Select a working condition where the maximum coal flow is 100kg/s. The 7th phase shift is chosen, and the value of δ is 7π/4. At this point, the TVRCF is determined. The decomposition method generates the gangue queue, which contains the arrival time interval, position coordinates and mass of 800 gangues. Each TVRCF cycles the decomposition method 200 times, such that the generated 200 gangue queues are a gangue queue data subset. Finally, the gangue queue data subset obtains the RGB image data subset.

CGSRS scheme set, scene consistency, and RGB sample set of full-condition TVRCF. The robot scheme contains the robot number and the rule combination. We did simple coding. When the robot number is 1, the rule combines three kinds, which are (1, 1), (1, 2), (1, 3). Taking (1, 2) as an example, 1 describes the robot number, and 2 describes the scheduling rule of the robot. Among them, 1 describes the FIFO. 2 describes the MF. 3 describes the SPT. The multi-task allocation strategy is modified once every CGSRS scheme is determined. Note that, according to the scene consistency, the belt speed of the CGSRS scheme must be the same as that of the TVRCF. The full-condition TVRCF describes the seven maximum coal flow. There are eight phase shifts for each maximum coal flow. There are 200 gangue queues for each phase shift. So, the multi-task allocation strategy needs to calculate 56 gangue queue data subsets or the evaluation indexes of 11,200 gangue queues. Finally, the label of each RGB image sample is generated according to these evaluation indexes.

TVRCF with multi-scene and full working condition. According to the gangue content and maximum value of coal flow, the specific calculation process of phase translation is as follows.

$${Q}_{\text{G}}(\tau )=\mu \left(\tau \right)\cdot {Q}_{\text{max}}\cdot \text{sin}\left[\tau +\delta \right]+\text{C}$$

(22)

The decomposition method of TVRCF. The t_Gi, y_Gi, and m_Gi of gangue need to be normalized. It is equivalent to the dimensional data normalization in Fig. 1. The normalized values will map to the red, green, and blue channels. The range is all [0, 255]. The normalization process is as follows.

$${Value}_{\text{norm}}^{{\text{G}}_{i}}=\frac{{Value}^{{\text{G}}_{i}}-{Value}_{\text{min}}^{\text{G}}}{{Value}_{\text{max}}^{\text{G}}-{Value}_{\text{min}}^{\text{G}}}$$

(23)

Value^Gi describes the dimensional data of the i-th gangue, such as t_Gi, y_Gi, and m_Gi. Value^G_max describes the maximum value of dimensional data. Value^G_min describes the minimum value of dimensional data. Value^Gi_norm describes the normalized dimensional data of the i-th gangue.

$${\text{RGB}}_{\text{Label}}(k)=k\cdot {\text{e}}^{13}+{\varepsilon }_{\text{Q}}\cdot {\text{e}}^{12}+{\varepsilon }_{\text{N}}\cdot {\text{e}}^{8}+\left(\frac{\sum {\varepsilon }_{{\text{M}}_{j}}}{j}\right)\cdot {\text{e}}^{4} k=\text{1,2},...,200$$

(24)

To sum up, according to TVRCF of multi-scene and full working condition, the RGB image sample set that the CGSRS scheme set can construct is shown in the following table (Table 1).

Table 1 Description of RGB image sample set based on CGSRS schemes with multi-scene and all-condition TVRCF.

CGSRS scheme evaluation method based on DenseNet

The information on the gangue queue’s time, space and feature dimensions is more complex. Shallow networks struggle to capture key features in the data. Although deep networks can learn richer features, improving complexity reduces generalization ability. Therefore, the DenseNet network, which is based on a dense block structure, is used in this paper. Among them, the direct connection between each layer and all previous layers is realized through the dense connection mechanism, and its output is passed to all subsequent layers, which improves the reuse and propagation of features²⁷. SR, TCSR, and RUR are the network’s outputs. Figure 2 shows that the RGB sample set of the full-condition TVRCF in one scene contains 11,200 gangue queues. Firstly, the samples are randomly sorted, and then the training, test, and validation sets are constructed according to the ratio of 0.9:0.05:0.05. Figure 3 shows the network structure diagram of the CGSRS scheme evaluation model based on DenseNet.

Fig. 2

The construction flow chart of the sample set based on the decomposition method.

Fig. 3

Network structure diagram of CGSRS scheme evaluation model based on DenseNet.

It can be seen from Fig. 3. The DenseNet network in this paper consists of an initial convolution block, six densely connected Dense blocks, five transition blocks, a global average pooling layer (AdaptiveAvgPool2d), a flat layer (Flatten) and a fully connected layer (Linear).

The initial convolution block is used to extract the low-level features of the input image. It takes 3-channel input (e.g., an RGB image) and outputs 32-channel feature maps. This block contains a convolution layer with a convolution kernel size of 3 × 3, a batch normalization layer, a ReLU activation function, and a max pooling layer with a pooling kernel size 3 × 3.

Each Dense block internally contains 4 convolution layers, each of which sequentially performs batch normalization (BatchNorm2d), ReLU activation function, and 3 × 3 convolution operation (Conv2d), where the padding of the convolution operation is 1 to keep the feature map size constant. In each Dense block, the number of input channels is computed dynamically and is equal to the number of initial input channels plus the sum of the number of output channels of all previous Dense blocks. This design allows the output of each layer to be concatenated with the output of all previous layers in the channel dimension, enabling the accumulation and reuse of features.

The transition block is located between two Dense blocks and usually contains a 1 × 1 convolution layer and a 2 × 2 average pooling layer for progressively reducing the number of feature maps. The global average pooling layer reduces the spatial dimension of the feature maps to 1 × 1, resulting in one global description for each feature map. The flat layer flattens the 2D feature map into a 1D vector, which a fully connected layer maps to the final output space. The output layer uses a modified Sigmoid activation function for regression prediction.

Adam optimizer is used during training, and the learning rate is set at 0.0001. The model performance was evaluated by calculating the RMSE between SR, TCSR, and RUR. The mapping relationship between RUR and RGB image samples is the most complex among the sample labels. To ensure the accuracy of the overall prediction results, we weighted the sum of each loss in the proportion of 0.45, 0.45, and 0.10 to optimize the model parameters and minimize the prediction error.

Simulation experiments

The experiment employs the Windows 10 operating system, Pytorch2.1.0 + cu118 deep learning development framework, and Python as the programming language for development. The CPU utilized is Intel Core i5-10400F, while the GPU employed belongs to NVIDIA 4060.

Parameter settings

According to the CGSRS scheme, the multi-task allocation strategy simulation is carried out in MATLAB R2021b. The specifications of robots are the same. The end-effector is 400 mm from the belt plane to avoid collisions between the robot and the gangue. The other parameters are shown in Table 2.

Table 2 Specifications of robots.

CGSRS scheme evaluation model

Figure 2 shows that the RGB sample set of each scene contains 11,200 gangue queues. The RGB sample set is divided into training, test, and validation sets according to the proportion. In the CGSRS scheme evaluation model of each scene, 500 gangue queues of the validation set were randomly selected for the experiment.

RMSE of CGSRS scheme evaluation model

RMSE describes the stability of the CGSRS scheme evaluation model. The CGSRS scheme evaluation model experiment of multi-scene and the specific results are as follows.

Figure 4 shows the RMSE of the CGSRS scheme evaluation model in multi-scene. Figure 4a–c show the results of SR-RMSE, TCSR-RMSE, and RUR-RMSE, respectively. Among them, SR-RMSE and TCSR-RMSE are less than 0.15. RUR-RMSE is less than 0.1. It indicates that the CGSRS scheme evaluation model is stable. Figure 4d shows a scene where the belt speed is 0.1 m/s. The deviation value of the CGSRS scheme evaluation model and multi-task allocation strategy is distributed between − 0.4 and 0.6. and there is no skewness. It shows good symmetry. Among them, the deviation values of CGSRS schemes such as (2, 2), (3, 2), and (3, 5) are long-tailed. It shows that the rule combination affects the multi-task allocation results of the CGSRS scheme. It conforms to the objective law of multi-task allocation strategy. In summary, the CGSRS scheme evaluation model has good stability and symmetry.

Fig. 4

RMSE of CGSRS scheme evaluation model in multi-scene. (a) SR-RMSE. (b) TCSR-RMSE. (c) RUR-RMSE. (d) CGSRS schemes in belt speed = 0.1 m/s.

Acc of CGSRS scheme evaluation model

The Acc describes the accuracy of the CGSRS scheme evaluation model. The CGSRS scheme evaluation model experiment of multi-scene and the specific results are as follows.

Figure 5 shows the Acc of the CGSRS scheme evaluation model in multi-scene. Figure 5a–c show the results of SR-Acc, TCSR-Acc, and RUR-Acc, respectively. Among them, SR-Acc and TCSR-Acc are greater than 0.85. RUR-Acc is greater than 0.9. This indicates that the CGSRS scheme evaluation model is accurate. The maximum Acc and minimum Acc of the CGSRS scheme (3, 2) are obvious. Table 1. Description of RGB image sample set based on CGSRS schemes with multi-scene and all-condition TVRCF shows that this is affected by the FIFO rule. Figure 5d shows the Acc of CGSRS scheme (3, 2) in multi-scene. It can be seen that SR-Acc and TCSR-Acc are smaller than RUR-Acc, but both are about 0.87. Such accuracy meets the application requirement of the CGSRS scheme evaluation model in multi-scene and full-condition TVRCF. Considering that the belt speed in Fig. 5d is 0.4 m/s and 0.5 m/s, the SR-Acc, TCSR-Acc, and RUR-Acc are poor. We show the predicted data of 500 gangues for SR, TCSR and RUR in Fig. 5e,f, respectively. It can be seen that the fitting curve of the predicted data is good, but the very few data are poor. It is because of the instability of FIFO.

Fig. 5

The Acc of CGSRS scheme evaluation model in multi-scene. (a) SR-Acc of multi-scene. (b) TCSR-Acc of multi-scene. (c) RUR-Acc of multi-scene. (d) Acc of CGSRS (3, 2) in multi-scene. (e) Data in belt speed = 0.4 m/s and CGSRS scheme (3, 2). (f) Data in belt speed = 0.5 m/s and CGSRS scheme (3, 2).

Time complexity of CGSRS scheme evaluation model

The time complexity describes the computation time of the CGSRS scheme evaluation model and multi-task allocation strategy. The computation time is relatively stable. This paper selects 50 RGB samples for a single set of tests. For example, in a scene with a belt speed of 0.1 m/s, the CGSRS scheme (1, 1) has 11,200 RGB samples, and the mean value of 50 RGB samples is randomly sampled for the time complexity test. As shown in Fig. 6, it is the time complexity of the CGSRS scheme evaluation model and multi-task allocation strategy.

Fig. 6

The time complexity of the CGSRS scheme evaluation model and multi-task allocation strategy.

Obviously, in the multi-scene and full-condition TVRCF, the time complexity of the CGSRS scheme evaluation model is far less than that of the multi-task allocation strategy. Moreover, CGSRS schemes and multi-scene only slightly affect this time complexity.

Experiment and result analysis

Experiment 1: Reliability of solutions, RGB image data sampled from gangue queue

The CGSRS scheme evaluation model is applied to the multi-task allocation of gangue queues. The primary problem is inconsistent quantity in the gangue queue and the RGB image. Therefore, in the TVRCF of the multi-scene and full working condition, 800 gangues of the RGB image are randomly sampled from the gangue queue to experiment with the stability of the solution. We discuss the scene where the belt speed is 0.5 m/s. The maximum coal flow is 60 kg/s and 120 kg/s, respectively. As shown in Table 3 describes the gangue queue and RGB image data. Among them, the case of RGB image data is coded according to the scene number and the working condition number .

Table 3 Gangue queue and RGB image data.

In this experiment, solution 2 of the multi-task allocation strategy calculates the gangue queue. Solution 1 of the CGSRS scheme evaluation model predicts RGB image data. Whether there is a difference between the two solutions depends on the stability of the solution. According to Table 1, the experimental results of all CGSRS schemes are as follows.

Figure 7 illustrates the solution of the multi-task allocation strategy and the CGSRS scheme evaluation model. Figure 7a shows the working condition of low coal flow, and solution 1 is the multi-task allocation strategy to calculate 1096 gangues. Solution 2 is the CGSRS scheme evaluation model predicting RGB image data. The larger difference is the CGSRS scheme (3, 2). According to Table 1, FIFO greatly influences the prediction results. Figure 7b shows the working condition of low coal flow, and solution 1 is the multi-task allocation strategy to calculate 1836 gangues. Solution 2 is the CGSRS scheme evaluation model predicting RGB image data. The larger difference is the CGSRS scheme (3, 5). According to Table 1, the rule combination of FIFO and MF greatly influences the prediction results.

Fig. 7

Solution of multi-task allocation strategy and CGSRS scheme evaluation model in belt speed 0.5 m/s. (a) Solutions in working condition Q_max = 60 kg/s. (b) Solutions in working condition Q_max = 120 kg/s.

In summary, with the RGB image data randomly sampled from the gangue queue, the solutions predicted by the CGSRS scheme evaluation model are in line with the solutions calculated by the multi-task allocation strategy. Combined with Fig. 2, we can see that the data of the RGB sample set comes from the TVRCF of one cycle, which contains all the information of the gangue queue. Therefore, the stability of the solution is not affected by random sampling.

Experiment 2: Methods comparison, Time complexity and solution

The RMSE, Acc and time complexity of the CGSRS scheme evaluation model have been discussed in “CGSRS scheme evaluation model”, and the results are good. Experiment 1 shows that the RGB image data are inconsistent with gangue quantity in the gangue queue, which does not affect the solution of the CGSRS scheme evaluation model. On this basis, the online multi-task allocation problem of CGSRS is discussed. It is assumed that the belt speed of the scene is 0.6 m/s, and the working condition is unknown. Ten gangue queues were randomly generated. The CGSRS scheme evaluation model and the multi-task allocation strategy were solved, respectively, and the two methods’ time complexity and solution results were compared.

Figure 8 is the flowchart of the method. The multi-task allocation strategy for CGSRS has been proposed in recent years². The process is to traverse the robot number and rule combination to calculate the optimal solution of the gangue queue, resulting from the multi-task allocation problem. The CGSRS scheme evaluation model in this paper cannot directly calculate the multi-task allocation results of the gangue queue. The specific process is that RGB images are obtained by randomly sampling the gangue queue. The predicted near-optimal solution is obtained by traversing all CGSRS scheme evaluation models. According to the CGSRS scheme of the near-optimal solution, the multi-task allocation strategy is adjusted according to the robot number and the rule combination, and the multi-task allocation results of the gangue queue are calculated. Different scenes have different preferences for solutions. In this experiment, the preference setting is SR.

Fig. 8

Flowchart of the methods.

According to Table 4, the solutions of example 1, example 8, example 9 and example 10 are the same, indicating that the CGSRS scheme evaluation method can predict the optimal robot number and rule combination. The solution is different from the other examples. The average difference in SR among the 10 examples is 0.0139. Only in case 7 is the SR of the multi-task allocation strategy 0.0839 higher than that of the CGSRS scheme evaluation model. Figure 9a,b show the multi-task allocation results of the multi-task allocation strategy and the CGSRS scheme evaluation method, respectively. The comparison of solutions shows that the CGSRS scheme evaluation method is stable. In 10 examples, the time complexity of the multi-task allocation strategy is 19 times that of the CGSRS scheme evaluation method. All in all, the solution of the CGSRS scheme evaluation method does not get worse, but the time complexity is significantly reduced.

Table 4 The solution results of the method comparison.

Fig. 9

Multi-task allocation result of Multi-task allocation strategy and CGSRS scheme evaluation method. (a) Allocation result of Multi-task allocation strategy. (b) Allocation result of CGSRS scheme evaluation method.

Conclusion

This paper proposes a Scheme evaluation method for CGSRS with time-varying multi-scenario based on deep learning. It aims to reduce the time complexity of solving multi-task allocation without changing the accuracy of the solution.

• (1) The basis of the CGSRS scheme evaluation method is the stability, accuracy and time complexity of the CGSRS scheme evaluation model. Regarding stability and accuracy, SR-RMSE and TCSR-RMSE are lower than 0.15, and RUR-RMSE is lower than 0.1. SR-Acc and TCSR-Acc are greater than 0.87, and RUR-ACC is greater than 0.9. This indicates that the solution predicted by the CGSRS scheme evaluation model is close to the solution calculated by the multi-task allocation strategy. In addition, the time complexity of the CGSRS scheme evaluation model is much less than that of the multi-task allocation strategy, and it is very stable. This paper applies the DenseNet network to evaluate the CGSRS scheme, and the experimental results show it is feasible. It provides more possibilities for the research of multi-task allocation problems.

• (2) The results of Experiment 1 show that the data inconsistency has no impact on the CGSRS scheme evaluation model, which is crucial for the method application. The results of Experiment 2 show that in the online multi-task allocation problem of CGSRS, the solution results of the proposed method are close to the multi-task allocation strategy. The time complexity is 19 times less than that of the multi-task allocation strategy. It allows CGSRS to be applied to scenes with higher belt speeds. In addition, the CGSRS scheme evaluation method is the first attempt to apply deep learning to the CGSRS multi-task allocation problem. It provides a methodological underpinning to the field and is an important contribution to this paper.

• (3) The CGSRS scheme evaluation method is applied to the TVRCF of multi-scene and full working conditions, and the time complexity is greatly reduced. It should be noted that the regression prediction of the DenseNet network is for a limited number of scenes and working conditions. It is suitable for regular coal preparation scenes, and its application in other fields still needs further research. In addition, the effect of rule combination on the CGSRS scheme evaluation method is objective, which needs follow-up research to improve. Finally, this paper proposes the CGSRS scheme evaluation method, but a better application requires in-depth research.