Introduction

Quantum computers promise to solve some computational problems much faster than classical computers in the future. However, large-scale fault-tolerant quantum computers are still years away. On noisy intermediate-scale quantum (NISQ) devices1, quantum noise strictly limits the depth of reliable circuits, which makes many quantum algorithms unrealistic, e.g., Shor algorithm for factorization2, Harrow–Hassidim–Lloyd algorithm for solving linear systems of equations3. Nevertheless, there exist quantum algorithms that are well suited for NISQ devices and may achieve quantum advantage with practical applications, such as the variational quantum algorithms4,5,6,7,8,9,10,11,12, the quantum imaginary time evolution13,14,15, quantum annealing16.

The quantum imaginary time evolution (QITE) is a promising near-term algorithm to find the ground state of a given Hamiltonian. It has also been applied to prepare thermal states, simulate open quantum systems, and calculate finite temperature properties17,18,19. A pure quantum state is said to be k-UGS if it is the unique ground state of a k-local Hamiltonian \(\hat{H}=\mathop{\sum }\nolimits_{j = 1}^{m}\hat{h}[j]\), where each local term \(\hat{h}[j]\) acts on at most k neighboring qubits. Any k-UGS state can be uniquely determined by its k-local reduced density matrices among pure states (which is called k-UDP) or even among all states (which is called k-UDA)20,21. The QITE algorithm is well suited for preparing k-UGS states with a relatively small k. We start from an initial state \(\left|{{{\Psi }}}_{{{{{{{{\rm{init}}}}}}}}}\right\rangle\), which is non-orthogonal to the ground state of the target Hamiltonian. The final state after long-time imaginary time evolution

$$\mathop{\lim }\limits_{\beta \to \infty }{e}^{-\beta \hat{H}}\left|{{{\Psi }}}_{{{{{{{{\rm{init}}}}}}}}}\right\rangle$$
(1)

has very high fidelity with the k-UGS state. If the ground state of \(\hat{H}\) is degenerate, the final state still falls into the ground state space. Trotter–Suzuki decomposition can simulate the evolution,

$${e}^{-\beta \hat{H}}={\left({e}^{-{{\Delta }}\tau \hat{h}[1]}{e}^{-{{\Delta }}\tau \hat{h}[2]}\ldots {e}^{-{{\Delta }}\tau \hat{h}[m]}\right)}^{n}+{{{{{{{\mathcal{O}}}}}}}}({{\Delta }}{\tau }^{2}),$$
(2)

where Δτ is the step interval, \(n=\frac{\beta }{{{\Delta }}\tau }\) is the number of Trotter step. Trotter error subsumes terms of order Δτ2 and higher. On NISQ devices, Trotter error is difficult to reduce due to the circuit depth limits and Trotterized simulation cannot be implemented accurately22.

Since we can only implement unitary operations on a quantum computer, the main idea of the QITE is to replace each non-unitary step \({e}^{-{{\Delta }}\tau \hat{h}[j]}\) by a unitary evolution \({e}^{-i{{\Delta }}\tau \hat{A}[j]}\) such that

$$\Big|\overline{{{\Psi }}}^{\prime} \Big\rangle =\frac{{e}^{-{{\Delta }}\tau \hat{h}[j]}}{\sqrt{\left\langle {{\Psi }}\right|{e}^{-2{{\Delta }}\tau \hat{h}[j]}\left|{{\Psi }}\right\rangle }}\left|{{\Psi }}\right\rangle \approx {e}^{-i{{\Delta }}\tau \hat{A}[j]}\left|{{\Psi }}\right\rangle ,$$
(3)

where \(\left|{{\Psi }}\right\rangle\) is the state before this step. \(\hat{A}[j]\) acts on D neighboring qubits and can be determined by measurements on \(\left|{{\Psi }}\right\rangle\). For details of the local approximation see Supplementary Note 1. If the domain size D equals the system size N, there always exists \(\hat{A}[j]\), such that the approximation sign of Eq. (3) becomes an equal sign. However, exp(D) local gates are required to implement a generic D-qubit unitary, and we also need to measure exp(D) observables to determine \(\hat{A}[j]\). The exponential resource of measurements and computation makes a large domain size D unfeasible, and we can only use a small one on real devices. This brings the local approximation (LA) error.

Trotter error and the LA error are two daunting challenges in the QITE. These algorithmic errors accumulate with the increase of steps n, which severely weakens the practicability of the QITE. On NISQ computers, a circuit with too many noisy gates is unreliable, and the final measurements give no helpful information. Therefore we cannot use a small step interval Δτ to reduce Trotter errors since this would increase the circuit depth, and noise would dominate the final state. The number of Trotter steps is a tradeoff between quantum noise and Trotter error. For the QITE with large-size systems, we need more Trotter steps and larger domain sizes, which seems hopeless on current devices. There exist some techniques to alleviate the problem, refs. 23,24,25 illustrated some variants of the QITE algorithm with shallower circuits. Refs. 19,26 used Hamiltonian symmetries, error mitigation, and randomized compiling to reduce the required quantum resources and improve the fidelity.

Reinforcement learning (RL) is an area of machine learning concerned with how intelligent agents interact with an environment to perform well in a specific task. It achieved great success in classical games27,28,29,30,31,32 and has been employed in quantum computing problems, such as quantum control33,34,35,36,37,38, quantum circuit optimization39,40,41, the quantum approximate optimization42,43, and quantum annealing44. Quantum computing, in turn, can enhance classical reinforcement learning45,46.

In this work, we propose a deep reinforcement learning-based method to steer the QITE and mitigate algorithmic errors. In our method, we regard the ordering of local terms in the QITE as the environment and train an intelligent agent to take actions (i.e., exchange adjacent terms) to minimize the final state energy. RL is well suited for this task since the state and action-space can be pretty large. We verified the validity of our method with the transverse-field Ising model and the Sherrington–Kirkpatrick model. The RL agent can mitigate most algorithmic errors and decrease the final state energy. Our work pushes the QITE algorithm to more practical applications in the NISQ era.

Results and discussion

In the following, we apply our method to the transverse-field Ising model and the Sherrington–Kirkpatrick (SK) model. A QITE circuit, the experimental setup, and a schematic of the SK model are given in Fig. 1a–c.

Fig. 1: Theoretical and experimental setup.
figure 1

a Quantum circuit of the quantum imaginary time evolution (QITE) for the transverse-field Ising model with 4 qubits and n Trotter steps. \({U}_{\hat{P}}^{j}\) represents the unitary operator that approximates \({e}^{-{{\Delta }}\tau \hat{P}}\) for the local Hamiltonian \(\hat{P}\) in the j-th Trotter step. b Molecule structure and nuclei parameters of the nuclear magnetic resonance processor. The molecule has four Carbon atoms C1, C2, C3, and C4. Diagonal entries of the table are the chemical shifts in Hz, off-diagonal entries of the table are the J-couplings between two corresponding nuclei. The T2 row gives the relaxation time of each nucleus. c A six-vertex complete graph with weighted edges. Different shades of gray represent different couplings in the Sherrington–Kirkpatrick model.

Full size image

Transverse-field Ising model

We first consider the one-dimensional transverse-field Ising model. With no assumption about the underlying structure, we initialize all qubits in the product state \(\left|{{{\Psi }}}_{{{{{{{{\rm{init}}}}}}}}}\right\rangle ={(\left|0\right\rangle +\left|1\right\rangle)}^{\otimes N}/\sqrt{{2}^{N}}\). The Hamiltonian can be written as

$${\hat{H}}^{{{{{{{{\rm{TFI}}}}}}}}}=-\mathop{\sum}\limits_{j}(J{\hat{Z}}_{j}{\hat{Z}}_{j+1}+h{\hat{X}}_{j}).$$
(4)

In the following, we choose J = h = 1. The system is in the gapless phase. For finite-size systems, the ground state of \({\hat{H}}^{{{{{{{{\rm{TFI}}}}}}}}}\) is 2-UGS, therefore 2-UDP and 2-UDA.

In the standard QITE, the ordering of local terms in each Trotter step is the same, e.g., we put commuting terms next to each other and repeat the ordering \({\hat{X}}_{1},\ldots ,{\hat{X}}_{N},{\hat{Z}}_{0}{\hat{Z}}_{1},{\hat{Z}}_{1}{\hat{Z}}_{2},\ldots ,{\hat{Z}}_{N-1}{\hat{Z}}_{N}\). The quantum circuit of the standard QITE with four qubits is shown in Fig. 1a. Inspired by the randomization technique to speed up quantum simulation47,48, we also consider a randomized QITE scheme where we randomly shuffle the ordering in each Trotter step. There is no large quality difference between randomizations, and we pick a moderate one. In the RL-steered QITE, the reward is based on the expectation value of the output state \(\left|{{{\Psi }}}_{{{{{{{\mathrm{f}}}}}}}}\right\rangle\) on the target Hamiltonian

$$E=\left\langle {{{\Psi }}}_{{{{{{{\mathrm{f}}}}}}}}\right|{\hat{H}}^{{{{{{{{\rm{TFI}}}}}}}}}\left|{{{\Psi }}}_{{{{{{{\mathrm{f}}}}}}}}\right\rangle .$$
(5)

The lower the energy, the higher the reward. The RL agent updates the orderings step by step.

For any given β, the RL agent can steer the QITE path and maximize the reward. We fix the system size N = 4, the number of Trotter steps n = 4, and the domain size D = 2. A numerical comparison of energy/fidelity obtained by the standard, the randomized, and the RL-steered QITE schemes for different β values is shown in Fig. 2a, b. The RL-steered path here is β-dependent. Throughout this paper, we use the fidelity defined by \(F(\rho ,\sigma)={{{{{{\mathrm{Tr}}}}}}}\,\sqrt{{\sigma }^{1/2}\rho {\sigma }^{1/2}}\).

Fig. 2: Different QITE schemes for the transverse-field Ising model.
figure 2

Filled markers represent the numerical data; unfilled markers represent the experimental data. The experimental errors were mitigated by virtual distillation. Blue diamonds/dotted lines represent the standard quantum imaginary time evolution (QITE); green triangles/dashed lines represent the randomized QITE; red circles/lines represent the reinforcement learning (RL)-steered QITE. a Energy versus β, the black dashed line represents the ground state energy. b Fidelity versus β. c Algorithmic errors during the evolution.

Full size image

When β is small, RL cannot markedly decrease the energy since the total quantum resource is limited. With the increase of β, the imaginary time evolution target state \(|{\overline{{{\Psi }}}}_{\,{{{{{{\mathrm{f}}}}}}}\,}^{\prime}\rangle ={e}^{-\beta {\hat{H}}^{{{{{{{{\rm{TFI}}}}}}}}}}|{{{\Psi }}}_{{{{{{{{\rm{init}}}}}}}}}\rangle\) approaches the ground state, therefore the obtained energy of all paths decrease in the beginning. However, algorithmic errors increase with β, and this factor becomes dominant after a critical point, the energy of the standard/randomized QITE increases when β > 1/3. Accordingly, the fidelity increases first, then decreases. The RL-steered QITE outperforms the standard/randomized QITE for all β values. Algorithmic errors in this path are canceled out. The fidelity between the output state and the ground state constantly grows to F > 0.996. The gap between the ground state energy and the minimum achievable energy of the standard QITE is 0.053, and that of the RL-QITE is only 0.016. For a detailed optimized path see Supplementary Note 2 (Supplementary Table 1).

Further, we implement the same unitary evolutions on a 4-qubit liquid state nuclear magnetic resonance (NMR) quantum processor49. We carry out the experiments with a 300-MHz Bruker AVANCE III spectrometer. The processor is 13C-labeled trans-crotonic acid in a 7-T magnetic field. The resonance frequency for 13C nuclei is about 75 MHz. The Hamiltonian of the system in a rotating frame is

$${\hat{H}}_{{{{{{{\mathrm{S}}}}}}}}=\mathop{\sum }\limits_{j=1}^{4}\pi {\nu }_{j}{\hat{Z}}_{j}+\mathop{\sum }\limits_{1\le i < j\le 4}^{4}\frac{\pi }{2}{J}_{ij}{\hat{Z}}_{i}{\hat{Z}}_{j},$$
(6)

where νj is the chemical shift of the j-th nuclei, Jij is the J-coupling between the i-th and j-th nuclei, \({\hat{Z}}_{j}\) is the Pauli matrix σz acting on the j-th nuclei. All the parameters can be found in Fig. 1b. The quantum operations are realized by irradiating radiofrequency pulses on the system. We optimize the pulses over the fluctuation of the chemical shifts of the nuclei with the technique of gradient ascent pulse engineering50. The experiment is divided into three steps: (i) preparing the system into a pseudo-pure state using the temporal average technique51; (ii) applying the quantum operations; (iii) performing measurements52.

Denote the NMR output state as ρ, whose density matrix can be obtained through quantum state tomography. ρ is a highly mixed state since quantum noise is inevitable. We use the virtual distillation technique to suppress the noise53,54. The dominant eigenvector of ρ, \(\mathop{\lim }\nolimits_{M\to \infty }{\rho }^{M}/\,{{\mbox{Tr}}}\,({\rho }^{M})\), can be extracted numerically. Its expectation value on \({\hat{H}}^{{{{{{{{\rm{TFI}}}}}}}}}\) and its fidelity with the ground state are shown in Fig. 2a, b with unfilled markers. Consistent with our numerical results, the RL-steered path significantly outperforms the other two for large β. For postprocessing of NMR data see Supplementary Note 3.

In our simulation, we have four orderings to optimize and 28 local unitary operations to implement. Denote \(\left|{{{\Psi }}}_{k}\right\rangle\) as the state after the k-th operation, \(|{\overline{{{\Psi }}}}_{k}^{\prime}\rangle\) as the temporal target state with the ideal imaginary time evolution. The instantaneous algorithmic error during the evolution can be characterized by the squared Euclidean distance between \(|{{{\Psi }}}_{k}\rangle\) and \(|{\overline{{{\Psi }}}}_{k}^{\prime}\rangle\),

$${\epsilon }_{{{{{{{{\rm{alg}}}}}}}}}=\parallel \Big|{{{\Psi }}}_{k}\Big\rangle -\Big|{\overline{{{\Psi }}}}_{k}^{\prime}\Big\rangle {\parallel }^{2}.$$
(7)

For β = 0.9, Fig. 2c shows ϵalg as a function of evolution step k. Although ϵalg fluctuates in all paths, it accumulates obviously in the standard QITE and eventually climbs to ϵalg = 0.082. The randomized QITE performs slightly better and ends with ϵalg = 0.030. The RL-steered QITE is optimal, the trend of ϵalg shows no accumulation and drops to ϵalg = 0.007 in the end. Although we cannot directly estimate ϵalg in experiments, we can minimize it via maximizing the reward function.

One question that arises is whether we can enhance the QITE algorithm with RL for larger systems? Now we apply our approach to the transverse-field Ising model with system sizes N = 2, 3, …, 8 to demonstrate the scalability. Still, we consider the QITE with 4 Trotter steps. Denote the target state with “evolution time” β as

$$\left|{{{\Psi }}}_{\,{{{{{{\mathrm{f}}}}}}}\,}^{\prime}(\beta)\right\rangle =\frac{{e}^{-\beta {\hat{H}}^{{{{{{{{\rm{TFI}}}}}}}}}}}{\sqrt{\left\langle {{{\Psi }}}_{{{{{{{{\rm{init}}}}}}}}}\right|{e}^{-2\beta {\hat{H}}^{{{{{{{{\rm{TFI}}}}}}}}}}\left|{{{\Psi }}}_{{{{{{{{\rm{init}}}}}}}}}\right\rangle }}\left|{{{\Psi }}}_{{{{{{{{\rm{init}}}}}}}}}\right\rangle .$$
(8)

In the following, we use an adaptive β for different N such that the expectation value \(\langle {{{\Psi }}}_{\,{{{{{{\mathrm{f}}}}}}}\,}^{\prime}(\beta)|{\hat{H}}^{{{{{{{{\rm{TFI}}}}}}}}}t|{{{\Psi }}}_{\,{{{{{{\mathrm{f}}}}}}}\,}^{\prime}(\beta)\rangle\) is always higher than the ground state energy of \({\hat{H}}^{{{{{{{{\rm{TFI}}}}}}}}}\) by 1 × 10−3. The results are illustrated in Fig. 3, the RL agent can efficiently decrease the energy and increase the fidelity between the final state and the ground state for all system sizes. The ratio of the RL-steered energy (ERL) to the standard QITE energy (Estd) is also given. This ratio increases steadily with the number of qubits. Note that the neural networks we use here only contain four hidden layers. The hyperparameters were tuned for the N = 4 case. We apply the same neural networks to larger N, the required number of training epochs does not increase obviously. If we want to increase the fidelity further for N > 4, we can use more Trotter steps and tune the hyperparameters accordingly. There is little doubt, however, the training process will be more time-consuming. For comparison between the QITE and quantum annealing see Supplementary Note 4.

Fig. 3: Scaling of different QITE schemes.
figure 3

Blue diamonds represent the standard quantum imaginary time evolution (QITE); green triangles represent the randomized QITE; red circles represent the reinforcement learning (RL)-steered QITE. a Energy versus system size, and the energy ratio ERL/Estd versus system size (inset). The black dash-dotted line represents the ground state energy. b Fidelity versus system size.

Full size image

Sherrington–Kirkpatrick model

The second model we apply our method to is the Sherrington–Kirkpatrick (SK) model55, a spin-glass model with long-range frustrated ferromagnetic and antiferromagnetic couplings. Finding a ground state of the SK model is NP-hard56. On NISQ devices, solving the SK model can be regarded as a special Max-Cut problem and dealt with by the quantum approximate optimization algorithm5,57. Here we use the QITE to prepare the ground state of the SK model. Compared with the quantum approximate optimization algorithm, the QITE does not need to sample a bunch of initial points and implement classical optimization with exponentially small gradients58.

Consider a six-vertex complete graph shown in Fig. 1c. The SK model Hamiltonian can be written as

$${\hat{H}}^{{{{{{{{\rm{SK}}}}}}}}}=\mathop{\sum}\limits_{i < j}{J}_{ij}{\hat{Z}}_{i}{\hat{Z}}_{j},$$
(9)

we independently sample Jij are from a uniform distribution Jij ~ U(−1, 1).

Since \(\hat{Z}\hat{Z}\)-terms commute, there is no Trotter error in Eq. (2) for the SK model. The ground state of \({\hat{H}}^{{{{{{{{\rm{SK}}}}}}}}}\) is twofold degenerate. The QITE algorithm can prepare one of the ground states. We fix β = 5, n = 6, D = 2, sample Jij and train the agent to steer the QITE path. Define the probability of finding a ground state of \({\hat{H}}^{{{{{{{{\rm{SK}}}}}}}}}\) through measurements as Pgs. Energy and Pgs as functions of β are shown in Fig. 4. Remember that the RL-steered path here was only optimized for a specific β value (i.e., β = 5) since we want to verify the dependence of the ordering on β.

Fig. 4: Different QITE schemes for the Sherrington–Kirkpatrick model.
figure 4

Blue dotted lines represent the standard quantum imaginary time evolution (QITE); green dashed lines represent the randomized QITE; red lines represent the reinforcement learning (RL)-steered QITE. a Energy versus the imaginary time β. The black dash-dotted line represents the ground state energy. b Success probability versus the imaginary time β.

Full size image

For each path, we observe a sudden switch from a high probability of success to a low one when 4 < β < 5. The reason is that for some states when the step interval exceeds a critical value, there is an explosion of LA error which utterly ruins QITE. After the critical value, the QITE algorithm loses its stability, and the energy E fluctuates violently. The randomized QITE performs even worse than the standard one, where the highest success probability is only 0.60. In the RL-steered QITE, Pgs falls down a deep “gorge” but can recover soon. When β = 5, most algorithmic errors disappeared and Pgs = 0.9964. In comparison, the standard QITE ends with Pgs = 0.0002 and the randomized QITE ends with Pgs = 0.0568, they dropped sharply and cannot be recovered even if we further improve β. For detailed couplings and QITE path see Supplementary Note 2 (Supplementary Tables 2 and 3). For additional numerical results of the SK model see Supplementary Note 5. For the control landscape see Supplementary Note 6.

Conclusions

We have proposed an RL-based framework to steer the QITE algorithm for preparing a k-UGS state. The RL agent can find the subtle evolution path to avoid error accumulation. We compared our method with the standard and the randomized QITE; numerical and experimental results demonstrate a clear improvement. The RL-designed path requires a smaller domain size and fewer Trotter steps to achieve satisfying performance for both the transverse-field Ising model and the SK model. We also noticed that randomization cannot enhance the QITE consistently, although it plays a positive role in quantum simulation. For the SK model, with the increase of the total imaginary time β, a switch from a high success probability to almost 0 exists. The accumulated error may ruin the QITE algorithm all of a sudden instead of gradually, which indicates the importance of an appropriate β for high-dimensional systems. The RL-based method is a winning combination of machine learning and quantum computing. Even though we investigated only relatively small systems, the scheme can be directly extended to larger systems. The number of neurons in the output layer of the RL agent only grows linearly with system size N. A relevant problem worth considering is how to apply the QITE (or the RL-steered QITE) for preparing a quantum state that is k-UDP or k-UDA but not k-UGS.

RL has a bright prospect in the NISQ era. In the future, one may use RL to enhance Trotterized/variational quantum simulation59,60,61,62 similarly, but the reward function design will be more challenging. Near-term quantum computing and classical machine learning methods may benefit each other in many ways. Their interplay is worth studying further.

Methods

The RL process is essentially a finite Markov decision process63. This process is described as a state-action-reward sequence, a state st at time t is transmitted into a new state st+1 together with giving a scalar reward Rt+1 at time t + 1 by the action at with the transmission probability p(st+1; Rt+1∣st; at). In a finite Markov decision process, the state set, the action set, and the reward sets are finite. The total discounted return at time t is

$${G}_{t}=\mathop{\sum }\limits_{k=0}^{\infty }{\gamma }^{k}{R}_{t+k+1},$$
(10)

where γ is the discount rate and 0 ≤ γ ≤ 1.

The goal of RL is to maximize the total discounted return for each state and action selected by the policy π, which is specified by a conditional probability of action a for each state s, denoted as π(a∣s).

In this work, we use distributed proximal policy optimization (DPPO)64,65, a model-free reinforcement learning algorithm with the actor-critic architecture. The agent has several distributed evaluators, and each evaluator consists of two components: an actor-network that computes a policy π, according to which the actions are probabilistically chosen; a critic-network that computes the state value V(s), which is an estimate of the total discounted return from state s and the following policy π. Using multiple evaluators can break the unwanted correlations between data samples and make the training process more stable. The RL agent updates the neural network weights synchronously. For more details of DPPO see Supplementary Note 7.

The objective of the agent is to maximize the cumulative reward under a parameterized policy πθ:

$$J\left({\pi }_{{{{{{{{\boldsymbol{\theta }}}}}}}}}\right)={{\mathbb{E}}}_{{\pi }_{{{{{{{{\boldsymbol{\theta }}}}}}}}}}\left[\mathop{\sum }\limits_{t=0}^{\infty }{\gamma }^{t}R\left({s}_{t}\right)\right].$$
(11)

In our task, the environment state is the ordering of local terms in each Trotter step, and the state space size is (m!)n. The agent observes the full state space at the learning stage, i.e., we deal with a fully observed Markov process. We define the action set by whether or not to exchange two adjacent operations in Eq. (2). Note that even if two local terms \(\hat{h}[j]\) and \(\hat{h}[j+1]\) commute, their unitary approximations \({e}^{-i{{\Delta }}\tau \hat{A}[j]}\) and \({e}^{-i{{\Delta }}\tau \hat{A}[j+1]}\) are state-dependent and may not commute. The ordering of commuting local terms still matters. For a local Hamiltonian with m terms, there are m − 1 actions for each Trotter step. Any permutation on m elements can be decomposed to a product of \({{{{{{{\mathcal{O}}}}}}}}({m}^{2})\) adjacent transpositions. Therefore our action set is universal, and a well-trained agent can rapidly steer the ordering to the optimum. The agent takes actions in sequence, the size of the action-space is 2n(m−1). A deep neural network with n(m − 1) output neurons determines the action probabilities. We iteratively train the agent from measurement results on the output state \(\left|{{{\Psi }}}_{{{{{{{\mathrm{f}}}}}}}}\right\rangle\), the agent updates the path to maximize its total reward. Figure 5 shows the diagram of our method.

Fig. 5: Schematic of steering the quantum imaginary time evolution with reinforcement learning.
figure 5

The colored symbols represent single-qubit (blue dots) and two-qubit (green rectangles) non-unitary operations \(\{{e}^{-{{\Delta }}\tau \hat{h}[j]}\}\). The reinforcement learning (RL) agent, realized by neural networks, interacts with different environments (Env 1, Env 2, …, Env 12) and optimizes the operation order to minimize the output state energy.

Full size image

The reward of the agent received in each step is

$${R}_{t}=\left\{\begin{array}{ll}0,\quad &t\in \{0,1,\ldots ,{N}_{d}-1\}\\ {{{{{{{\mathcal{R}}}}}}}},\quad &t={N}_{d}\end{array}\right.$$
(12)

where Nd is the time delay to get the reward, \({{{{{{{\mathcal{R}}}}}}}}\) is the modified reward function. In particular, we define \({{{{{{{\mathcal{R}}}}}}}}\) as

$${{{{{{{\mathcal{R}}}}}}}}=\left\{\begin{array}{ll}-1,\quad &\,{{\mbox{if}}}\,E\le {E}_{{{{{{{{\rm{std}}}}}}}}}\\ -1/\log [{{{{{{\mathrm{clip}}}}}}}\,((E/{E}_{{{{{{{{\rm{std}}}}}}}}}-1),0.01,1.99)],\quad &\,{{\mbox{otherwise}}}\,\end{array}\right.$$
(13)

where E is the output energy given by our RL-steered path, Estd is the energy given by the standard repetition path without optimization. In order to avoid divergence of the reward, we use a clip function to clip the value of (E/Estd − 1) within a range (0.01, 1.99).