Addressing general measurements in quantum Monte Carlo

Abstract

Quantum Monte Carlo is one of the most promising approaches for dealing with large-scale quantum many-body systems. It has played an extremely important role in understanding strongly correlated physics. However, two fundamental problems, namely the sign problem and general measurement issues, have seriously hampered its scope of application. We propose a universal scheme to tackle the problems of general measurement. The target observables are expressed as the ratio of two types of partition functions $\langle O\rangle=\overline{Z}/Z$, where $\overline{Z}=tr({Oe}^{-\beta H})$ and $Z=tr({e}^{-\beta H})$. These two partition functions can be estimated separately within the reweight-annealing frame, and then be connected by an easily solvable reference point. We have successfully applied this scheme to XXZ model and transverse field Ising model, from 1D to 2D systems, from two-body to multi-body correlations and even non-local disorder operators, and from equal-time to imaginary-time correlations. The reweighting path is not limited to physical parameters, but also works for space and time. Essentially, this scheme solves the long-standing problem of calculating the overlap between different distribution functions in mathematical statistics, which can be widely used in statistical problems, such as quantum many-body computation, big data and machine learning.

Introduction

Quantum Monte Carlo (QMC) is a highly promising numerical method without approximations for large-scale or high-dimensional quantum many-body systems, capable of simulating complex systems with an exponential degree of freedom while maintaining polynomial computation complexity^{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25}. Despite the maturity of QMC techniques after decades of development^{15,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41}, there remain two essential challenges that greatly limit the application of QMC. The first is the notorious sign problem^{34,35,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58}, and the second is the issue of general (off-diagonal) measurements^{7,10,16,30,59}.

In this work, we will focus on the enduring challenge of measuring general (off-diagonal) observables. The target is how to extract as more as information from the QMC samplings. Unlike other numerical methods, QMC cannot directly obtain the wave-function of ground state. Typically, the evaluation of a physical quantity 〈O〉 in QMC is derived as follows: $\langle O\rangle=tr({Oe}^{-\beta H})/Z$, where $Z=tr({e}^{-\beta H})$ is the partition function (PF), β is the inverse temperature and H is the Hamiltonian. For simplicity, we define $\overline{Z}=tr({Oe}^{-\beta H})$, hence $\langle O\rangle=\overline{Z}/Z$.

In a standard QMC framework, the partition function Z can be generally decomposed into the sum of all the weights, i.e. Z = ∑_iW_i. If the operator O can be treated as a number O_i under the configuration of W_i, which corresponds to a diagonal measurement, the physical quantity can be readily estimated in the form

$$\langle O\rangle=\frac{\overline{Z}}{Z}=\frac{{\sum }_{i}{O}_{i}{W}_{i}}{Z}$$

(1)

In this way, the value O_i can be directly obtained when we sample the configurations W_i of the PF, making diagonal measurements straightforward in the QMC framework. In the case of diagonal measurement, it is clear that two PFs, $\overline{Z}={\sum }_{i}{O}_{i}{W}_{i}$ and Z = ∑_iW_i, share the same set of configurations {W_i}, but differ in their associated values, with O_i for $\overline{Z}$ and 1 for Z. Consequently, sampling the configurations {W_i} is sufficient to capture the expectation value $\langle O\rangle=\overline{Z}/Z$.

However, the situation would deteriorate significantly during off-diagonal measurements. Off-diagonal operators typically alter the existing configurations {W_i} of Z = ∑_iW_i, resulting in new configurations $\{{W}_{i}^{{\prime} }\}$ for $\overline{Z}={\sum }_{i}{W}_{i}^{{\prime} }$ that are entirely distinct from the original set {W_i}. This implies that we are unable to obtain samples $\{{W}_{i}^{{\prime} }\}$ within the framework of conventional QMC methods, which are designed to sample from {W_i}. As shown in Fig. 1(a), two PFs no longer share the same configurations, making it impossible to simulate their ratio directly as in the diagonal case. Furthermore, it is usually impossible to design updates between {W_i} and $\{{W}_{i}^{{\prime} }\}$ in QMC algorithms (If you can realize the updates between {W_i} and $\{{W}_{i}^{{\prime} }\}$, the ratio $\overline{Z}/Z$ then can be obtained, such as the QMC algorithm for entanglement entropy⁶⁰). This represents the fundamental challenge in the off-diagonal measurements.

Fig. 1: Schematic of the bipartite reweight-annealing process.

Direct evaluation of the ratio $\overline{Z}({J}_{0})/Z({J}_{0})$ is generally infeasible, since the corresponding weight distributions at J₀ exhibit a lack of overlap, as sketched in (a). Instead, two independent reweight-annealing (RA) processes are performed: the red path corresponds to $\overline{Z}(J)/\overline{Z}({J}_{0})$, and the blue path corresponds to Z(J)/Z(J₀), as indicated by the colored RA process labels. Along each path, the system is gradually evolved in the annealing direction through small parameter shifts ${J}^{{\prime} }=J+\delta J$, ensuring sufficient overlap between adjacent distributions, as illustrated in (b) and (c). Once the annealing paths reach an easily solvable reference point, the target ratio $\overline{Z}({J}_{0})/Z({J}_{0})$ can be reconstructed.

In some special cases, certain off-diagonal observables can be extracted in ingenious ways. For instance, determinant QMC (DQMC) can efficiently calculate two-body Green’s functions, where higher-order correlators can be derived using Wick’s theorem^11,31,35,61. This works particularly well in fermionic systems, where off-diagonal Green’s functions such as $\langle {c}_{i}{c}_{j}^{\dagger }\rangle$ are naturally embedded in the sampling weight. We now fix the measurement problem in other Monte Carlo methods, such as diagrammatic QMC^62,63,64, world-line QMC and stochastic series expansion (SSE) formulations, in which off-diagonal measurements pose a much more severe challenge. Despite numerous efforts in the past, two-body Green’s functions can be obtained only in certain cases within the framework of worm-like QMC^{9,65,66,67,68,69,70,71,72,73}. The reason is that the configurations in the worm-like update process can be treated as samplings of the two-point Green’s function. However, multi-body Green’s functions remain challenging to be extracted even with this specialized approach and the worm-like algorithm only works in several models. In addition, if the off-diagonal operator to be measured is a part of the Hamiltonian, it can be estimated through the sampling process⁷⁴. As an instance, 〈S^x〉 can be measured in a transverse field Ising model (TFIM)^75,76. Another example is that, in the stochastic series expansion (SSE) method, the energy value can be calculated directly by counting the number of operators in the space-time configurations^2,7,30. Although the importance of off-diagonal observables in quantum systems, there is currently no general method for measuring arbitrary operators in QMC, even though a lot of effort has been devoted to it over the past decades.

Recently, a newly proposed method – reweight-annealing (RA)⁷⁷ has been successfully applied to determine the ratio of two same-type PFs at different parameters. In the reweight-annealing method, as shown in Fig. 1(b) and (c), the PF at the parameter ${J}^{{\prime} }$ can be estimated using the value of PF at another parameter J by resetting the weights.

$$\frac{Z({J}^{{\prime} })}{Z(J)}={\left\langle \frac{W({J}^{{\prime} })}{W(J)}\right\rangle }_{J}$$

(2)

This equation states that the ratio $Z({J}^{{\prime} })/Z(J)$ is obtained by averaging $W({J}^{{\prime} })/W(J)$ over configurations sampled from the ensemble at J (i.e.〈⋅〉_J). However, this equation works well only when the distributions $Z({J}^{{\prime} })$ and Z(J) are adjacent. In this context, the importance sampling can be maintained⁷⁸. Therefore, if the target parameters J and ${J}^{{\prime} }$ are far away from each other, a series of intermediate parameters {J_i} need to be inserted to split the reweighting process by gradually moving from J to ${J}^{{\prime} }$. This can be expressed as $Z({J}^{{\prime} })/Z(J)=Z({J}^{{\prime} })/Z({J}_{1})\times Z({J}_{1})/Z({J}_{2})\times \ldots Z({J}_{i})/Z({J}_{i+1})\ldots \times Z({J}_{n})/Z(J)$. Since the entire process involves annealing from one parameter to another with iterative reweighting, it is dubbed as “reweight-annealing"⁷⁷. The similar spirit of reweighting also has been developed in the high-energy physics and other fields^{79,80,81,82,83,84}. Once a reference point Z(J) is known, $Z({J}^{{\prime} })$ can be calculated through the ratio. It has been proved that the computation complexity of the RA method is polynomial if the ratio of two closest Z(J) and $Z({J}^{{\prime} })$ is fixed in the division strategies⁷⁷. More specifically, in the QMC simulation, maintaining the ratio of two closest partition functions $Z({J}^{{\prime} })/Z(J)$ with ${\mathcal{O}}(1)$ range (i.e. 0.1 ~ 10) is essential–not only to preserve the polynomial computation complexity under importance sampling⁷⁷, but also to ensure that system error remains controllable (system error analysis can be found in Supplementary Note 7). In addition, the number of slices should be increased near the phase transition to maintain the ratio with ${\mathcal{O}}(1)$ range, since the distribution of the partition function changes more rapidly in this regime. Motivated by the reweighting scheme, we propose a novel scheme termed “bipartite reweight-annealing (BRA)" method to address the challenges of general measurements in QMC simulations. We will present several examples to demonstrate its feasibility and versatility.

Results

Bipartite reweight-annealing framework

In fact, we realize that the reweighting scheme is not only limited to the standard PF Z(J) but can be applied to any distribution that varies with the related parameters. In practice, an off-diagonal observable can be treated as the ratio of two types of PFs $\langle O\rangle=\overline{Z}/Z$, where $\overline{Z}=tr({Oe}^{-\beta H})$. This insight inspires us to reweight different kinds of PFs (the numerator $\overline{Z}(J)$ and denominator Z(J)) respectively, as Fig. 1(b) and (c) show. The key idea is that we firstly calculate the ratios $\overline{Z}({J}^{{\prime} })/\overline{Z}(J)$ and $Z({J}^{{\prime} })/Z(J)$, and if we have a reference point $\overline{Z}(J)/Z(J)$ which is easily solvable (as displayed in Fig. 1), then the target measurement $\langle O({J}^{{\prime} })\rangle$ can be estimated in this approach:

$$\langle O({J}^{{\prime} })\rangle=\frac{\overline{Z}({J}^{{\prime} })}{Z({J}^{{\prime} })}=\frac{\overline{Z}(J)}{Z(J)}\times \frac{\overline{Z}({J}^{{\prime} })}{\overline{Z}(J)}\times \frac{Z(J)}{Z({J}^{{\prime} })}$$

(3)

where $\overline{Z}(J)/Z(J)$ is the known reference point, $\overline{Z}({J}^{{\prime} })/\overline{Z}(J)$ and $Z(J)/Z({J}^{{\prime} })$ can be calculated by reweighting. Likewise, maintaining the ratio within an ${\mathcal{O}}(1)$ range is imperative (detail can be found in Supplementary Note 7).

This BRA scheme avoids the intractable problem of calculating the ratio between two entirely different PFs (Fig. 1(a)) by translating it into a solvable framework. It is highly general and can be applied to almost all physical quantities. In the following sections, we will employ this scheme to demonstrate several off-diagonal measurements that previously were rather difficult, even impossible to be calculated in QMC. Moreover, scanning the observables along the path of physical parameter to trace the phase diagram becomes natural and efficient in the BRA frame. Actually, we will show the annealing path is not limited to the physical parameter only, but also works for the degree of freedom in both space and time.

Equal-time off-diagonal correlations

As an example, we consider the Hamiltonian of the spin-1/2 XXZ model, which is given by:

$${H}_{XXZ}=\mathop{\sum }\limits_{\langle i,j\rangle }\left[\frac{1}{2}({S}_{i}^{+}{S}_{j}^{-}+{S}_{i}^{-}{S}_{j}^{+})+\Delta {S}_{i}^{z}{S}_{j}^{z}\right]$$

(4)

where 〈i, j〉 denotes the nearest neighbors, Δ is the parameter that controls the anisotropy. The Hamiltonian can be simulated using the directed loop algorithm of the SSE method (detail can be found in Supplementary Note 1)^9,71,85,86. In this model, two-point off-diagonal operators $\langle {S}_{i}^{x}{S}_{j}^{x}\rangle=\langle {S}_{i}^{y}{S}_{j}^{y}\rangle=(\langle {S}_{i}^{+}{S}_{j}^{-}\rangle+\langle {S}_{i}^{-}{S}_{j}^{+}\rangle )/4$, which can be measured using worm-like sampling trick^10,68,70. However, measuring a general off-diagonal correlation function is significantly more challenging.

Here, taking correlation of S^x operators as an example, we show how to measure it via varying the physical parameter Δ in our scheme,

$${\langle {S}_{i}^{x}{S}_{j}^{x}\rangle }_{\Delta }=\frac{tr({S}_{i}^{x}{S}_{j}^{x}{e}^{-\beta H})}{tr({e}^{-\beta H})}=\frac{\overline{Z}(\Delta )}{Z(\Delta )}$$

(5)

where $\overline{Z}(\Delta )$ represents a general partition function with extra off-diagonal operators inserted, distinguished from a normal partition function without these extra off-diagonal operators. The calculation of $\langle {S}_{i}^{y}{S}_{j}^{y}\rangle$ is the same as $\langle {S}_{i}^{x}{S}_{j}^{x}\rangle$ in this frame, as explained in the Supplement note 1.

Firstly, we consider an obvious reference point of this model: ${\Delta }^{{\prime} }=1$, known as the “Heisenberg point" and possessing SU(2) symmetry. This implies isotropic spin-spin correlations with $\langle {S}_{i}^{z}{S}_{j}^{z}\rangle=\langle {S}_{i}^{x}{S}_{j}^{x}\rangle$. Moreover, since $\langle {S}_{i}^{z}{S}_{j}^{z}\rangle$ is accessible through diagonal measurements in a standard QMC framework, the focus shifts to measuring the ratio of the partition functions. For convenience, we define that $\overline{Z}r=\overline{Z}(\Delta )/\overline{Z}({\Delta }^{{\prime} })$ and $Zr=Z(\Delta )/Z({\Delta }^{{\prime} })$. Then the Eq. (5) can be rewritten as

$${\langle {S}_{i}^{x}{S}_{j}^{x}\rangle }_{\Delta }=\overline{Z}r/Zr\times {\langle {S}_{i}^{z}{S}_{j}^{z}\rangle }_{{\Delta }^{{\prime} }=1}$$

(6)

In this way, the correlation of S^x operators can be easily calculated as Fig. 2 shows (detail can be found in Supplementary Note 2).

Fig. 2: Equal-time off-diagonal correlation measurement via the reweight-annealing framework.

a Ratios of two-point correlations ${C}_{ij}=\langle {S}_{i}^{x}{S}_{j}^{x}\rangle$ as a function of the Ising coupling strength Δ for L = 10 with β = 20. b Ratios of four-point correlations ${C}_{ijkl}=\langle {S}_{i}^{x}{S}_{j}^{x}{S}_{k}^{x}{S}_{l}^{x}\rangle$ in the one-dimensional chain. c Ratios of representative two-point (C_ij) and four-point (C_ijkl) correlations on a 4 × 2 lattice with β = 8. d Four-point correlation ${C}_{12,L/2,L/2+1}=\langle {S}_{1}^{x}{S}_{2}^{x}{S}_{L/2}^{x}{S}_{L/2+1}^{x}\rangle$ on 8 × 8 and 20 × 20 square lattices with β = 2L. a–c include comparisons with exact diagonalization (ED) results. Error bars ( ± 1σ) from SSE simulation denote the standard error of the mean obtained from the Monte Carlo bins. All calculations are performed on lattices with periodic boundary conditions (PBC).

The QMC results are also compared with the exact diagonalization (ED) in order to demonstrate the reliability of this scheme. Figure 2 shows the calculation results from ED and BRA. The subfigures (a) and (b) exhibit two-point correlations and four-point correlations, represented by ${C}_{2}(r)=\langle {S}_{i}^{x}{S}_{i+r}^{x}\rangle$ and ${C}_{4}=\langle {S}_{i}^{x}{S}_{j}^{x}{S}_{k}^{x}{S}_{l}^{x}\rangle$ in an XXZ chain with L = 10 and β = 20. Similar simulation results of 2D lattice with L_x = 4, L_y = 2, β = 8 are shown in the subfigure (c). The black line represents the ED results which match well with the QMC data. We have plotted only a few points on the graph for clarity, while the actual simulation data points of BRA are densely distributed.

One may feel that the SU(2) symmetry at ${\Delta }^{{\prime} }=1$ is a strict condition which is not general for an arbitrary model. Actually, it is convenient to introduce an auxiliary Hamiltonian H₀ with friendly symmetry or easily solvable property. As what quantum annealing does^87,88,89, we can set the BRA path as tH + (1 − t)H₀ and anneal from t = 0 to t = 1. This approach allows us to obtain the observable of the target Hamiltonian H.

System size and distance growth by annealing

Another choice for the reference point is to measure the observable through the ED method in small size, then anneal the small system to large size.

In this approach, the system size L and distance r between ${S}_{1}^{x}$ and ${S}_{1+r}^{x}$ can be considered as BRA parameters. For instance, we can choose the $\langle {S}_{1}^{x}{S}_{2}^{x}\rangle$ for L₀ = 4 as a reference point, and then we obtain the $\langle {S}_{1}^{x}{S}_{2}^{x}\rangle$ for larger system L via adding the remaining sites L − L₀ to the original chain, as shown in the Fig. 3(a). In this procedure, the interaction J₂ is tuned to couple 4 sites with L − 4 sites. When we fix the system size and choose the $\langle {S}_{1}^{x}{S}_{2}^{x}\rangle$ as a reference point, we can obtain the $\langle {S}_{1}^{x}{S}_{1+r}^{x}\rangle$ (r > 1) via adding some sites to the area between ${S}_{1}^{x}$ and ${S}_{2}^{x}$ and removing some sites at the end of this chain. We need to tune the coupling J₂ from 0⁺ to 1 and also gradually adjust the coupling J₃ from 1 to 0⁺, in order to keeping the chain length unchanged as displayed in the Fig. 3(b). A consistent match between the QMC and ED results for off-diagonal correlations is found in Supplementary Note 5. This method certainly can be extended to the simulation of large systems. As shown in Fig. 3, we obtain the off-diagonal correlation ${C}^{xx}(r)=\langle {S}_{1}^{x}{S}_{1+r}^{x}\rangle$ using the reweighting method of changing system size (benchmark with the worm-trick method can be found in Supplementary Note 8). In Fig. 3(b), ∣C^xx(r)∣ has power-law decay, which reflects the physical feature of Luttinger liquid. As Δ decreases, the power-law parameter becomes smaller, which indicates the ${S}_{i}^{x}{S}_{j}^{x}$ correlation is enhanced and ${S}_{i}^{z}{S}_{j}^{z}$ correlation is weakened. Besides, we utilize the method of annealing Δ, which was introduced in the preceding section, with fixed large-size L = 48 to obtain the curves of ∣C^xx(r)∣ (dashed line in Fig. 3(b)), which agrees well with the results through annealing L and r.

Fig. 3: Spatial coupling parameters Annealing.

The off-diagonal correlation measurement via lattice reweight-annealing method in the 1D XXZ model with L = 48. a The lattice diagram for annealing along the system size L. We incrementally tune the coupling J₂ from 0⁺ to 1. b The lattice diagram for annealing along the distance r between ${S}_{1}^{x}$ and ${S}_{1+r}^{x}$. We firstly gradually adjust the coupling J₂ from 0⁺ to 1, then we gradually tune the coupling J₃ from 1 to 0⁺. c Two point off-diagonal correlations for system size L = 48 obtained from subplots (a) and (b) annealing method. The dashed lines represent the simulation for annealing Ising coupling Δ with fixed ${S}_{1}^{x}{S}_{1+r}^{x}$. Both the hollow symbols and the dashed line represent QMC results, displayed with error bars denoting the mean value and standard error.

Separability of measurements

The second scheme, which involves annealing from a small system to larger system, inspires us to explore the separability of the general measurement in a large system. Without loss of generality, we consider a scenario where a large system is composed of two decoupled smaller subsystems as shown in Fig. 4. This approach can be easily extended to systems with multiple parts. In the decoupled case, the density matrix of the total system is the tensor product of the two density matrices, i.e. ρ = ρ_A ⊗ ρ_B. Typically, we encounter two kinds of measured operators, O_A ⊗ O_B and O_A + O_B, they satisfy

$${\langle {O}_{A}\otimes {O}_{B}\rangle }_{A\cup B}= \frac{{\rm{tr}}(({\rho }_{A}\otimes {\rho }_{B})({O}_{A}\otimes {O}_{B}))}{{\rm{tr}}({\rho }_{A}\otimes {\rho }_{B})}\\= \frac{{\rm{tr}}({\rho }_{A}{O}_{A}){\rm{tr}}({\rho }_{B}{O}_{B})}{{\rm{tr}}({\rho }_{A}){\rm{tr}}({\rho }_{B})}\\= {\langle {O}_{A}\rangle }_{A}{\langle {O}_{B}\rangle }_{B}$$

(7)

and

$${\langle {O}_{A}+{O}_{B}\rangle }_{A\cup B}= \frac{{\rm{tr}}(({\rho }_{A}\otimes {\rho }_{B})({O}_{A}+{O}_{B}))}{{\rm{tr}}({\rho }_{A}\otimes {\rho }_{B})}\\= \frac{{\rm{tr}}(({\rho }_{A}\otimes {\rho }_{B})({O}_{A}\otimes {{\mathbb{I}}}_{B}))}{{\rm{tr}}({\rho }_{A}\otimes {\rho }_{B})}\\ +\frac{{\rm{tr}}\left(\right.({\rho }_{A}\otimes {\rho }_{B})({{\mathbb{I}}}_{A}\otimes {O}_{B})}{{\rm{tr}}({\rho }_{A}\otimes {\rho }_{B})}\\= {\langle {O}_{A}\rangle }_{A}+{\langle {O}_{B}\rangle }_{B}$$

(8)

where 〈…〉_A∪B denotes the observable is measured in the total system A ∪ B and the coupling between A and B is zero. 〈…〉_A(B) denotes the measurement in the subsystem A (B).

Fig. 4: Separability of multipartite off-diagonal observables.

a When a large system is decomposed into several parts without coupling, the measured observable can also be separated into the product of independent components. b The off-diagonal correlations obtained via the annealing from two small part A and B. Here ${C}_{4}=\langle {S}_{1}^{x}{S}_{2}^{x}{S}_{L/2+1}^{x}{S}_{L/2+2}^{x}\rangle$. The ${S}_{1}^{x}{S}_{2}^{x}$ is set on the part A, and the ${S}_{L/2+1}^{x}{S}_{L/2+2}^{x}$ is set on the part B (i = L/2 + 1). The colorful dots are QMC results with standard error as error bars. And the dashed lines are the pure ED results.

Based on the above two equations, we can firstly decompose a large system into several independent parts without coupling and measure the observables of each part via ED. By taking the ED result as a reference point, we then employ QMC to reweight the coupling between each parts from zero to the target value. Consequently, the final observable in the total system can be obtained in this way.

For example, we assign ${S}_{1}^{x}{S}_{2}^{x}$ operator to subsystem A and another ${S}_{i}^{x}{S}_{i+1}^{x}$ operator to subsystem B. The expectation value ${\langle {S}_{1}^{x}{S}_{2}^{x}\rangle }_{A}$ and ${\langle {S}_{i}^{x}{S}_{i+1}^{x}\rangle }_{B}$ can be obtained via ED since the system size of A or B is small. Subsequently, we incrementally adjust the coupling J_AB between A and B to obtain the correlation $\langle {S}_{1}^{x}{S}_{2}^{x}{S}_{i}^{x}{S}_{i+1}^{x}\rangle$. As depicted in Fig. 4, we utilize the above annealing method to obtain the four-point off-diagonal correlation with different system size, and the reference points are obtained with small system size ${L}^{{\prime} }=L/2$ via ED. The QMC results are in excellent agreement with the pure ED results, which demonstrates the reliability of this method. From a technical standpoint, it is related to the annealing of L and r of the chain in the previous section, but this section more clearly demonstrates the power of this method from small to large sizes. In the next section, we will use this approach to calculate disorder operators in 2D systems.

Disorder operator in 2D TFIM

Here we investigate the off-diagonal measurement for the transverse Ising model (TFIM). The Hamiltonian is given as follows,

$${H}_{TFIM}=-J\mathop{\sum }\limits_{\langle i,j\rangle }{\sigma }_{i}^{z}{\sigma }_{j}^{z}-h\mathop{\sum }\limits_{i}{\sigma }_{i}^{x}$$

(9)

where σ^z/x is the Pauli spin-1/2 matrix and 〈i, j〉 means the nearest-neighbor coupling. h > 0 is transverse field term and J > 0 is the ferromagnetic term⁶. Because the TFIM only preserves Z₂ symmetry, we choose the J = 0⁺ and h = 1 as a reference point. When J = 0, the reference point $\langle {\sigma }_{i}^{x}{\sigma }_{j}^{x}\rangle=1$ since all the σ^x = 1. In the simulation, we can choose J → 0⁺ which makes $\langle {\sigma }_{i}^{x}{\sigma }_{j}^{x}\rangle$ very close to 1. The BRA formula can be expressed as $\frac{\overline{Z}(J)}{Z(J)}=\overline{Z}r/Zr\times {\langle {\sigma }_{i}^{x}{\sigma }_{j}^{x}\rangle }_{J={0}^{+}}$, where $\overline{Z}r=\overline{Z}(J)/\overline{Z}({J}^{{\prime} }={0}^{+})$ and $Zr=Z(J)/Z({J}^{{\prime} }={0}^{+})$. If we want to measure the many-body off-diagonal observables, we just need to change the $\overline{Z}(J)={\langle {\sigma }_{i}^{x}{\sigma }_{j}^{x}\rangle }_{J}$ into ${\langle {\sigma }_{1}^{x}{\sigma }_{2}^{x}\ldots {\sigma }_{n}^{x}\rangle }_{J}$. For TFIM, the QMC results in small system sizes are also well consistent with the ED^90,91, detail can be found in Supplementary Note 6.

We then mainly focus on the disorder operator of 2D TFIM on a square lattice. The disorder operator is a non-local operator which can reveal the high-form symmetry breaking and conformal field theory (CFT) information in quantum many-body systems^{92,93,94,95,96,97,98,99,100,101,102}. For 2D TFIM, we define the disorder operator $\langle X\rangle=\langle {\prod }_{i\in M}{\sigma }_{i}^{x}\rangle$ to detect the non-local information, where M is a R × R square area in the lattice. Its perimeter is l = 4R and it contains R² off-diagonal operators. This disorder operator, a multi-body off-diagonal observable, was only well measured in the QMC based on σ^x basis in the past, which is challenging to obtain directly in the σ^z basis⁹². Although the operator σ^x is contained in the TFIM Hamiltonian and can be measured in the σ^z basis in principle^27,74, it suffers from rather large fluctuations due to the requirement of a product of a series of σ^x in an area. It requires that the series of σ^x operators must appear connectedly in the time-space manifold in the σ^z basis, which is a low-probability event.

This difficulty can be overcome via BRA method. As depicted in Fig. 5, we have successfully obtained the disorder operator with different perimeters l in the paramagnetic (PM) phase and ferromagnetic (FM) phase (benchmark with the directed calculation in σ_x basis can be found in Supplementary Note 9). Here we set h = 1 and the critical point becomes J = 0.3285^73,92. For convenience, we firstly utilize the separability method in the above section to measure the disorder operator at J/h = 0.18. Taking it as a reference point, we then obtain the disorder operators for different J/h via annealing along J. In the PM phase, the disorder operator satisfies the perimeter law 〈X〉 ~ e^−al, which is consistent with the CFT prediction. In the FM phase, the disorder operator satisfies the area law $\langle X\rangle \sim {e}^{-b{l}^{2}}$, which reveals the presence of high-form symmetry⁹².

Fig. 5: The disorder operator 〈X〉 measurement in the 2D TFIM.

Setting L = 16, β = 16 and h = 1. The dashed lines are the fitting curves. a Scaling behaviors of 〈X〉 in the paramagnetic phase. b Scaling behaviors of 〈X〉 in the ferromagnetic phase. Each error bar represents the statistical one-standard-error estimates.

Imaginary-time off-diagonal correlations

Our goal becomes to extend our method to imaginary time correlation functions involving off-diagonal operators. Our discussions will concentrate on the framework of path-integral-like QMC. The first way based on the physical parameter reweighting is straightforward, which is similar to the method we have employed in the above sections. By fixing two operators at distinct points in imaginary time τ, we have observed the evolution of the imaginary-time correlation function $\langle {S}_{i}^{x}(\tau ){S}_{j}^{x}(0)\rangle$ with varying parameter Δ, as depicted in Fig. 6. This is achieved by evaluating the correlation function at several distinct imaginary-time points: τ = 0.1, τ = 1.0, τ = 3, and τ = 5. Notably, when β = 10, τ = β/2 = 5 corresponds to the maximum separation in imaginary time. The simulated values, directly comparable as $\overline{Z}r/Zr$, demonstrates excellent agreement with the ED results as shown in the subfigure (a) of Fig. 6. For a larger size L = 32 with β = 64, as Δ is tuned from 1 to 0, the imaginary-time off-diagonal correlation $\langle {S}_{i}^{x}(\tau ){S}_{i+1}^{x}(0)\rangle$ gradually becomes larger, which is analogous to the equal-time cases.

Fig. 6: The QMC results of the two-point imaginary-time correlation $\langle {S}_{i}^{x}(\tau ){S}_{i+1}^{x}(0)\rangle$.

a The ratio for fixed two-point imaginary-time correlations as the parameter Δ varies in the XXZ chain, with L = 8 and β = 10. For clarity, we plotted only 15 parameter points from the dataset, each matching the ED results (black line). All data points are calculated starting from the Heisenberg condition Δ = 1. b The imaginary-time off-diagonal correlation in the XXZ chain with L = 32 and β = 64 for τ = β/4 and τ = β/8. The associated error bars ( ± 1σ) represent the standard error.

Nontrivially, we perform the BRA measurement along the imaginary-time axis, where the distance between two inserted operators increases linearly during the annealing process, as illustrated in Fig. 7(a). For example, we focus on the measurement of 〈O(τ)O(0)〉 (the operators are indicated by gradient of colors) currently inserted at time zero and τ, the corresponding partition function for this configuration is $\overline{Z}(\tau )$. We aim to derive $\overline{Z}({\tau }^{{\prime} })$ for the off-diagonal operator at ${\tau }^{{\prime} }$ using the reweighting technique. Different from the above schemes for reweighting in which the old/new weight uses a same configuration, the measured operators O(τ) and $O({\tau }^{{\prime} })$ represent different configurations here. The solution is to construct an extended ensemble $\overline{Z}(\tau )\cup \overline{Z}({\tau }^{{\prime} })$, where $\overline{Z}(\tau )$ and $\overline{Z}({\tau }^{{\prime} })$ are the measured ensembles containing operators O(τ) and $O({\tau }^{{\prime} })$, as shown in Fig. 7(a). In this frame, the ratio $\overline{Z}({\tau }^{{\prime} })/\overline{Z}(\tau )$ can be estimated by the ratio of sampling numbers ${N}_{\overline{Z}({\tau }^{{\prime} })}/{N}_{\overline{Z}(\tau )}$, where the number ${N}_{\overline{Z}(\tau )}$ or ${N}_{\overline{Z}({\tau }^{{\prime} })}$ denotes how many times the sampling belongs to the ensemble $\overline{Z}(\tau )$ or $\overline{Z}({\tau }^{{\prime} })$. The similar spirit has been used to calculate the entanglement entropy in QMC⁶⁰. More details about this scheme are explained in the Supplementary Note 3.

Fig. 7: The diagram and results of the imaginary-time BRA method.

a illustrates the schematic of imaginary-time BRA process. The O operators are depicted by a gradient of colors, with one instance inserted and fixed at the imaginary time τ = 0, and the other moving within the time axis. If successfully moved, it corresponds physically to a transition from the imaginary time point τ to a new time point ${\tau }^{{\prime} }$. The time difference is denoted by $\Delta \tau={\tau }^{{\prime} }-\tau$. b The simulation results for the XXZ chain with L = 20 using imaginary-time BRA method are presented. The main plot displays the weight ratios with standard error for a fixed inverse temperature β = 40 and varying Δ = 0.1, 0.5, and 1.0. The small inset shows the weight ratios with error bars for fixed Δ = 0.1 and varying β = 2L, 3L, 4L, 5L. These results have not yet been multiplied by the reference values of $\langle {S}_{1}^{x}(0){S}_{2}^{x}(0)\rangle$. It can be observed that when β is sufficiently large, the furthest correlation $\langle {S}_{1}^{x}(\tau=\beta /2){S}_{2}^{x}(0)\rangle$ decays to nearly zero.

We present the numerical outcomes for the XXZ chain in Fig. 7(b). Our analysis has focused on the behavior of the weight ratio $\overline{Z}r$, across three distinct coupling strengths: Δ = 0.1, Δ = 0.5, and the Heisenberg condition Δ = 1.0. We observe that $\overline{Z}r$ initiates from the same starting point for all three curves, with the Heisenberg coupling exhibiting a more rapid decay which reflects the energy gaps in related cases. As Δ decreases, the imaginary-time off-diagonal correlation decays more slowly, indicating the S^x imaginary-time correlation is enhanced that is similar to the equal-time case. Moreover, the inset illustrates that, at Δ = 0.1, larger β makes the ratio $\overline{Z}r$ closer to zero via reducing the finite size effect in imaginary-time direction.

Furthermore, we can obtain the spectrum of operators from the momentum imaginary-time correlations via stochastic analytical continuation (SAC)^38,103,104. The momentum imaginary-time correlation is defined as ${G}^{\alpha \alpha }({\bf{q}},\tau )=\frac{1}{L}{\sum }_{i,j}{e}^{-i{\bf{q}}\cdot ({{\bf{r}}}_{i}-{{\bf{r}}}_{j})}\langle {s}_{i}^{\alpha }(\tau ){s}_{j}^{\alpha }(0)\rangle (\alpha=x,y,z)$. All the real-space off-diagonal imaginary-time correlation can be captured by the above imaginary-time BRA method, which is used to stimulate the excitation spectrum S^αα(q, ω). As shown in Fig. 8, the off-diagonal spectrum has sharper lower boundary with weak continuum, which is different from the diagonal spectrum that has strong spinon continuum on the upper boundary (ED results can be found in Supplementary Note 4). Since Δ = 0.1 here is close to zero, the difference of the diagonal and off-diagonal spectra can be understood qualitatively from the limit Δ = 0. When Δ = 0, the off-diagonal excitation can be solved by the Jordan-Wigner transformation, which is related to a single-mode dispersion of free fermion, thus its excitation is sharp. Meanwhile, the diagonal spectrum S^zz corresponds to two fermion operators, which contributes a continuum therefore. The results demonstrate that our BRA method can be successfully applied to extract the off-diagonal spectrum, which also reveals the different excitation modes compared to the diagonal spectrum with the anisotropic phase.

Fig. 8: The spin excitation spectrum obtained from the SAC method with L = 20, Δ = 0.1 and β = 80.

The imaginary-time correlations are extracted from the above BRA method. a The diagonal operator spectrum S^zz(q, ω). b The off-diagonal operator spectrum S^xx(q, ω).

Discussion

In addition to the sign problem in the original ensemble Z (denominator), the numerator $\overline{Z}$ may also exhibit a sign problem. It involves another sign problem within this BRA measurement scheme because we have to calculate the ratio of $\overline{Z}$ with different parameters. For example, when calculating the operator ${\sigma }^{y}=-i| \uparrow \rangle \langle \downarrow |+i| \downarrow \rangle \langle \uparrow |$, it introduces an extra sign of i or  − i into the weight, contrasting with the case of ${\sigma }^{x}=| \uparrow \rangle \langle \downarrow |+| \downarrow \rangle \langle \uparrow |$. If we attempt to reweight the general PF containing the measured operator σ^y, denoted as ${\overline{Z}}_{y}$, the simulation of ratio ${\overline{Z}}_{y}({J}^{{\prime} })/{\overline{Z}}_{y}(J)$ would encounter sign problem. A simple way is to calculate the ratio of ${\overline{Z}}_{y}/{\overline{Z}}_{x}$, where ${\overline{Z}}_{x}$ is the ensemble with the measured operator σ^y replaced by σ^x. Note that ${\overline{Z}}_{y}={\sum }_{i}{W}_{i}$ and ${\overline{Z}}_{x}={\sum }_{i}| {W}_{i}|$. As is commonly used in calculating sign value^45,47,48,105, ${\overline{Z}}_{y}$ represents the sign system and ${\overline{Z}}_{x}$ is the reference system. The ratio ${\overline{Z}}_{y}/{\overline{Z}}_{x}$ can then be extracted by sampling the reference system ${\overline{Z}}_{x}={\sum }_{i}| {W}_{i}|$, averaging the sign of each configuration in the sign system (${\overline{Z}}_{y}={\sum }_{i}{W}_{i}={\sum }_{i}{sign}_{i}| {W}_{i}|$), and ultimately obtaining ${\overline{Z}}_{y}/{\overline{Z}}_{x}=\langle sign\rangle$. Finally, the target observable ${\overline{Z}}_{y}/Z$ can be derived via ${\overline{Z}}_{y}/{\overline{Z}}_{x}\times {\overline{Z}}_{x}/Z$.

In summary, we propose a variety of detailed schemes in the frame of bipartite reweight-annealing to achieve universal measurement by QMC simulation. Typically, we perform annealing along a physical parameter for the PFs $\overline{Z}(J)$ and Z(J) independently, then connect them via an easily solvable point such as $\overline{Z}({J}^{{\prime} })/Z({J}^{{\prime} })$. Thereafter, this concept has been extended to annealing of system size and imaginary time. For example, it is easy to employ ED to calculate the observables in each independent parts and anneal their couplings to construct a large system and solve the target measurement problem. The dynamical behaviors of off-diagonal operators have also been addressed in this work. Off-diagonal spectrum is no longer a natural moat in the quantum many-body computation. Within this framework, the long-standing problem for the measurement of QMC has been addressed in a general way.

Essentially, we solve the problem of calculating the overlap between different distribution functions, which is a fundamental challenge in mathematical statistics. The spirit of BRA can be easily generalized to the measurement of entanglement^106,107,108 and other statistical problems, such as machine learning^109,110,111.

Methods

Our work proposes a general measurement framework based on the SSE-QMC method, the principle of which has been introduced in the main text. The XXZ model and TFIM model involved employ directed-Loop algorithm and cluster SSE algorithm (detail can be found in Supplementary Note 1 and Note 6), respectively, which will be found in detail in the supplementary information.