Observation of plaquette fluctuations in the spin-1/2 honeycomb lattice

Abstract

Quantum spin liquids are materials that feature quantum entangled spin correlations and avoid magnetic long-range order at T = 0 K. Particularly interesting are two-dimensional honeycomb spin lattices where a plethora of exotic quantum spin liquids have been predicted. Here, we experimentally study an effective S = 1/2 Heisenberg honeycomb lattice with competing nearest and next-nearest-neighbour interactions. We demonstrate that YbBr₃ avoids order down to at least T = 100 mK and features a dynamic spin–spin correlation function with broad continuum scattering typical of quantum spin liquids near a quantum critical point. The continuum in the spin spectrum is consistent with plaquette type fluctuations predicted by theory. Our study is the experimental demonstration that strong quantum fluctuations can exist on the honeycomb lattice even in the absence of Kitaev-type interactions, and opens a new perspective on quantum spin liquids.

Introduction

Magnetism arises because of the quantum mechanical nature of the electron spin, yet for the understanding of many materials, particularly those used in today’s applications, a classical approach is sufficient. Materials with strong quantum fluctuations are rare, but attract significant research attention since they hold enormous potential for future technologies¹ that make use of the long-range entanglement for quantum communication^2,3. Fault-tolerant quantum computers are proposed to operate with anyon quasi-particles² which exist in a class of quantum spin liquids^4,5.

Quantum spin liquids (QSL) are caused by quantum fluctuations which reduce the size of the ordered magnetic moment of static magnetic structures and can affect the dynamics of the spin excitations. This happens in the S = 1/2 frustrated antiferromagnetic square lattice, with competing nearest and next-nearest-neighbour interactions, J₁, J₂, where the zone boundary spin-waves develop a dispersion due to the presence of quantum dimer-type fluctuations between nearest neighbours⁶. These fluctuations are similar to the resonant valence bond fluctuations predicted in the frustrated triangular lattice⁷, which are believed to be relevant for high-temperature superconductors⁸. Frustration can be induced by competing interactions and depending on their relative strength, incommensurate magnetic phases, valence bond solids with periodic ordering of local quantum states, or QSLs with different symmetry are theoretically predicted^{4,9,10,11,12,13,14,15}. In particular, it is expected that frustration enforces a quantum phase transition at which fractionalization of magnons into deconfined spinons occurs¹⁶.

It has been a challenge to identify and understand appropriate model systems to study QSLs. In general, lowering the dimension will increase quantum fluctuations. In one-dimension QSLs have been identified in antiferromagnetic (AF) spin chains. Case in point are KCuF₃¹⁷ and Cu(C₆D₅COO)₂ ⋅ 3D₂O¹⁸. In two- and three-dimensions, quantum fluctuations can be enhanced by frustration, and there are several routes to achieve this: The inherent geometrical frustration of kagome¹⁹, triangular²⁰, spinel²¹ and pyrochlore²² lattices may prohibit long-range ordering at low temperatures. Another promising candidate is the honeycomb lattice which has received relatively little attention until Kitaev’s work⁴ when it was realized that bond-dependent anisotropic interactions can stabilize a new form of QSL whose properties are known exactly. Representatives materials are α-RuCl₃²³, Li₂IrO₃²⁴ and H₃LiIr₂O₆²⁵ which show signatures of spin correlations due to quantum entanglement.

Quantum fluctuations are enhanced in the honeycomb lattice compared to the square lattice since the number of neighbours of each spin is lower, thus placing it closer to the quantum limit. When next-nearest-neighbour frustrating exchange interactions are sufficiently large compared to the nearest-neighbour exchange, theories predict a quantum phase transition from a Néel ground state into a quantum entangled state. However, there is no consensus on the nature of this ground state: Theories predict either a QSL^11,12 or a plaquette valence bond crystal (pVBC)^13,14,15 with different magnetic excitations which include spinons¹¹, rotons¹⁵ or plaquette fluctuations²⁶.

Here, we study the magnetic properties of the trihalide two-dimensional compound YbBr₃ that forms a realization of the undistorted S = 1/2 honeycomb lattice with frustrated interactions. Short-range magnetic correlations between the Yb moments develop below T ≈ 3 K, but the correlation length is only of the order of the size of an elementary honeycomb plaquette at T = 100 mK, consistent with a QSL ground state. Despite this short correlation length, inelastic neutron measurements reveal well-defined dispersive low energy magnetic excitations close to the Brillouin zone centre. At high energies and at the zone boundary, we observe a continuum of excitations that we interpret as quantum fluctuations on an elementary hexagonal plaquette.

Results

Crystal structure and susceptibility

YbBr₃ crystallizes with the BiI₃ layer structure in the rhombohedral space group $R\bar{3}$ (148), where the Yb ions form perfect two-dimensional (2D) honeycomb lattices perpendicular to the c-axis, as shown in Fig. 1. The atomic positions are given in Table 1 in Supplementary Note 1. The temperature dependence of the magnetic susceptibility has a broad maximum around T = 3 K, but as shown below, there is no evidence for long-range magnetic order down to at least T = 100 mK. In the low-temperature regime below 10 K we observe χ^a ≈ 1.3χ^c which reflects a small easy-plane anisotropy.

Fig. 1: Magnetic susceptibility, crystal-electric field and crystal structure.

a View along [210] on the unit cell of YbBr₃. b Yb³⁺ honeycomb layer. c Temperature dependence of the magnetic susceptibility χ of YbBr₃ for field orientations along the a- and c-axes. Solid lines are the calculated single-ion susceptibilities based on the crystal-field (CEF)-parameters (see Supplementary Note 2). Inset: Measured low-temperature susceptibility showing a rounded peak around T = 2.75 K. Error bars are standard deviations but are smaller than the data points.

The rare-earth ion Yb³⁺ features a J = 7/2 ground-state multiplet that is split by the crystal-electric field (CEF), giving rise to a total of four Kramers doublets with the three excited CEF levels being observable via neutron scattering. The first excited level is observed at ~15 meV (see Supplementary Fig. 1) and the ground-state doublet is an effective S = 1/2 state. From an analysis of the measured susceptibility and the inelastic neutron data, we obtain the CEF parameters listed in Supplementary Note 2. They result in ground state expectation values of 〈J_⊥〉 = 1.2 and 〈J_∥〉 = 0.8 where the subscript indicates spin orientations measured relative to the c-axis.

Magnetic ground state

Figure 2a shows the neutron diffraction pattern of the energy integrated magnetic scattering of Yb₃ that was determined as the difference between diffraction patterns taken at T = 100 mK and T = 10 K in order to eliminate the contributions of nuclear scattering. No magnetic Bragg peaks are visible in the diffraction pattern, demonstrating that YbBr₃ avoids magnetic order down to at least this temperature.

Fig. 2: Magnetic diffuse scattering and correlation length.

a Magnetic diffuse scattering in YbBr₃ in the [h, k, 0] plane at T = 100 mK, after subtraction of the nuclear Bragg contribution. b Calculated magnetic diffuse scattering based on the spin-wave model including exchange and dipolar interactions. c Cut through the diffuse scattering along the (h, 0, 0) direction. The line is a fit to the data with a Lorentzian function convoluted with the instrumental resolution approximated by a Gaussian. [Note that the presence of paramagnetic scattering at 10 K leads to a negative background in the 100 mK data after subtraction.] Error bars are standard deviations. d Constant-energy scan for $\hbar$ω = 0.4 meV in YbBr₃ at T = 250 mK showing well-defined low energy excitations. The solid line represents the computed inelastic neutron scattering cross-section. Observed small peaks are due to spurious scattering and are not included in the model calculation. Error bars are standard deviations.

Diffuse magnetic scattering is centred at (1, 0, 0) and equivalent wave-vectors, which implies that the short-range correlations are described by a propagation vector Q₀ = (0, 0, 0). Figure 2b shows the diffuse scattering as obtained from the 2D spin-wave theory described below, which reproduces both position and intensity of the observed diffuse scattering.

Figure 2c shows a cut along the Q = (q, 0, 0) direction which reveals diffuse scattering with Lorentzian line shape that reflects short-range magnetic order²⁷. From a fit to the neutron intensity I ∝ κ²/(q² + κ²), we determine an in-plane correlation length between the Yb moments of ξ = 1/κ ≈ 10 Å at T = 100 mK, comparable to the fourth nearest-neighbour distance of 10.66 Å which is ~1.25 times the diameter of an Yb₆-hexagon plaquette.

Magnetic excitations

We measured well-defined magnetic excitations at T = 250 mK along three cuts in the hexagonal plane. Within experimental resolution, we observed a single excitation branch and no spin gap at the zone centre. As shown in the constant-energy-scans in Fig. 2d and in Fig. 3, the magnetic excitations are sharp close to the Brillouin zone centre. One of the key results of this study is the observation of a broadening of the spectrum when the dispersion approaches the zone boundary, as shown in Fig. 3. In fact, the inelastic neutron spectrum close to the zone boundary exhibits a continuum which extends to over twice the energy of the well-defined magnetic excitation. While low-lying excitations are sharp, these broad excitations are only observed at higher energies.

Fig. 3: Magnetic excitations along high-symmetry directions.

False colour plot of the observed (top) and calculated (bottom) inelastic neutron cross-section of the magnetic excitations in YbBr₃ at T = 0.25 K. The intensity is shown on a logarithmic scale. Note the existence of a continuum of excitations around (1/2, 1/2, 0) and (1/2, −1, 0) which is not described by spin waves and is indicative of plaquette fluctuations (see Fig. 5).

While it may appear surprising that we observe well-defined excitations even in the presence of a correlation length of merely 10 Å, this agrees with the predictions of Schwinger-Boson²⁸ and modified spin-wave²⁹ theories which show that spin waves can propagate in low-dimensional systems with short-range Néel order. The well-defined excitations in YbBr₃ can be described by an effective S = 1/2 Hamiltonian including nearest and next-nearest-neighbour Heisenberg exchange coupling, and dipolar interactions between the CEF ground-state doublets,

$$H=-\frac{1}{2}\mathop {\sum}\limits_{i,j}\mathop {\sum}\limits_{\alpha ,\beta }{{\mathcal{J}}}_{\alpha ,\beta }(i,j){S}_{i}^{\alpha }{S}_{j}^{\beta },$$

(1)

where ${{\mathcal{J}}}_{\alpha ,\beta }(i,j)={g}_{\alpha }^{2}{\delta }_{\mathrm{\alpha \beta }\,}J(i,j)+{g}_{\alpha }{g}_{\beta }{D}_{\alpha ,\beta }(i,j)$, with α, β = x, y, z cartesian coordinates of the hexagonal cell, and ${S}_{i}^{\alpha }$ is the α-component of a spin-1/2 operator at site i. Here, J(i, j) are the exchange coupling constants between distinct sites i and j, while D(i, j) denotes the dipolar interactions. For the calculation of the spin-wave dispersion, we use the random-phase approximation (RPA) around the Néel state with spins in the hexagonal plane and S = 1/2 (see Supplementary Note 3). Our measurements allow the determination of the exchange couplings, while the dipolar coupling is fixed by the magnetic moment. As shown in Fig. 3, we find good agreement between measured and calculated spin-wave dispersions. The nearest- and next-nearest-neighbour exchange interactions J₁, J₂ are obtained from a least-square fit to the data. We obtained g²J₁ = −0.69(8) meV and g²J₂ = −0.09(2) meV that correspond to a calculated ground state with Q₀ = (0, 0, 0) (see Supplementary Note 4). We note that our spin-wave theory does not describe all aspects of our experimental results: It predicts an optical branch for values of the easy-plane anisotropy that corresponds to the measured susceptibility (see Fig. 1), while we do not find experimental evidence for such a second branch. Also, it does not explain the existence of an excitation continuum as we shall discuss next.

Continuum of excitations

As shown in Fig. 3, the magnetic excitation spectrum also features weaker broad scattering at energies where the optical branch is expected. This is particularly evident near the M-points at (0.5, 0.5, 0) and (0.5, −1, 0), where the excitations extend to 0.8–1 meV and are reminiscent of scattering observed in other low-dimensional antiferromagnets^30,31. In most materials, spin-waves are long-lived excitations that are resolution-limited as a function of energy. When the spin waves are damped or interact with other spin-waves they have a finite life-time and the line-shape of the dynamical structure factor S(Q, ω) broadens³². We have simulated the line-shape of S(Q, ω) derived from our model and convoluted it with the resolution of the spectrometer obtained from the Takin software³³ (see ‘Methods’). While the spin-wave model adequately explains the dispersion and intensity distribution close to the Brillouin zone centres, it does not reproduce the inelastic neutron line-shape close to the maximum of the dispersion of the spin-wave branch as shown in Fig. 4.

Fig. 4: Excitation continuum near the Brillouin zone boundary.

Observed and simulated magnon spectra based on the spin-wave model explained in the text. The lines are the results of a Monte–Carlo simulation of the RPA result convoluted with the instrumental resolution function. In the RPA calculation, a linewidth of ϵ = 0.05 meV is assumed. The shaded area indicates the continuum of excitations. Error bars are standard deviations.

Discussion

Although YbBr₃ only exhibits short-range magnetic order, the dispersion of the sharp magnetic excitations can be well described by a spin-1/2 Heisenberg Hamiltonian with easy-plane anisotropy and dipolar interactions. For a honeycomb lattice classical theories predict instability of the Néel state for J₂/J₁ ≈ 0.1^10,29 and quantum fluctuation in linear spin-wave theory destroy long-range Néel order. We note that other theoretical approaches find that quantum fluctuations may stabilize the Néel phase up to somewhat higher ratios of J₂/J₁. These approaches include Schwinger-Boson approach¹¹, variational wave functions^15,34 and exact diagonalization¹³ which all yield a critical ratio J₂/J₁ ≈ 0.2. Since we find only magnetic short-range correlations between the Yb moments, we conclude that YbBr₃ must be in close proximity of such a quantum phase transition.

In YbBr₃ the Yb-ion has a large magnetic moment of the order of 2μ_B and therefore the dipolar interactions cannot be neglected. At the classical level, one can show that they favour antiferromagnetic Néel order with the spins along the c-axis³⁵ enabled by a spin gap at the zone centre of ~200 μeV. This spin gap caused by the dipolar interaction is reduced by the CEF easy-plane anisotropy which contributes to a destabilization of the Néel state at finite temperature (see Fig. 2. in Supplementary Note 3). At g_crit ≈ g_zz/g_xx ≡ 0.985 the spin gap closes and quantum fluctuations will be enhanced. Below that value, the spins rotate into the basal plane. Linear spin-wave theory predicts that easy-plane anisotropy entails a lifting of the degeneracy of the two spin-wave branches at the zone centre, and the splitting increases with increasing anisotropy. A large anisotropy in YbBr₃ would then become measurable since the branch separation becomes large enough to be resolved. A computation of S(Q, ω) at g_crit is shown in Fig. 3 and describes the observed dispersion and intensities of the sharp excitations very well.

Experimentally, we have observed neither a splitting of spin waves nor a spin gap within the available energy resolution. This suggests that the absence of long-range order in YbBr₃ at T = 100 mK is caused by the competition between easy-plane anisotropy which favours spins in the plane, and dipolar interactions that favour Ising order along the c-axis. These opposing trends will enhance quantum fluctuations which places YbBr₃ close to the quantum critical point towards a QSL of the spin-1/2 Heisenberg Hamiltonian on the honeycomb lattice.

Our experiment provides clear evidence for the presence of a continuum of excitations at high energies in YbBr₃. We observe that the intensity of the continuum is stronger at the M’ points along (h, −1, 0) and (h, h, 0) directions whereas it is weak along (0, k, 0) and at the Γ and ${\Gamma }^{\prime}$ points, in contrast to calculations of the two-magnon cross-section for the Heisenberg Hamiltonian (see Supplementary Fig. 3). We found, as shown in Fig. 5, that this modulation of the neutron intensity associated with the continuum can be reproduced by a RPA calculation for a hexamer plaquette with the exchange parameters obtained from the spin-wave calculations (see ‘Methods’). This picture of local excitations in YbBr₃ is supported by an analogous calculation of the magnetic susceptibility which shows a broad maximum at T ≃ 4 K (see Fig. 4 in Supplementary Note 5). Similar excitations associated with small spin clusters were also observed in the spinel lattice³⁶. Our neutron measurements are also in agreement with recent Monte–Carlo calculations of the dynamical structure factor for the frustrated honeycomb lattice¹⁵ that show a deconfined two-spinon continuum³⁷ with enhanced intensity at the zone boundary due to proximity of a quantum critical point.

Fig. 5: Calculated neutron form factor for a plaquette.

The neutron form factor of an Yb₆ hexagon calculated within the random-phase approximation at 0.5 meV and assuming Neel order on the plaquette. The directions of the neutron measurement are indicated by white arrows: cut I corresponds to (h, h, 0), cut II is (0, −k, 0) and cut III is (h, −1, 0). High-symmetry points are labelled similar to Fig. 3. Basis vectors of the reciprocal lattice are denoted as a*, b*.

In summary, we have shown that the magnetic ground state of YbBr₃ exhibits only short-range order well below the maximum in the static susceptibility. Analysis of the dispersion of the magnetic excitations reveals competition between the nearest-neighbour and next-nearest-neighbour exchange interactions, but we did not observe the mode softening at the K-point which has been predicted¹⁵ for large J₂. This could be related to the large experimental uncertainty in the value of J₂. Also, it is known that a large value of J₂ is not necessary for the existence of fractional excitations in 2D systems³⁸. However, an unfrustrated Heisenberg model (J₂ = 0) would for our value of J₁ result in a long correlation length at T = 100 mK that corresponds to a narrow resolution-limited peak which is at variance with our observations³⁹. We observed a continuum of excitations with the spectrum of excitations extending to approximately twice the energy of the position of the maximum in S(Q, ω). The neutron inelastic intensity due to the continuum follows the modulation expected for the fluctuations of a honeycomb spin plaquette. Our results demonstrate that YbBr₃ is a two-dimensional S = 1/2 system on the honeycomb lattice with spin-liquid properties without Kitaev-type interactions. The observation of the continuum associated with localized plaquette excitations supports the view of a deconfined quantum critical point⁴⁰ in the frustrated honeycomb lattice, in agreement with results from coupled-cluster methods, density matrix renormalization group calculations and Monte–Carlo simulations^14,15,41. Our measurements set a quantitative benchmark for future theoretical work.

Methods

Crystal growth and sample preparation

An YbBr₃ single crystal of cylindrical shape (15-mm diameter, 18-mm height) was grown from the melt in a sealed silica ampoule by the Bridgman method, as previously described for ErBr₃⁴². YbBr₃ was prepared from Yb₂O₃ (6N, Metall Rare Earth Ltd.) by the NH₄Br method⁴³ and sublimed for purification. All handling of the hygroscopic material was done under dry and O-free conditions in glove boxes or closed containers.

Magnetic susceptibility

The magnetic susceptibility was determined with a MPMS SQUID system (Quantum Design).

Neutron scattering experiments

The neutron experiments were performed at the Swiss Spallation Neutron Source (SINQ) utilizing different instruments. On all instruments, filters were used to reduce contamination of the beam by higher-order neutron wavelengths.

The crystal structure of YbBr₃ was refined using diffraction data collected with the high-resolution powder diffractometer HRPT at the wavelength of λ = 1.494 Å at room temperature. The crystal structure and lattice parameters were refined with Fullprof.

The magnetic ground state was investigated with the multi-counter diffractometer DMC at the wavelength λ = 2.4576 Å which integrates fluctuations up to a maximum of ~13.5 meV. The measured neutron intensity is proportional to the equal time spin–spin correlation function.

The crystal-field splitting of the Yb³⁺ ions was determined on the thermal three-axis spectrometer EIGER operated in the constant final-energy mode with k_f = 2.662 Å⁻¹ at T = 1.5 K and ∣Q∣ = 1.5 Å⁻¹. With that configuration the energy resolution is 0.8 meV.

The dispersion of the magnon excitations is bound by $\hbar$ω(q) < 1 meV in YbBr₃ which required the use of cold neutrons that provide an improved energy resolution. Therefore the measurements of the spin waves were performed with the TASP three-axis spectrometer using k_f = 1.3 Å⁻¹ which resulted in an energy resolution of 80 μeV. To maximize the intensity, the measurements were performed without collimators in the beam and the analyser was horizontally focusing.

Magnetic excitations

We analysed the dispersion of the magnetic excitations with a Heisenberg Hamiltonian,

$${H}_{{\rm{h}}}=-\frac{1}{2}\mathop {\sum}\limits_{i,j}\mathop {\sum}\limits_{\alpha }{g}_{\alpha }^{2}\ J(i,j)\ {S}_{i}^{\alpha }{S}_{j}^{\alpha }.$$

(2)

J(i, j) are the exchange constants between sites i and j, to be determined experimentally, the anisotropic g-factors reflect the cystal-field anisotropy where α = x, y, z denotes Cartesian coordinates, and ${S}_{i}^{\alpha }$ denotes the α component of a spin-1/2 operator at site i. For Heisenberg interactions g_x = g_y = g_z ≡ g. Because the magnetic moment of Yb³⁺ is large, we also consider the dipolar interactions,

$${H}_{{\rm{dip}}}=-\frac{{\mu }_{0}{\mu }_{B}^{2}}{8\pi }\mathop {\sum}\limits_{i,j}\mathop {\sum}\limits_{\alpha ,\beta }{g}_{\alpha }{g}_{\beta }{D}_{\alpha ,\beta }(\mathrm{ij}\,){S}_{i}^{\alpha }{S}_{j}^{\beta },$$

(3)

with

$${D}_{\alpha ,\beta }(\mathrm{ij}\,)=\frac{3{({R}_{\mathrm{ij}\,})}_{\alpha }{({R}_{\mathrm{ij}\,})}_{\beta }}{{{\bf{R}}}_{\mathrm{ij}\,}^{5}}-\frac{1}{{{\bf{R}}}_{\mathrm{ij}\,}^{3}}{\delta }_{\alpha ,\beta }.$$

(4)

where R_ij ≡ R_j − R_i is the relative position vector between the j’th and i’th ion.

The dispersion of magnetic excitations was calculated within the RPA where the spin waves appear as poles in the dynamical tensor $\overline{\overline{\chi }}({\bf{q}},\omega )$,

$$\overline{\overline{\chi }}({\bf{q}},\omega )={[\overline{\overline{1}}-{\overline{\overline{\chi }}}_{0}(\omega )\overline{\overline{M}}({\bf{q}})]}^{-1}{\overline{\overline{\chi }}}_{0}(\omega )$$

(5)

with $\overline{\overline{M}}({\bf{q}})$ the Fourier transform of the exchange and dipolar interactions and ${\overline{\overline{\chi }}}_{0}(\omega )$ the single-ion susceptibility. The neutron cross-section is proportional to the imaginary part of the dynamical susceptibility⁴⁴,

$$\frac{{d}^{2}\sigma }{d{\Omega} dE\,}\propto \ \mathop {\sum}\limits_{\alpha ,\beta }\left({\delta }_{\alpha ,\beta }-\frac{{Q}_{\alpha }{Q}_{\beta }}{| {\bf{Q}}{| }^{2}}\right)\ {S}^{\alpha ,\beta }({\bf{Q}},\omega ),$$

(6)

where we defined the dynamical structure factor,

$${S}^{\alpha ,\beta }({\bf{Q}},\omega )=\frac{1}{\pi }\frac{1}{1-\exp (-\hslash \omega/{k}_{B}T)}\mathop {\sum}\limits_{u,v}\ \Im {\chi }_{u,v}^{\alpha ,\beta }({\bf{Q}},\omega ).$$

(7)

Here Q denotes the scattering vector, and u, v labels the Yb ions in the magnetic cell. To analyse the data, the scattering cross-section was convoluted with the resolution of the spectrometer using Popovici method implemented in Takin³³, and the corresponding Monte–Carlo results are shown in Fig. 4.