The nature of low-temperature spin-freezing in frustrated Kitaev magnets

Abstract

The subtle interplay between competing degrees of freedom, anisotropy, and spin correlations in frustrated Kitaev magnets offers an ideal platform to host non-trivial spin freezing and exotic low-energy excitations. We elucidate low-temperature spin freezing as evidenced by thermodynamics, NMR, and inelastic neutron scattering (INS) results in frustrated Kitaev magnets adopting Halperin and Saslow (HS) and spin jam frameworks. The temperature dependence of specific heat (C_m) shows a broad maximum, indicating short-range spin correlations, while the T² dependence of C_m below the spin-glass temperature (T_g) suggests gapless excitation spectra. The aging and memory effect experiments suggest a non-hierarchical free energy distribution, which differs from the hierarchical organization of conventional spin freezing. The NMR spin-lattice relaxation rate follows a power law behavior below T_g, suggesting exotic spin excitation spectra that are supported by the INS. The INS susceptibility is proportional to the energy transfer, corroborating the emergence of topological spin freezing, which can be explained by the HS hydrodynamic modes. HS modes account for instigating non-Abelian defect propagation, thereby inducing a spin jam state in the low-T regime in Kitaev magnets. Our work captures the essence of topological spin freezing, characterized by macroscopic ground state degeneracy, short-range spin correlations, and linearly dispersive low-energy excitations in frustrated Kitaev magnets.

Introduction

The synergistic interplay between competing degrees of freedom and anisotropy conspire with frustration-induced quantum fluctuations, offers a viable basis to realize an exotic quantum state with low energy fractional excitations, promising to address some of the enduring themes in quantum condensed matter^1,2,3. The non-trivial ground states of frustrated quantum magnets at T → 0, characterized by smooth energy landscapes, can preclude localization in condensed matter⁴. Quenched disorder such as atomic vacancies or randomness in exchange interactions in frustrated quantum magnets plays a vital role in deforming smooth energy landscapes into rugged energy ones, thereby leading to localization and the emergence of glassiness characterized by slow spin dynamics⁵. The emergence of glassiness of topological nature in the low-temperature limit of disorder-free frustrated Mott insulators is attributed to macroscopic ground state degeneracy, unusual spin correlations, and exotic low energy excitations⁶. Landau theory of symmetry breaking is quite successful in describing the conventional phase transitions associated with local order parameters. However, Landau’s theory is inadequate in capturing the central essence of quantum and topological phases such as quantum spin liquids, quantum criticality, high-temperature superconductivity, and the fractional Hall effect that are driven by non-thermal control parameters and interplay between emergent degrees of freedom and electron correlations. The concept of topological order⁷ was introduced to elucidate the phase transition of such unconventional quantum states under the mathematical formulation of topology. Topologically ordered states are robust against perturbations and are characterized by topological degeneracy, fractional excitations, emergent gauge fields, non-local entanglement, and finite topological entropy at zero temperature that have far-reaching implications in advancing our understanding of fundamental physics and set a stage for the next-generation technologies⁷.

A quantum spin liquid state (QSL) is a highly entangled state of quantum matter without a symmetry-breaking phase transition down to absolute zero temperature and is characterized by fractional excitations and long range entanglement^3,8. QSL was proposed for S = 1/2 moments embodying a Heisenberg triangular lattice⁸; however, the triangular lattice with nearest-neighbor isotropic Heisenberg exchange interaction host magnetically ordered state. Later, it was demonstrated that the next-nearest neighbor exchange interaction, magnetic anisotropy, and higher-order interactions can stabilize QSL in frustrated magnets. In this context, the spin-orbit driven frustrated honeycomb magnets with bond-dependent highly anisotropic interactions between J_eff = 1/2 degrees of freedom (see Fig. 1a, b) provide a natural habitat to host the celebrated Kitaev QSL state characterized by a spectrum of deconfined fractional excitations such as Majorana fermions². Moreover, the external perturbations, such as magnetic field, break time-reversal symmetry and stabilize a spin gap, supporting the excitation of non-Abelian anyons in the QSL state^2,9. The exactly solvable Kitaev Hamiltonian consists of spin-orbit driven bond-dependent nearest neighbor Ising interactions between J_eff = 1/2 moments generated owing to crystal electric field and spin-orbit coupling (see Fig. 1a, b) and is represented as ${{{\mathscr{H}}}}=-{\sum }_{ < ij > }{K}_{\gamma }{S}_{i}^{\gamma }{S}_{j}^{\gamma }$, where γ = x, y, z and K_γ is the bond dependent Kitaev exchange with the nearest neighbor sites i and j as shown in Fig. 1a². The emblematic Kitaev model hosts a solution where the spin-1/2 is fractionalized into four fermions, and alternatively, one can write the Hamiltonian as ${{{\mathscr{H}}}}=-\frac{1}{4}{\sum }_{i,j}{K}_{\gamma }{b}_{i}^{\gamma }{b}_{j}^{\gamma }{c}_{i}{c}_{j}$. Subsequently, a set of four Majorana fermions emerges with three immobile fermions {b_x, b_y, b_j} and one mobile fermion c⁹. The manifestation of Majorana fermions engenders the emergence of a Z₂ gauge field. Recently, an emergent glass-like state was observed in the celebrated Kitaev magnet α-RuCl₃ in the intermediate magnetic field limit, as evidenced by the non-linear magnetic susceptibility¹⁰ wherein the enhancement of non-linear susceptibility around the spin-glass temperature is related to the fluctuation of magnetic moments in the host spin-lattice⁴. The density-matrix renormalization group method at zero temperature in the intermediate field suggests the emergence of glassiness owing to the slowing down of Z₂ fluxes in the proximity of the U(1) spin liquid region¹¹. In many-body localized systems, the interplay of many-body interactions and disorder gives rise to an emerging integrability that puts a strong constraint on thermalization¹². Recent calculations on one-dimensional spin-1 Kitaev spin chain show the fragmentation of the Hilbert space into unequal disconnected subspaces¹³. In integrable spin systems, the sheer abundance of conserved quantities limits the ability of an initial state to fully navigate all feasible configurations in the Hilbert space and lacks self-thermalization in isolation, leading to the emergence of weak ergodicity breaking in Kitaev materials^13,14. In honeycomb Kitaev model, the conserved quantities, for instance, take the form of flux operators defined around each hexagon, expressed as a product of six spin operators defined as ${W}_{1{\mbox{-}}6}={2}^{6}{S}_{1}^{z}{S}_{2}^{x}{S}_{3}^{y}{S}_{4}^{z}{S}_{5}^{x}{S}_{6}^{y}$ (see Fig. 1a)¹⁵. The interplay between the integrability¹⁶ inherent in the Kitaev model and the presence of quenched disorder¹⁷, resulting in many-body localization, can lead to the intrinsic glassiness in frustrated honeycomb lattices.

Fig. 1: Frustrated Kitaev magnets—an emblematic model.

a A schematic representation of a frustrated Kitaev magnet decorated on a honeycomb lattice with bond-dependent anisotropic exchange interactions. b The combined effect of crystal electric field (CEF) and spin-orbit coupling in the 4d⁵ and 5d⁵ based quantum magnets realizes a low-energy J_eff = 1/2 state, which has the potential to host Kitaev physics. c A schematic of the temperature dependence of magnetic susceptibility, depicting ZFC-FC splitting below the spin-glass temperature in the honeycomb material Li₂RhO₃ (reproduced with permission from ref. ⁴⁴, ⓒ2017 American Physical Society). The inset shows a schematic representation of a free-energy landscape in a frustrated spin glass, where the solid lines correspond to possible free-energy configurations in zero-field cooling, while the dashed convex envelope represents the equilibrium free-energy configuration in the presence of a magnetic field.

Understanding the origin of spin-freezing in frustrated magnets is of paramount importance, which may provide vital clues for the experimental realization of topological states including the elusive QSL state and associated fractional quantum numbers. Frustration leads to massive ground state degeneracy and low-energy excitations with topological characteristics that are reflected as non-trivial behavior of magnetization, specific heat, NMR relaxation rate, and imaginary part of dynamic spin susceptibility in inelastic neutron scattering in frustrated spin glass in contrast to conventional spin-glass materials¹⁸. The experimental search for the topological origin of spin freezing began with the observation of non-trivial spin glass behavior in the magnetopumbites SrCr_9pGa_12−9pO₁₉¹⁹ (SCGO) and spinel Ba₂Sn₂ZnCr_7pGa_10-7pO₂₂²⁰ (BSZCGO) decorated on a kagome lattice. In contrast to dilute magnetic alloys, these frustrated magnets feature densely populated magnetic ions potential to exhibit characteristic features that are strikingly different from their canonical counterparts. The manifestation of a glassy state is characterized by the splitting of ZFC-FC magnetic susceptibility (see Fig. 1c) in these systems, which is ascribed to quenched disorder and frustration-induced quantum fluctuations. This unique topological state is commonly referred to as a ‘spin jam’^21,22. In such geometrically frustrated systems, the mechanism of “order-by-disorder” driven by quantum fluctuations perturbs the classical ground state degeneracy leading to a metastable state within the complex and rugged energy landscape⁴. The spin jam state can be well explained by Halperin-Saslow (HS) mode. Halperin and Saslow proposed a hydrodynamic mode in the background of frozen spins, and a linear dispersion relation is predicted below the freezing temperature²³. HS framework effectively captures the essence of low-temperature spin freezing in the S = 1 triangular lattice antiferromagnet Ni₂Ga₂S₄ with easy-plane magnetic anisotropy, wherein short-range spin correlations, non-trivial spin fluctuations, and anisotropic spin dynamics govern the low-temperature magnetism^24,25,26. In 2D frustrated quantum magnets, the T² behavior of magnetic specific heat (C_m ∝ T²) can be described by the mean-field low-energy excitations termed as ‘spaghetti modes’, characterized by a length scale L₀ extending upto a order of 10² number of spins that provide a huge energy barrier to tunnel from one local minima to other^21,22,27,28. Anderson proposed a scaling behavior where the free energy fluctuation which is the sum of the interaction energy at a generalized plane boundary, separating large blocks of spins, varies as the square root of the boundary area i.e., $\langle \overline{E} \rangle \propto {A}^{1/2}$²⁹. In the spin jam state, finite-length spin folds are present, and the energy fluctuations originate from the interaction between these spin folds, giving rise to metastable states that drive the system to a glassy phase. In this context, the experimental realization and elucidation of unconventional ground states such as spin-glass, associated low-energy excitations and their interactions in spin-orbit driven anisotropic frustrated Kitaev quantum materials decorated on 2D honeycomb and its 3D analog known as hyperhoneycomb lattices may provide key ingredients for establishing paradigmatic models, as well as for the experimental realization, detection, and manipulation of exotic quasi-particles that set a stage for topological quantum computing². However, most frustrated Kitaev magnets exhibit long-range magnetic order due to additional terms in the spin Hamiltonian or stacking faults, which impose a strong constraint on the true experimental realization of the elusive Kitaev spin liquid and exotic Majorana fermions. Similarly, some pristine and doped Kitaev magnets demonstrate spin-glass behavior with an anomalous thermodynamic response, which is strikingly different from that of conventional spin glasses. Nevertheless, there is no detailed analysis to comprehend the nontrivial spin-freezing behavior in frustrated Kitaev magnets. A thorough elucidation of such unconventional spin glass behavior in Kitaev magnets may provide crucial insights for the unambiguous experimental realization of exotic quantum states and quasiparticle excitations in the exactly solvable Kitaev model^9,30,31.

Herein, we provide a comprehensive account demonstrating the unusual spin freezing behavior based on thermodynamic, NMR, and INS results on a complementary scale in a few selected frustrated Kitaev magnets. The low temperature spin freezing phenomenon is interpreted within the framework of the Halperin-Saslow theory, which is relevant for unconventional spin glass behavior, and also incorporates the concept of spin jam. In this framework, the HS modes follow a linear dispersion, and these hydrodynamic modes are entangled to atomic spins that probe the behavior of spins decorated on a frustrated spin lattice at low temperatures. Our phenomenological interpretation of experimental results in frustrated Kitaev magnets on honeycomb lattice captures the crux of relevant low-energy excitations in the spin glass state. Furthermore, we provide a comparative account concerning the effect of doping on the freezing temperature in conventional spin glass materials and non-trivial spin glass that is observed in frustrated Kitaev magnets. Our analysis and subsequent discussions based on HS hydrodynamic modes reveal that exotic low-energy excitations in frustrated Kitaev magnets are most likely of topological origin, which is entirely different from the conventional spin glass materials. In essence, the HS formalism has wider applicability in a broad class of frustrated magnets. We also propose a phase diagram taking into account the free energy landscape and observed experimental signatures of Kitaev magnets with distinct magnetic phases, which are highly relevant for the exploration of promising class of frustrated quantum materials for the faithful realization of topological and quantum states with exotic quasi-particle excitations.

Results

Formalism of Halperin and Saslow (HS) mode

Halperin and Saslow proposed a nonequilibrium hydrodynamic state for spin glass and helical spin ordering, which completely breaks the O(3) symmetry (locally)^23,32. In order to encounter the non-equilibrium states, the magnetization density ${m}_{\alpha }(\vec{r})$ corresponding to three slowly varying rotational angles ${\theta }_{\alpha }(\vec{r})$, α = 1, 2, 3 was introduced. The free energy cost because of long-wavelength fluctuation of ${\theta }_{\alpha }(\vec{r})$ and low-frequency fluctuation of ${m}_{\alpha }(\vec{r})$ is expressed by

$$\bigtriangleup F[\vec{m},\vec{\theta }]=\frac{1}{2}{\sum}_{\alpha =1}^{3}\int\,{d}^{3}r\left({\chi }^{-1}{m}_{\alpha }^{2}+{\rho }_{{{{\rm{s}}}}}| \vec{\nabla }{\theta }_{\alpha }{| }^{2}\right)$$

(1)

where χ and ρ_s are magnetic susceptibility and spin-stiffness, respectively. The spin stiffness characterizes the change in free energy of a frustrated magnet when it undergoes a modulation of spin texture. The spin-texture is identified by short-range spin correlation and oscillating spin density profile owing to topological defects in frustrated magnets. In the spin glass state, the two variables θ_α and m_α satisfy the commutation relation

$$\left[{\theta }_{\alpha }(\vec{r}),{m}_{\beta }(\vec{{r}^{{\prime} }})\right]=i\hslash \gamma {\delta }_{\alpha \beta }\delta (\vec{r}-\vec{{r}^{{\prime} }})$$

(2)

where γ = gμ_B/ℏ is the gyromagnetic ratio and α, β are the cartesian components correspond to magnetization density along x, y and z directions. By applying Heisenberg’s equation of motion, which is associated with the commutation bracket stated in Eq. (2), in conjunction with Eq. (1), two coupled equations of motion that govern the dynamics of m_α and θ_α are represented as²³

$$\frac{\partial {\theta }_{\alpha }(\vec{r})}{\partial t}=\gamma \frac{\delta (\bigtriangleup F)}{\delta {m}_{\alpha }(\vec{r})}=\gamma {m}_{\alpha }(\vec{r}){\chi }^{-1}$$

(3)

$$\frac{\partial {m}_{\alpha }(\vec{r})}{\partial t}=-\gamma \frac{\delta (\bigtriangleup F)}{\delta {\theta }_{\alpha }(\vec{r})}=\gamma {\rho }_{{{{\rm{s}}}}}{\nabla }^{2}\theta (\vec{r})$$

(4)

By solving the aforementioned pair of coupled differential equations, one can obtain the solutions that describe the dispersion relations for the propagation of low frequency spin wave, which are given by ω = ± ck, where the wave propagation velocity c turns out to be $c=\gamma {\left({\rho }_{{{{\rm{s}}}}}/\chi \right)}^{1/2}$. The spin dynamics in densely populated frustrated spin glass materials may instigate two interesting scenarios (i) excitation pertaining to small-amplitude motions around equilibrium or quasi-equilibrium states, known as spin waves, and (ii) barrier mode excitation that involves the large amplitude motions to overcome the energy of metastable state owing to topological defects³³. Gapless spin wave modes can be realized in materials wherein magnetic interactions maintain symmetry under rotations in an n-dimensional spin space³². Phenomenologically, below the freezing temperature T_g, the HS modes exhibit gapless excitation spectrum. The behavior is similar to the observation of sound waves in glasses wherein the fluctuation of random forces is averaged out in the long-wavelength limit³³. Remarkably, the bosonic excitations exhibit distinctive power law characteristics in the magnetic specific heat, namely C_m ~ T^D/μ, where D represents the dimension of the spin-lattice and μ signifies the exponent governing energy dispersion. It should be noted that the observed behavior in magnetic specific heat holds true exclusively for the propagation of undamped modes³⁴. Consequently, in scenarios involving the linear HS mode, the magnetic-specific heat should lead to a T² behavior in two-dimensional spin-lattice. Similarly, this theory suggests the power-law behavior of nuclear magnetic resonance spin-lattice relaxation rates and linearly dispersive modes, with the imaginary part of the dynamic susceptibility being proportional to the energy transfer in neutron scattering experiments conducted below the spin-glass temperature^27,35,36. This behavior encapsulates the intricate interplay between competing degrees of freedom, anisotropy, macroscopic ground state degeneracy, short range spin correlations, and low-energy excitation, that are manifested in thermodynamic, NMR and INS responses in frustrated magnets, including Kitaev materials.

Thermodynamics of Kitaev magnets

Competing interaction, spin-orbit driven anisotropy, and quantum fluctuations play a vital role in the manifestation of topological quantum states in Kitaev materials. Magnetic susceptibility refers to the response of magnetization in the presence of an external magnetic field that provides crucial information concerning the ground state properties of frustrated magnets. In a weak magnetic field, a bifurcation of susceptibility recorded in ZFC and FC modes below the freezing temperature is a characteristic feature of spin glass materials. In ZFC, one can observe a linear response of susceptibility (χ_lr) due to the fluctuation of magnetization in a particular metastable state characterized by long relaxation time (see Fig. 1c). In a given applied magnetic field below the freezing temperature, the spin state exhibits long-lived stability. However, with the gradual increase in temperature, the susceptibility shows an asymptotic enhancement. While in the FC case, as we cool the material from high temperature to low temperature, the system likely transitions to one of the lowest free energy states, representing an average of all ensembles, and the susceptibility can be denoted as χ_eq as depicted in Fig. 1c. In terms of the Edward-Anderson spin-glass order parameter, which is associated with the remanent properties of the material under study, the two susceptibilities can be expressed as: ${\chi }_{{{{\rm{lr}}}}}=\left(\frac{1-{q}_{{{{\rm{EA}}}}}}{T}\right)$ and χ_eq = ∫ dqP(q)(1 − q)³⁷. The spin-glass order parameter q_EA is defined as the thermal and disorder averaging of the square of the magnetic moment, which can be mathematically expressed as ${q}_{{{{\rm{EA}}}}}=\overline{{ \langle {S}_{i} \rangle }^{2}}$³⁷. The magnetic susceptibility measured in ZFC and FC modes can be represented in terms of free energy as³⁸

$${\chi }_{{{{\rm{eq}}}}}=\frac{1}{\left(\frac{{\partial }^{2}{F}_{{{{\rm{eq}}}}}}{\partial {M}^{2}}\right)},\quad {\chi }_{{{{\rm{lr}}}}}=\frac{1}{\left(\frac{{\partial }^{2}{F}_{{{{\rm{lr}}}}}}{\partial {M}^{2}}\right)}$$

(5)

Here χ_eq is the susceptibility corresponds to the free energy at equilibrium (F_eq), which is obtained following the field cooling protocol and is depicted by the dashed convex envelope in the inset of Fig. 1c. While χ_lr is the result of magnetization in the zero field cooled (ZFC) mode, which develops in one of the possible metastable states (F_lr) enclosed within the quasi-equilibrium free energy state. Since the double derivative is the measure of curvature, it is obvious that χ_eq > χ_lr. This deviation from linear response theory arises due to the extensive exploration of all the phase points in the phase space in field cooling mode. In contrast, under zero field cooling conditions, the system explores only a subset of the phase space, a phenomenon termed as broken ergodicity characterized by long relaxation time compared to the timescale of the experiment³⁹. In the simplest approximation, the difference between the susceptibility measured following two protocols is given by: ${\chi }_{{{{\rm{eq}}}}}(H)-{\chi }_{{{{\rm{lr}}}}}(H)={\left(\frac{{d}^{2}{F}_{{{{\rm{m}}}}}}{d{M}^{2}}\right)}^{-1}=\frac{d{M}_{{{{\rm{r}}}}}}{dH}$, where M_r is the remanent magnetization.

Recently, unconventional spin freezing behavior has been observed in Kitaev materials, driven by perturbations such as external magnetic fields^10,11, applied pressure⁴⁰, doping^41,42,43, or quenched disorder⁴⁴. The frustrated Kitaev magnet Li₂RhO₃, where 4d⁵ Rh⁴⁺ ions constitute a honeycomb spin-lattice with an effective moment of J_eff = 1/2, shows non-trivial spin-freezing with unusual low-energy excitations as reflected in the field magnetic susceptibility, specific heat and NMR relaxation results (Figs. 2a, 3a and 4)^44,45,46. Notably, in the case of d⁵ ions in an octahedral environment in 4d and 5d transition metals, the crystal electric field (CEF) splits the energy levels of the t_2g and the e_g orbitals as shown in Fig. 1b⁴⁷. Furthermore, the spin-orbit coupling splits the unquenched t_2g orbital into Kramer’s doublet ground state with J_eff = 1/2 and quartet state with J_eff = 3/2 (see Fig. 1b). For the system with a half-filled Kramer’s doublet, the on-site Coulombic interaction induces a gap in the half-filled band, resulting in a weak spin-orbital Mott insulator³⁰. Experimental observations, in tandem with theoretical calculations, indicate that Li₂RhO₃ exhibits Mott insulating behavior, characterized by an energy gap Δ ~ 80 meV⁴⁵. It shows spin glass behavior around 6 K with remanent magnetization M_r = 1.4 × 10⁻³μ_B/Rh⁴⁺ at 2 K in the presence of an applied magnetic field μ₀H = 100 Oe⁴⁵. The first-order derivative of the magnetic susceptibility in Li₂RhO₃, recorded in both zero-field-cooled (ZFC) and field-cooled (FC) modes, is illustrated in Fig. 2b and its inset, respectively. The inflection point in the ZFC mode, where the slope changes sign $\left(\frac{d\chi }{dT}=0\right)$, corresponds to the glass transition temperature in a given magnetic field. This spin freezing behavior is also observed in several 4d and 5d based frustrated Kitaev magnets including Cr doped α-RuCl₃, Ti and Ru doped A₂IrO₃ (A = Na, Li) as presented in this work 2, which suggests the existence of a common spin-freezing mechanism in this class of frustrated magnets. The non-magnetic Ti doping at the Ir site in the extensively studied Kitaev magnet Na₂IrO₃ induces spin freezing and T_g decreases upon increasing Ti concentration, as shown in Fig. 2c, which is in stark contrast to that observed in unfrustrated spin-glass materials.

Fig. 2: Magnetic susceptibility of frustrated Kitaev magnets.

a The temperature dependence of magnetic susceptibility recorded in zero-field and field-cooled modes in a few selected frustrated Kitaev magnets decorated on a honeycomb lattice. The magnetic susceptibility of Ru_0.9Cr_0.1Cl₃ and Na₂Ir_0.89Ti_0.11O₃ is scaled by factors of 0.13 and 1.7, respectively, relative to the original data reported in the literature. b The field-dependent first-order derivative of zero-field-cooled magnetic susceptibility. The inset features a field-cooled first-order derivative of magnetic susceptibility of Li₂RhO₃. The arrow marks correspond to the spin-glass transition temperature. c Field-cooled and zero-field-cooled magnetic susceptibility of Na₂Ir_xTi_1−xO₃ for x = 0.05, 0.11, 0.17, and 0.26 scaled by factors of 2, 1.7, 1.3, and 1, respectively. The data illustrate the progressive suppression of T_g upon increasing doping concentration. Panels (a) and (c) are reproduced with permission from refs. ^41,42,43,44, ⓒ 2017, 2014, 2019, 2015 American Physical Society. Panel (b) is reproduced with permission from ref. ⁴⁵, ⓒ 2013 American Physical Society.

Specific heat experiment is an excellent probe to shed insights into the ground state and associated low-energy quasi-particle excitations in Kitaev magnets. The magnetic specific heat of Li₂RhO₃ (C_m), obtained after subtracting the lattice contribution using the specific heat of non-magnetic analog Li₂SnO₃, shows T² dependence (see Fig. 3a) in the low temperature limit below the spin glass temperature suggesting the presence of unconventional low-energy excitations^44,45. It may be noted that such T² dependence is peculiar in the antiferromagnetic two-dimensional Goldstone modes as C ~ T^D, where D is the dimensionality of the spin-lattice. In the Kitaev magnet Li₂RhO₃, the temperature dependence of magnetic specific heat is robust against the external magnetic field as shown in the inset of Fig. 3a, measured up to 9 T. The C_m(H, T)/T vs. T⁴⁴ shows the change of curvature at about the broad maximum T_broad = 10 K, which ensures that all field derivative of C_m(H, T)/T vanishes at T_broad. Considering ∂²C_m/∂²H = ∂²χ/∂²T, it is expected that the first-order temperature derivative of the field-cooled magnetic susceptibility exhibits an extremum around T_broad⁴⁸. However, it is the minima observed in dχ/dT as depicted in the inset of Fig. 2b (dashed line) for all magnetic fields up to 1.5 T occur below T_broad, derived from the specific heat data. Since specific heat is more sensitive to low-energy hydrodynamic modes and short range spin correlations, this attribute might provide a plausible explanation for the disparate features observed in the two measurements or different experimental techniques with varied characteristic time scales probe a bit different spin dynamics. This scenario is in sharp contrast to that observed in conventional spin glass, where the system cannot allocate enough time to respond to the dynamically evolving magnetic field and temperature fluctuations in specific heat measurements and is less sensitive to glassiness. Furthermore, another reason behind the disparity could be attributed to the distinct energy scales involved⁴⁹. The energy scale T_broad refers to a hidden energy scale in geometrically frustrated systems associated with the development of short-range correlations among atomic spins as the temperature decreases. It is considered an inherent property of the pure compound and is typically an order of magnitude smaller than the Weiss constant θ_CW. The significance of T_broad lies in its suggestion that a glass transition may occur even in a disorder-free system, indicating a fundamental characteristic of the material’s magnetic behavior⁴⁹.

Fig. 3: Magnetic specific heat of frustrated Kitaev magnets.

a The temperature dependence of magnetic specific heat in selected Kitaev materials, with the red solid line depicting a power-law fit (C_m ~ T²) below the spin-glass temperature. The inset shows the field-independent behavior of magnetic specific heat in Li₂RhO₃ on a double log-scale. By examining the trend of C_m/T as a function of temperature, we aim to elucidate the unique thermodynamic properties and underlying physics that govern these Kitaev magnets. b The temperature dependence of the magnetic specific heat of Kitaev magnets with spin stiffness fitting below T_g, as discussed in the text, offers insights into the ground state properties. Reproduced with permission from refs. ^41,42,43,44, ⓒ 2017, 2014, 2019, 2015 American Physical Society.

In Li₂RhO₃, the magnetic entropy recovered at 45 K is much lower ($0.35\,{\mbox{R}}\,\ln 2$) than that expected for J_eff = 1/2 moments suggesting huge ground state degeneracy and short-range spin correlations in agreement with the presence of a broad maximum in specific heat and NMR spin-lattice relaxation rate⁴⁴. The specific heat rules out the presence of any Schottky contribution, and the observed behavior of magnetic specific heat in Li₂RhO₃ is at variance with that found in conventional spin glass materials wherein C_m ~ T⁵⁰. Remarkably, similar C_m ~ T² behavior has been observed in several Kitaev materials such as Cr doped α-RuCl₃⁴¹, Ti doped Li₂IrO₃⁴², Ru doped Na₂IrO₃⁴³ below the spin-freezing temperature much below the characteristic exchange interaction as shown in Fig. 3a. To shed insights into this unconventional low-energy excitations as reflected in the specific heat of the spin-orbit driven frustrated Kitaev magnets with a goal to demonstrate a commonality in this class of materials, we have analyzed the specific heat of a few representative Kitaev quantum magnets following Halperin-Saslow formalism. The coherent propagation of HS mode can be captured with a finite spin-stiffness constant in the spin texture with a characteristic length scale L₀. Within this finite spin-texture, the low-energy spin excitation in 2D spin-lattice gives rise to the magnetic specific heat $\frac{{C}_{{{{\rm{m}}}}}}{R}=\left[\frac{3\sqrt{3}\zeta (3)}{2\pi }\right]{\left(\frac{a{k}_{{{{\rm{B}}}}}T}{\hslash {{{\mathscr{D}}}}}\right)}^{2}-\left(\frac{\sqrt{3}\pi }{2}\right){\left(\frac{a}{{L}_{0}}\right)}^{2}$ at temperature T < T_g < < ∣θ_CW∣, where θ_CW refers to the energy scale of exchange interaction and ${{{\mathscr{D}}}}$ is the spin stiffness constant associated with the free-energy^51,52. It is worth noting that we employ the spin stiffness constant ${{{\mathscr{D}}}}$ for dimensional normalization, distinct from the notation ρ_s used previously for the same quantity and L₀ is the characteristic length scale over which the Goldstone modes are well defined. The first term corresponds to the quadratic temperature dependency of C_m in 2D antiferromagnetic spin wave, and the second term is the size-dependent negative shifting of specific heat valid for a 2D gapless linear dispersive mode with frequency $\frac{\hslash \omega }{{k}_{{{{\rm{B}}}}}} \, < \, T \, < \, | {\theta }_{{{{\rm{CW}}}}}|$⁵². Considering only the antiferromagnetic ordering at T ~ ∣θ_CW∣, the spin spin stiffness constant can be calculated from the relation ${{{{\mathscr{D}}}}}_{0}^{2}=\left[3\sqrt{3}\zeta (3)/4\pi \right]{(a{k}_{{{{\rm{B}}}}}{\theta }_{{{{\rm{CW}}}}}/\hslash )}^{2}/\ln (2S+1)$⁵¹. The fit of the low temperature experimental specific heat data below T_g of Li₂RhO₃ with the expression relevant for elucidating the low-energy excitations in the HS mode yields, ${{{{\mathscr{D}}}}}_{0} \sim 1960$ m s⁻¹ and ${{{\mathscr{D}}}} \sim 1091$ m s⁻¹ with spin texture length L₀ ~ 40 nm and corresponding fit is shown in Fig. 3b. This framework has been extended to Cr doped α-RuCl₃, Ti doped Li₂IrO₃, Ru doped Na₂IrO₃ Kitaev materials and the resulting parameters are presented in Table 1 exemplifying non-trivial spin-freezing mechanism in this class of spin-orbit driven frustrated quantum materials. The reduction in stiffness constant ${{{\mathscr{D}}}}$ compared with ${{{{\mathscr{D}}}}}_{0}$ is ascribed to softening owing to magnetic frustration²⁷.

Table 1 Relevant parameters extracted from the analysis of magnetic susceptibility and specific heat following the Halperin-Saslow framework for selected frustrated honeycomb Kitaev magnets exhibiting non-trivial spin-freezing behavior

Next, the T² dependence of magnetic-specific heat, as shown in Fig. 3a in the Kitaev magnets wherein the magnetic frustration is mediated by spin-orbit driven anisotropic bond-dependent exchange interaction, can be explained by assuming three non-degenerate hydrodynamic modes in the presence of the magnetic field. With the application of magnetic field H, the degeneracy of hydrodynamic mode gaps out, and the frequency of each mode is expressed as²⁷:${\omega }_{\pm }=\pm \frac{g{\mu }_{{{{\rm{B}}}}}H}{2}+\sqrt{{\left(\frac{g{\mu }_{{{{\rm{B}}}}}H}{2}\right)}^{2}+{(ck)}^{2}}$ where g is the Landé g-factor, c is the velocity of hydrodynamic modes and k is the wave vector. In the limit of $\left(\frac{g{\mu }_{{{{\rm{B}}}}}H}{2}\right) \, > \, > \, ck$, the Taylor expansion yields ${\omega }_{+}\approx g{\mu }_{{{{\rm{B}}}}}H+\frac{{c}^{2}{k}^{2}}{g{\mu }_{{{{\rm{B}}}}}H}$ and ${\omega }_{-}\approx \frac{{c}^{2}{k}^{2}}{g{\mu }_{{{{\rm{B}}}}}H}$. While ω₀ = ck corresponds to field-independent excitations showing linear dispersion relation. The quadratic polarization ω₋ compensates for the gapped mode ω₊ to some extent at low energy. This compensation accounts for the observed negligible deviation from the T² behavior of C_m at low temperature below the spin-glass temperature. The proportionality constant A (C_m = AT²) is given by: $A=\frac{3\zeta (3){k}_{{{{\rm{B}}}}}^{2}RV}{\pi d{\hslash }^{2}}{\sum }_{i}\frac{1}{{c}_{i}}$, where V is volume of the unit cell, d is the spacing between the successive layers in the honeycomb lattice, and c_i is the velocity of spin wave mode along three spatial directions. Apart from the HS mode, the robustness of specific heat against an external magnetic field below the freezing temperature, can be associated with pseudo-Goldstone modes fostered by noncollinear antiferromagnetic ordering⁵³. An applied external magnetic field reduces the symmetry of the Hamiltonian from O(3) to O(2), decreasing the number of Goldstone modes from three to one. Despite this symmetry reduction, the noncollinear antiferromagnetic ground state retains an “accidental” degeneracy, unrelated to the O(2) symmetry of the Hamiltonian, similar to that in zero-field conditions⁵⁴. The ground-state manifold preserves three adjustable continuous parameters, including one true Goldstone mode from symmetry and two pseudo-Goldstone modes not linked to the Hamiltonian symmetry. These pseudo-Goldstone modes play a crucial role in maintaining the T² behavior of the low-temperature specific heat under applied magnetic fields^53,54. Thus, the specific heat remains robust against external perturbations. Noteworthy accidental degeneracy and pseudo-Goldstone modes may become approximate in quantum systems due to quantum effects⁵³. A broad peak in C_m/T is observed around temperature T = 10 K, indicating the persistence of short-range spin correlations in Li₂RhO₃⁴⁴. In an isotropic unfrustrated system, a λ-type peak in specific heat is typically associated with the onset of long-range magnetic ordering which can be suppressed completely by applying a critical field H_C(0) ~ k_B∣θ_CW∣/gμ_B. So one would expect a complete suppression of the peak in specific heat by applying a critical magnetic field of approximately H_C(0) ~ k_BT_peak/gμ_B = 7.4 T if the peak at T = 10 K was associated with long range magnetic ordering in Li₂RhO₃⁴⁴. However, it is worth noting that, even in an applied field of 9 T, there is no suppression of the observed peak in Li₂RhO₃. A similar scenario is also observed in Cr-doped α-RuCl₃, Ti doped Li₂IrO₃, Ru doped Na₂IrO₃. The broad peak in specific heat and T² behavior of C_m below the spin-glass temperature (see Fig. 3) are robust against external magnetic field, which is associated with the presence of short-range spin correlations and abundant low energy excitations in these frustrated magnets^41,42,43. Second, it can be the coexistence of glassiness with spin singlets formation around the energy scale of glass transition T_g⁴⁸. The low-energy state might be dominated by these moment free singlets. It may represent a specific type of spin correlation characterized by a Gaussian-shaped relaxation profile in muon spin resonance (μSR) experiment involving short-lived singlets whose life-time is significantly shorter than that of the muon^48,55. For instance, in the frustrated trillium spin-lattice K₂Ni₂(SO₄)₃, this scenario involves proximity to a quantum critical point (QCP) between an ordered phase and a quantum fluctuation-dominant phase, with K₂Ni₂(SO₄)₃ near the ordered side⁵⁵. It escapes the glassiness region; however, another trillium lattice, KSrFe₂(PO₄)₃ with high spin (S = 5/2), exhibits the emergence of glassiness characterized by a power-law specific heat C_m ~ T^2.33 below spin-glass temperature⁵⁶. The trillium lattice material KSrFe₂(PO₄)₃ that crystallizes in the cubic space group forms a diamond spin-lattice through the combination of nearest-neighbor J₁ = − 4.87 K and next-nearest-neighbor J₂ = − 2.67 K couplings. The power law behavior of specific heat C_m ~ T^2.33 in diamond lattice antiferromagnets arises from the quantum treatment of thermal fluctuations, particularly when considering the behavior of the fluctuation stiffness parameter κ_T(q) at low temperatures⁵⁷. In these systems, κ_T(q) is modified non-perturbatively by thermal fluctuations, leading to a non-analytic temperature-independent factor that vanishes only at specific spiral wavevectors  ± Q, corresponding to the emergence of Goldstone modes. This results in a non-analytic temperature dependence in the specific heat. While the classical specific heat scales as C_classical(T) = A + BT^1/3 at low temperatures, the quantum treatment, which involves quantization of the spin-wave modes to obtain the magnon spectrum, predicts a specific heat power law with an exponent of 7/3: C_quantum(T) ~ T^7/357. This behavior reflects the intricate interplay between thermal fluctuations, the fluctuation stiffness κ_T(q), and the non-analytic temperature dependence, highlighting the unique quantum behavior and thermal properties of the spin system under consideration.

In the HS framework, the magnetic susceptibility χ_m, defined as the experimental value of the ZFC susceptibility as the temperature approaches zero, and C_m/T² below T_g, can be linked to yield two characteristic energy scales denoted as E₁ and E₂, which are expressed as follows:

$${E}_{1}=\frac{2{g}^{2}{\mu }_{{{{\rm{B}}}}}^{2}S(S+1){N}_{{{{\rm{A}}}}}}{z{\chi }_{{{{\rm{m}}}}}}={k}_{B}{\tau }_{1}$$

(6)

The energy scale E₂ is derived from the relation^23,27:

$$\frac{{C}_{{{{\rm{m}}}}}}{{N}_{{{{\rm{A}}}}}\nu }=\frac{3\zeta (3)}{\pi d}\frac{{k}_{{{{\rm{B}}}}}^{3}{T}^{2}}{{\hslash }^{2}}{\sum}_{i}\left(\frac{1}{{c}_{i}^{2}}\right)-\frac{3{k}_{{{{\rm{B}}}}}\pi }{{L}_{0}^{2}d}$$

(7)

where C_m is the magnetic specific heat, $\nu =\frac{\sqrt{3}{a}^{2}d}{2}$ is the unit cell volume, d is the spacing between decoupled spin-lattice layers, a is the bond length between magnetic moments in a plane, and L₀ is the characteristic length scale for Goldstone modes. Neglecting the second term and considering three Goldstone modes (n_p = 3):

$$\frac{{C}_{{{{\rm{m}}}}}}{{N}_{{{{\rm{A}}}}}\nu }=\frac{3\zeta (3)}{\pi d}\frac{{k}_{{{{\rm{B}}}}}^{3}{T}^{2}}{{\hslash }^{2}}\frac{3}{{c}^{2}}$$

(8)

Using $c=\gamma {\left(\frac{{\rho }_{{{{\rm{s}}}}}}{{\chi }_{{{{\rm{m}}}}}}\right)}^{1/2}$ in equation (4):

$$\frac{{\rho }_{{{{\rm{s}}}}}}{{N}_{{{{\rm{A}}}}}\nu }=\frac{3{n}_{{{{\rm{p}}}}}\zeta (3)}{\pi d}\frac{{k}_{{{{\rm{B}}}}}^{3}}{{g}^{2}{\mu }_{{{{\rm{B}}}}}^{2}}\frac{{\chi }_{{{{\rm{m}}}}}}{{C}_{{{{\rm{m}}}}}/{T}^{2}}$$

For in-plane interactions, this gives the energy scale E₂:

$${E}_{2}=\frac{{\rho }_{{{{\rm{s}}}}}d}{{N}_{{{{\rm{A}}}}}\nu }=\frac{3{n}_{{{{\rm{p}}}}}\zeta (3)}{\pi }\frac{{k}_{{{{\rm{B}}}}}^{3}}{{g}^{2}{\mu }_{{{{\rm{B}}}}}^{2}}\frac{{\chi }_{{{{\rm{m}}}}}}{{C}_{{{{\rm{m}}}}}/{T}^{2}}$$

$${E}_{2}=\frac{3{n}_{{{{\rm{p}}}}}\zeta (3)}{\pi }\frac{{k}_{{{{\rm{B}}}}}^{3}}{{g}^{2}{\mu }_{{{{\rm{B}}}}}^{2}}\frac{{\chi }_{{{{\rm{m}}}}}}{{C}_{{{{\rm{m}}}}}/{T}^{2}}={k}_{{{{\rm{B}}}}}{\tau }_{2}$$

(9)

Here z is the co-ordination number i.e., the number of nearest neighbor magnetic ions, which is 3 for the honeycomb lattice, and n_p is the number of degenerate hydrodynamic modes. The two energy scales E₁ and E₂, are linked to the exchange interaction energy and spin stiffness, which are in turn comparable to two distinct temperature scales, ∣θ_CW∣ and T_g, respectively. The external perturbations, such as doping at the magnetic site of Kitaev magnets, could lead to novel physical phenomena such as spin freezing in the case of Ru_1−xCr_xCl₃ or Na₂Ir_1−xRu_xO₃. In such a scenario, the relevant energy scale can be extracted following modified expressions relevant for E₁ and E₂. The derived relevant energy scales for some selected Kitaev quantum materials are tabulated in Table 1, which suggests a common underlying mechanism governing unconventional spin-freezing phenomena in frustrated Kitaev magnets.

In quantum materials with ferromagnetic interactions, the undamped linear dispersion of spin wave mode leads to a T³ behavior in specific heat below the spin glass temperature³⁴. Such behavior is observed in the 3D variant of Kitaev honeycomb magnet, namely, the hyperhoneycomb lattice β-ZnIrO₃ and β-MgIrO₃, where the dominant exchange interaction between effective spin J_eff = 1/2 of Ir⁴⁺ moments is ferromagnetic with θ_CW ~ 45.6 K and 56 K, respectively^58,59,60. In this hyperhoneycomb spin-lattice, the Z₂ flux operator W, is a conserved quantity defined as a loop operator that encompasses precisely ten lattice sites. Both 3D Kitaev materials, β-ZnIrO₃ and β-MgIrO₃, demonstrate a distinctive spin freezing phenomenon, characterized by the onset of weak anomaly in thermodynamic experiments at temperatures around 12 K and 22 K, respectively^58,59,60. The magnetic contribution to the specific heat exhibits a notable T³ dependency below the spin-glass temperature, indicating a linear dispersion behavior, which was initially postulated to elucidate the hydrodynamic mode of spin waves in ferromagnetic materials³². For ferromagnetic long-range ordering, the spin wave excitations lead to quadratic dispersion relation so the linear dispersion is due to planar ferromagnetic excitations. The T³ behavior remains intact for β-ZnIrO₃ in the presence of an applied magnetic field upto 5 T, which is due to the compensation of ω₊ with ω₋ and the propagation of linear mode (ω₀) accounts for the field independent behavior of specific heat in the presence of an external magnetic field⁵⁸.

Next, we demonstrate how the Kitaev spin freezing of topological origin manifested by macroscopic ground state degeneracy and linearly dispersive low energy spin excitations, significantly differs from conventional spin-glass materials. In a seminal paper⁵⁰, Anderson et al. introduced the idea of two-level system (TL), which successfully describes the linear behavior of specific heat in conventional spin glass. At low temperatures, only tunneling systems with energy differences ΔE close to k_BT contribute significantly to the specific heat. The specific heat scales linearly with temperature T because the density of states for ΔE is approximately constant around zero energy difference. This behavior aligns with the Anderson-Varma-Halperin (AVH) mechanism, which requires the presence of localized tunneling states. For tunneling to occur, the energy barrier separating two local minima must not be too large, enabling transitions between states that are energetically degenerate within k_BT. The specific heat contribution from a cluster spin glass, incorporating these tunneling effects, can be expressed as ${C}_{{{{\rm{TL}}}}}=\frac{{\pi }^{2}}{6}{k}_{{{{\rm{B}}}}}^{2}Tn(0),$ where n(0) represents the density of states⁵⁰. This density of states accounts for the linear specific heat behavior, with the energy scale ΔE ~ k_BT_g. The absence of the AVH mechanism in 2D Kitaev magnets, despite spin freezing, stems from their quantum and topologically constrained nature. The AVH mechanism relies on a broad, random distribution of local magnetic fields (H_loc) with a nonzero density of states at H_loc = 0, enabling linear specific heat via localized zero-point tunneling^61,62. In contrast, the Kitaev Hamiltonian $H=-{\sum }_{\langle i,j\rangle }{J}_{\gamma }{S}_{i}^{\gamma }{S}_{j}^{\gamma }$ imposes anisotropic, bond-dependent interactions that create a structured energy landscape dominated by emergent flux excitations and gapless excitations^2,63. Enhanced quantum fluctuations in 2D suppress classical trapping, while topological constraints preclude the singular field distribution at H_loc = 0⁶⁴. Thus, Kitaev magnets epitomize a fundamentally different paradigm, where spin freezing emerges from quantum coherence and geometrical frustration, not the random field physics central to AVH. In principle, the contribution to the specific heat arises from both spin jam and spin glass cluster mechanisms⁶⁵. As such, the specific heat can be expressed as a linear combination: C = f ⋅ C_HS + (1 − f) ⋅ C_TL, where C_HS represents the specific heat contribution from the spin jam component, while C_TL is that due to the spin glass cluster, and f is a weighting factor corresponds to the relative proportion of each contribution. The magnetic susceptibility and specific heat provide a consistent picture in accord with the HS framework manifesting a common spin-freezing mechanism in a few selected Kitaev magnets presented here.

Local probe techniques

Nuclear magnetic resonance and neutron scattering are two excellent microscopic techniques to shed insights into non-trivial spin glass characterized by short-range spin correlations and unconventional low energy spin excitations in frustrated Kiatev magnets. The NMR spectra broaden upon lowering temperature below the spin glass temperature. The relative NMR linewidth defined as the linewidth divided by the resonance field (δH = ΔH/H) at two NMR frequencies 70 MHz (μ₀H ~ 4.23T) and 114 MHz (μ₀H ~ 6.89T) exhibits no field dependency below the spin-glass temperature (Fig. 4a of ref. ⁴⁴). This transition is marked by a gradual development of frozen moments, where intermediate temperatures display both static and dynamic components. In general, the magnetic broadening in ΔH reflects the distribution of internal magnetic fields. In paramagnetic systems, these fields originate from induced moments that vary linearly with the applied magnetic field H. However, during the transition to a frozen state with a local static moment, the width ΔH is expected to remain largely unaffected by changes in H that is consistent with magnetic susceptibility⁶⁶. The NMR line width is independent of temperature in the paramagnetic region T > > θ_CW, however, the enhancement of NMR line width in the intermediate temperature range well above the spin glass transition suggests the predominance of short-range spin correlations that are in agreement with thermodynamic results. NMR spin-lattice relaxation rate 1/T₁, that tracks low-energy spectra in frustrated magnets follows T^2.2 (Fig. 4) behavior down to 1.8 K in Li₂RhO₃ suggesting the persistence of low energy excitations below spin-glass temperature T_g ~ 6 K. The deviation of the exponent of the power law in the temperature dependence of 1/T₁ from the expected 1/T₁ ~ T³ behavior might be related to the distribution of local hyperfine fields at the probing nuclear site reflected as broad NMR spectra and/or anisotropic spin fluctuations at low temperatures^24,35,44. The broad maximum around 10 K in 1/T₁, well above T_g, infers the persistence of short-range spin correlations, which is consistent with the broad peak in the magnetic specific heat. The absence of a significant loss in NMR signal intensity and the power law behavior of NMR spin-lattice relaxation rate further support that Li₂RhO₃ is not a conventional spin-glass material. The spin-lattice relaxation rate, $\frac{1}{{T}_{1}}$, is related to the spin correlation as the wave vector averaged dynamical susceptibility ${\chi }_{{{{\rm{m}}}}}^{{\prime} }({{{\bf{q}}}},\omega )$, $\frac{1}{{T}_{1}}\propto {\sum }_{{{{\bf{q}}}}}| {A}_{{{{\rm{hf}}}}}({{{\bf{q}}}}){| }^{2}\frac{{\chi }_{{{{\rm{m}}}}}^{{\prime} }({{{\bf{q}}}},\omega )}{\omega }$, which probes low energy spin excitations in frustrated magnets. For instance, the frustrated triangular lattice antiferromagnet NiGa₂S₄ exhibits an anisotropic spin dynamics and power-law T³ dependence below 1 K in NMR NMR spin-lattice relaxation rate, 1/T₁²⁴. Since the specific heat coefficient C_m/T is proportional to the thermal average of the density of states, ${C}_{{{{\rm{m}}}}}/T\propto \overline{N({E}_{{{{\rm{F}}}}})}$, and $\overline{N({E}_{{{{\rm{F}}}}})}\propto T$ due to the linear dispersion arising from antiferromagnetic spin-wave excitations in a 2D spin-lattice, it follows that C_m ∝ T². Furthermore, the nuclear spin-relaxation rate 1/T₁ is proportional to the squared density of states, yielding $1/{T}_{1}\propto T\,{\overline{N({E}_{{{{\rm{F}}}}})}}^{2}$. Given $\overline{N({E}_{{{{\rm{F}}}}})}\propto T$, this results in the cubic temperature dependence 1/T₁ ∝ T³ followed by a broad peak. The damping of spin wave excitation gives rise to a dynamic susceptibility of the form ${\chi }^{{\prime} }(k,\omega )=\frac{\omega \chi {D}_{{{{\rm{s}}}}}{k}^{2}}{2}\left[\frac{1}{{(\omega -ck)}^{2}+{({D}_{{{{\rm{s}}}}}{k}^{2})}^{2}}+\frac{1}{{(\omega +ck)}^{2}+{({D}_{{{{\rm{s}}}}}{k}^{2})}^{2}}\right]$ with D_s is the spin diffusion constant³⁴. In the topologically protected state, the barrier mode propagation involves the diffusion of quadruplet, not pair, flips of defect variables by spin operators. So, the barrier mode is thermally suppressed, and in the limit of D_s → 0, the imaginary part of the dynamic susceptibility becomes ${\chi }^{{\prime} }=\frac{\pi }{2}\omega \chi \left[\delta (\omega -ck)+\delta (\omega +ck)\right]$. This results in the propagation of undamped spin wave modes that, in conjunction with linear modes, yield a power-law dependence in specific heat, which reflects the gapless nature of the low-energy excitations³⁴. In a similar vein, the frustrated spin-$\frac{1}{2}$ system embodying a square lattice, Sr₂CuTe_0.5W_0.5O₆, exhibits glassy behavior below 1.7 K³⁶. Below the spin-glass temperature, it is evidenced from the time-of-flight neutron scattering measurements that the imaginary part of the dynamic susceptibility proportional to the energy transfer, χ′′(q, ω) ~ ℏω, and the Goldstone modes emerge for ℏω < k_BT_g³⁶, and the specific heat shows quadratic behavior below T_g⁶⁷, which indicates the linear dispersion of low-lying magnon excitations. NMR shares an interface with neutron scattering, and both experimental techniques track similar spin dynamics in frustrated magnets.

Fig. 4: NMR of frustrated Kitaev magnet.

The temperature dependence of NMR relaxation rate 1/T₁ in Li₂RhO₃ taken in two different magnetic fields shows a T^2.2 dependence. Reproduced with permission from ref. ⁴⁴, ⓒ 2017 American Physical Society.

The imaginary part of the dynamic susceptibility, χ′′(ω) obtained from inelastic neutron scattering, offers a crucial distinction HS mode-driven spin jam in frustrated spin glass including Kitaev magnets and canonical spin glass behavior on a complementary scale. This can be expressed as: ${\chi }^{{\prime} }(\omega )\propto \left({\tan }^{-1}\left(\frac{\omega }{{\Gamma }_{\min }}\right)+\alpha \cdot \left[\frac{\omega }{{\omega }^{2}+{\Gamma }_{L}^{2}}\right]\right)$. The first term, represented by the arctangent function, characterizes HS modes that arise due to quantum fluctuations and are associated with a broad distribution of spin relaxation rates, where ${\Gamma }_{\min }$ denotes the lower energy scale of dissipation with α a weighting factor^21,36. Remarkably, these long-wavelength collective excitations dominate in materials near pristine conditions with minimal disorder, such as SrCr_9pGa_12−9pO₁₉ at high spin densities (p > 0.8)²¹. In this regime, the system exhibits coherent low-energy spin dynamics that harden with disorder, while the quadratic specific heat behavior (C_m ~ T²) reflects the frozen but collective nature of HS-like excitations akin to Goldstone modes. In contrast, the Lorentzian term $\frac{\omega }{{\omega }^{2}+{\Gamma }_{L}^{2}}$ reflects localized glassy dynamics within short-range correlated spin clusters, characteristic of canonical spin glasses, where Γ_L sets the characteristic energy at which the susceptibility χ′′ peaks^68,69. As disorder increases, for instance, through nonmagnetic doping in SrCr_9pGa_12−9pO₁₉, the Lorentzian contribution becomes increasingly significant, signaling a crossover to a conventional spin-glass state dominated by localized excitations²¹. The competition between spin jam and conventional spin glass highlights the delicate interplay between quantum fluctuations and disorder in frustrated systems. At high spin densities, the dynamics remain HS-dominated with long-range coherence, while increased disorder facilitates a gradual transition to a localized, glassy regime^21,49. This physical insight not only clarifies the evolution of spin dynamics in systems under doping and disorder but also provides a unifying framework for understanding emergent behaviors in frustrated magnets.

Elastic neutron scattering data of Kitaev magnet Na₂Ir_0.89Ti_0.11O₃ reveals profound insights into the underlying physics of topological spin freezing. The wave vector dependence of magnetic structure factor below the spin-glass temperature highlights the emergence of prominent magnetic correlations at low temperatures, indicative of freezing into a topologically nontrivial state⁶. This is evidenced by the elastic scattering intensity peak centered at the wave vector Q = 0.87 Å⁻¹, corresponding to short-range spin correlations with a characteristic length scale ξ, given by $\xi =\frac{1}{{{{\rm{FWHM}}}}}$⁷⁰, where the full width at half maximum (FWHM) was obtained from the digitized graph from the original literature and is found to be FWHM = 0.3 Å⁻¹ and yielding ξ ≈ 3.3 Å⁶. Using this correlation length, we define a thermal velocity scale ${v}_{{{{\rm{th}}}}}=\frac{\xi {k}_{{{{\rm{B}}}}}T}{\hslash } \sim 65$ m s^−1 71 at 1.5 K, provides a characteristic scale for the propagation of thermal fluctuations over the correlation length which is significantly lower than typical spin wave velocities listed in Table 1. These results underscore the unique magnetic structure factor of this Kitaev magnet, starkly contrasting with the featureless profile of conventional spin glasses like Cu_1−xMn with low Mn(2%) concentration^6,69. The neutron scattering results of Cu_1−xMn(2%) reveal a broad, featureless magnetic structure factor indicative of long-range Ruderman-Kittel-Kasuya-Yosida (RKKY) type disordered spin interactions between dilute magnetic impurities. However, with increasing Mn concentration in Cu_1−xMn and in other canonical spin glasses, inelastic neutron scattering reveals a broad low-energy peak, identified as the magnetic boson peak^68,69,72. The hierarchical energy landscape of spin glasses leads to localized excitations within the metastable states^5,33, where larger spin clusters generate low-energy modes, and smaller clusters produce higher-energy excitations, exhibiting Bose scaling and a broad spectrum with a high-energy tail⁶⁹. An intriguing direction for future investigation lies in uncovering a potential connection between the emergence of the magnetic boson peak in canonical spin glasses and the emergence of spin wave modes in topological spin freezing, which may reveal an underlying transition or unifying framework connecting these distinct phenomena.

This corroborates the thermodynamic, NMR results, and subsequent analysis presented here. A detailed comparison of NMR and neutron scattering results in frustrated magnets may provide profound insights in this context, which is beyond the scope of the present study. Nonetheless, the current brief elucidation of dynamic spin susceptibility in a representative Kitaev magnet offers an exciting impetus for future research directions in frustrated Kitaev magnets.

Effect of doping on spin freezing

The influence of external perturbations, such as chemical pressure and magnetic field, is quite striking and often drives quantum phase transitions such as Bose-Einstein condensation, spin liquids, superconductivity, and non-trivial spin-glass in frustrated magnets. In canonical spin glass systems, the introduction of dopants has been observed to evoke a pronounced and significant influence on the underlying physical mechanism at play. It is known that minute amount of magnetic dopants such as Mn and Fe into non-magnetic host materials, such as Cu_1−xMn_x, Au_1−xFe_x, and Ag_1−xMn_x, there is a substantial increment in the spin-glass temperature upon increasing the doping concentration. This enhancement of T_g is characterized by a marked and significant scaling response that is contingent upon the concentration of dopants, as depicted in Fig. 5a⁷³. In frustrated magnets, the spin-glass temperature in a spin jam state depends on both the energy of barrier mode (E_B) necessary for the collective spin flipping and the spin correlation length ξ(T), which can be expressed as T_g ∝ F(E_B, ξ)²². The spin-glass temperature T_g remains unchanged upon substitution of non-magnetic impurities in frustrated magnets such as SCGO and BCGO until the separation between non-magnetic impurities becomes comparable to ξ^74,75.

Fig. 5: Perturbation-induced novel quantum states in frustrated Kitaev magnets.

a The spin-glass temperature T_g as a function of doping concentration x for canonical spin glass materials AuFe, CuMn, and AgMn, as well as for frustrated Kitaev spin glasses. References to these data points are reproduced from refs. ^41,42,43, ⓒ 2014, 2019, 2015 American Physical Society, and ref. ⁷³, ⓒ 2019 Elsevier. b A schematic triangular phase diagram of selected Kitaev materials, distinguishing unique characteristics based on their low-energy, non-trivial excitations and energy landscapes. Zig-zag and spiral ordering are associated with well-defined energy minima, indicative of classical magnetic order. Topological spin freezing is characterized by a rugged energy landscape, reflecting glassy dynamics arising from frustration and disorder effects. The Kitaev spin liquid phase accommodates both Dirac cone and flat band dispersions, which emerge under different conditions as signatures of the spin liquid state (see refs. ^{41,42,43,44,58,59,60,80,84,97,98,99,100,101,102,103,104} for more details.).

The effect of doping was investigated in Kitaev materials Ru_1−xCr_xCl₃, Na₂Ir_1−xRu_xO₃ and Na₂Ir_1−xTi_xO₃ decorated on a honeycomb lattice, and by varying the doping parameter x, it was observed that within a percolation threshold, the doping is less robust compared to the canonical spin glass^42,43. The emergence of topological quantum state akin to Kitaev QSL in the presence of external magnetic field in the celebrated honeycomb lattice α-RuCl₃ is quite remarkable⁷⁶. The well studied Kitaev magnet Na₂IrO₃ shows a zig-zag antiferromagnetically ordered state at temperature T_N = 15 K⁷⁷ and a few percentages of Ru⁴⁺ doping at Ir⁴⁺ site gives rise to a topologically protected spin-glass state⁴³. In the context of Kitaev materials, topological freezing refers to the emergence of a glassy state from an underlying topologically ordered phase, such as the Kitaev spin liquid. The term “topological” in this case is not merely a descriptor of the spin freezing but reflects the fact that the freezing emerges from a substrate with inherent topological properties, such as non-trivial ground-state degeneracy and fractionalized excitations that proposed in Kitaev spin liquid^33,78. The substitution of Ru⁴⁺ at Ir⁴⁺ site preserves the magnetic exchange path that suggests the emergence of glassiness can be intrinsic. This could manifest as a spin jam state, given its correlation with intrinsic glassy states²². As presented in Fig. 5a, the spin-glass temperature does not change drastically for the Ru⁴⁺ doping concentration 0.1 ≤ x ≤ 0.3 in the single crystal of Na₂Ir_1−xRu_xO₃. Upon increasing the amount of Ru⁴⁺ doping concentrations at Ir⁴⁺ site in Na₂Ir_1−xRu_xO₃ and the non-magnetic Ti⁴⁺ at Ir⁴⁺ site in Na₂Ir_1−xTi_xO₃, the spin-glass temperature decreases smoothly. The observed behavior of spin-freezing temperature upon doping concentration is consistent with spin jam theory^6,36. Interestingly, upon increasing doping concentration, we observe an increase in χ(T_g) (see Fig. 2c), while the T_g value is suppressed which, in turn, imposes the constraint dχ(T_g)/dT < 0. The observed trend differs from the conventional spin glasses, wherein an escalation in doping concentration typically yields an increase in both χ(T_g) and T_g. This behavior adheres to the underlying relationship expressed by dχ(T_g)/dT > 0 in the case of conventional spin glass⁴⁹. It is worth noting that the extensively studied Kitaev material α-RuCl₃ hosts a glassy phase upon substitution of S = 3/2 Cr³⁺ ions at Jeff = 1/2 Ru⁴⁺ site and the glassy phase remains stable for values of x exceeding 0.1 in α-Ru_1−xCr_xCl₃⁴¹. In contrast, nonmagnetic doping of Ir³⁺ (5d⁶: J_eff = 0) in α-Ru_1−xIr_xCl₃ stabilizes a quantum disordered QSL like state for 0.16 ≤ x ≤ 0.4⁷⁹. The data collapse behavior in specific heat is observed as a function of the magnetic field, and it is achieved by plotting CH^1−γ/T vs T/H, where γ (=0.19) represents the critical exponent typical for random singlets in the quantum disordered state⁸⁰. The observed features suggest that frustrated Kitaev magnets are quite sensitive to external stimuli in undergoing a phase transition leading to the emergence of a plethora of topological states driven by perturbations in a controlled manner. Taking into account the intriguing phenomena detected in Kitaev magnets owing to a subtle interplay between competing degrees of freedom, their free energy landscape and effect of external perturbations on the underlying physical phenomena, we proposed a comprehensive phase diagram as shown in Fig. 5b that reflects immense promise for the experimental realization of topological and quantum states such as Kitaev QSL with elusive Majorana fermions in this class of frustrated quantum materials.

A proposed phase diagram for Kitaev materials

To construct a phase diagram for Kitaev magnets, which captures transitions between complex magnetic ordering, topological spin freezing, and Kitaev spin liquid like behavior, we account for three primary parameters: applied magnetic field, hydrostatic pressure, and chemical pressure. These external factors allow us to tune the interactions and probe different phases of the material. The general Hamiltonian governing a Kitaev model is given by: ${{{\mathscr{H}}}}={\sum }_{\langle ij\rangle }\left(-{K}_{\gamma }{S}_{i}^{\gamma }{S}_{j}^{\gamma }+J{{{{\bf{S}}}}}_{{{{\bf{i}}}}}\cdot {{{{\bf{S}}}}}_{{{{\bf{j}}}}}+\Gamma \left({S}_{i}^{\alpha }{S}_{j}^{\beta }+{S}_{i}^{\beta }{S}_{j}^{\alpha }\right)\right)$, where K_γ represents the anisotropic Kitaev exchange along different bond directions γ, J is the Heisenberg exchange interaction, and Γ denotes the off-diagonal symmetric exchange involving spin components α and β⁹. These interactions are crucial for describing the intricate balance between quantum spin liquid and magnetically ordered phases, as well as topological spin freezing, influenced by external perturbations. In real Kitaev materials, such as α-RuCl₃⁸¹ and γ-Li₂IrO₃⁸², the ideal quantum spin liquid ground state predicted by the pure Kitaev model is disrupted by additional perturbations, such as Heisenberg exchange, off-diagonal symmetric exchange, and further-neighbor interactions. These deviations from the pure Kitaev model give rise to long-range magnetic order at low temperatures such as zigzag or spiral patterns. The intricate balance between these interactions not only stabilizes these ordered phases but also offers a deeper understanding of the glassy behaviors and emergent phases in Kitaev magnets.

The emergence of glassiness in Kitaev materials, particularly in systems like α-RuCl₃, arises from the interplay between high Z₂ vortex density, magnetic fields, and the intrinsic topological constraints of the spin-liquid phase^11,31. As the magnetic field exceeds a critical value, the system transitions into a dense vortex regime where the proliferation of Z₂ fluxes generates local frustration, impeding long-range order and trapping the system in numerous metastable states^31,83. These flux excitations, characterized by the flux operator W_p, exhibit slow dynamics that mirror the energy landscape of classical spin glasses but are distinctly topological. This emergent freezing phase retains quasi-long-range correlations with a substantial flux correlation length, despite the absence of crystallization¹¹. The autocorrelation function 〈W_p(t)W_p(0)〉, which decays slowly, reflects the sluggish dynamics of the glassy phase. The spin-lattice relaxation rate $\frac{1}{{T}_{1}}$ is related to the fluctuating local magnetic field h_loc(t) of electronic origin at the nuclear site, coupled to the nuclear spins as: $\frac{1}{{T}_{1}}\propto \int_{0}^{\infty }\langle {h}_{{{{\rm{loc}}}}}(t){h}_{{{{\rm{loc}}}}}(0)\rangle \,dt$, where 〈h_loc(t)h_loc(0)〉 is the autocorrelation function of the local magnetic field⁴⁴. In a conventional magnetic material, this correlation decays rapidly as spin fluctuations die out. However, in this emergent frozen state, the decay slows significantly due to the quasi-static nature of Z₂ fluxes or Majorana modes. Consequently, the NMR signal, rather than freezing completely, will result in a finite relaxation rate down to the lowest experimentally accessible temperatures. This behavior can be probed directly through NMR experiments, providing valuable insight into the glassy dynamics and persistent flux correlations. The enhanced second- and third-order susceptibilities in this phase¹⁰, along with the slow autocorrelation decay, highlight the non-Gaussian and chaotic nature of the fluctuations¹¹. The system becomes trapped in local minima, forming a glassy phase dominated by flux operators rather than spins, making Kitaev magnets a unique platform to explore glassy behavior driven by quantum fluctuations, non-trivial excitation spectra, and short-range spin correlations.

The substitution of Ag (which is larger and less electronegative than Li) tends to expand the lattice due to the increased ionic radius leading to the decrease of chemical pressure in going from Li₂RhO₃ to Ag₃LiRh₂O₆. The transition from Li₂RhO₃ to Ag₃LiRh₂O₆ involves a topochemical reaction that introduces Ag ions, resulting in a trigonal distortion of the octahedral coordination around Rh⁸⁴. This structural modification enhances the competition between spin-orbit coupling and crystal field effects, leading to the emergence of a new magnetic ground state characterized by robust antiferromagnetic ordering. Consequently, Ag₃LiRh₂O₆ exhibits significantly stronger magnetic interactions compared to the spin-glass behavior observed in Li₂RhO₃⁸⁴. The intricate interplay between quantum fluctuations, spin rigidity, and exchange frustration in magnetic systems gives rise to a rich tapestry of emergent phases, elegantly captured in a triangular phase diagram (Fig. 5b). The vertices of this diagram represent distinct physical states: the Kitaev spin liquid, characterized by fractionalized excitations and long-range quantum entanglement; topological freezing, where disorder and frustration arrest spin dynamics into a glassy, localized state; and zig-zag or spiral ordering, defined by classical long-range magnetic order. The edges connecting these phases are governed by parameters such as strength of quantum fluctuation(Q), spin stiffness, chemical pressure, hydrostatic pressure, and exchange frustration. Understanding these transitions not only deepens our knowledge of magnetic frustration but also offers insights into topological phases, spin dynamics, and disorder effects. Spin stiffness representing the rigidity of the spin configurations, is defined as: $\rho =\frac{1}{N}{\sum }_{q}\left(\frac{{\partial }^{2}E(q)}{\partial {q}^{2}}\right),$ where E(q) is the spin wave dispersion. Systems with high stiffness stabilize classical long-range orders, such as zig-zag or spiral configurations, whereas reduced stiffness enables localized or frozen states. Quantum fluctuations, arising from zero-point motion, renormalize this stiffness through corrections (Δρ(Q)), yielding an effective stiffness ρ_eff = ρ₀ − Δρ(Q)^85,86. Physically, strong quantum fluctuations (Δρ(Q) ~ ρ₀) in the Kitaev spin liquid phase collapse ρ_eff, leading to a dynamic, entangled state without classical long-range order. In the topological freezing phase, intermediate stiffness (ρ_eff > 0) reflects a balance between frustration and fluctuations, stabilizing localized, non-coherent spin dynamics²³. Conversely, in the zig-zag or spiral ordering phase, high stiffness (ρ_eff ≈ ρ₀) enables robust classical order, with quantum fluctuations playing only a perturbative role³². This interplay between spin stiffness and quantum fluctuations underscores the emergence of diverse magnetic phases and their transitions in Kitaev systems, highlighting the intricate coupling of quantum and classical effects.

In the phase diagram (Fig. 5b inset), we highlight that Kitaev magnets can exhibit two distinct low-energy features: a Dirac band or a flat band, depending on the specific parameter regime of the Hamiltonian. This duality arises from the delicate balance between the gauge symmetries and energy dispersion within the Kitaev model^9,31,63,87. In the uniform flux configuration (i.e., W_p = +1 for all plaquettes), the low-energy spectrum exhibits gapless Dirac points, reminiscent of graphene’s Dirac cone structure. However, when next-nearest-neighbor interactions are introduced, they break time-reversal symmetry, thereby gapping out the Dirac points and leading to topologically non-trivial Chern bands characterized by a quantized Chern number^9,31. In specific gauge configurations, the energy dispersion flattens, leading to flat bands. The emergence of flat bands in Kitaev magnets originates from the cancellation of kinetic energy contributions of Majorana fermions, driven by the destructive interference of hopping matrix elements. The hopping matrix T_ij, which governs the amplitude and phase of fermion hopping between sites i and j, is intrinsically coupled to the background ${{\mathbb{Z}}}_{2}$ gauge fields³¹. Decomposing T_ij into symmetric and antisymmetric components reveals their distinct roles in shaping the energy dispersion. When the antisymmetric component dominates, the interference of hopping phases leads to a suppression of the effective kinetic energy, flattening the band structure over specific regions of the Brillouin zone and resulting in highly degenerate energy states. This can be captured by the flattening of the energy dispersion ϵ(k) over the Brillouin zone, where the second derivative ∂²ϵ(k)/∂k² ≈ 0, indicating negligible curvature and thus a flat band³¹. In the classical limit, the flat energy landscape is associated with to the degeneracy of ground states in the large-spin (S) Kitaev model. The total energy for each dimer configuration, which minimizes the energy, scales as  − JSN/2, and the number of these configurations grows exponentially with the number of lattice sites N, scaling as (1.662)^N63. These degenerate ground states are discrete in spin space but are connected by continuous “valleys” in energy, producing a flat band-like behavior. Even when quantum fluctuations are included via a spin-wave analysis (which introduces corrections in powers of 1/S), the system retains this flatness, reflecting the resilience of the classical degeneracy to quantum corrections^63,87. Hence, the flat band observed in both the quantum and classical regimes of the Kitaev model reflects deep connections between gauge symmetries, frustration, and the system’s topological structure. The experimental observation of flat-band energy dispersion in the promising J_eff = 1/2 Kitaev magnet Na₂Co₂TeO₆ is a step forward in this direction^88,89,90. The existence of flat-band energy dispersion is supported by the prevalence of significant ground state degeneracy, short - range spin correlations, and gapless excitation spectra, as probed by thermodynamics, NMR, and INS experiments on the Kitaev materials presented here. This scenario elucidates the innate connection between topology, electron correlation, and symmetry, signaling the rich potential of this class of frustrated magnets in realizing novel organizing principles and quantum phenomena^1,2,3,9.

Discussion

The honeycomb magnet stabilizes on a bipartite lattice, and the Heisenberg exchange interaction between spins leads to a Néel state at low temperatures. However, spin-orbit driven anisotropic bond-dependent frustrated Kitaev exchange interaction between J_eff = 1/2 spins decorated on a honeycomb lattice can host a quantum spin liquid state with deconfined fractional excitations^2,3. The frustrated Kitaev magnets on a honeycomb lattice are potential candidates to host non-trivial spin-freezing of topological origin¹¹. The order function formalism associated with remanent properties of frustrated magnets may be of paramount importance in elucidating a broad class of disordered systems including spin glasses³⁸. The low-temperature spin freezing behavior observed in pristine and doped frustrated Kitaev magnets Li₂RhO₃, Na₂Ir_1−xTi_xO₃, Ru_1−xCr_xCl₃, Li₂Ir_1−xTi_xO₃, and Na₂Ir_1−xRu_xO₃ exhibit characteristic features including low-energy excitations akin to the hydrodynamic modes with gapless spectra proposed for frustrated spin glass materials²³. The presence of a linearly dispersing mode is evident in the low-temperature thermodynamic, NMR, and neutron scattering experiments. The T² behavior of magnetic specific heat below the freezing temperature remains independent of the magnetic field, which is quite different from that observed in conventional spin-glass materials. The higher value of the characteristic energy scale, τ₂, with respect to the freezing temperature T_g (Table 1), may be due to the underestimation of next-nearest neighbor interactions in the HS framework; however, this can be addressed by increasing the number of nearest neighbors z in honeycomb spin-lattice from three to some effective higher number.

In general, the conventional spin glass materials do not support the spin wave excitations where the spin stiffness ρ_s = 0. Spin wave excitations arise from clusters of spins, each of which exhibits finite bounds with finite spin stiffness constant. These spin clusters undergo independent spin freezing that is attributed to jamming effects²². The observed spin fluctuations in frustrated Kitaev magnets, as probed by neutron scattering measurements, down to freezing temperatures, likely originating from the spins located at the boundaries of spin clusters⁹¹. Taking into account Anderson’s proposal regarding the areal scaling of free energy fluctuations at the boundaries of clusters, one can expect the emergence of a rugged energy landscape beneath an initially flat energy landscape^4,29. It is worth noting that the HS mode is coupled with both electronic and nuclear magnetic moments due to hyperfine coupling between the nucleus and the surrounding electronic moments²⁷. Understanding the nature of spin correlations and fluctuations of hyperfine fields in the ground state of the frustrated Kitaev magnet is highly significant, as it may underpin elusive Kitaev QSL and associated low energy excitations. In this context, microscopic techniques such as NMR provide an excellent probe for tracking the intrinsic magnetic susceptibility, nature of spin correlations, and associated unconventional low-energy excitations in the ground state of these frustrated magnets^3,44. The power law behavior of the NMR spin-lattice relaxation rate supports thermodynamic results in these Kitaev magnets.

Unlike the hierarchical organization of free energy observed in conventional spin glasses, a non-hierarchical structure is posited, supported by evidence of memory and aging effects in frustrated kagome structures such as SCGO, BCGO, and Kitaev magnets like Li₂RhO₃ and Na₂Ir_0.89Ti_0.11O₃^6,28. The aging behavior exhibits diminished prominence, necessitating a large time scale observations to manifest it. The presence of a subtle memory effect implies the development of a nonhierarchical landscape featuring a wide, nearly flat, but rough bottom^21,28. The non-Abelian nature of defect propagation in anisotropic kagome was proposed for the memory effect in SCGO like nearly defect-free kagome materials⁴. The defect-free hyperhoneycomb β-ZnIrO₃ and β-MgIrO₃ Kitaev magnets with dominant ferromagnetic exchange interaction exhibit intrinsic glassiness, necessitating further theoretical investigation. The HS framework provides interesting insights into the spin correlations and spin-freezing mechanism in 2D and 3D Kitaev magnets. The parameters derived from the analysis of magnetic susceptibility and specific heat results following HS framework on a few selected Kitaev magnets are documented in Table 1. The low-energy excitations of the spin jam state are primarily governed by the HS mode, wherein spin wave excitations emerge from the unconventional spin-freezing dynamics. This is in stark contrast to conventional spin glasses characterized by localized two-level energy systems, where the excitations occur through the barrier mode. Our analysis suggests that the HS model is quite apt and generic in elucidating the low temperature spin freezing mechanism in a large class of frustrated quantum magnets²³.

Overall, while the semiclassical framework of the HS formalism and spin-jam theory may initially seem incompatible with the extreme quantum spins of Kitaev magnets studied in this paper, the experimental evidence highlights the need for further theoretical development to fully elucidate the underlying physics. In the quantum regime, quantum fluctuations due to frustration can lead to a multitude of metastable states and quantum tunneling effects, differing from the semiclassical picture. This complexity is illustrated in the spin-1/2 Heisenberg-Kitaev kagome magnet, where anisotropic Kitaev interactions drive the quantum-to-classical crossover⁹². The spin-1/2 system, Sr₂CuTe_0.5W_0.5O₆ exhibits glassy behavior below T_g ≈ 1.7 K. The imaginary part of dynamic susceptibility χ′′ ∝ ℏω for ℏω < k_BT_g, suggesting low-energy Goldstone modes³⁶. The specific heat shows a quadratic temperature dependence, C ∝ T², below the spin-freezing temperature, indicating linear dispersion of low-lying magnon excitations⁶⁷. Additionally, the specific heat exhibits a broad maximum around 1.7 K., with field-independent behavior up to 9 T. These experimental findings suggest the presence of a quantum analog of spin jam in this diluted and low-spin quantum material, which could provide deeper insights into the nature of disordered quantum magnets and their excitations. The randomness leaves the emergent gauge fields as distinct degrees of freedom and prevents the system from undergoing order by disorder, leading to the formation of a frozen state with spontaneously broken symmetry under global spin rotations³⁶. This study provides insights into how weak exchange randomness influences the emergence of hydrodynamic modes in low-spin frustrated magnets⁹³, shedding light on the intricate interplay between emergent gauge fields and low-energy excitations in these systems. The rare-earth triangular oxide TmMgGaO₄ features Tm³⁺ ions with a 4f¹² electron configuration and a spin-orbit moment J = 6 with strong Ising anisotropy⁹⁴. Crystal-electric-field splitting separates these ions into a non-Kramers doublet, well separated from other energy levels by about 400 K, allowing the doublet to be treated as an effective spin-1/2 (J_eff = 1/2) system^94,95. In this quantum Ising magnet, the presence of quenched disorders, specifically weak non-magnetic disorders due to Ga and Mg site sharing, gives rise to the Halperin-Saslow mode. It is proposed that weak quenched disorder transforms the emergent 2D Berezinskii–Kosterlitz–Thouless (BKT) phase and the critical region into a gauge glass⁹⁶. This transformation leads to the emergence of the Halperin-Saslow mode associated with the gauge glass in the quantum Ising magnet TmMgGaO₄. In applying a hydrodynamic approach to model the disordered Kitaev materials, we recognize that this method serves as a phenomenological tool rather than a microscopic theory. However, the hydrodynamic framework effectively captures the essential low-temperature behaviors, including gapless nature of low-energy excitations that emerge in the presence of disorder. This approach allows us to describe the collective modes and associated thermodynamic responses that arise from the complex interplay of spin-orbit coupling, Mott physics, and disorder without requiring detailed microstate calculations. As such, our model represents a simplified yet scientifically grounded interpretation of the magnetic dynamics observed in these materials.

The possible scenarios that can elucidate the observed unconventional low temperature behavior of frustrated Kitaev magnets are: (i) In the framework of the Kitaev model, the distinctive spin-orbit driven bond-dependent anisotropic interactions engender a set of conserved quantities intimately associated with the spins, often denoted as integrals of motion⁹. The existence of this substantial ensemble of conserved quantities such as Z₂ flux operator W can facilitate the emergence of weakly broken ergodicity, thereby leading to the intrinsic manifestation of glassy behavior in the proximity of a clean system without doping¹⁴. (ii) In the absence of ZFC-FC bifurcation, the analysis of higher-order susceptibility can provide valuable insights into the intrinsic spin-glass phase, manifesting distinctive anomalies in the vicinity of the glass transition. (iii) Remarkably, intrinsic glassiness represents a plausible magnetic phase that can endure within the realm of quantum fluctuations and topological defects that may underpin elusive Kitaev QSL.

Conclusions

Frustrated Kitaev quantum materials are potential candidates for hosting intriguing quantum phenomena, such as topological spin glasses and nontrivial low-energy excitations. Detailed insights into these phenomena through complementary experimental techniques may provide vital clues to the sought-after Kitaev QSL. Our comprehensive analysis and discussion, based on Halperin-Saslow hydrodynamic linearly dispersive modes, elegantly capture the essence of the spin-freezing mechanism, which closely aligns with what is expected in the spin-jam framework. The power law behavior of magnetic specific heat at low temperatures in a few representative Kitaev magnets decorated on 2D and 3D spin lattices manifests the persistence of abundant low-energy topological excitations in conjunction with macroscopic ground state degeneracy, slow spin dynamics, and short-range spin correlations, which is in sharp contrast to that observed in canonical spin-glass magnets. In addition, our analysis of magnetic specific heat below the spin-freezing temperature in Kitaev spin glass materials following the HS framework suggests the dominant role of low-energy excitations. Our results, based on the comparative account of thermodynamic NMR, and neutron scattering experiments in a complementary scale demonstrate that frustrated Kitaev materials are promising candidates for exhibiting an intrinsic glassiness, the genesis of which can be attributed to the effects of quantum fluctuations, low energy excitations, and the imposition of topological constraints. The distinctive character of the free energy landscape and non-trivial behavior of frustrated Kitaev magnets enable us to establish a rich phase diagram that may provide a new direction for further exploration of 2D and 3D Kitaev magnets’, which are potential contenders for hosting quantum and topological states with elusive fractional excitations. The measurement of non-linear susceptibility as a direct probe of the Edward-Anderson order parameter may shed insights into the free energy landscape and non-trivial low temperature spin-freezing in frustrated Kitaev magnets. A detailed inelastic neutron scattering and low-temperature NMR experiments on the high-quality single crystals, conducted below the spin-glass temperature, may provide profound insights into the low-energy excitations in this class of frustrated magnets. Such insights are pivotal for understanding topological order and for detecting and manipulating exotic fractional excitations in Kitaev magnets. The intriguing question of whether frustrated Kitaev magnets can exhibit spin-freezing without quenched disorder prompts further investigation. Additionally, the prevalence of spin-glass in proximate Kitaev quantum spin liquid candidates invokes further studies in this direction. Such efforts may broaden our understanding of complex magnetic systems, establish realistic Hamiltonian, and pave the way to gain new insights into the behavior of frustrated magnets under the influence of various external perturbations such as doping, pressure, and applied magnetic fields. These are crucial for underpinning novel topological and quantum states and associated low-energy excitations, as well as the organizing principles for designing Kitaev quantum materials with optimal functionality.

The phenomenological scenario proposed in this work is simple, yet it captures some of the key attributes of macroscopic ground state degeneracy, low-energy excitations, and short-range correlations, as manifested in complementary thermodynamic, NMR, and neutron scattering experiments in emblematic frustrated Kitaev magnets that host non-trivial spin-freezing phenomena. This invokes comprehensive microscopic theoretical models for a deeper understanding of the complex many-body state of promising frustrated magnets that may eventually lead to the faithful experimental realization of novel quantum phenomena with exotic quasi-particle excitations, which are essential for addressing some of the fundamental questions in condensed matter and represent a step forward for next-generation technologies to meet the challenges of ever-increasing demands for energy, data storage, and computing.

Methods

τ ₁ and τ ₂ for doped Kitaev magnets

In the presence of doping with two magnetic elements of the same site, the two temperature scales can be expressed as:

$${E}_{1}=\frac{2{(x{g}_{2}{\mu }_{B})}^{2}{S}_{2}({S}_{2}+1){N}_{A}}{z{\chi }_{M}}+\frac{2{((1-x){g}_{1}{\mu }_{B})}^{2}{S}_{1}({S}_{1}+1){N}_{A}}{z{\chi }_{M}}={k}_{B}{\tau }_{1}$$

(10)

and

$${E}_{2}=\frac{3{n}_{p}\zeta (3)}{\pi }\frac{{k}_{B}^{3}}{{(x{g}_{2}{\mu }_{B}+(1-x){g}_{1}{\mu }_{B})}^{2}}\frac{{\chi }_{M}}{{C}_{m}/{T}^{2}}={k}_{B}{\tau }_{2}.$$

(11)