Optical detection of bond-dependent and frustrated spin in the two-dimensional cobalt-based honeycomb antiferromagnet Cu₃Co₂SbO₆

Abstract

Two-dimensional honeycomb antiferromagnets are promising materials class for realizing Kitaev quantum spin liquids. The signature of these materials includes anisotropic bond-dependent magnetic responses and persistent fluctuations in paramagnetic regime. Here, we propose Cu₃Co₂SbO₆ heterostructures as an intriguing candidate, wherein bond-dependent and frustrated spins interact with optical excitons. First-principles spin Hamiltonian calculations and in-plane anisotropic critical fields suggest strong frustration and dominant Kitaev exchange interactions. Optical spectroscopy reveals exciton coupled to frustrated magnetism, enabling optical detection of spin states. Spin-exciton coupling displays anisotropic responses to light polarization along the bond-parallel and the bond-perpendicular directions, highlighting Kitaev interactions and persistent short-range spin correlations above twice the Néel temperatures. The robustness of short-range spin fluctuations under magnetic fields underscores the stability of the spin-fluctuation region. Our results establish Cu₃Co₂SbO₆ as an attractive candidate for exploring quantum spin liquid, where the spin Hamiltonian and quasiparticle excitations can be probed and potentially controlled by light.

Introduction

While at least 16,300 materials have been discovered/proposed as magnetic¹, fewer than 10 materials have been suggested as potential candidates for Kitaev quantum spin liquid (QSL)^2,3. Kitaev QSL represents a rare, exactly solvable ground state of the Kitaev two-dimensional honeycomb model⁴, characterized by massive quantum entanglement maintained without long-range order even at zero-temperature. Their excitations are described by fractionalized Majorana fermions and non-Abelian anyons, providing a promising avenue for achieving fault-tolerant quantum computation through the non-Abelian braiding process⁵. However, despite their intriguing properties, the realization of Kitaev QSL remains extremely challenging. This exotic model can be implemented in edge-shared octahedral systems with strong spin-orbit coupling⁶. Therefore, 4 d/5 d transition metal systems, such as α-RuCl₃^7,8,9,10 and Na₂IrO₃^11,12,13, are suggested as possible hosts for Kitaev QSL. Despite the emergence of long-range antiferromagnetic ordering at low temperature (T), these candidate materials still exhibit signatures indicative of proximity to Kitaev QSL. These signs include half-integer thermal Hall¹⁰, broad continuum in inelastic neturon⁸ and Raman scattering¹⁴, anisotropic bond-dependent responses¹⁵ and high-T spin fluctuations¹⁶.

Recent proposals have highlighted high-spin 3d⁷ cobalt-based honeycomb materials as potential candidates for Kitaev QSL^{17,18,19,20,21}. These suggestions are based on the ground state characterized by the effective total angular momentum J_eff = 1/2, which comprising the effective orbital momentum L = 1 and total spin momentum S = 3/2 of high-spin 3d⁷ electronic configuration. This ground state occurs despite its relatively smaller spin-orbit coupling compared to 4 d/5 d transition metal oxides. Furthermore, the localized nature of Co 3 d orbitals might suppress undesired next-nearest neighbor and direct exchange interactions. Various experimental and theoretical studies have provided mixed conclusions on the feasibility of Kitaev QSL in these 3d⁷ cobalt-based honeycomb systems. For instance, the antiferromagnetic 1/3 order, the Γ-M characteristics of the magnetic ordering wave vectors, and in-plane anisotropy in critical magnetic field of Na₃Co₂SbO₆ are well described as a bond-dependent Kitaev Hamiltonian with a positive off-diagonal term Γ₁^15,22,23. Using time-domain terahertz spectroscopy on BaCo₂(AsO₄)₂, a broad magnetic continuum, a signature of fractionalization, has been observed²⁴. Recent theoretical and experimental investigations, however, suggest that Co-Co direct hopping is too substantial to ignore, particularly in BaCo₂(AsO₄)₂²⁵_, leading to an isotropic XXZ-J₁-J₃ model, not Kitaev model¹⁵. Therefore, appropriate spin Hamiltonian for cobalt-based honeycomb antiferromagnet is remain elusive (Fig. 1a).

Fig. 1: Schematics of spin exchange interaction, spin-exciton coupling and the phase diagram of Cu₃Co₂SbO₆.

a Layered crystal structures of Cu₃Co₂SbO₆ with alternative Cu⁺ and (Co_2/3Sb_1/3)O₂^- layers. (Left), The possible dominant magnetic spin exchange interaction of (Co_2/3Sb_1/3)O₂^- layers has two-dimensional honeycomb structures. The blue circles indicate the cobalt atom. The blue, red, and green solid lines are anisotropic Kitaev spin exchange interactions along each direction. The black and sky arrows show the nearest and third-nearest neighbor isotropic Heisenberg spin exchange interaction, respectively. (Right), The schematic of exciton formation of Cu₃Co₂SbO₆. Within incident light, there is formation of excitons between Cu⁺ and (Co_2/3Sb_1/3)O₂^- layers, which interact with the spin exchange interaction shown in left panel. b Suggested magnetic phase diagram of Cu₃Co₂SbO₆ with H along the bond-parallel direction (PM: paramagnetic region, AFM antiferromagnetic region, SP spin-polarized region). The T_N and T_H is obtained from the χ(T) measurement and optical spectroscopy, respectively. Unconventional spin fluctuating regime was observed between low-T AFM and high-T PM phases.

Despite being scarcely studied, Cu₃Co₂SbO₆ can be an intriguing material system to investigate spin Hamiltonian and frustration in Co-based honeycomb oxides. Cu₃Co₂SbO₆ features an alternatively stacked structure comprising a Cu⁺ layer and a (Co_2/3Sb_1/3O₂)^- layer, characterized by honeycomb edge-sharing CoO₆ octahedra (Fig. 1a and Supplementary Fig. 1). The O-Cu-O dumbbell structure aligns Co/Sb atoms directly above those in subsequent layers, effectively eliminating the unwanted two-fold anisotropy²⁶. While this compound exhibits zigzag ordering at 16 K, the magnetic entropy releases occur twice, at 16 K and 60 K, with approximately half of the total entropy released around 40 K²⁷. The heat capacity between these two releases is linearly related to T, similar to experimental observation in other Kitaev QSL candidate α-RuCl₃ and theoretical calculation^28,29. However, the presence of inevitable magnetic impurities in bulk samples has constrained further studies on its magnetic properties and relevance to Kitaev physics²⁷.

Here, we propose Cu₃Co₂SbO₆ heterostructures as intriguing frustrated magnet wherein there’s an interplay between bond-dependent/frustrated spins and optical excitons. From the ab initio calculation, we find strong magnetic frustration with dominant Kitaev spin interaction. Although single-phase Cu₃Co₂SbO₆ does not exist in nature²⁷, we achieved its synthesis through heterostructure epitaxy on both ZnO and MgAl₂O₄ substrates. A pronounced in-plane anisotropy in critical magnetic fields, H_C, between bond-parallel and bond-perpendicular directions was observed, consistent with the presence of anisotropic Kitaev term¹⁵. A distinctive characteristic of Cu₃Co₂SbO₆ compared to other Kitaev systems is the formation of excitons between Cu⁺ and (Co_2/3Sb_1/3O₂)^- layers and their strong interaction with quasi-2D frustrated magnetism (Fig. 1a). Through optical spectroscopy, we identified a strong exciton near 4 eV, exhibiting peculiar spin-exciton coupling. We observed a clear anomaly not only at Néel temperature T_N ≈ 16 K, but also an additional temperature at T_H ≈ 40 K in the raw ellipsometry parameters, spectral weight (SW), and peak position of the excitonic peak. The occurrence of T_H, which is indicative of non-zero short-range spin-spin correlation functions far above the T_N, implies the presence of strong frustrated exchange interactions, a key ingredient of QSL. Additionally, the SW transfer shows noticeable anisotropy between light polarization along the bond-parallel and the bond-perpendicular directions, indicating considerable anisotropic Kitaev exchange interaction. The T_H showed little magnetic field (H) dependence, underscoring the robustness of the spin fluctuation region. Based on H-dependent χ and optical spectroscopy, we constructed a T-H phase diagram (Fig. 1b), which highlights the presence of unconventional spin fluctuation regime between antiferromagnetic and conventional paramagnetic phases.

Results

First-principles calculation for spin Hamiltonian of Cu₃Co₂SbO₆

First-principles calculations demonstrated that Cu₃Co₂SbO₆ retains a leading ferromagnetic Kitaev term and moderate non-Kitaev terms, which leads to substantial magnetic frustration. The spin Hamiltonian was computed by leveraging a recently developed fourth-order strong coupling perturbation theory²⁵. Details on the tight-binding parameters can be found in the supplementary information (Supplementary note 1). We found that the spin Hamiltonian of this compound has a leading ferromagnetic Kitaev interaction (K = −2.207 meV), antiferromagnetic third-neighbor nearest Heisenberg interaction (J₃ = 1.664 meV), ferromagnetic nearest-neighbor Heisenberg interaction (J₁ = −1.040 meV), and additional symmetric anisotropy interaction terms (Г = −0.243 meV) in the order of energy (Fig. 1a and Table 1). As both Kitaev and J₃ exchange interaction induce magnetic frustration with J₁, our first-principles calculations suggest that Cu₃Co₂SbO₆ is a frustrated magnet.

Table 1 Nearest- and third-neighbor exchange interactions from ab-initio calculation results (in meV)

Synthesis of single-crystalline, epitaxial Cu₃Co₂SbO₆ thin film

We successfully synthesized single-phase Cu₃Co₂SbO₆, which does not naturally exist, via epitaxy on ZnO and MgAl₂O₄ substrates. During the synthesis of bulk samples, substantial magnetic impurities (Co,Sb)₃O₄ were inevitably formed due to the similarity between ordered stacking temperature (1250 °C) and decomposition temperature (1260 °C). This has hindered the understanding of intrinsic properties of Cu₃Co₂SbO₆^27,30. However, in thin film geometry, the epitaxial relationship with the substrate and its energetic synthesis mechanism can result in different consequences from bulk synthesis³¹. Single-phase, epitaxial Cu₃Co₂SbO₆ film was confirmed by X-ray diffraction, χ(T), and transmission electron microscopy (Supplementary Fig. 2–4). Notably, χ(T) of bulk Cu₃Co₂SbO₆ shows a strong ferromagnetic transition near 60 K due to the presence of Co₅SbO₈ impurities²⁷. In contrast, our χ(T) of the Cu₃Co₂SbO₆ thin film exclusively exhibits an antiferromagnetic transition near 16 K, a distinctive signature of the formation of single-phase Cu₃Co₂SbO₆. We have confirmed that all physical properties of fully relaxed Cu₃Co₂SbO₆ are independent of the substrate (Supplementary Fig. 2, 4, 5).

Anisotropic H _C between bond-parallel and bond-perpendicular directions in Cu₃Co₂SbO₆

For a deeper understanding of the magnetic ground state of Cu₃Co₂SbO₆, we analyzed the magnetic anisotropy of its thin films. Despite the presence of six-fold twin domains in the thin film, the bond-parallel and bond-perpendicular directions can be uniquely determined. We show magnetic susceptibility of Cu₃Co₂SbO₆ film on ZnO substrate with H parallel to the bond (χ_bond//), perpendicular to the bond (χ_bond⊥), and perpendicular to the ab plane (χ_c), as displayed in Fig. 2a–c and Supplementary Fig. 6. Clear sharp kinks were evident in χ_bond//(T) and χ_bond⊥(T) near 16 K, while the change in χ_c(T) was relatively weaker and broader, indicating a dominant spin direction in the ab plane. As H increased, the kinks in χ_bond//(T) and χ_bond⊥(T) shifted to lower T and became less discernible, whereas the ones in χ_c(T) remained robust even at the highest H. Figure 2d–f show contour graphs of dχ/dT across different H and T for the vertical and horizontal axis, respectively. These graphs visualize the paramagnetic (blue region) and antiferromagnetic (red region) phases. While in-plane H significantly reduces the T_N, the out-of-plane H hardly affects the magnetic properties. Similar observations have been frequently reported in quasi-2D magnets^24,32,33.

Fig. 2: Bond-dependent antiferromagnetism in Cu₃Co₂SbO₆.

χ(T) of Cu₃Co₂SbO₆ under different orientations of applied H along a the bond-parallel direction, b the bond-perpendicular direction, and c, perpendicular to honeycomb planes. Full datasets are listed in Supplementary Fig. 6. The T and H dependent contour plot of derivative susceptibility, dχ/dT, along d the bond-parallel direction, e the bond-perpendicular direction, and f, perpendicular to honeycomb planes. The antiferromagnetic and paramagnetic regions are indicated by red and blue areas, respectively. g An overlapped graph of χ_bond//(T) and χ_bond⊥(T) presenting in-plane magnetic anisotropy under various H. h M-H and dM/dH curves with H along the bond-parallel and bond-perpendicular directions. Open and closed triangles (circles) indicate the bond-parallel and bond-perpendicular directions M-H (dM/dH) curves, respectively. The error bar is smaller than the size of point. The vertical dashed lines indicate the H_C for bond-parallel and bond perpendicular directions. These results support the presence of considerable anisotropic Kitaev term.

In our study, we not only identified a considerable anisotropy between the in-plane and out-of-plane responses but also observed a distinct anisotropy in H_C between the bond-parallel and bond-perpendicular directions. To clarify the difference between the bond-parallel and bond-perpendicular directions, we plotted χ_bond//(T) and χ_bond⊥(T) together as shown in Fig. 2g. At lower field H = 10 kOe, χ_bond//(T) and χ_bond⊥(T) overlapped almost perfectly. However, as we increased the H, χ_bond//(T) and χ_bond⊥(T) began to deviate from each other. The upturn in low T for χ_bond//(T) is more evident than in χ_bond⊥(T), suggesting the antiferromagnetic state is more fragile to H in the bond-parallel direction. Figure 2h displays the M-H and dM/dH curves for both directions at 2 K. The peak structures in dM/dH typically indicate the spin-flip transition. The H_C are approximately 45 ± 2.5 kOe and 60 ± 2.5 kOe for the bond-parallel and bond-perpendicular directions, respectively, further corroborating that the antiferromagnetic state is more vulnerable in the bond-parallel direction. This is consistent with the suggested zigzag antiferromagnetic ordering in bulk, where spins align along the bond-perpendicular direction³².

The observed in-plane anisotropic H_C below T_N confirms the presence of considerable anisotropic Kitaev spin exchange interaction, consistent with theoretical calculation. The appropriate spin Hamiltonian for high-spin 3d⁷ cobalt-based materials —whether it is the XXZ-J₁-J₃ or Generalized Heisenberg-Kitaev (GHK) model—remains a subject of ongoing debate. A key distinction between these two models is the presence of in-plane magnetic anisotropy of H_C¹⁵. Specifically, the GHK model has different H_C between bond-parallel and bond-perpendicular directions in the ab plane, while the XXZ-J₁-J₃ model retains full rotational symmetry, despite a slight orthorhombic distortion. For instance, in α-RuCl_3, the H_C differs between bond-parallel and bond-perpendicular direction³⁴. Conversely, the XXZ-J₁-J₃ model, applied to BaCo₂(AsO₄)₂, exhibits negligible in-plane anisotropy in its H_C for field-induced transitions¹⁵. The evident in-plane anisotropic H_C of Cu₃Co₂SbO₆ coincides with the theoretical result which exhibits predominant Kitaev spin exchange interactions (Table 1).

Coupling between exciton and frustrated spin in two-dimensional honeycomb layer of Cu₃Co₂SbO₆

A unique feature of the antiferromagnetism observed in Cu₃Co₂SbO₆, distinguishing it from other Co-based honeycomb systems, is its strong interaction with excitons. Figure 3a displays the real part of the optical conductivity, σ₁(ω), of a Cu₃Co₂SbO₆ film on MgAl₂O₄ substrate at 6 K obtained through ellipsometry. Within this energy range, we distinguished four major optical excitations; notably, one near 4 eV exhibits anomalously high intensity, indicative of excitonic transition. To analyze these optical transitions, we conducted band structure calculations by employing density functional theory plus on-site Coulomb interactions (DFT + U). Figure 3b displays the orbital-projected density of states of Cu₃Co₂SbO₆, where the first two peaks are identified as Cu 3 d → Co 3 d t_2g, and Cu 3 d → Co 3 d e_g, respectively, while other peaks at higher energy correspond to charge-transfer excitations from O 2p states. Note that transitions from Cu 3 d to Sb 4 s are optically forbidden due to their hybridization with orthogonal O 2p orbitals (Supplementary Fig. 7–9). The considerable intensity difference between Cu 3 d → Co 3 d t_2g transitions and Cu 3 d → Co 3 d e_g transitions can be attributed to the smaller orbital overlap between Cu 3 d and Co t_2g orbitals compared to that between Cu 3 d and Co e_g orbitals (Supplementary Note 2). Therefore, the most intense transition can be assigned to the Cu 3 d → Co 3 d e_g exciton.

Fig. 3: Optical exciton and spin-exciton coupling in Cu₃Co₂SbO₆.

a σ₁(ω) at 6 K with polarization along the bond-perpendicular direction (open circles) and Lorentz-Gaussian oscillator fitting (solid lines) for Cu₃Co₂SbO₆. The peak is assigned based on orbital-projected density of states in Fig. 3b. b the orbital-projected density of state of bond-perpendicular oriented zigzag ordering Cu₃Co₂SbO₆ obtained from DFT + U. The horizontal dashed line is Fermi level. The solid arrows indicate the optical transition from Cu 3 d to Co 3 d (shown in Fig. 3a). The T-dependent c, the ellipsometry data Ψ d, exciton peak position, e peak intensity, and f scattering rate. The error bars in c correspond to standard deviations derived from random noise, while error bars in d, e and f represent the figure of merit, calculated as the standard 90% confidence limit multiplied by square root of the mean-squared error. The red line in Fig. 3d is Bose-Einstein statistics fitting functions implemented at high T data, generically found in spectra due to phonon contribution of thermal broadening. The black arrows indicate the T_N and T_H. The distinct kink below T_N in all fitting parameters implies the existence of coupling between the exciton and antiferromagnetic ordering. The clear additional anomaly in Ψ and peak position at T_H indicates the presence of non-zero short-range spin-spin correlation functions above T_N.

The T-dependence of the excitonic transition reveals an unconventional spin-exciton coupling, manifested not only through T_N but also additionally through T_H ~ 40 K. Figure 3c exhibits the raw ellipsometry parameters, Ψ, which represents the amplitude ratio between reflected p- and s- polarized light as a function of T at exciton peak. Here, s- polarized light is perpendicular with Co-Co bond direction. Naively speaking, it reflects the T-dependent absorption coefficient at the exciton energy (Supplementary Fig. 10). Without employing any model fitting, we identified two discernible features: a kink around T_N ~ 16 K and an additional anomaly near T_H ~ 40 K, which is more than twice of T_N. This unconventional kink implies a robust spin-exciton coupling that remains even above T_N. For a quantitative analysis, we conducted a simultaneous fitting of σ₁(ω) and ε₁(ω) using Lorentz-Gaussian oscillator models (Fig. 3a and Supplementary Fig. 11). Figure 3d-f display the peak position, peak intensity, and scattering rate of the excitonic transition as a function of T, respectively. All these fitting parameters show clear anomaly at T_N through the spin-exciton coupling. In particular, the peak’s blueshift begins around T_H, not T_N, consistent with the observed T-dependence in Ψ.

The origin of T_H can be understood from the SW redistribution between exitonic transitions at lower energy and charge-transfer transitions at higher energy. Figure 4a exhibits the polarization-dependent σ₁(ω), s-polarized light along the bond-parallel (E_bond//) and bond-perpendicular directions (E_bond⊥), of Cu₃Co₂SbO₆ at 6 and 300 K, featuring a clear isosbestic point (equal absorption) ℏω_iso near 4.3 eV (marked by a red open circle). The integration of σ₁(ω) up to this point reflects the effective number of electrons contributing to excitonic transitions, expressed as N_eff (ω=ω_iso, T) = 2m₀V/πe²${\int }_{0}^{\omega } {\sigma }_{1}(\omega ^{\prime} ){{\rm{d}}}\omega ^{\prime}$, where m₀, V, and e represent the free electron mass, the unit cell volume, and the electron charge, respectively. Figure 4b shows N_eff (ω=ω_iso, T), which exhibits two notable anomalies at T_N and T_H regardless of polarization orientation, consistent with T-dependent Ψ and peak energy. A decrease of N_eff (ω=ω_iso, T) indicates a transfer of SW from lower-energy excitonic transitions to higher-energy charge-transfer transitions. Figure 4c, d, and e display the bond-parallel direction N_eff (ω, 40 K) - N_eff (ω, 75 K), N_eff (ω, 16 K) - N_eff (ω, 40 K), and N_eff (ω, 6 K) - N_eff (ω, 16 K), which represent T-evolution of SW transfer above T_H, between T_H and T_N, and below T_N, respectively. Above T_H, there is no SW transfer across the isosbestic point, suggesting the preservation of excitonic SW. In contrast, SW shifts from lower (red regime) to higher energies (blue regime) below both T_H and T_N. Remarkably, the overall shape of SW transfer below T_N and T_H is almost identical, strongly suggesting that the origin of T_H is also magnetic.

Fig. 4: SW transfer from different temperature range to explore the origin of T_H and bond-dependent spin exchange interaction.

a The σ₁(ω) of Cu₃Co₂SbO₆ at 6 K and room temperature. The red circle indicates the isosbestic point. Solid and dotted lines are σ₁(ω) along the bond-perpendicular (E_bond⊥) and the bond-parallel directions (E_bond//), respectively. The inset visualizes the relationship between the polarization and bond directions of Cu₃Co₂SbO₆. b T-dependence of effective number of electrons accord with excitonic transition, N_eff (ω=ω_iso, T) = 2m₀V/πe²${\int }_{0}^{{\omega }_{{iso}}} {\sigma }_{1}(\omega ^{\prime} ){{\rm{d}}}\omega ^{\prime}$. The vertical lines display both characteristic temperature T_N and T_H. Black closed squares and red open squares indicate the bond-parallel and bond-perpendicular polarizations, respectively. The difference effective number of electrons along each direction between three different temperature range c, f above T_H (40 K–75 K), d, g between T_N and T_H (16 K–40 K), and e, h below T_N (6 K–16 K). Red and blue contours, divided by the isosbestic point, represent SW transfer before and after the isosbestic point, suggesting that the origin of T_H is linked to the magnetic structure. The quantitative differences between the bond-parallel and bond-perpendicular directions indicate a considerable anisotropic Kitaev spin exchange interaction.

An anomaly in the optical exciton observed at T_H suggests the presence of non-zero short-range spin-spin correlation functions driven by spin fluctuations, indicating strong frustration. Various research on optical transitions in Mott and charge-transfer insulators have demonstrated that spin Hamiltonians described by nearest-neighbor spin-spin correlations along the polarization direction led to SW redistribution and renormalization of excitation energy^35,36,37,38.

$$\frac{{a}_{0}{\hslash }}{{e}^{2}}\int _{0}^{\infty }{\sigma }_{1}^{\left({m}\right),\gamma }\left({\omega} \right)d\omega=-\pi \left\langle {H}_{m}^{\gamma }\right\rangle$$

(1)

Here, $\gamma$ denotes the polarization direction, m is the excited state, ${a}_{0}$ is the distance between two magnetic ions, and $\left\langle {H}_{m}^{\gamma }\right\rangle$ represents the superexchange energy of the m-state transition under $\gamma$-polarization³⁶. Owing to bilinear nature of spin Hamiltonian, such SW and energy renormalization is generally proportional to |M_sublattice | ² under the mean field approximation³⁹, where M_sublattice represents magnetization of magnetic sublattices. A comprehensive explanation on the observed SW transfer and peak shift below T_N, therefore, can be provided with increasing M_sublattice with decreasing T. However, the observed SW transfer and peak shift between T_N and T_H is intriguing and beyond the mean field regime, especially given that M_sublattice being zero above T_N. These can be attributed to the non-zero short-range spin-spin correlation functions due to spin-fluctuations. Such non-zero short-range spin-spin correlation functions above T_N have been extensively observed in frustrated system and are considered indicative of frustration and quantum fluctuation^{7,28,40,41,42,43}.

The presence of non-zero short-range spin-spin correlations above T_N and the appearance of T_H suggest considerable magnetic frustration. In a frustrated system, since the spin configuration of ground-state fails to meet all spin exchange demands, long-range order tends to manifest at temperatures substantially below the energy scale of these interactions. As a result, persistent energy gain from the spin exchange interactions supports the formation of short-range spin structure even above the ordering temperature. Although thermal energy at T between T_N and T_H is too elevated to establish long-range magnetic order, it is not sufficient to overcome the energy benefits provided by frustrated spin interaction.

In addition to the qualitative changes in SW, there is a quantitative difference in SW redistribution depending on the direction of polarization, further supporting the presence of a sizable anisotropic Kitaev spin Hamiltonian. When comparing Fig. 4c (for the bond-parallel direction) and Fig. 4f (for the bond-perpendicular direction), it is evident that the SW transfer is nearly identical for both directions above T_H. In contrast, a much more pronounced SW redistribution occurs in the bond-parallel direction below T_H and T_N (as shown in Figs. 4d, e, g, and h). This quantitative divergence reflects a difference in the strength of the superexchange interactions along the bond-parallel and bond-perpendicular directions, as described by Eq. (1). Based on our superexchange model for Cu₃Co₂SbO₆ (Table 1), the dominant energy terms are K, J₃, and J₁, when arranged in order of magnitude. Since the two isotropic Heisenberg interactions, J₁ and J_3, cannot produce in-plane anisotropy, the observed anisotropic SW transfer between bond-parallel and bond-perpendicular directions clearly indicates the presence of anisotropic Kitaev spin exchange. Furthermore, the fact that this anisotropy appears at T_H, rather than T_N, suggests that the Kitaev term is comparable in magnitude to other spin exchange interactions.

Magneto-optic experiment and robustness of spin fluctuating region under magnetic field in Cu₃Co₂SbO₆

The observed spin-exciton coupling is further supported by peak shift and enhanced spectral-weight under H which suppress the antiferromagnetic ordering. Figure 5a, b and S12 represent the absorbance spectrum with various H for each direction at 1.6 K. In the absorbance spectrum, we observed a sharp peak near 4.2 eV, which is consistent with σ₁(ω) shown in Fig. 3a. The slight discrepancy in peak energy between σ₁(ω) and absorbance spectrum can be attributed to the variations in experimental method (Supplementary Fig. 13). To offer a quantitative understanding of the changes in absorption with respect to H, we fitted the absorption spectrum using three Lorentzian functions (Supplementary Fig. 14). Figures 5c, d indicate the peak energy and integrated intensity between two isosbestic point (Supplementary Fig. 14) for each H direction. Increasing H induces a distinct redshift in peak energy and increases the intensity both in bond-parallel and bond-perpendicular directions. Given that in-plane H suppresses the antiferromagnetic order, the peak energy and the intensity evolve in the opposite direction of lowering T. Conversely, out-of-plane H (Supplementary Fig. 12) leaves both the peak position and intensity unchanged. This observation corroborates the robust antiferromagnetism in that axis, as shown Fig. 2c.

Fig. 5: Modulation of exciton peak energy and intensity driven by external magnetic field H and robustness of T_H.

The absorbance of Cu₃Co₂SbO₆ under various applied H along a the bond-parallel direction and b the bond-perpendicular direction. Red and blue arrows represent the direction of applied H and incident light, respectively. Full datasets are listed in Supplementary Fig. 12 The redshift of absorbance in in-plane H is attributed to the suppression of antiferromagnetic ordering. The exciton peak position and intensity as a function of H c, along the bond-parallel direction and d along the bond-perpendicular direction. The black open circles, red closed circles, red solid lines, and red dashed vertical lines indicate the peak position, intensity, fitting function for the peak position, and critical magnetic field obtained from M-H curve respectively. The fitting function provides the critical H_C, consistent with observed spin-flip transition. eT-dependent exciton peak position under different H. The different trends of 0 kOe, 40 kOe, and 70 kOe data reflect the antiferromagnetic, intermediate, and ferromagnetic phase, respectively. While the sign of spin-spin correlation function has changed with applied H, the peak shift emerges around similar T_H in both H = 0 and 70 kOe cases. f H-dependent exciton peak position under various T. The peak shift at 10 K, where antiferromagnetic ordering remains, shows a similar behavior at 2 K. Despite the disappearance of antiferromagnetic ordering at 20 K and 30 K, the persistent peak shift suggests finite spin-spin correlation functions. The disappearance of the peak shift at 40 K, again consistent with T-dependent peak shift in Fig. 3d. The error bars in c, d, e and f correspond to standard error obtained from Lorentz fitting of absorbance spectra.

The observed peak energy and intensity of the excitonic transition under H also show bond-dependent anisotropy, consistent with different spin-flip transitions. The intensity of the peak, for example, begins to show an abrupt increase above 40 kOe (60 kOe) with H along the bond-parallel (bond-perpendicular) direction. These fields are close to the observed spin-flip transition fields (dashed lines) in Fig. 2h. In terms of the peak position, we expect it will be proportional to the number of spins that flip to the ferromagnetic configuration throughout the linear Zeeman term for the classical antiferromagnetic ordered state at sufficiently low T. Since even in the presence of frustrated magnetic exchanges, the occurrence of magnetic ordering allows us to consider the spins classically. The flipping of a single spin can be described as the creation of spin-1 boson. Therefore, the phenomenological equation for peak energy can be expressed as follows.

$$\Delta E\left(H\right)=-\frac{A}{\exp \left(-\frac{{H}_{C}}{H}\right)-1}$$

(2)

where ΔE(H), H_C, and A are the redshift of the peak, the H required for spin-flip transition, and a constant, respectively. The solid lines in Figs. 5c, d represent the best fitting results with H along bond-parallel and bond-perpendicular directions, respectively. Note that H_C is found to be 44 ± 6.7 kOe (66 ± 5.0 kOe) along the bond-parallel (bond-perpendicular) direction, providing excellent coincidence with the H for the spin-flip transition observed in Fig. 2h (45 ± 2.5 kOe and 60 ± 2.5 kOe). These results clarify a strong coupling between antiferromagnetism and exciton, establishing Cu₃Co₂SbO₆ as a unique platform for the investigation and potential manipulation of quantum magnetism via light.

We found that T_H remains unchanged under H, implying the robustness of the spin-fluctuation region. Figure 5e shows the T-dependent peak position obtained through absorbance under varying H. To discern the contrast in peak shifts on either side of H_C, we focus on bond-parallel directional case, which has a relatively lower H_C. Without an applied H, both the absorbance and σ₁(ω) reveal a similar redshift trend. This trend, however, reverses with H of 70 kOe, suggesting a phase transition to a spin-polarized (SP) state and the sign change of the spin-spin correlation function. In both cases, the peak position begins to change around 30 ~ 40 K, emphasizing the robustness of T_H. The peak position becomes T-independent with applied H of 40 kOe, close to the spin-flip field H_C, representing a balance of antiferromagnetic and ferromagnetic spin-spin correlation functions. Figure 5f illustrates H-dependence of absorbance peak at a various T. At 10 K, given the persistence of antiferromagnetic ordering, the trend mirrors that of Fig. 5c. Notably, the presence of peak shift exists even at 20 and 30 K, where the antiferromagnetic ordering has vanished. The peak shift become almost H-independent above 40 K consistent with T_H of 40 K observed in T-dependent σ₁(ω).

Although the spin fluctuating region below T_H is unclear yet, one potential explanation might be a spin fractionalized region, commonly found in a Kitaev QSL system. In α-RuCl₃, it is reported that the second releases of magnetic entropy and finite spin-spin correlation function at and below a characteristic temperature T_H, which is significantly higher than the T_N, respectively²⁸. Similarly, Cu₃Co₂SbO₆ also exhibits two instances of magnetic entropy releases, with the T at which half of the magnetic entropy is released being 40 K, a value close to the T_H we measured²⁷. This spin fluctuation region below T_H is understood as an unconventional Kitaev paramagnetic phase due to the occurrence of spin fractionalization. In addition, recent theoretical study on the GHK model, one of the possible spin Hamiltonians for the high-spin 3d⁷ cobalt-based honeycomb materials, has resembled a similar fluctuating regime with finite spin-spin correlation functions below the conventional paramagnetic phase^16,29. Moreover, they reported that the T_H is barely changed by external H, which is similar to our observation in Cu₃Co₂SbO₆. Further studies for understanding the spin fluctuation region are highly required.

In summary, we have revealed bond-dependent antiferromagnetism, the exciton coupled to its magnetic ground state, and an unconventional anisotropic spin fluctuation region between antiferromagnetic and paramagnetic phases in Cu₃Co₂SbO₆. The observed bond-dependent antiferromagnetism and spin fluctuation region imply that Cu₃Co₂SbO₆ can be a promising starting materials to realized QSL phase. Additionally, the interaction between exciton and fluctuating spin offers Cu₃Co₂SbO₆ with a unique platform to detect, realize, and manipulate the spin liquid using light. For example, transient spin liquid states can be achieved using a time-resolved pump-probe experiment as it perturbs its spin Hamiltonian through spin-exciton coupling^44,45. The potential to generate and control fractional excitation via light through the exciton-spin interaction could be explored if the desired Kitaev QSL phase can be stabilized in Cu₃Co₂SbO₆. Lastly, we believe that our experimental approach bridges two areas previously seem as incompatible: heterostructures and quantum magnetism. While there is growing interest in applying heterostructure methods to Kitaev QSL studies^12,19, applicable experiments have been rare due to the extremely small volume available for detecting spin-spin correlation functions. However, our approach via light, unaffected by volume constraints, provides promising methodology to merge two distinct territories.

Methods

Sample preparation

High-quality Cu₃Co₂SbO₆ thin films were synthesized using pulsed laser deposition. The O-faced ZnO [0001] and MgAl₂O₄ [111] substrates were annealed for 2 h at 1100 ^◦C and for 10 h at 1200 ^◦C in ambient pressure, respectively, to improve the surface roughness and crystal quality. We made a target followed the previously reported synthesis of polycrystalline Cu₃Co₂SbO₆ powder using the solid-state reaction method²⁷. The base pressure remained under 1 × 10^-6Torr. The optimized growth conditions were as follows: substrate temperature T = 800 ^◦C, oxygen partial pressure P = 10 mTorr, energy of the KrF Excimer laser (λ = 248 nm) E = 1.3 J/cm², laser repetition rate = 10 Hz, and the distance between the target and substrate was set at 50 mm. Cooling was performed under the same as grown pressure after the deposition was completed. No sample degradation was observed, even when the target and the synthesized thin film were stored at room temperature and under ambient pressure.

Characterization of lattice structure and film thickness

By using a D8 Discovery high-resolution X-ray diffraction (Bruker), high resolution X-ray diffraction data of Cu₃Co₂SbO₆ thin film were collected at room temperature using a wavelength of 1.5406 Å. Using the 0D mode of Lynxeye detector, θ-2θ scan was performed at 0.005° intervals from 10° to 80° in 2θ and at a scan speed of 0.016° s^–1. The rocking curves were performed at 0.005° intervals from ± 1.5° of 004 peak of Cu₃Co₂SbO₆ with 0.5 step/sec. An azimuthal φ-scans of Cu₃Co₂SbO₆ thin film were conducted at angle χ = 54.7356° with respect to ZnO (0001) surface. The collected angle range is from −210° to 150° with 0.02° increments and 0.5 step/sec scan speed for ZnO substrate and Cu₃Co₂SbO₆ thin film. Note that counterclockwise rotation is positive.

Magnetic susceptibility

The temperature dependence of the zero-field-cooled and field-cooled d.c. magnetization of 14.2 mg (13.6 mg) Cu₃Co₂SbO₆/ZnO (Cu₃Co₂SbO₆/MgAl₂O₄) thin film, with dimension of 3.3 mm (bond-parallel) × 2.5 mm (bond-perpendicular) × 0.33 mm (0.5 mm for MgAl₂O₄) (out-of-plane), was measured in each direction using a superconducting quantum interference device (Quantum Design). The measurements were conducted by attaching the samples to the quartz sample mounting post with GE varnish. Similarly, the magnetic properties of 15.2 mg (13.6 mg) bare ZnO (MgAl₂O₄), of the same dimension, were obtained. The magnetization of Cu₃Co₂SbO₆ was calculated by subtracting the mass-normalized substrate magnetic susceptibility. To compare relatively small difference in χ_bond//(T) and χ_bond⊥(T), temperature- and field-independent background contribution from environment is further subtracted based on high-temperature Curie-Weiss fitting. This process is valid with the known fact that bulk Cu₃Co₂SbO₆ has a negligible constant background contribution χ₀ in magnetic susceptibility²⁷. We observed that the magnetic properties of Cu₃Co₂SbO₆ grown both substrates are qualitatively identical, particularly for anisotropic H_C (Supplementary Fig. 4). However, we found that the MgAl₂O₄ exhibits a significantly larger paramagnetic signal from defect dipole moments at low temperature compared to ZnO, distorting the subtracted data; therefore, we exhibit the magnetic data of Cu₃Co₂SbO₆ on ZnO substrate in our main manuscript.

Scanning transmission electron microscopy measurements

For transmission electron microscopy (TEM) analysis, a cross-section specimen of approximately 50 nm was fabricated using a focused ion beam (FIB) machine (Helios G4, FEI) from a 20 nm Cu₃Co₂SbO₆ film grown on a ZnO substrate. The interface structure was then measured using a Titan Double Cs corrected TEM (Titan cubed G2 60–300, FEI) with high-angle annular dark field scanning TEM (HAADF-STEM) mode. The microscope was operated at 300 kV accelerating voltage with the beam convergence semi-angle of 25.2 mrad. The inner and outer angles of the HADDF detector were set as 38 and 200 mrad, respectively. Several 1024 × 1024 images of the film-substrate interface were acquired with 4 µs dwell time per pixel. The pixel size was 6.48 pm for Supplementary Fig. 2e, 12.96 pm for Supplementary Fig. 3a-b, and 25.91 pm for Supplementary Fig. 3c. The total electron dose was about 4.64 × 10⁵ electrons Å⁻² for Supplementary Fig. 2e, 1.18 × 10⁵ electrons Å⁻² for Supplementary Fig. 3a, 1.13 × 10⁵ electrons Å⁻² for Supplementary Fig. 3b, and 2.94 × 10⁴ electrons Å⁻² for Supplementary Fig. 3c.

Electronic structure calculations

Electronic structure calculations were performed via employing Vienna Ab initio Simulation Package (VASP) within projector augmented wave formalism⁴⁶. Crystal structure of Cu₃Co₂SbO₆ was optimized with the choice of PBEsol exchange-correlational functional⁴⁷, in addition invoking an effective onsite Coulomb potential U_eff = 4 eV on d orbitals of Co and Cu through Dudarev approach⁴⁸ and Zigzag-type antiferromagnetic order. The plane wave cutoff and size of k-mesh were set to 500 eV and 8 × 8 × 4, respectively. For optimization of crystal structure, force and energy convergence criteria were set to 10⁻⁴ and 10⁻⁹eV, respectively. The spin-polarized electronic band structure was calculated using PBEsol+U with U = 6 eV and Néel-type order. Furthermore, non-spin-polarized electronic structure was computed, which was used later to estimate hopping parameters with employing maximally localized Wannier functions (MLWF) method⁴⁹ as implemented in WANNIER90⁵⁰. For the estimation of the magnetic exchange interactions we constructed two Wannierized tight-binding models with d orbitals of Co only and including full Co d and O p orbitals.

Ellipsometry and optical conductivity

The optical conductivity of 20 nm Cu₃Co₂SbO₆ was obtained using an M-2000 ellipsometer (J. A. Woolam Co.). The ellipsometry parameters, Ψ and Δ, of Cu₃Co₂SbO₆ were measured over the energy range of 0.74 to 6.46 eV (5900 to 52000 cm⁻¹) at 60° incident angles and temperatures (6 to 300 K). Each measurement had a duration of 200 s. Here, Ψ represents the amplitude ratio of the reflected p- and s-waves, while Δ represents the phase shift between the two waves. We also obtained Ψ and Δ of the MgAl₂O₄ and ZnO under the same incident angles and temperatures. We determined the optical constants of Cu₃Co₂SbO₆ layers by constructing thin film models including intermixing layer and surface roughness. For low-temperature measurement, we calibrated the window effect to determine the Δ offset by using a 25 nm SiO₂/Si wafer. To prevent ice formation on the sample surface, we baked out a chamber to obtain the base pressure below 5 × 10⁻⁹Torr. All samples were attached with silver epoxy to oxygen-free copper cones to prevent reflections from the backside of the sample. We observed that the optical properties of Cu₃Co₂SbO₆ grown both substrates are qualitatively identical, including T_N and T_H (Supplementary Fig. 5). However, the strong absorption of ZnO near 3.3 eV distorted the spectra. To this reason, we exhibit the optical data of Cu₃Co₂SbO₆ deposited on MgAl₂O₄ in our main manuscript.

Transmittance under applied magnetic field

The transmittance spectra of Cu₃Co₂SbO₆ thin film on MgAl₂O₄ substrate were measured using an unpolarized Deuterium light source (SLS204, Thorlabs) and a charge-coupled device (CCD) spectrometer (CCS200, Thorlabs). To reduce background noise, a dark measurement was subtracted from all raw spectra. The transmission experiments under the external magnetic fields were conducted using a magneto-optic chamber (SpectromagPT, Oxford Instruments), which has 4 optical ports to facilitate optical experiments in both vertical and horizontal directions relative to the external magnetic fields. Due to the strong absorption of ZnO, we cannot obtain the absorption spectrum of Cu₃Co₂SbO₆ film on ZnO substrate.