Spin correlations in the nematic quantum disordered state of FeSe

Abstract

The quantum-disordered state in FeSe, intertwined with superconductivity and nematicity, has been a research focus in iron-based superconductors. However, the intrinsic spin excitations across the entire Brillouin zone in detwinned FeSe, crucial for understanding its magnetism and superconductivity, have remained unresolved. Using inelastic neutron scattering, we reveal that stripe spin excitations (Q = (1, 0)/(0, 1)) exhibit the C₂ symmetry, while Néel spin excitations (Q = (1, 1)) retain C₄ symmetry within the nematic state. Temperature-dependent differences between Q = (1, 0) and (0, 1) spin excitations above the structural transition unambiguously reveals the nematic quantum disordered state. Comparison with NaFeAs suggests the Néel excitations originate from enhanced 3d_xy orbital correlations. Modeling the stripe dispersions using a J₁-K-J₂ Heisenberg Hamiltonian, we establish a spin-interaction phase diagram, positioning FeSe near a crossover regime between the antiferroquadrupolar, Néel, and stripe orders. Our results provide key insights into the microscopic spin interactions and their role in the intertwined orders in iron-based superconductors.

Introduction

The parent compounds of iron pnictide superconductors exhibit a tetragonal-to-orthorhombic structural phase transition at T_s and then form collinear long-range antiferromagnetic (AF) stripe order at the wave vector Q_AF = (1, 0) below T_N (T_N≤ T_s)^1,2. At temperatures below T_s, iron pnictides establish an electronic nematic phase where the electronic and magnetic properties along the Q = (1, 0) direction are different from those along the (0, 1) direction^{3,4,5,6,7,8,9,10,11,12,13}. With increasing electron or hole doping, both T_N and T_s decrease and vanish near optimal superconductivity but AF and nematic fluctuations remain, thus suggesting their important role for superconductivity^2,3,6,7. While most iron-based superconductors follow this paradigm, iron chalcogenide FeSe is different¹⁴. With a simple layered structure, the stoichiomtric FeSe undergoes a tetragonal-to-orthorhombic structural (nematic) transition at T_s = 90 K, and then becomes superconducting at T_c ≈ 8 K without static AF order (Fig. 1a)^15,16,17,18. Since nematic phase transitions for iron pnictides and FeSe occur below room temperature with small orthorhombic lattice distortions, samples are twinned below T_s where the (1, 0) and (0, 1) directions are mixed and cannot be separated in most spectroscopic measurements^9,12. While inelastic neutron scattering (INS) experiments on twinned iron pnictides such BaFe₂As₂ have spin excitations stemming from the stripe ordering Q_AF = (1, 0)/(0, 1) positions consistent with expectations¹⁹, similar measurements on twinned FeSe reveal spin excitations at the stripe (Q_AF = (1, 0)/(0, 1)) and Néel (Q_N = (1, 1)) ordering wave vectors²⁰. The absence of static AF order in FeSe was interpreted as arising from the magnetic frustration due to competing stripe and Néel interactions^21,22,23.

Fig. 1: Sample characterization, detwinning device and summary of the key results.

a Temperature-dependent resistivity of FeSe single crystal. The left-upper inset illustrates the crystal structure of FeSe in tetragonal notation; the right-lower inset shows the orthorhombic unit cell in the Fe plane. J₁, J₂, and J₃ represent the nearest-neighbor- (NN), next-NN, and next-next-NN exchange interactions between Fe ions. K denotes the biquadratic interaction between the adjacent Fe ions. b The uniaxial-strain device based on the differential thermal expansion coefficients between the Invar alloy (Fe_0.64Ni_0.36) frame and the aluminum sheet. c, d One-dimensional constant-energy cuts (E = 55 ± 10 meV) of the spin excitations along [1, K] and [H, 1] directions collected at the SEQUOIA (c) and 4SEASONS (d) time-of-flight spectrometers using incident energy of 147.5 meV and 80 meV, respectively. The inset of (d) depicts the positions of the stripe and Néel spin excitations in [H, K] space, and the trajectories (blue and red dashed lines with arrowheads) for the 1D cuts shown in (c, d). The vertical bars indicate the statistical errors of 1 s.d. in (c, d). e, f Static (green squares) and fluctuating (red diamonds) magnetic moment (f), and the electron correlation (m^*/m_band) at the 3d_xy orbital (f) as a function of the anion height h_FeX. The data points in (f) are from ref. ³². g Spin-excitation dispersions for BaFe₂As₂ (green diamonds), NaFeAs (red squares), and FeSe (black circles). The data points in the range (0, 0)–(0.5, 0) were measured with RIXS^25,47. The data points for BaFe₂As₂ and NaFeAs obtained with INS are extracted from refs. ^13,31. The green, red, and black solid curves are the fittings of the dispersions with the J₁-K-J₂ model. The gray-shaded area marks the (Q, E) region where the spin excitations are heavily damped. The black dashed line schematically shows that the spin-excitation dispersion turns up and deviates from the fitting. The horizontal and vertical bars indicate the momentum and energy integration range, respectively. h The ratios K/J₁ and J₂/J₁ for the fittings of the dispersions for BaFe₂As₂ (ref. ¹³), NaFeAs (ref. ³¹), and FeSe. The INS data of BaFe₂As₂, NaFeAs, and FeSe were collected at T = 7 K, 5 K, and 5 K, respectively. The uncertainties were estimated according to the confidence interval of the fittings in (h).

Previous INS studies on FeSe have revealed broad spin excitations centered around Q_N = (1, 1) that merge with those stemming from Q_AF = (1, 0)/(0, 1) at E ≳ 70 meV in twinned samples²⁰. However, these measurements lacked the clarity needed to fully resolve the intrinsic spin excitation spectrum of FeSe, primarily due to the presence of twin domains and limitations in background determination. These challenges have hindered a detailed understanding of the spin excitations, including their energy dependence and symmetry properties. To overcome these obstacles and better understand the unusual magnetic state of FeSe, it is imperative to investigate its spin excitations in a twin-free sample, employing improved experimental setups such as enhanced background determination to map the full spin excitation spectrum. In previous work, we have carried out INS experiments on partially detwinned FeSe by gluing them on uniaxial strained BaFe₂As₂ substrates^13,24. However, spin waves from the BaFe₂As₂ substrates overwhelm the magnetic signal from FeSe for spin excitation energies E ≳ 10 meV and temperature across T_s²⁴. Resonant inelastic X-ray scattering (RIXS) experiments on the FeSe crystal detwinned by the same method are capable of avoiding the substrate spin waves. These measurements unveil a large spin excitation anisotropy up to energy of E ~ 200 meV below T_s²⁵; however, the limited momentum transfer of the Fe L₃ RIXS ($| {{{{\bf{q}}}}}_{\parallel }| \lesssim 0.5\frac{2\pi }{{a}_{{{{\rm{o}}}}}}$) means that one cannot access Q_AF and Q_N positions in these measurements.

In this work, we use INS to obtain clear intrinsic spin-excitation spectrum of FeSe across the entire Brillouin zone (BZ), enabled by a newly designed low-background uniaxial-strain detwinning device (Fig. 1b) and careful background determination from an empty device²⁶. Our results identified the symmetries of the stripe and the Néel spin excitations (Fig. 1c, d), as well as their energy-momentum dispersions and the temperature dependence across the nematic phase transition. By analyzing the features of the spin excitations in FeSe, NaFeAs, and BaFe₂As₂, we find that the Néel excitations are somewhat independent of the stripe excitations and should be driven by the enhanced electron correlations of the d_xy orbital (Fig. 1e, f). Through fitting the stripe-excitation dispersions using a Heisenberg Hamiltonian with biquadratic interaction (J₁-K-J₂)^27,28, we establish a spin-interaction phase diagram for FeSe, NaFeAs, and BaFe₂As₂. As the low-energy stripe excitations in FeSe exhibit a linear-in-energy spectral weight and nematic spin correlations that can be described in an antiferroquadrupolar (AFQ) ordering regime^22,29, we argue that FeSe is positioned close to a crossover regime where the AFQ, Néel, and stripe orders intersect²⁶. The results provide an experimental basis for establishing a microscopic theoretical model to describe the origin and the intertwining of the emergent orders in iron-based superconductors (Fig. 1f–h).

Results

Uniaxial-strain device to detwin a large amount of FeSe single crystals

We describe in Fig. 1a–d the characterization of the sample and the detwinning efficiency of the uniaxial-strain device. Figure 1a shows the resistivity of FeSe as a function of temperature. The two clear features on the resistivity curve (marked by black arrows) correspond to the nematic and the superconducting transitions at T_s = 90 K and T_c = 8.5 K, respectively. Figure 1b displays a photo of Sample #1 for the INS experiment, which consists of five separate uniaxial-strain devices stacked along the c-axis containing  ~1500 pieces (~1.61 grams) of thin FeSe crystals co-aligned along the tetragonal [1, 1, 0] direction. The FeSe crystals are attached to 0.2 mm-thick aluminum alloy (6061) sheets using type-M CYTOP. The uniaxial-strain device is designed based on the differential thermal expansion coefficients between the Invar alloy (α ≈ −2 × 10⁻⁶/K) outer frame and the aluminum (α ≈ −24 × 10⁻⁶/K) sheet fixed on the frame²⁶. While cooling, the thermal expansion difference between the Invar-alloy frame and the aluminum sheet can generate a temperature-dependent anisotropic strain, reaching ε = ε_V − ε_H ≈ 0.6% at base temperature, which is large enough to detwin FeSe with orthorhombic lattice distortion δ = [(a_o − b_o)]/[(a_o+ b_o)] ≈ 0.27% (corresponding to ε ≈ 0.54) at T < < T_s²⁶. Neutron diffraction measurements on 26 pieces of FeSe crystals (25 mg) glued on one uniaxial-strain device reveals a detwinning ratio $\eta=\frac{{P}_{1}-{P}_{2}}{{P}_{1}+{P}_{2}}\approx 71\%$²⁶, where P₁ ≈ 85.5% and P₂ ≈ 14.5% represent the relative population of the two kinds of twin domains. For the samples used for INS experiments, we use the low-energy spin excitations at E = 4 ± 1 meV and 10 ± 2 meV to estimate the detwinning ratio and find the samples are highly detwinned with η ≈ 58% (P₁/P₂ ≈ 3.75)—79% of the domains are aligned along one direction (for details see the Supplemental Material)²⁶.

Spin excitations of detwinned FeSe at low temperature

Figure 1c shows one-dimensional (1D) constant-energy cuts with E = 55 ± 10 meV along the [1, K] (red diamonds) and [H, 1] (blue squares) directions measured on Sample #1. The same cuts on Sample #2 are presented in Fig. 1d. In both panels, the integrated intensity of the stripe excitations around (1, 0) is  ~ 3 times of that around (0, 1), indicating a large spin-excitation anisotropy. For comparison, the coincidence of the peaks at (1, −1) and (−1, 1) in Fig. 1c, d indicates the Néel excitations at (±1, ±1) are identical in lineshape and intensity. This confirms the C₂ symmetry of the stripe excitations reported in previous INS and RIXS studies^24,25 and demonstrates that the Néel spin excitations are C₄ symmetric in the nematic state.

Figure 2 reveals the energy and wave-vector dependence of the spin excitations of detwinned FeSe measured at base temperature. Figures 2a–d show the spin excitations for E_i = 147.5 meV projected onto (Q, E) planes with Q along the [1, K], [H, 0], [H, 1], and [0, K] directions, respectively. The scattering of an empty strain device (with CYTOP) measured under the same conditions has been subtracted from the data, leading to the clear spin excitation dispersions below E = 100 meV in Fig. 2a–d (see the Supplemental Material)²⁶. Despite the absence of a stripe order, spin excitations arising from the stripe-order wavevector (1, 0) are much stronger than the excitations emanating from the (0, 1) and the C₄-symmetric Néel spin excitations at the (±1, ±1) positions (Fig. 2a–d). The energy-dispersion branches along the [1, K] direction of the stripe excitations are very sharp (Fig. 2a), while the stripe excitations in Fig. 2b, d along the longitudinal [H, 0] and [0, K] direction appear non-dispersive and damped. Similar anisotropic damping of the stripe excitations between the [1, K] and [H, 0] directions was observed in CaFe₂As₂³⁰, BaFe₂As₂¹³, and NaFeAs³¹. While a clear dispersion along [H, 0] can persist to E ≈ 150 meV in CaFe₂As₂ and E ≈ 100 meV for BaFe₂As₂, it was damped so quickly that no dispersive feature can be resolved in NaFeAs. The highly anisotropic damping of the stripe excitations in FeSe is much akin to that in NaFeAs. As we will discuss later, the much stronger anisotropic damping in FeSe and NaFeAs could be attributed to the large anion height (h_FeX) in these two materials that control the electron-correlation magnitude in the d_xy orbital (Fig. 1e)^32,33.

Fig. 2: Two-dimensional slices of the magnetic excitations in detwinned FeSe.

a–d Energy-vs-momentum slices of the magnetic excitations across Q = (1, 0), (0, 1) and (1, 1) measured with E_i = 147.5 meV at T = 5 K. e–n Constant-energy slices of the magnetic excitations, with E_i = 21 meV (e), 36 meV (f, g), 80 meV (h), 147.5 meV (i–j), and 295 meV (k–n) at based temperatures. For the E_i = 147.5 meV data (a–d), and (i, j), the scattering signal of an empty strain device with CYTOP has been subtracted as a background. A ∣Q∣-dependent background has been subtracted from the data shown in (e–h)²⁰, and energy-dependent flat backgrounds are subtracted from the data (k–n).

Figure 2e–n are constant-energy intensity maps of the spin excitations in the [H, K] plane, confirming again the C₂ and C₄ symmetry of the stripe and the Néel excitations, respectively. The stripe spin excitations at (1, 0) are isotropic in momentum space at the low-energy range E ≲ 15 meV (Fig. 2e, f), but exhibit anisotropic dispersion and damping at E ≳ 20 meV. They propagate well along the [1, K] direction but damp quickly along the [H, 0] direction, consistent with the energy-momentum slices in Fig. 2a–d. The Néel excitations are visible at E ≳ 40 meV (Fig. 2h–j) and merge with the stripe excitations at E ≳ 90 meV. The anisotropy between the stripe excitations at (1, q₁) and (q₂, 1) persists to E = 120 ± 10 meV and vanishes at higher energies, leaving four-fold symmetric broad scattering at around (±1, ±1) in Fig. 2m (E = 140 ± 10 meV) and 2(n) (E = 160  ± 10 meV).

The intensity difference between the spin excitations at Q₁ = (1, q₁) and Q₂ = (q₂, 1) in the nematic state, is a manifestation of the electronic nematicity in the spin-spin correlations, termed nematic spin correlations, can be defined as ψ(E) = [S(Q₁, E) − S(Q₂, E)]/[S(Q₁, E) + S(Q₂, E)] (or defined via the imaginary part of the dynamic susceptibility χ^″(Q, E) in the same way)¹³. It was indirectly probed by RIXS in the limited region of the first BZ at the Γ point²⁵. Here we conclusively determine the energy scale of the nematic spin correlations as E ≈ 120 ± 10 meV, roughly consistent with that (E ≈ 200 meV) determined in the RIXS study of detwinned FeSe²⁵.

Temperature dependence of the spin excitations

The nematic spin correlation ψ(E, T) has been suggested to represent the nematic order parameter in the spin channel³. Figure 3a–d shows the temperature dependence of the stripe and the Néel spin excitations in the energy range E = 55 ± 7 meV, measured at T = 90, 105, 120, and 150 K (see the Supplemental Material for details)²⁶. The integrated intensities as a function of temperature are shown in Fig. 3e. The Néel excitations with smaller spectral weight increase slightly from 5 K to 150 K. On warming from T = 5 K to 150 K, the stripe excitations at (1, 0) decrease, whereas the excitations at (0, 1) increase gradually, leading to a diminishing ψ(E). No sudden change occurs for the stripe and the Néel excitations at the unstrained T_s = 90 K, possibly because the structural transition is smeared out under the uniaxial strain³⁴. Note that the uniaxial strain decreases gradually with increasing temperature and could be reduced by  ~40% at T = 150 K.

Fig. 3: Temperature dependence of the magnetic excitations in detwinned FeSe.

Constant energy slices of the magnetic excitations for detwinned FeSe single crystals with E = 55 ± 7 meV at T = 90 K, 105 K, 120 K, and 150 K. The background scattering collected from an empty strain device with CYTOP has been subtracted from the data (a)–(d). e Temperature dependence of the integrated intensities of the magnetic excitations at Q = (1, 0) (denoted by I₁₀), (0, 1) (denoted by I₀₁), and (1, 1) with E = 55 ± 7 meV. f Comparison of local susceptibility χ^″(E) (momentum averaged χ^″(Q, E)) for detwinned FeSe single crystals at Q = (1, 0) and (0, 1), measured at T = 5 K (open symbols) and 150 K (filled symbols). The horizontal and vertical error bars indicate the energy integration range for calculating χ^″(E) and the statistical errors of one standard deviation. The dashed curves in (e, f) are guides to the eyes. The data were collected on SEQUOIA with E_i = 147.5 meV. The horizontal and vertical bars in (e, f) indicate the energy integration range, and the statistical errors of 1 s.d., respectively.

Although the persistence of ψ(E) at T > T_s under uniaxial strain (ε ~ 0.2%–0.4%) in the nematic fluctuation regime was previously inferred from a RIXS study of FeSe_1−xS_x³⁵, the present study provides the first direct observation of the temperature dependence of nematic spin correlations across T_s. This behavior aligns with theoretical predictions in ref. ³ and highlights the establishment of the nematic quantum spin-disordered state. Naively, the presence of preemptive nematic spin correlations implies a spin-driven origin of nematicity. However, a recent X-ray linear dichroism study on FeSe proposed that spontaneous orbital polarization could instead serve as the primary nematic order parameter^36,37. Taking together that the stripe spin excitations are closely tied to the d_xz/d_yz orbitals, the entangled spin- and orbital-degrees of freedom may both play a pivotal role in driving electronic nematicity.

Figure 3f shows the temperature-dependent χ ″(E) for the energy range E ≲ 90 meV measured on sample #1 at SEQUOIA. The nematic stripe spin correlation ([χ ″(Q₁, E) − χ ″(Q₂, E)]/[χ ″(Q₁, E) + χ ″(Q₂, E)]) retains its magnitude at energies up to 90 meV at T = 5 K and almost vanishes at T = 150 K well above the unstrained T_s under a moderate uniaxial strain (Fig. S10 in the Supplemental Material)²⁶.

Analysis of the spin excitations in FeSe

A surprising discovery in the spin excitations is the linear-in-energy spectral weight (local dynamic susceptibility χ^″(E) ∝ E) at the energy range E ≲ 60 meV measured at T < < T_s and the temperature (T = 150 K) well above T_s, distinct from that in BaFe₂As₂ and NaFeAs (see the Supplemental Material for more details)²⁶. Some of us considered an S = 1 generalized bilinear-biquadratic model on a square lattice and proposed that an (π, 0) AFQ state could describe the magnetism in bulk FeSe^22,29. The linear energy dependence of the low-energy χ^″(E) was an essential prediction/feature originating from the linearly dispersive Goldstone mode in the AFQ ordered phase, providing a clue to understanding the magnetic ground state, as we will discuss later.

Twin-domain averaged χ^″(E) for the full energy range measured on Sample #2 at 4SEASONS (Fig. S9) confirmed the χ^″(E) measured on a twined sample²⁰. The integral of the total spectral weight of χ^″(E) generates the fluctuating moments $< {m}^{2} > =7.45\pm 0.25\,{\mu }_{{{{\rm{B}}}}}^{2}$ at T = 6 K, and $6.23\pm 0.34\,{\mu }_{{{{\rm{B}}}}}^{2}$ at T = 120 K. Following the magnetic moment sum rule $< {m}^{2} > ={g}^{2}{\mu }_{{{{\rm{B}}}}}^{2}S(S+1)$ and with g = 2², we get S ≈ 0.95 ± 0.06 for T  = 6 K and S ≈ 0.84 ± 0.08 for T = 120 K, corroborating the S = 1 localized spin scenario for FeSe²⁰.

To obtain a quantitative understanding of the spin excitations, we plot in Fig. 4a–e the constant-energy momentum cuts across (1, 0), (0, 1), and (1, 1) along the [1, K] and [H, 1] directions, with E = 20 ± 10, 50 ± 10, 70 ± 10, 90 ± 10, and 120 ± 10 meV, and in Fig. 4f–j the momentum cuts along a diagonal [2 −K, K] direction with E = 45 ± 5, 65 ± 5, 75 ± 5, 85 ± 5, and 95 ± 5 meV. The fit of the constant-energy momentum cuts with multi-Gaussian functions generates the energy dispersions of the stripe and the Néel spin excitations, which are plotted onto the E-vs-[1, K] slices in Fig. 4l, m, where the stripe and the Néel excitations are resolved for energies up to E ≈ 125 meV. We note that the Néel excitations touch the stripe excitation branch around E ≈ 90 meV while the latter persist to E ≈ 130 meV and dominate the spectral weight, indicating that they could be separate excitations arising from the competing stripe and Néel magnetic interactions²⁰.

Fig. 4: One-dimensional cuts and energy dispersion of the magnetic excitations in detwinned FeSe.

a–j Constant-energy cuts along the [1, K], [H, 1], and [2 − K, K] directions across the Q = (1, 0) and (0, 1) (a–e) and Q = (1, 1) (f–j) for detwinned FeSe single crystals at base temperature. The integral intervals perpendicular to the cut directions are H = 1 ± 0.2, K = 1 ± 0.2, and Δq_⊥ = 0.28 r.l.u., respectively. k Constant-Q cuts (E_i = 147.5 meV, T = 5 K) at Q = (1, q ± 0.05) (q = 0, 0.1, . . . , 0.6). The data are fitted by a general damped harmonic oscillator model (solid curves). l, m Magnetic excitation dispersions along [1, K] directions projected onto energy and momentum planes measured on (l) Sample #2 with E_i = 295 meV, T = 6 K, and (m) Sample #1 with E_i = 147.5 meV, T = 5 K. The black circle symbols represent the magnetic excitation dispersions at Q = (1, 0) and (1, 1), which are the Gaussian function fitting results of one-dimensional constant-energy cuts. The white squares and pink diamonds in (m) correspond to the damping factor γ(q)/2 and undamped energy E₀, respectively, as determined by damped harmonic oscillator fitting in (k). The vertical bars indicate the statistical errors of 1 s.d.

In Fig. 4k, we use a general damped harmonic oscillator function to describe ${\chi }^{{\prime}\prime}({{{\bf{Q}}}},E)=A\,\frac{2\,\gamma \,E{E}_{0}}{{\left({E}^{2}-{E}_{0}^{2}\right)}^{2}+{(\gamma E)}^{2}}$ and fit the energy cuts at Q = (1, q ± 0.05) (q = 0, 0.1, . . . , 0.6) with S(Q, E) = χ^″(Q, E)f ²(Q) (solid curves in Fig. 4k), where E₀(q) is the undamped energy, γ(q)/2 is the damping rate, and f(Q) the magnetic form factor of Fe²⁺. The resulting damping rates γ/2 (white open squares in Fig. 4m) are much smaller than the undamped energies (magenta open diamonds in Fig. 4l), indicating the stripe excitations are only slightly damped for q ≤ 0.7. Compared with the broad excitations in previous studies^20,25 and subsequent theoretical works^38,39,40, the new results provide unprecedented strong evidence supporting the localized spin picture in FeSe. Thus, our results demonstrate that the S = 1 localized spin picture is a correct starting point for describing the magnetism in FeSe, and it is likely that the magnetic state in FeSe is intimately connected with the AFQ regime.

The Néel spin excitations

In iron-based superconductors, the Néel spin excitation at (±1, ±1) is not unique to FeSe, but had already been observed in NaFeAs hosting a static stripe order (T_N ≈ 45 < T_s ≈ 58 K) with the ordered magnetic moment M = 0.17  ± 0.034 μ_B/Fe and the total fluctuating moment $< {m}^{2} > =3.6\,{\mu }_{{{{\rm{B}}}}}^{2}$ (S ≈ 0.57)^31,41,42. Figure 5 shows the comparison of the Néel spin excitations in NaFeAs and FeSe, measured with E_i = 150 meV on the ARCS spectrometer, and E_i = 147.5 meV on the SEQUOIA spectrometer, respectively. Figure 5a, b shows magnetic excitations projected onto the [H, K] plane with E = 55 ± 7 meV, in which similar Néel excitations at (1, 1) are visible in both NaFeAs (Fig. 5a) and FeSe (Fig. 5b). The Néel excitations projected onto the E-vs-[2 −K, K] plane (Fig. 5c, d) also exhibit similar “V”-shaped dispersion. Furthermore, the constant-energy cuts across (1, 1) along the [2 − K, K] direction at E = 55 ± 7 meV (Fig. 5e) and E = 80 ± 10 meV (Fig. 5f), and the energy cuts at the Q = (1, 1) show that the Néel excitations in NaFeAs and FeSe are almost identical.

Fig. 5: Comparison of the Néel excitations in NaFeAs and FeSe.

Constant-energy slices of the spin excitations in twinned NaFeAs (E = 55 ± 7 meV) (a), collected with E_i  = 150 meV and detwinned FeSe (E = 55 ± 7 meV) (b), collected with E_i = 147.5 meV. Spin-excitation dispersion along the transverse direction [2 − K, K] (dashed rectangles in (a)-(b)) across the Q = (1, 1) in twinned NaFeAs (c) and detwinned FeSe (d). Constant energy cuts along the [2 − K, K] direction for NaFeAs (black open circles) and FeSe (red open diamonds) with E = 55 ± 7 meV (e), and E = 80 ± 10 meV (f). The integral interval perpendicular to [2 − K, K] is Δq_⊥ = 0.28. g Energy cuts at Q = (1 ± 0.15, 1 ± 0.15) (marked by the blue dashed square in Fig. 5b) for NaFeAs and FeSe. The error bars in (e–g) indicate one standard deviation of the neutron intensity.

In addition, the stripe excitations of FeSe and NaFeAs exhibit similar anisotropic damping along the [1, K] (transverse) and the [H, 0] (longitudinal) directions, and similar much lower band top along the [1, K] direction (E ~ 100 meV) than BaFe₂As₂ (E ~ 200 meV)^31,42. Figure 1e, f plots the systematic changes of the ordered moment ($< m >$), the fluctuating moment ($< {m}^{2} >$), and the electron correlation (m^*/m_band) of these three compounds as a function of the anion height Fe-X (X = As, Se) (h_FeX), suggesting that the larger h_FeX in NaFeAs and FeSe (compared to BaFe₂As₂) enhances the electron correlations in the d_xy orbital^31,32. This has been used to explain small bandwidth along [1, K] in FeSe and NaFeAs^31,32,42. As the Néel excitations are absent in BaFe₂As₂ and related materials, it is natural to speculate that the Néel excitations are also associated with the d_xy orbital with enhanced electron correlations.

In electron-doped NaFe_1−xCo_xAs, while the low-energy stripe excitations associated with Fermi surface nesting evolve with the topology changes of the Fermi surfaces driven by electron doping, the relatively high energy (E ≳ 50 meV) stripe excitations, as well as the Néel excitations, remain essentially unchanged across the whole superconducting regime (x = 0 − 0.11)⁴². These results suggest that superconductivity is coupled to the low-energy stripe excitations but not the Néel excitations. We note that the iron pnictogen height (h_FeAs) in NaFe_1−xCo_xAs decreased by less than 1% from x = 0 to 0.11⁴³, consistent with the invariance of the Néel excitations. In addition, several studies suggest that the stripe order could be “restored” in FeSe under P ≳ 1 GPa hydrostatic pressure that reduces the anion height slightly^23,44,45,46. Therefore, the h_FeX could drive the ground state of FeSe across a phase boundary associated with the stripe order, and the pressured FeSe (P ≳ 1 GPa) with stripe order fills the gap between FeSe and NaFeAs in Fig. 1f, g.

The spin-interaction phase diagram

To achieve a better understanding of the magnetic ground state in FeSe, we use a minimal S = 1 J₁-K-J₂ model to fit the energy dispersions of the stripe spin excitations in FeSe, NaFeAs, and BaFe₂As₂ in Fig. 1(e). To improve the accuracy of the fittings, we include the dispersion near the Γ point measured with RIXS (data points between (0, 0) and (0, 0.5)^25,47. The fitting of the dispersion for BaFe₂As₂ is a reference to show the validity of the fitting strategy. Previously, the fitting of the S(Q, E) in twinned BaFe₂As₂ with the J_1a-J_1b-J₂ (-J_c) model generates J_1a = 59.2, J_1b = −9.2 and J₂ = 13.6 meV, corresponding to J₁ = 25 meV, K = 17.1 meV (J_1a= J₁+ 2K, J_1b= J₁-2K), J₂/J₁ ≈ 0.54 and K/J₁ ≈ 0.68 in the J-K model. In comparison, the fitting of the energy dispersion of BaFe₂As₂ in Fig. 1e gives J₁ = 23.5 ± 2.1812 meV, K = 16.1 ± 0.4407 meV, and J₂ = 15 ± 0.5385 meV (J₂/J₁≈ 0.64 and K/J₁ ≈ 0.69), agrees well with the fitting of the S(Q, E) in twinned BaFe₂As₂.

For FeSe (NaFeAs), the fitting of the energy dispersion provides J₁= 29.9 ± 2.3545 (28.8  ± 2.3646) meV, K = 11.9  ± 0.7465 (11.2 ± 0.667) meV, and J₂= 11.0 ± 0.8812 (13.4  ± 0.7974) meV, leading to J₂/J₁ ≈ 0.37(0.47) and K/J₁ ≈ 0.40(0.39). The J₂/J₁ of FeSe is close to that (J₂/J₁= 0.413) reported in ref. ⁴⁸. FeSe shows slightly larger J₁ and K and smaller bandwidth than NaFeAs, further indicating the electron correlation in FeSe is slightly stronger than the other compounds (Fig. 1f).

We plot in Fig. 1h the trend in the magnetic interactions J₂/J₁ and K/J₁ of FeSe, NaFeAs, and BaFe₂As₂. It is well known that BaFe₂As₂ is deep in the stripe-ordering region. NaFeAs hosting a weak stripe order accompanied by the Néel excitations should already be close to the crossover between the stripe and the Néel regime. For FeSe, as the AFQ model can describe the linear-in-energy χ^″(E) for E ≲ 60 meV and C₂ symmetry of the stripe excitations, it should be near the AFQ ordering regime. Meanwhile, FeSe is close to NaFeAs in the phase diagram and also exhibits the Néel excitations. Thus, it should also be close to both the stripe and the Néel ordering regimes. All these key features can be qualitatively described in a zero-temperature phase diagram containing the stripe order, Néel order, and AFQ order generated by a S = 1 bilinear-biquadratic J₁-J₂-K model, as shown in Fig. 1h⁴⁹. We find that: (1) both K/J₁ and J₂/J₁ are essential in tuning the magnetic ground states; (2) J₂/J₁ plays a key role in driving the ground state from the stripe ordering region to the AFQ regime; and (3) FeSe is positioned close to a crossover regime where the AFQ, Néel, and stripe orders intersect⁴⁹.

It is worth noting that this simplified J₁-K-J₂ model may not provide a final and complete understanding of the evolution of magnetic ground state and interactions due to the complexity of these materials. Extending the model to include further neighboring exchange couplings, such as J₃, K₂, K₃, etc., could describe both the Néel and the stripe excitations simultaneously, and could generate more intriguing AF phases, which might occur in the crossover regime (white shaded areas) in Fig. 1h. In this study, however, we focus on the stripe excitations and aim to elucidate the origins of the Néel excitations using the minimal J₁-K-J₂ model. This approach provides a straightforward and effective phase diagram (Fig. 1h) that captures the evolution of spin interactions and magnetic ground states in iron-based superconductors.

In summary, our INS results on the uniaxial-strain detwinned FeSe clarify the symmetry of the stripe and the Néel spin excitations, characterize the nematic spin correlations, determine the magnetic interactions, and establish the evolution of the magnetic ground state in iron-based superconductors. The uniaxial-strain device suitable for INS developed in this work could also detwin or even apply uniaxial strain on similar magnetic materials with layered structures.

Methods

Sample preparation

The FeSe single crystals used in the present study were grown using the chemical vapor transport method under a permanent gradient of temperature (~400 −330 °C) in the KCl-AlCl₃ flux. The direction of the self-cleaving edges of the FeSe crystals is tetragonal [1, 0, 0], as determined with a Laue camera. We co-aligned and glued  ~1500 pieces (m₁ ≈ 1.61 grams) of thin FeSe crystals onto ten (20 × 22 mm²) areas (the front and back sides of 5 aluminum sheets) along the tetragonal [1, 1, 0] direction using the hydrogen-free CYTOP, which were installed onto the invar alloy frames to form the Sample #1 used in the neutron scattering experiments on SEQUOIA and MAPS time-of-flight spectrometers. We also prepared a Sample #2 following the same way, which contained  ~1000 pieces (m₂ ≈ 1.24 grams) of thin FeSe crystals and was used in the neutron scattering experiment on the Fermi chopper spectrometer 4SEASONS.

Strain characterization

The orthorhombic lattice distortion in FeSe is δ ≈ 0.27%^44,50, corresponding to anisotropic strain ε ≈ 2δ ≈ 0.54% in a fully detwinned sample. To characterize the magnitude and distribution of the strains on aluminum sheets and FeSe crystals, we performed finite element analysis, measured the strains using resistive and optical virtual strain gauges. We have also tested the detwinning efficiency of FeSe crystals mounted on the strain device by checking resistivity anisotropy and nuclear Bragg peaks (see the Supplemental Material for details). Though the applied strain ε(T) does not perfectly correspond to 2δ(T), our analysis and discussion based on the configuration of the device and the strain transfer mechanism show that the samples can be highly detwinned and additional strain effects beyond detwinning are negligible (see the Supplemental Material for details)^51,52,53.

Neutron scattering experiments

The INS experiments were performed on the SEQUOIA⁵⁴, 4SEASONS⁵⁵, and MAPS⁵⁶ time-of-flight spectrometers at the Spallation Neutron Source at the Oak Ridge National Laboratory (ORNL), Materials and Life Science Experimental Facility (MLF) at the Japan Proton Accelerator Research Complex (J-PARC), and the ISIS spallation neutron source, Rutherford Appleton Laboratory (RAL), respectively. The measurements at the Fine-Resolution Fermi Chopper Spectrometer SEQUOIA were performed with E_i = 147.5 meV, in high-flux mode with chopper frequency f = 300 Hz. The energy resolution of this setup is ΔE ≈ 10 meV at E = 0, ΔE ≈ 7 meV at E ≈ 50 meV, and ΔE ≈ 4 meV for E ≈ 100 meV. We defined the wave vector Q in reciprocal space as Q = Ha^* + Kb^* + Lc^*, where H, K, L are Miller indices and a^* = $\hat{{{{\bf{a}}}}}2\pi /{a}_{o}$, b^* = $\hat{{{{\bf{b}}}}}2\pi /{b}_{o}$, and c^* =  $\hat{{{{\bf{c}}}}}2\pi /c$ are reciprocal lattice unit (r.l.u.) vectors with a_o ≈ 5.33 Å, b_o ≈ 5.31 Å and c ≈ 5.49 Å.

Theory method

We have applied the flavor wave theory in the SU(3) representation to calculate the spin dynamics of the S = 1 J₁-K-J₂ model. The exchange parameters are extracted from fitting the experimental dispersion data. This is a method similar to the linear spin wave theory, but with the ability to take into account the quantum effects of quadrupolar ordering and fluctuations. More details can be found in refs. ^22,45.