Interplay of canted antiferromagnetism and nematic order in Mott insulating Sr₂Ir_1-xRh_xO₄

Abstract

Sr₂IrO₄ is one of the prime candidates for realizing exotic quantum spin orders owing to the subtle combination of spin-orbit coupling and electron correlation. Sensitive local magnetization measurement can serve as a powerful tool to study these kinds of systems with multiple competing spin orders since the comprehensive study of the spatially-varying magnetic responses provides crucial information on their energetics. Here, using sensitive magneto-optical Kerr effect measurements and spin Hamiltonian model calculations, we show that Sr₂IrO₄ has non-trivial domain structures which cannot be understood by conventional antiferromagnetism. This unconventional magnetic response exhibits broken symmetry along the Ir-O-Ir bond direction and is enhanced upon spin-flip transition or Rh-doping. Our analysis, based on possible stacking patterns of spins, shows that introduction of an additional rotational-symmetry breaking is essential to describe the magnetic behavior of Sr₂Ir_1-xRh_xO₄, providing strong evidence for a nematic hidden order phase in this highly correlated spin-orbit Mott insulator.

Introduction

The layered perovskite Sr₂IrO₄ has drawn significant attention due to its resemblance to cuprate superconductors in structural, electrical and magnetic properties^1,2,3,4,5,6. Similar to the effect of doping in cuprates, the electrical and magnetic properties of Sr₂IrO₄ also change significantly upon doping, suggesting a potential to observe unconventional superconductivity in this system. Besides these similarities, Sr₂IrO₄ shows spin-orbit coupling (SOC) and electron correlation driven Mott insulating antiferromagnetism (AFM)^2,7,8 and is a prominent candidate to realize an ideal spin-half Heisenberg model in a square lattice^4,6,9,10. Via strain, doping and current, the strong SOC can be utilized for tuning of the AFM phase^{11,12,13,14,15,16}, potentially leading to applications, such as spintronics. Particularly, slight Rh-doping is believed to decrease SOC¹⁷, suppressing the AFM order and enlarging a “hidden” ordered (HO) phase above the Neel temperature (T_N) with an unknown order parameter whose coupling to the most of detection techniques are significantly weak, so deemed as hidden^18,19,20,21. In the phase diagram, this HO phase resembles the pseudogap phase in cuprates, which surrounds the Mott insulating phase, raising questions about the nature of this phase and which kind of symmetry is broken. For undoped and slightly Rh-doped Sr₂IrO₄, second-harmonic generation (SHG), magnetic torque and polarized neutron diffraction studies proposed a loop-current order as an explanation of the HO phase^18,19,20, suggesting that exotic orbital effects may be reflected on the properties of the AFM phase due to strong SOC.

In order to understand the relationship between the HO phase and the AFM phase, an accurate characterization of the AFM phase is critical. Based on the concept of the spin-orbit coupled Mott insulator, the spin configurations of Sr₂IrO₄ are described by effective ½-spins (J_eff = 1/2) centered at Ir sites²². These pseudospins (red arrows in Fig. 1b) are confined to the basal plane of IrO₂, and the relative angles between pseudospins remain fixed regardless of the external magnetic field due to strong intralayer exchange interaction and the Dzyaloshinskii-Moriya interaction²³, which allows the net moment of each layer to be regarded as a single magnetic moment (black arrows in Fig. 1b) with a constant magnitude. The ground state configurations in zero magnetic field limit, however, lead to the cancellation of the total moment with layer moment configurations predominantly along the a-axis ($\bar{a}{aa}\bar{a}$) or b-axis ($\bar{b}\bar{b}{bb}$), depicted in Fig. 1b (AFM-Ⅰ phase)^22,24. Upon applying a magnetic field, owing to the relatively weak interlayer couplings, the moments of the individual layers rotate and can develop a net magnetization via the Zeeman interaction. Beyond the critical field H_C, all these moments abruptly flip to align in the same direction, resulting in weak ferromagnetic (WFM) behavior in Sr₂IrO₄, as shown in Fig. 1c (AFM-Ⅱ phase)²². Various magnetic measurements have been conducted to investigate symmetry of the spin responses in the canted AFM phase^{25,26,27,28,29,30,31}, and significant deviations from the four-fold rotational symmetric magnetic responses anticipated from the tetragonal crystal structure are often observed^31,32. The spin behaviors in this system are still yet to be fully understood but most previous experiments have relied on measurements of bulk properties, where inherent difficulties in resolving local orders and eliminating geometrical effects of a crystalline sample complicate the interpretation of the already complex quantum phenomena.

Fig. 1: Crystal structure and magnetic configuration of undoped Sr₂IrO₄.

a The crystal structure of Sr₂IrO₄, which belongs to the tetragonal I4₁/acd space group, is composed of canted IrO₆ octahedra^22,53. The unit cell consists of four iridium-oxide layers. b (Top view) Ground state configurations in the AFM-Ⅰ phase. c Potential magnetic configurations in the AFM-Ⅱ phase for H > H_C where H_C is the critical field. The thin red arrow indicates the pseudospin at each Ir site, the thick black arrow indicates the net moment of each layer, and the thick blue arrow indicates the net magnetization M in a unit cell. d Schematic of the MOKE measurement. θ_K,L (θ_K,T) represents the longitudinal (transverse) Kerr angle, corresponding to the magnetization component parallel (perpendicular) to the external magnetic field H.

In this paper, we report scanning three-dimensional vector magneto-optical Kerr effect (MOKE) measurements of Sr₂Ir_1-xRh_xO₄, that resolve the comprehensive local magnetic properties within individual domains. In undoped samples, the local responses exhibit peculiar two-fold rotational symmetry upon the spin-flip, with the direction of the symmetry axis mainly rotated 45° from the crystalline axes but varying spatially across a sample, suggesting a spontaneously broken rotational symmetry driven by an order coexisting with the conventional AFM phase. This observation is further corroborated with measurement of Rh-doped samples, where two types of domains with the respective easy-axes pointing in the directions whose angles are again 45° rotated from the crystalline axes, which is possibly related to the nematic HO phase.

Results

MOKE measurement of undoped Sr₂IrO₄

The basal-plane rotational symmetry of the AFM phase in Sr₂IrO₄ is analyzed by measuring its local magnetization vector while sweeping the magnetic fields in-plane. To perform this experiment, we used a zero-area Sagnac interferometer to accurately determine the vector components of the local magnetization, by measuring both the longitudinal and transverse Kerr effects (Fig. 1d and methods). As the magnetic field was always applied parallel to the plane of incidence, the longitudinal (transverse) Kerr signal represents the magnetization component parallel (perpendicular) to the magnetic field. With a sample mount rotating about the surface normal of the crystal, the relative direction of the in-plane magnetic field to the crystal axes of the sample could be controlled.

Initially, the magnetization vectors at selected positions (as denoted in Fig. 2a) were measured as a function of the magnetic field. Figure 2b shows the experimental results at position P5 when the magnetic field is applied along the a-axis. Combining the longitudinal (black line) and transverse (red line) Kerr data, we can determine the full magnetization vector and represent its components in any crystallographic coordinate. Note that the jump in the magnetization at the critical field μ₀H_C ~ 0.2 T is the metamagnetic transition, in which the moments of the layers flip and point along the same direction²². Unexpectedly, the transverse signal becomes non-zero beyond the transition, indicating that the direction of magnetization is not along the magnetic field even applied along the a-axis. This implies that the symmetry of the magnetic response is different from the symmetry given by the crystallographic axes anymore. To get a better picture of this phenomenon, we plotted the critical field μ₀H_C as a function of magnetic field direction at the positions in Fig. 2c. The responses show two-fold rotational symmetry at all positions, but the symmetry direction, which is defined by the direction having the maximum value of H_C, is different for each position. Since the ground state spin configuration of Sr₂IrO₄ is known to be either $\bar{a}{aa}\bar{a}$ or $\bar{b}\bar{b}{bb}$, the extrema of H_C are expected to be on either a- or b-axis but, at P4, P5, and P6, they are observed to be along the x- or y-axis, which is the Ir-O-Ir bond direction (see also Supplementary Fig. 5).

Fig. 2: Local magnetic behavior of undoped Sr₂IrO₄.

a Optical microscope image of the sample. b Magnetization M at position P5 in (a) as a function of the magnetic field applied along the a-axis. ${M}_{{{\rm{L}}}}$ (${M}_{{{\rm{T}}}}$) represents magnetization component parallel (perpendicular) to the external magnetic field H. The magnetization is normalized by the value of M_S, which is the saturation magnetization when all layer moments are fully aligned. The raw data can be found in Supplementary Fig. 4a. The shaded region represents H < H_C, and the unshaded region represents H > H_C, where H_C is the critical field. c Magnetization magnitude |M| (color map) as a function of an in-plane magnetic field in a polar plot, at three different positions as indicated. Critical fields H_C are indicated by black circles. Detailed magnetic responses at (d) P1, (e) P4, and (f) P5. The blue arrows display net magnetizations M at various in-plane magnetic fields whose values are in the polar maps. For illustration, the arrows for H < H_C are magnified four times, while for H > H_C, a single arrow is plotted per field direction. Insets in (d–f) depict the schematics of the uniaxial anisotropic energy terms extracted from the model calculation. Increasing external magnetic field induces the spin-flip, accompanied by a change in the energetically favorable axes. All measurements were performed at T = 5 K.

For further analysis, the detailed view of the local magnetizations at positions P1, P4, and P5, which display representative behaviors, are shown in Fig. 2d–f, respectively. In each panel, the blue arrows represent the magnetization vectors measured under the external magnetic fields whose coordinates are shown in a polar plot. In Fig. 2d, we show the data at position P1, which is one of the most conventional and thus the simplest to interpret. The magnetization vectors for H < H_C lie noticeably parallel to the b-axis throughout the directions of the magnetic field until the spin-flip transition occurs. When a small magnetic field is applied in the AFM-I phase, each layer moment has no option but tilts slightly. As a result, any net magnetization would be perpendicular to the orientation of the layer moments of the ground state. Based on this argument, the observed net magnetization along the b-axis should be ascribed to the magnetic response of a region with $\bar{a}{aa}\bar{a}$ (left panel in Fig. 1b) – or its Kramers pair $a\bar{a}\bar{a}a$ – and rules out other potential ground state spin configurations – such as $\bar{b}\bar{b}{bb}$ – where domains that break the symmetry of $\bar{a}{aa}\bar{a}$ and $\bar{b}\bar{b}{bb}$ configurations were reported and attributed to crystal twin boundaries^32,33,34. Beyond the spin-flip (H > H_C), all the layer moments point to nearly the same directions due to the Zeeman energy and thus follow the magnetic fields more freely, but there still exists preference to align along the a-axis. In our model Hamiltonian analysis, the responses at P1 below and above the spin-flip are well explained by the single Hamiltonian term ${K}_{{ab}}$ (methods). Qualitatively similarly, the magnetic behaviors at positions P2 and P3 (Supplementary Fig. 5b, c) show nearly conventional behaviors of $\bar{b}\bar{b}{bb}$ and $\bar{a}{aa}\bar{a}$, respectively, but with anomalous distortion in the shape of H_C.

The effect of distortion becomes stronger at position P4. In Fig. 2e, the magnetization below the spin-flip transition is along the b-axis also similar to the case of position P1, which infers that the ground state configuration is along the a-axis. However, after the spin-flip transition, the magnetization prefers to align along the x-axis, unexpected from the given crystalline symmetry. Without any a priori reason for spins to tend to align along the x- or y-axis, this seems to suggest spontaneously-broken rotational symmetry and the existence of a nematic order that couples with the WFM moment and provides uniaxial preference to the Ir-O-Ir bond direction. Further corroborating the effect, the data at position P5 and P6 are plotted in Fig. 2f and Supplementary Fig. 5d, respectively. First, the magnetic behaviors for H < H_C are those expected from a region with the ground state configurations $\bar{b}\bar{b}{bb}$ and $\bar{a}{aa}\bar{a}$ nearly degenerate or coexisting. Nevertheless, when H > H_C, the effect of coupling with the bond-directional order discussed at the position P4 is again observed, evidencing that this order represents an independent mechanism distinct from the effect that locks the moments to one of the tetragonal crystal axes.

The bond-directional order can be associated with an energy term that gives preference along the x- or y-axis and couples strongly with moments after the spin-flip transition. To implement phenomenologically these observations in the model calculations (methods), we introduce the following additional Hamiltonian term:

$${H}_{{xy}}={-N}_{{xy}}\left({\left(\frac{1}{4}\mathop{\sum }\limits_{i=1}^{4}\cos {\alpha }_{i}\right)}^{\!2}-{\left(\frac{1}{4}\mathop{\sum }\limits_{i=1}^{4}\sin {\alpha }_{i}\right)}^{\!2}\right)$$

(1)

where α_i refers to the angle of the layer moment of the i-th layer in a unit cell with respect to the x-axis. Since ${H}_{{xy}}$ is proportional to $-{N}_{{xy}}({{M}_{x}}^{2}-{{M}_{y}}^{2})$ where ${M}_{x}$ (${M}_{y}$) is the magnetization component along the x- (y-) axis, it has little effect prior to the spin-flip transition. After the transition, ${H}_{{xy}}$ acts to align the net magnetization along the x- or y-axis. The Hamiltonian ${H}_{{xy}}$ can be interpreted as a result of the exchange interaction between the pseudospins and the background electronic subsystem with anisotropic susceptibility along the Ir-O-Ir bond direction (Supplementary Note 4). By introducing ${H}_{{xy}}$, we quantitatively reproduce the critical features of the data (Supplementary Fig. 7), to the level we are strongly convinced that the correct model Hamiltonian with minimal terms is used. Thus, this model supports the picture that the WFM moment in the AFM-Ⅱ phase, which suddenly develops above the spin-flip transition, triggers strong interaction between the Ir spins and the anisotropic nematic order, resulting in the emergent two-fold symmetry selectively observed in the AFM-Ⅱ phase. We note that introducing a finite value of the pseudo-Jahn-Teller (pseudo-JT) effect is also required to describe the sudden jump of the magnetization³⁵. By comparing the experimental results with the model calculations (Fig. 2d–f and Supplementary Fig. 7), we determined that the xy type orthorhombic distortion (denoted as Γ₁ in methods) is dominant in the undoped sample^24,35.

As discussed, the spin-flip transition represents a point that separates the low-field limit where the a-b symmetry breaking effect is dominant and the high-field limit where the moments strongly couple to the bond-directional nematic order. To study the spatial distribution of these effects, we present scanning MOKE images of the undoped sample in Fig. 3. First of all, we measured remanent magnetization in the absence of an external magnetic field to distinguish non-ideal regions, as shown in Fig. 3a; the sample shows zero magnetization as expected for the AFM phase except for a small region with finite remanent magnetization which is coincident with the position P0 in Fig. 2a. This region displays a WFM response along the x-axis with T_N of ~200 K (Supplementary Figs. 9 and 10), similar to that of a weakly doped Sr₂IrO₄, and we attribute it to originate from the oxygen deficiency³⁶. Increasing the magnetic field, the sample shows a unique spatial pattern of magnetization as shown in Fig. 3b. Then, to investigate the spatial distribution of the distinct broken rotational symmetries, such as a-, b-, x- and y-axis preferences discussed in Fig. 2, we devise an efficient measure for visualization with a differential susceptibility tensor ${\chi }_{{ij}}={\partial M}_{i}/\partial {H}_{j}{|}_{{H}_{0}}$($i,j=a,b$), where the quantity is evaluated at H = H₀. For H < H_C, where the phases preferring the a-axis ($\bar{a}{aa}\bar{a}$) and the b-axis ($\bar{b}\bar{b}{bb}$) are to be distinguished, ${\chi }_{{aa}}$ evaluated at H₀ = 0 is plotted in Fig. 3c. The region of non-zero values is where $\bar{b}\bar{b}{bb}$ configuration is present while the region of near zero values means that the area allows $\bar{a}{aa}\bar{a}$ as the only allowed configuration. For H > H_C, due to the emergence of coupling to a bond-directional order, the symmetry breakings along a-, b-, x- and y-axes are to be distinguished. We use ${\chi }_{{ba}}$ evaluated at H₀ > H_C in Fig. 3d, where the positive (negative) value indicates that the nematic order is along the x-axis (y-axis). Note that if a phase has symmetry breaking along the a- or b-axis with weak bond-directional order, ${\chi }_{{ba}}$ would be zero, which is visible as yellow-colored regions in Fig. 3d (Supplementary Note 8). A comparison of two images indicates that they both have domain structures. The a-b symmetry breaking in H < H_C is known to be mostly dictated by crystal twinning^32,33,34; on the other hand, the image for x-y symmetry breaking at H > H_C apparently shows a different spatial pattern across the sample (see also Supplementary Fig. 11), supporting that a unique mechanism plays a role after the spin-flip transition. The observation of the macroscopic-sized regions with a sudden and distinctive change of rotational symmetry to the x- or y-axis above H_C evidences the formation of domains of a bond-directional order, which cannot be accounted for by any conventional AFM picture of Sr₂IrO₄.

Fig. 3: Scanning MOKE image of undoped Sr₂IrO₄.

Magnetization image of the sample at (a) μ₀H = 0 T and (b) μ₀H = 0.15 T. Arrows indicate the direction and magnitude of the local magnetization M, normalized by the value of the saturation magnetization M_S for which all layer moments are fully aligned. Differential magnetic susceptibility images. c ${\chi }_{{aa}}$ for H < H_C and (d) ${\chi }_{{ba}}$ for H > H_C where H_C is the critical field. The magnetic field is applied along the a-axis. Grey areas indicate weak ferromagnetic (WFM) regions in the absence of an external magnetic field. The dashed lines indicate the physical edge of the sample.

MOKE measurement of Rh-doped Sr₂IrO₄ (Sr₂Ir_1-xRh_xO₄)

We have established that the coupling to the bond-directional nematic order appears only in the AFM-Ⅱ phase of undoped Sr₂IrO₄, after the spin-flip transition. The AFM-Ⅱ phase is also known to be induced by slight Rh-doping^37,38,39, and we thus measured Rh-doped Sr₂IrO₄ to determine whether the spin-flipped state shows the same bond-directional magnetic behavior.

To identify the rotational symmetry of Rh-doped Sr₂IrO₄, we performed the rotating-field MOKE measurements at a representative position in a 3% Rh-doped sample. As shown in Fig. 4a, the magnitude remained to be nearly constant for a wide range of magnetic field strength and angles, indicating a saturation of moments; this confirms that all the moments of layers are already flipped and point to the same direction, which is the characteristic of the AFM-Ⅱ phase. More importantly, it shows two-fold rotational symmetric magnetic responses with the symmetry axis along the x-axis, evidencing that the spin-flipped state exhibits the same bond-directional nematic behavior as undoped Sr₂IrO₄. In Fig. 4b, we plotted ${M}_{{{\rm{T}}}}/{M}_{{{\rm{L}}}}$ (a quantity to represent the misalignment of the magnetization vector with respect to the magnetic field direction) as a function of $\phi$, and compared this with the calculation based on the same model Hamiltonian used in the undoped case. The calculation reproduces very accurately all features of the experimental data, showing the easy axis 45° rotated from the tetragonal crystal axes. It becomes apparent that the bond-directional order influences dominantly in the AFM-Ⅱ phases of all samples, regardless of doping level (Supplementary Figs. 9 and 12).

Fig. 4: MOKE measurement and model calculation of the 3% Rh-doped Sr₂IrO₄.

a Quiver plot of local magnetic responses, where blue arrows indicate the magnetization vectors M. b Comparison between experimental data (circles) and model calculation (lines) under external magnetic fields of 0.3 T (red), 0.5 T (green), and 0.7 T (blue). ${M}_{{{\rm{L}}}}$ (${M}_{{{\rm{T}}}}$) represents magnetization component parallel (perpendicular) to the external magnetic field H. $\phi$ is the angle of the magnetic field vector with respect to the a-axis. In the model calculation, we used parameters of J_1c = −13.0 μeV, J_2c = 3.0 μeV, N_xy = 1.9 μeV, Γ₂ = 1.2 μeV, and an effective parameter E_Th = 0.4 μeV, which was introduced to describe magnetic hysteresis (see methods for details of each parameter). c Magnetization direction image at zero magnetic field, with respect to the x-axis. Magnetizations are aligned along the x-axis (denoted as X) or the y-axis (denoted as Y). The pink (blue) arrow indicates the direction of the remanent magnetization in domain X (Y). The configuration in the dashed red box represents the magnetic configuration of domain X, where the thin red arrow indicates the pseudospin at each Ir site, the thick black arrow indicates the net moment of each layer, and the thick blue arrow indicates the net magnetization M in a unit cell. An optical microscope image of the sample is shown on the right. All measurements were performed at T = 5 K.

The most critical feature is that the magnetization behaves similar to a ferromagnet with an easy axis along the x-axis. This behavior is captured in the model calculation, which we attribute to Rh-doping-induced sign-change in interlayer interactions³⁸ and the bond-directional order with positive N_xy (see Eq. 1). As shown in Fig. 4c, the positive N_xy term enforces the ground state configuration to be ${xxxx}$, excluding any possibility of a domain mixed of conventionally-argued ${aaaa}$ and $\bar{b}\bar{b}\bar{b}\bar{b}$. It is worthy to note that we also had to introduce the x²-y² type orthorhombic distortion (denoted as Γ₂ in methods) to closely reproduce the kink indicated by a green arrow in Supplementary Fig. 13a, b. This term confirms the previous conjecture that the ground state magnetic ordering affects the nature of the pseudo-JT effect³⁵; i.e., the $\bar{a}{aa}\bar{a}$ or $\bar{b}\bar{b}{bb}$ state in undoped Sr₂IrO₄ is related with lattice distortion along the a- and b-axes (xy type), whereas the ${xxxx}$ or ${yyyy}$ state in Rh-doped Sr₂IrO₄ is associated with lattice distortion along the x- and y-axes (x²-y² type).

To study the spatial distribution of the nematic order, zero-magnetic-field scanning MOKE images of Rh-doped samples of 3% and 5% doping are shown in Fig. 4c and Supplementary Fig. 12a, respectively. Indeed, the magnetizations are distributed exclusively along the x- or y-axis, and the images display distinctive domain structures with two mutually orthogonal axes (denoted as X and Y). In the model calculation, the magnetic field response of any region can be modeled using the N_xy term, simply by changing its sign (Supplementary Fig. 12b–i). In contrast to previous Bragg peak studies^37,38, the observation of X and Y domains indicates that the magnetization in doped samples is not confined to the a- or b-axis. It should be noted, however, that the previous measurements provide information averaged over the substantial area that represents properties of a whole sample, which importantly differ from our study that directly measures local characteristics, in the presence of the spatially-varying bond-directional order.

Discussion

This work represents systematic local profiling of the full 3D magnetization of spin orders, measured across the whole surface of high-quality single crystals Sr₂Ir_1-xRh_xO₄. The data acquired by sweeping in-plane magnetic fields provide a comprehensive picture of the energetics of spins in this canted AFM phase and reveal the previously unknown features and hidden broken-symmetries, especially through the response above the spin-flip transition.

While our analysis provides a strong case for electron nematicity in this system, the microscopic mechanism that could drive the nematicity is still unspecified. The current work however places stringent constraints on theoretical models that attempt to implement the nematicity. To our knowledge, there are two major proposals in the literature for a nematic order that can be present in this system; a loop-current order^18,19,20 and a spin nematic order⁴⁰. The existence of the bond-directional loop-current order, which preserves translational symmetry but is expected to break inversion, time-reversal and rotational symmetry, could give rise to nematicity of the coexisting electron system but is only partially consistent with our observations; spontaneous breaking of time-reversal symmetry is not observed in any of samples with our sensitive Sagnac interferometer above T_N (see Kim, H. et al.⁴⁰ and Supplementary Fig. 10). On the other hand, the recently discovered spin nematic phase preserves time-reversal symmetry⁴⁰. The quadrupolar order of the spin nematicity and the dipolar order of the canted AFM phase mutually constrain their directions with each other⁴⁰ and thus, in our measurement in H > H_C, we effectively rotate the ferromagnetic component of the AFM phase and the ferro-quadrupolar component of spin nematicity together by rotating the external magnetic field. Since the quadrupole is known to prefer to point to the bond directions^40,41,42, which is intuitively understood based on the fact that a quadrupole of the spin nematic phase is formed by two spins at the nearest neighbor sites sharing the Ir-O-Ir bond, the energetic preference of the mutually-constraint dipole to align to one of the bond directions is naturally explained. The bond-directional response of the canted AFM dipoles, combined with the observed preservation of time-reversal symmetry, thus provides evidence of the spin nematic phase as the mechanism behind the observed nematicity.

It would be worth noting that we find the bond-directional phase to remain very similar upon thermal cycling to the room temperature. It may imply that external effects—such as strain or structural inhomogeneity, while they are not identifiable in X-ray diffraction, energy-dispersive X-ray spectroscopy, and atomic force microscopy data (see Supplementary Note 2)—can influence the nematic behavior; they may contribute to the local realization of domains as a pinning mechanism via the local variation of the pseudo-spin coupling strengths. But we would like to point out that this spatially distributed domain structure cannot be explained primarily by sample strains¹³ or by a peculiar surface magnetization suggested in the SHG experiment³². First, there is no reason to believe that directions of strain from extrinsic sources (i.e., different thermal contractions between the sample and grease) would align along particular high-symmetric crystal axes, such as a-, b-, x- and y-axis, and be distributed over the length scale of 100 μm and 1000 μm, which is the typical size of a domain observed (Fig. 4c and Supplementary Fig. 12a). Second, there is no indication of suppression of T_N in any of the crystals we measured. In addition, our optical measurement accesses at least hundreds of unit cells from the surface of a crystal, estimated based on the optical penetration depth, which excludes any possibility of a signal originating from a certain surface termination or reconstruction. We also note a recent study that suggested Ir-O-Ir bond-directional spin orientation in insulating SrIrO₃ monolayer film while the microscopic mechanism is not proposed⁴³.

Methods

Sample preparation

The Sr₂Ir_1-xRh_xO₄ single crystals were grown using the flux method. Stoichiometric quantities of SrCO₃, IrO₂, and Rh₂O₃ powders were thoroughly mixed with SrCl₂ flux. The powder-to-flux molar ratio was 1:7, based on previously reported synthesis methods^36,44,45. The mixed materials were placed in a platinum crucible. The growth procedure involved heating the mixture to 1300 °C and then slowly cooling it to 800 °C at a rate of 2 °C per hour, followed by furnace cooling. The doping concentration of each sample was confirmed using an energy-dispersive X-ray spectroscopy machine (Hitachi S-3400N). The measured doping concentrations for the x = 0.03 and x = 0.05 samples were 0.0369 and 0.0550, respectively, showing no significant deviation from the nominal ones. Neel temperature measured by MOKE further confirms the doping level (Supplementary Fig. 10).

Magneto-optical Kerr effect measurement

We use the normal-incidence (oblique-incidence) zero-area Sagnac interferometer operating at 1550 nm wavelength to measure local out-of-plane (in-plane) magnetic properties^46,47,48. This interferometry-based scheme provides superior stability and reliability for detecting magnetic signals from a sample⁴⁹, as it rejects all reciprocal effects, such as linear birefringence and optical activity, in both the optics and the sample. For the normal-incidence MOKE measurement, we didn’t find any discernible polar Kerr signal except for WFM domain boundaries or near the sample edges under the noise limit, confirming spins lying on the IrO₂ plane. For the oblique-incidence MOKE measurement, a beam of light is incident on a sample with an angle of ${\theta }_{{{\rm{inc}}}}$ ~ 15° and measures simultaneous longitudinal and transverse Kerr signals with the zero-area Sagnac interferometer^47,48. For the measurements of in-plane magnetization components (${M}_{{{\rm{L}}}}$, ${M}_{{{\rm{T}}}}$), we utilized oblique-incidence Sagnac interferometer. As shown in Supplementary Fig. 1, we installed the whole optical elements on a 3-axis motorized stage for the scanning Kerr measurements. To measure longitudinal Kerr effect (which is proportional to ${M}_{{{\rm{L}}}}$), we use a half-wave plate as waveplate 1 with an angle of 22.5° (, which is one half of 45°) from the fast axis of the polarization-maintaining fiber and a quarter-wave plate as waveplate 2 parallel to the fast axis. To measure the transverse Kerr effect (which is proportional to ${M}_{{{\rm{T}}}}$), the waveplate 1 is (physically) removed and the waveplate 2 is rotated 45°.

As shown in Fig. 1d, magnitude |M| (Eq. 2) and direction $\varOmega$ (Eq. 3) of the local magnetization are calculated by measuring both longitudinal Kerr angle ${\theta }_{{{\rm{K}}},{{\rm{L}}}}={\mathrm{Re}}(2\sin {\theta }_{{{\rm{inc}}}}Q/({\varepsilon }_{{{\rm{s}}}}-1)){M}_{{{\rm{L}}}}$ and transverse Kerr angle ${\theta }_{{{\rm{K}}},{{\rm{T}}}}={\mathrm{Re}}(2\sin {\theta }_{{{\rm{inc}}}}Q/({\varepsilon }_{{{\rm{s}}}}-1)){M}_{{{\rm{T}}}}$⁴⁸:

$$|{{\bf{M}}}|=\sqrt{{{M}_{{{\rm{L}}}}}^{2}+{{M}_{{{\rm{T}}}}}^{2}}\propto \sqrt{{{\theta }_{{{\rm{K}}},{{\rm{L}}}}}^{2}+{{\theta }_{{{\rm{K}}},{{\rm{T}}}}}^{2}}$$

(2)

$$\varOmega ={\tan }^{-1}({\theta }_{{{\rm{K}}},{{\rm{T}}}}/{\theta }_{{{\rm{K}}},{{\rm{L}}}})$$

(3)

where Voigt parameter $Q$ and dielectric constant ${\varepsilon }_{{{\rm{s}}}}$ at wavelengths of 1550 nm are material dependent parameters. In the current study, we apply the in-plane magnetic field along the plane-of-incidence; thus, ${M}_{{{\rm{L}}}}$ (${M}_{{{\rm{T}}}}$) represents the in-plane magnetization component of a sample that is parallel (perpendicular) to the plane-of-incidence and the magnetic field. Then we rotate the sample, to control the angle between the magnetic field and the crystallographic axes of the sample. A beam spot is approximately 50 μm in diameter. We confirm that the optical heating is smaller than 1 K as the incident optical power is maintained below 1 mW. For all data presented, we subtract temperature independent background signal from optical windows which is linearly proportional to the strength of the external magnetic field (Supplementary Note 11).

We have performed empirical calibration procedures to transform the Kerr angle to magnetization as follows: We used previously measured bulk magnetization data of a sample with Magnetic Property Measurement System⁵⁰ and compare with the bulk Kerr signal obtained by numerically averaging local Kerr signals over a sample. However, the Kerr-to-magnetization conversion factors obtained by the procedure are intrinsically prone to sample dependent errors originating from a specific sample geometry and wavelength dependence of Kerr rotations etc. Thus, in the figures, we choose to present the magnetization values normalized to the saturation magnetization of each sample, which would deliver physical meanings of the measured data more clearly.

Model Hamiltonian

The Hamiltonian for the first- and second-nearest-interlayer interactions is written in the form²⁴:

$${H}_{{{\rm{inter}}}}=\frac{1}{N}\mathop{\sum }\limits_{ < {ij} > }{J}_{1{{\rm{c}}}}{S}_{i}\cdot {S}_{j}\pm {\Delta }_{{{\rm{c}}}}({S}_{i}^{a}{S}_{j}^{a}-{S}_{i}^{b}{S}_{j}^{b})+\frac{1}{N}\mathop{\sum }\limits_{ < {ij} > }{J}_{2{{\rm{c}}}}{S}_{i}\cdot {S}_{j},$$

(4)

where the first (second) <i j> refers to the nearest pseudospin pairs in the first (second) adjacent layers, N refers to the number of Ir-atoms, and a and b denote the directions along the crystal axes. Note that one pseudospin S_i has eight (two) nearest pseudospins S_j in the first (second) two nearest layers. J_1c and J_2c determine the exchange energy of the corresponding pseudospin pairs. Δ_c accounts for orbital characteristics of pseudospins and gives the direction dependence in the nearest interlayer coupling^24,51.

Additionally, the strong spin-orbit coupling in Sr₂IrO₄ allows the pseudospins to be coupled to lattice vibrations, which is called pseudo-JT effect. The Hamiltonian of the pseudo-JT effect is written in the form^24,35:

$${H}_{{{\rm{JT}}}}=-{\varGamma }_{1}{\left(\frac{1}{N}\mathop{\sum }\limits_{ < {ij} > }{S}_{i}^{a}{S}_{j}^{a}-{S}_{i}^{b}{S}_{j}^{b}\right)}^{2}-{\varGamma }_{2}{\left(\frac{1}{N}\mathop{\sum }\limits_{ < {ij} > }{S}_{i}^{a}{S}_{j}^{b}+{S}_{i}^{b}{S}_{j}^{a}\right)}^{2},$$

(5)

where <i j> refers to the nearest intralayer pseudospin pairs. Note that Γ₁ and Γ₂ define the preference for xy and x²-y² type orthorhombic distortion of IrO₆ octahedra. The observation of metamagnetic transition in Sr₂IrO₄ can be explained with the pseudo-JT effect³⁵.

Using the Hamiltonian $H={H}_{{{\rm{inter}}}}+{H}_{{{\rm{JT}}}}$ from Eqs. 4, 5, the magnetic configurations of $\bar{a}{aa}\bar{a}$ and $\bar{b}\bar{b}{bb}$, shown in Fig. 1b, and their Kramers pairs are the most stable^22,24 (Supplementary Note 12). This description could support the observations of magnetic Bragg peaks implying the magnetic configurations^22,52. However, there are reports that moments didn’t have both $\bar{a}{aa}\bar{a}$ and $\bar{b}\bar{b}{bb}$ configurations as the ground state, but only one of them as the ground state^32,33,34, which cannot be explained by the four-fold Hamiltonian. To explain this, it is necessary to introduce a Hamiltonian that breaks the four-fold rotational symmetry and locks the moments to a unique in-plane crystal axis, which can be written in the form:

$${H}_{{{\rm{aniso}}}}=-\frac{1}{N}\mathop{\sum }\limits_{ < {ij} > }{K}_{{ab}}\left({S}_{i}^{a}{S}_{j}^{a}-{S}_{i}^{b}{S}_{j}^{b}\right),$$

(6)

where <i j> refers to the nearest intralayer pseudospin pairs.

We also introduced a Hamiltonian ${H}_{{xy}}$ to describe the new bond-directional order in Eq. 1. Based on these arguments, we calculated the energy E derived from the Hamiltonian $H={H}_{{{\rm{inter}}}}+{H}_{{{\rm{JT}}}}+{H}_{{{\rm{aniso}}}}+{H}_{{xy}}$ with Zeeman energy in Eq. 7.

$$E = \, \frac{{j}_{1{{\rm{c}}}}}{4}\mathop{\sum }\limits_{i=1}^{4}\cos \left({\alpha }_{i}-{\alpha }_{i+1}\right)+\frac{{j}_{2{{\rm{c}}}}}{4}\mathop{\sum }\limits_{i=1}^{4}\cos \left({\alpha }_{i}-{\alpha }_{i+2}\right)+\frac{{\delta }_{{{\rm{c}}}}}{4}\mathop{\sum }\limits_{i=1}^{4}{\left(-1\right)}^{i+1}\\ \sin \left({\alpha }_{i}+{\alpha }_{i+1}\right)-{\gamma }_{1}{\left(\frac{1}{4}\mathop{\sum }\limits_{i=1}^{4}\sin \left(2{\alpha }_{i}\right)\right)}^{2}-{\gamma }_{2}{\left(\frac{1}{4}\mathop{\sum }\limits_{i=1}^{4}\cos \left(2{\alpha }_{i}\right)\right)}^{2}\\ -\frac{{k}_{{ab}}}{4}\mathop{\sum }\limits_{i=1}^{4}\sin \left({2\alpha }_{i}\right)-{N}_{{xy}}\left\{{\left(\frac{1}{4}\mathop{\sum }\limits_{i=1}^{4}\cos {\alpha }_{i}\right)}^{2}-{\left(\frac{1}{4}\mathop{\sum }\limits_{i=1}^{4}\sin {\alpha }_{i}\right)}^{2}\right\}\\ -\frac{h}{4}\mathop{\sum }\limits_{i=1}^{4}\cos \left({\alpha }_{i}-\phi -\frac{{{\rm{\pi }}}}{4}\right),$$

(7)

where ${\alpha }_{1}(={\alpha }_{5})$, ${\alpha }_{2}(={\alpha }_{6})$, ${\alpha }_{3}$, ${\alpha }_{4}$ are the angles of the layer moments with respect to the x-axis. ϕ is the angle of the external magnetic field with respect to the a-axis. Note that j_1c = 4S²J_1csin²φ, j_2c = -S²J_2ccos2φ, δ_c = 4S²Δ_ccos²φ, γ₁ = 4S⁴Γ₁, γ₂ = 4S⁴Γ₂, k_ab = 2S²K_ab, h = gμ₀μ_BSHsinφ, with S = 1/2, φ = 12°, g = 2, and μ_B = 57.88 μeV T^-1. For more details on the model calculation, see Porras, J. et al.²⁴ and Liu, H. & Khaliullin, G³⁵. To induce hysteric behavior in the model calculation, we introduce a constant decision threshold E_Th. Similar to the rotating-field MOKE measurement, the model calculation is performed by sweeping the magnetic field from positive to negative and then from negative to positive. For the initial magnetic structure, we determine the spin configurations that minimize the energy of the four spin layers, whose energy E_G represents the global minimum energy. For subsequent magnetic structures, we compare E_G with the local minimum energy E_L. E_L is determined using the Nelder-Mead minimization method from the previous spin configuration. If E_L - E_G < E_Th (E_L - E_G > E_Th), the spin configuration whose energy is the local (global) minimum is chosen for the next spin configuration.