Introduction

Materials featuring a kagome lattice have attracted considerable attention due to their intriguing electronic structure and the emergence of novel states of matter1. Together with Dirac points arising at the K points2 and flat bands3,4,5,6, one of the most compelling features of the kagome electronic structure is its van Hove singularities (VHSs) at the M points, whose high density of states (DOS) can drive strongly correlated phenomena (Fig. 1c)1,7,8,9,10. Notably, each VHS at three distinct M points in a kagome lattice is predominantly governed by a different sublattice, resulting in a marginal on-site interaction vertex (referred to as sublattice interference)11. Combined with inter-site Coulomb interactions, this interference greatly enhances unconventional Fermi surface instability near van Hove filling8. Indeed, the presence of VHSs near EF has been shown to produce novel phenomena in kagome metals, such as charge–density wave (CDW) phases and unconventional superconductivity7,8,10,12,13,14,15,16,17,18,19,20.

Fig. 1: Structure and bulk properties of CeTi3Bi4.
figure 1

a Crystal structure of CeTi3Bi4 consisting of Ti kagome layers and 1D chains of Ce3+ ions aligned along the a-axis. b Side views of the crystal structure. Black solid lines in a, b illustrate the crystallographic unit cell. c Schematic electronic structure, dispersion, and density of states (DOS) for a kagome lattice, obtained from a simple 2D tight-binding model. d Temperature-dependent magnetic heat capacity of CeTi3Bi4 after subtracting non-magnetic components (see “Methods”) and corresponding magnetic entropy. e Temperature dependence of magnetization (M) along the a-axis, highlighting an antiferromagnetic transition around 3.4 K. f Highly anisotropic field-dependent magnetization in the magnetically ordered CeTi3Bi4 (T < TN).

Full size image

The integration of magnetism into a kagome metal further enriches the potential for exotic phases. Magnetism diversifies the topological characteristics of the Dirac cone, leading to phenomena such as exotic gap openings2,21, pair-creation of Weyl points22,23,24, and the anomalous Hall effect2,25,26. The coexistence of strongly correlated electronic behavior and spin degrees of freedom introduces a similar intriguing question: how does the high electronic DOS at EF (mutually) influence magnetic properties and generate unique magnetic states? One of the most compelling possibilities is that a nesting instability between VHSs could lead to diverging spin-spin correlations, resulting in a spin-density wave (SDW) state—a modulation of the magnetic moments—analogous to how the instability induces CDW through diverging charge correlations8,9,12. Notably, while SDWs have been observed in various magnetic systems27,28,29, SDWs driven by the VHSs of the kagome electronic structure remain, to our best knowledge, largely unexplored. An elegant way to address this challenge, while also opening additional avenues for investigation, is to add a distinct magnetic sublattice with well-localized moments between non-magnetic kagome layers, where the latter dominate the electronic structure near the Fermi level and control the collective magnetic behavior through the Ruderman–Kittel–Kasuya–Yosida (RKKY) mechanism. While many novel phenomena have been noted in magnetic kagome materials3,5,6,19,21,24,25,26, layered materials that structurally decouple the magnetic sublattice from the kagome motif can facilitate a more systematic exploration of their interplay.

The RTi3Bi4 (R = rare earth) family has recently emerged as a promising platform for pursuing the aforementioned ideas30,31,32,33. These compounds consist of R ions manifesting localized 4f magnetism and non-magnetic Ti kagome layers responsible for their metallicity (Fig. 1a). While the system appears nearly hexagonal due to the Ti kagome layers, its actual symmetry is orthorhombic due to the R sublattice (Fmmm space group). Notably, the magnetic R ions form 1D zig–zag chains along the a-axis (Fig. 1a), potentially hosting low-dimensional and anisotropic magnetic behavior30,31,32,33,34,35. Despite this symmetry reduction, the electronic structure near the Fermi level retains key features of the kagome lattice formed by Ti atoms. Recent angle-resolved photoemission spectroscopy (ARPES) and density functional theory (DFT) studies on several RTi3Bi4 compounds have revealed flat bands, VHSs, and Dirac points30,32,35,36,37,38,39, suggesting a potential interplay between spin degrees of freedom and the kagome electronic structure. Additionally, these layered compounds can be thinned down through mechanical exfoliation32, making them a promising platform for studying the two-dimensional limit of magnetism surrounded by a metallic kagome environment.

Among these compounds, CeTi3Bi4 stands out as particularly intriguing. The Ce3+ ions exhibit quantum magnetism arising from a well-isolated Kramers doublet ground state (Jeff = 1/2), as evidenced by the saturation of magnetic entropy at Rln(2) near the antiferromagnetic transition at TN = 3.2 K (Fig. 1d, e, also see Supplementary Fig. 1)30. While this aspect is often overshadowed by the heavy fermion behavior typically found in Ce-based metals40, Kondo physics can be relatively marginal when RKKY interactions dominate40,41, as is the case in CeTi3Bi4 (see Supplementary Note 4 for further explanation). Additionally, the spin–orbital coupled nature of its ground state, combined with the surrounding crystal-field environment, promotes strong magnetic anisotropy42. This is evident in the anisotropic M–H behavior of CeTi3Bi4 below TN, with a spin-flop transition observed for H//b (see Fig. 1f)30. Thus, CeTi3Bi4 provides a unique platform for investigating the interplay between anisotropic quantum magnetism and kagome electronic structure.

In this study, we present an extensive investigation of CeTi3Bi4 utilizing neutron scattering, ARPES, and DFT, and argue that CeTi3Bi4 represents a compelling case of unusual magnetic ground states driven by the VHS of the kagome electronic structure. Neutron diffraction measurements reveal an incommensurate SDW ground state of Jeff = 1/2 magnetic moments coexisting with a commensurate antiferromagnetic spin modulation. Their modulation wave vectors are nearly identical to that associated with the 2a × 2a CDW found in other kagome metals, where the VHS near the Fermi level has been suggested to be important17,18,19. This coexisting phase is observed across a majority of the temperature-field phase diagram, but transforms into a paramagnetic (or polarized) phase via an intermediate single-Q SDW state lacking the commensurate spin component, as temperature (or magnetic field) increases. Notably, both the commensurate and incommensurate modulations propagate perpendicular to the Ce3+ chains, indicating competing long-range magnetic correlations between well-separated nearest-neighbor (NN) and further-NN Ce3+ chains (d > 5.9 Å), likely mediated by conduction electrons. Interestingly, ARPES and DFT results identify VHSs near Ef at the M points of the pseudo-hexagonal reciprocal lattice, manifesting high DOS. The observed magnetic ordering wave vectors closely align with their separation vector, suggesting that a nesting instability between these singularities may have driven the observed magnetic ground state.

Results and discussion

Single-crystal neutron diffraction

The antiferromagnetic order in CeTi3Bi4 was determined via single-crystal neutron diffraction measurements conducted at multiple diffractometers (see “Methods”). All diffraction data in this study are referenced to the reciprocal lattice of the conventional orthorhombic unit cell (Fig. 1a; also see Supplementary Note 1). Figure 2a–c shows the diffraction profile of CeTi3Bi4 across a broad region of reciprocal space, both below and above TN. Above TN (Fig. 2a, c), nuclear reflections are observed at Qnuc = (h, k, l) with all-even or all-odd indices, following the selection rule of the FCC structure. Below TN, additional Bragg reflections appear at Q = Qnuc + (0, 1, 0), approximately two orders of magnitude weaker than the nuclear reflections (Fig. 2b, c). These signals become weaker at higher momentum transfer (Supplementary Fig. 5), which, along with their presence only below TN, confirms their magnetic origin.

Fig. 2: Co-existing commensurate antiferromagnetic and incommensurate spin density wave orders revealed by neutron diffraction.
figure 2

a, b Single-crystal neutron diffraction profiles of the (H0L) plane at 1.8 K (T < TN) and 5 K (T > TN). Right panels of a, b show the same plot as the left plots but scaled to magnify weak reflections. The enhanced background signal in b originates from liquid helium in the cryomagnet’s sample space and should not be interpreted as part of the sample’s signal. The dashed box in b highlights magnetic reflections. c Comparing the diffraction intensity scans along the L direction measured at 1.5 K (T < TN) and 5 K (T > TN). d Diffraction profiles on the (0KL) plane at 1.8 K. e Intensity scans along the K direction around a commensurate magnetic reflection, each collected from different beamlines and crystals. f Two magnetic ordering wave vectors that describe the observed magnetic Bragg peaks: commensurate QC = (0, 1, 0) and incommensurate QIC = (0, δ, 0) in reciprocal lattice units. Orange (gray) arrows in d–f point out magnetic reflections associated with QIC (QC). Note that δ is temperature-dependent (see Fig. 3). g Least-square refinement result of 74 nuclear reflections collected at ZEBRA. Error bars of the observed intensities are much smaller than data symbols. h, i Least-square refinement results of 24 commensurate and 47 incommensurate magnetic reflections collected at ZEBRA, respectively. j, k Magnetic structure solutions obtained from the analysis result in (h, i), respectively. In both solutions, the magnetic moments are uniaxially aligned to the b-axis. Error bars in c and e (h, i) represent the uncertainty in the measured intensity (fitted integrated intensity).

Full size image

Interestingly, diffraction profiles along the (0, k, 0) direction reveals additional incommensurate reflections near Q = Qnuc + QC below TN, where QC = (0, 1, 0) is the commensurate magnetic modulation described above. At 1.8 K, these reflections are separated from Q = Qnuc + QC by (0, ±0.06, 0) (r.l.u.), indicating a long-period spin modulation with a wavelength of ~17|b | ≃ 17.5 nm. Notably, these incommensurate reflections were consistently observed across multiple measurements using three different crystals and diffractometers (Fig. 2e). Like the commensurate magnetic reflections associated with QC, these incommensurate reflections weaken at higher Q, as expected for magnetic scattering (Supplementary Fig. 5). Together these observations suggest that CeTi3Bi4 hosts two intrinsic magnetic modulations in the ordered phase—one commensurate (QC) and one incommensurate [QIC = (0, ±0.94, 0)]—which is not likely due to sample inhomogeneity or crystal mosaic as they would result in measurement-dependent results (also see Supplementary Fig. 2). Figure 2f illustrates the schematic diffraction pattern from these two ordering wave vectors, while Supplementary Note 2 provides additional explanation about assigning QIC = (0, ±0.94, 0) instead of (0, ±0.06, 0).

Further analysis of the magnetic Bragg peak intensities clearly reveals the two spin configurations associated with QIC and QC, respectively. A key observation is the absence of both (0, 1, 0) and (0, 1 ±0.06, 0) magnetic peaks, confirmed by multiple neutron diffraction measurements (e.g., see Fig. 2d). This indicates that both spin modulations consist solely of magnetic moments aligned along the b-axis, consistent with the easy-axis anisotropy along this direction, as evidenced by the M–H curves in Fig. 1f. Importantly, the only way to incorporate the incommensurate modulation of QIC into this uniaxial spin configuration is by introducing a modulation of the local moment length, i.e., a SDW-type order. Additionally, the zero components of QIC and QC along directions perpendicular to b* suggest that the Ce3+ sublattices are ferromagnetically aligned along both the Ce3+ chain (// a) and out-of-plane (// c) directions.

The spin configurations shown in Fig. 2j, k are the only solutions compatible with these findings. This conclusion is further supported by our least-square refinement with 74 nuclear reflections and 24 (47) commensurate (incommensurate) magnetic reflections collected at ZEBRA, PSI (Fig. 2g–i). Indeed, the best agreement was found with the spin configurations depicted in Fig. 2j, k, while the other candidates yielded worse agreement factors (RF and RF2), particularly due to their predicted intensity at Q = (0, 1, 0) and (0, 1 ± 0.06, 0) (Supplementary Figs. 3 and 4). A similar refinement using data from WAND2 (HFIR) reached the same conclusion (Supplementary Fig. 6). This conclusively demonstrates a SDW ground state of the Ce3+ Jeff = 1/2 moments in CeTi3Bi4, which co-exists with a commensurate antiferromagnetic spin configuration.

The temperature dependence of the two magnetic modulations reveals a close correlation between them and the resultant intriguing phase diagram. Figure 3a shows neutron diffraction profiles around (0, 1, −8) and (0, 1, −10) at various temperatures across TN. While both QIC and QC persist up to 2.6 K, only QIC is found at 3.1 K. A detailed examination with fine temperature steps reveals two-step phase transitions (Fig. 3b–d; full datasets in Supplementary Fig. 7). The incommensurate modulation QIC = (0, \(\pm\)δ, 0) first emerges below T = 3.4 K ≃TN, with δ being temperature-dependent (Fig. 3d). This is followed by the appearance of the commensurate modulation QC at T2 = 3.0 K, below which QIC and QC coexist. This two-step transition process has been consistently observed across multiple diffraction measurements, confirming its intrinsic nature.

Fig. 3: Temperature and field dependence of the two distinct spin modulations.
figure 3

a, b Temperature-driven evolution of commensurate (QC, gray arrows) and incommensurate (QIC, orange arrows) magnetic reflections. c Temperature-dependent integrated intensities of the Qc and QIC peaks at/around (0, 1, −10). d Temperature dependence of δ for the incommensurate modulation QIC = (0, δ, 0). The two-step phase transition process at TN ≈ 3.4 K and T2 = 3.0 K is evident throughout (a–d). e Field-driven evolution of commensurate (QC, gray arrows) and incommensurate (QIC, orange arrows) magnetic reflections at T = 2.6 K. f Field-dependent integrated intensities of the Qc and QIC peaks at/around (0, 1, −10). g Field dependence of the ordering wave vector component δ. h Overall temperature-field (H // a) phase diagram summarizing the neutron diffraction measurement results. A color code shows dM/dH extracted from multiple M–H curves measured in the range of 1.8 K < T < 4 K with a 0.1 K step. The fine-step phase boundary between the polarized and intermediate single-Q phases was determined using these dM/dH curves. All neutron diffraction data shown in this figure were collected at WAND2. Error bars in c and f (d, g) represent the uncertainties in the fitted integrated intensity (fitted peak position).

Full size image

The temperature dependence of δ, or δ(T), merits further explanation. While δ(T) exhibits subtle but clear changes when only QIC is present (T2 < T < TN), it locks into a constant value δ ≃0.94 below T2 (Fig. 3b, d). This suggests that QIC and QC are not independent, implying possible double-Q nature of the magnetic ground state rather than phase separation. This is further indicated by heat capacity measurements with fine temperature steps, which show two distinct transitions approximately 0.3–0.4 K apart, consistent with neutron diffraction results (see Supplementary Fig. 1). Further comparative analysis of QIC and QC based on their temperature-dependent correlation lengths (see Supplementary Note 6 and Supplementary Fig. 8) also suggests an interdependence between these two modulations. Nevertheless, definitive evidence for the double-Q nature below T2 is still lacking, and we leave it as a future challenge.

Interestingly, the response of QIC and QC to an external magnetic field (H) is very similar to their temperature dependence. Figure 3e shows neutron diffraction profiles around (0, 1, −8) and (0, 1, −10) under H // a at various field strengths and T = 2.6 K, showing strong similarities to their temperature dependence in Fig. 3a (also see Supplementary Fig. 9). As the field increases, QC disappears first while QIC remains nearly intact (0 T < H < 2.5 T in Fig. 3f). Once QC is fully suppressed and the systems only holds a single modulation (single-Q), QIC is rapidly suppressed as the field increases further (2.5 T < H < 3.5 T in Fig. 3f). Notably, this suppression of the incommensurate component is accompanied by a faster decrease in δ (Fig. 2g), similar to its temperature dependence (Fig. 2d).

Figure 3h presents the overall temperature-field (H // a) phase diagram derived from our neutron diffraction and magnetization measurements. While both modulations coexist over a broad range, thermal fluctuations and external magnetic fields suppress QC more rapidly, resulting in an intermediate incommensurate single-Q phase before transitioning to either a paramagnetic or fully-polarized phase. Also, the relative intensity profile of the incommensurate magnetic Bragg peaks (QIC) remains nearly the same throughout the ordered antiferromagnetic phases below TN, with only a uniform intensity scaling due to changes in the size of the ordered moments (see Supplementary Fig. 10). Thus, the SDW-type modulation shown in Fig. 2k persists across the entire antiferromagnetic region in Fig. 3h, highlighting its dominance in CeTi3Bi4.

In Ce-based metallic systems, magnetic interactions between Ce3+ localized moments are generally mediated via conduction electrons through the RKKY mechanism. Indeed, the potential dominance of the RKKY mechanism in driving the observed magnetic phase diagram (Fig. 3h) is implied by previous studies43. In rare-earth metallic magnets with uniaxial anisotropy, two-step paramagnetic–incommensurate–commensurate transitions with temperature have been widely observed (see Table 1 in Ref. 43). This phenomenon has been modeled through competing exchange interactions arising from the long-ranged RKKY interactions at the level of mean-field theory43. In this model, the Fourier transformed interaction matrix J(Q) has a minimum at a commensurate wave vector for T = 0, meaning that incommensurate spin length modulation is unstable at T = 0. However, thermal fluctuations (T > 0) shift this minimum to an incommensurate wave vector, leading to intermediate incommensurate ordering between the commensurate and paramagnetic phases. Moreover, an external magnetic field appears to mimic the effect of thermal fluctuations, inducing two-step commensurate–incommensurate–ferromagnetic transitions43. This behavior resembles the phase diagram of CeTi3Bi4 (Fig. 3h), suggesting that applying the RKKY framework is appropriate for this system. Supplementary Note 5 provides additional arguments supporting the predominance of the RKKY mechanism in driving the QIC and QC modulations.

Despite the similar phase diagram, a unique feature found in CeTi3Bi4 is that the incommensurate SDW phase persists even after the appearance of the commensurate component (Qc), unlike the previous cases where incommensurate SDW modulation is intrinsically unstable for T = 043. The amplitude of QIC remains significant at the lowest measurement temperature (~1.8 K), as evident in the ordered moment sizes of QIC and QC obtained from least-square refinement (see Supplementary Table 4). This suggests that the conduction electron profile of CeTi3Bi4 possesses additional driving forces that favor the incommensurate SDW modulation, which were absent in previous examples43 of rare-earth metallic magnets. Notably, QC and QIC are nearly identical to the modulation wave vector associated with the 2a × 2a CDW found in other kagome materials17,18,19 (=\(\vec{\Gamma {\rm{M}}}\), see Fig. 4a or Supplementary Note 1), potentially indicating they share a common origin related to the kagome electronic structure.

Fig. 4: Calculated and measured electronic structure of CeTi3Bi4 near the Fermi energy EF.
figure 4

a First Brillouin zone of CeTi3Bi4 in the primitive orthorhombic convention, with labels of key high-symmetry Q points. For clarity, the Qc and QIC wave vectors are compared with the \(\overrightarrow{\Gamma {\rm{Y}}}\) wave vector. b, c Electronic structure dispersion of CeTi3Bi4 obtained from DFT calculations and ARPES measurements. d Enlarged ARPES intensity spectrum along the \(\bar{\Gamma }-{\bar{{\rm{M}}}}^{{\prime} }\) direction with a quadratic dispersion fit (pink dashed line). Orange arrows indicate the positions of maximum intensity. e Similar to d, but along the \(\bar{{\rm{K}}}-{\bar{{\rm{M}}}}^{{\prime} }\) direction. The spectra in d, e are symmetrized with respect to the reflection plane at the \({\bar{{\rm{M}}}}^{{\prime} }\) point. DFT calculations (gray dotted lines) are overlayed on the ARPES spectrum in (c, e). f Fermi surface measured by ARPES with an enlarged view (right panel) around two adjacent \({\bar{{\rm{M}}}}^{{\prime} }\) points, illustrating the (quasi-)connection between VHSs by Qc and QIC. The color scale in f represents the second derivative of the measured intensity (arb. units), while the original intensity spectrum is shown in Supplementary Fig. 11. An energy integration range of 5 meV was used to construct the ARPES Fermi surface plots. g Fermi surface cross-section from the DFT band structure. The color scale indicates the Fermi velocity (eV × Å). Orange and purple arrows in (f), or blue and purple arrows in (g), indicate the Qc and QIC wave vectors, with their length scales synchronized to the background plot in Å−1. All ARPES measurements were conducted using an incident photon energy of 65 eV.

Full size image

Electronic structure analysis

The arguments above led us to explore the Fermi surface topology of CeTi3Bi4, as the J(Q) profile shaped by RKKY interactions, which determines the magnetic ground state, should be encoded in the Fermiology. Thus, we analyzed the electronic structure close to EF using ARPES and DFT (Fig. 4). For the DFT calculations, we utilized the result from LaTi3Bi4 as it allows for investigating the uncorrelated electronic structure of LnTi3Bi4 near the Fermi level with significantly decreased complexity of the calculation due to the non-magnetic nature of La (see “Methods”). Additionally, we adopted pseudo-labels of high-symmetry Q points from a hexagonal framework (see Fig. 4a), as done in prior RTi3Bi4 studies30,32,37,38. See “Methods” for more explanations.

Both the DFT and ARPES measurements reveal that CeTi3Bi4 exhibits VHSs near the Fermi level, E = EF, at the M′ points–four of the six M points in a hexagonal framework that do not correspond to the orthorhombic Y point (=b*). Figure 4b, c shows the calculated and measured electronic structures near EF. Except for the α band forming a small pocket around the \(\bar{\Gamma }\) point, which likely has surface character (see Supplementary Note 7 and Fig. 13), our DFT band structure with a slight adjustment of EF to +0.043 eV provides a good description of the ARPES spectrum around EF. Some apparent discrepancies between the ARPES and DFT results below the Fermi energy (Fig. 4c) may stem from using LaTi3Bi4 in the calculations: the validity of this approximation becomes less reliable at lower energies, where the influence of Ce3+ electronic states gets more pronounced. Interestingly, strong spectral weight is observed at the \({\bar{{\rm{M}}}}^{{\prime} }\) point at the EF (Fig. 4c), where DFT reveals the presence of a VHS (red dashed box in Fig. 4b). A more careful analysis through the \(\bar{\Gamma }-{\bar{{\rm{M}}}}^{{\prime} }-\bar{\Gamma }\) (Fig. 4d) and \(\bar{{\rm{K}}}-{\bar{{\rm{M}}}}^{{\prime} }-\bar{{\rm{K}}}\) (Fig. 4e) cuts further confirms the saddle point nature of the band dispersion at \({\bar{{\rm{M}}}}^{{\prime} }\) near EF (see also Supplementary Fig. 12). A simple quadratic fit indicates the VHS is positioned at EF + 0.015 eV (Fig. 4d), very close to EF. Notably, the VHS manifests as a slightly extended line signal along the \(\bar{\Gamma }-{\bar{{\rm{M}}}}^{{\prime} }\) direction in the Fermi surface geometry (Figs. 4d, f), with the maximum intensity at the Q positions marked by orange triangles in Fig. 4d, slightly offset from \({\bar{{\rm{M}}}}^{{\prime} }\). This could be simply due to the slight upward shift of the VHS from EF (Fig. 4d), though a similar observation in NdTi3Bi4 was interpreted as higher-order VHS with quartic dispersion38. Regardless, this suggests not only a high DOS at \({\bar{{\rm{M}}}}^{{\prime} }\) but also around \({\bar{{\rm{M}}}}^{{\prime} }\) due to the proximity of the VHS.

Interestingly, the ordering wave vectors QIC and QC appear to correspond to the wave vectors connecting the high DOS at adjacent \({\bar{{\rm{M}}}}^{{\prime} }\) points (Fig. 4f, g), suggesting relevance between the magnetic order and a nesting instability between VHS in CeTi3Bi4. Due to the large DOS, VHS near EF can induce CDW or SDW instabilities, with their modulation period determined by a wave vector connecting these points7,8,9,12. In a perfect hexagonal structure, the VHSs at the six M points would result in three equivalent nesting vectors related by three-fold rotational symmetry (C3z), leading to the 2a × 2a type tri-directional density-wave modulation. However, in orthorhombic CeTi3Bi4, the VHS only appears at four \({\bar{{\rm{M}}}}^{{\prime} }\) points according to DFT (Fig. 4b), leaving a nesting vector aligned with the Γ–Y direction (// b*) as the only active one. Indeed, QC = (0, 1, 0) (|QC| ~ 0.61 Å−1) is consistent with this unique nesting vector, whereas QIC = (0, 0.94, 0) (|QIC| ~ 0.57 Å−1) is slightly shorter (Fig. 4a).

The extended high DOS along the \(\bar{\Gamma }-{\bar{{\rm{M}}}}^{{\prime} }\) direction, as suggested in Fig. 4d, may explain why QIC is as prominent as QC: a nesting path connecting the two orange triangles near \({\bar{{\rm{M}}}}^{{\prime} }\) could also be significant [orange (blue) arrows in Fig. 4f, g], which is marginally shorter than the ideal vector connecting the \({\bar{{\rm{M}}}}^{{\prime} }\) points (=QC). Notably, QIC nicely agrees with the separation vector of the two orange triangles. More generally, given the extended high DOS along the \(\bar{\Gamma }-{\bar{{\rm{M}}}}^{{\prime} }\) line (Fig. 4d), any wave vectors between (0, δ, 0) and (0, 1, 0), with δ greater than a certain threshold δ0 but less than 1, may connect Q positions near \({\bar{{\rm{M}}}}^{{\prime} }\) with high DOS (Fig. 4g). Such a subtle feature, determined by the fine band structure in the vicinity of \({\bar{{\rm{M}}}}^{{\prime} }\), may be responsible for the coexistence of QC and QIC (Fig. 3h) and the continuous variation of QIC by temperatures (Fig. 3d).

Overall, our comprehensive investigation using neutron diffraction, ARPES, and DFT conclusively confirms the incommensurate SDW modulation in the Ce3+ Jeff = 1/2 moments and suggests its relationship to the VHS of Ti-derived Kagome bands. The latter connection warrants more careful investigation, such as a temperature-dependent electronic structure study across TN and T2. Another important aspect is the impact of the Ce3+ characteristics—strong easy-axis magnetic anisotropy and quantum spin—on the observed ground state. The uniaxial magnetic anisotropy can promote the SDW by disfavoring alternative spin spiral orders without length modulation43 (see Supplementary Fig. 4), which could also arise from a nesting instability, as observed in NdAlSi44. However, if and how the quantum aspect of the Jeff = 1/2 ground state doublet impacts SDW formation remains unclear based on our current results. It may aid in the SDW modulation in that smaller spin size reduces the energy cost of modulating the ordered moments (which arises from Heisenberg interaction terms) but also exhibits stronger longitudinal quantum fluctuations. Further spectroscopic studies, which are sensitive to longitudinal spin fluctuations, would be illuminating.

Methods

Sample preparation

CeTi3Bi4 single crystals are grown through a bismuth self-flux. Elemental reagents of Ce (AMES), Ti (Alfa 99.9% powder), and Bi (Alfa 99.999% low-oxide shot) were combined at a 2:3:20 ratio into 2 mL Canfield crucibles fitted with a catch crucible and a porous frit45. The crucibles were sealed under approximately 0.7 atm of argon gas in fused silica ampoules. Each composition was heated to 1050 °C at a rate of 200 °C/hr. Samples were allowed to thermalize and homogenize at 1050 °C for 12–18 h before cooling to 500 °C at a rate of 2 °C/h. Excess bismuth was removed through centrifugation at 500 °C. Crystals are a lustrous silver with a hexagonal habit. The samples are mechanically soft and are easily scratched with a knife or wooden splint. They are layered in nature and readily exfoliate using adhesive tape. Samples with side lengths up to 1 cm are common. We note that samples are moderately stable in air and tolerate common solvents and adhesives (e.g., GE Varnish, isopropyl alcohol, toluene) well. However, the samples are not indefinitely stable and will degrade, tarnish, and spall if left in humid air for several days.

Bulk property measurements

Magnetization measurements of CeTi3Bi4 single crystals were performed in a 7 T Quantum Design Magnetic Property Measurement System (MPMS3) SQUID magnetometer in vibrating-sample magnetometry (VSM) mode. The crystallographic orientations were carefully determined prior to the measurements using Laue XRD30. Samples were mounted to quartz paddles using a small quantity of GE varnish or n-grease. All magnetization measurements were performed under zero-field-cooled conditions unless specified. Heat capacity measurements on CeTi3Bi4 single crystals between 300 K and 1.8 K were performed in a Quantum Design 9 T Dynacool Physical Property Measurement System (PPMS) equipped with the heat capacity option. Additional heat capacity measurements from 400 mK to 9 K utilized a 3He insert for the Quantum Design 14 T PPMS. LaTi3Bi4 was used as the reference nonmagnetic species.

Single-crystal neutron diffraction

Single-crystal neutron diffraction measurements were conducted using the ZEBRA thermal neutron diffractometer at the Swiss spallation neutron source (SINQ) and the triple-axis spectrometer HB-1A at the High Flux Isotope Reactor (HFIR). At ZEBRA, we mount the sample in a Joule-Thomson close-cycled cryostat and measured diffraction peaks using a four-circle geometry and λ = 2.305 Å neutrons produced via the (002) plane of pyrolytic graphite monochromator. At HB-1A, two CeTi3Bi4 crystals, oriented in the [H, 0, L] and [0, K, L] scattering plane respectively, were measured in a liquid helium cryostat using neutron wavelength of λ = 2.38 Å selected by a double-bounce pyrolitic-graphite (PG) monochromator system and a PG analyzer using PG (0 0 2).

We conducted single-crystal neutron diffraction under a magnetic field using the WAND2 diffractometer at HFIR and the thermal triple-axis spectrometer TAS-2 at the research reactor JRR-3 of the Japan Atomic Energy Agency (JAEA). At WAND2, the sample was aligned in the Mag-B vertical cryomagnet (0 < H < 4.5 T) and measured with the (0KL) plane aligned horizontally. We used an incident neutron wavelength of λ = 1.48 Å. For the TAS-2 experiment, the sample was loaded into a cryogen-free 10 T magnet to have the (0KL) horizontal scattering plane. The spectrometer was fixed to the elastic condition in a triple-axis mode with a wavelength of 2.36 Å. The collimation set of open-40’- 40’-40’ was employed together with a PG filter placed before the sample.

The rocking curves measured in all three diffraction experiments exhibit nearly resolution-limited Bragg peak linewidths (0.25°–0.35°), as shown in Supplementary Fig. 2.

Angle-resolved photoemission spectroscopy (ARPES)

Synchrotron-based ARPES measurements were performed at the Lawrence Berkeley National Laboratory’s Advanced Light Source (ALS) beamline 10.0.1.1 using a Scienta R4000 hemispherical electron analyzer. The ARPES spectra presented in the main text were measured using 65 eV of incident photon energy. The angular and energy resolutions were set to 0.2° and 15 meV, respectively. Measurements were conducted at 20 K under ultra-high vacuum conditions (~10−11 Torr). High-quality single crystals were cut and mounted on a copper post using silver epoxy paste, and ceramic posts were affixed to the samples. To prevent oxidation, all sample preparation took place in a glove box. After preparation, the samples were loaded into the main chamber, cooled, and pumped down for several hours before measurements.

As described in the main text, we used pseudo-labels of high-symmetry Q points from a hexagonal framework for the ARPES data analysis. Despite the orthorhombic symmetry, the in-plane distortion of the Ti kagome lattice is marginal, as indicated by the lattice parameter ratio \(\frac{b}{a}=\,1.741\simeq \sqrt{3}\) (see Supplementary Table 1). Thus, using high-symmetry Q points from a hexagonal framework remains effective in mapping the kagome band structure features onto the orthorhombic reciprocal space of CeTi3Bi4. The pseudo-hexagonal nature of the material is especially relevant to ARPES since the states at EF are derived predominantly from the nearly hexagonal kagome network (the orthorhombic character is introduced by the Ce3+ chain configuration, see Fig. 1a). As such, we use Q labels projected onto the quasi-hexagonal 2D reciprocal space (denoted with an overline, e.g., \(\bar{{\rm{M}}}\) in Fig. 4a) for the ARPES results.

Density functional theory (DFT) calculations

To understand the purely electronic features of the CeTi3Bi4 system, we examine the DFT calculated band structure for LaTi3Bi4. Utilizing the La-congener instead of CeTi3Bi4 significantly decreases the complexity originating from the partially filled Ce 4f shell, but yields qualitatively identical results around the Fermi energy as the Ln 4f orbitals are far from it. Indeed, consistent with many smaller kagome compounds (e.g., CsV3Sb546 and ScV6Sn647), the electronic structure of LnTi3Bi4 near EF is almost entirely dominated by the Ti kagome orbitals.

The band structure of LaTi3Bi4 has been calculated within DFT based approach48. The electronic exchange-correlations were described using generalized gradient approximation and the Perdew–Burke–Ernzerhof49 parametrization. The calculations were executed using the plane-wave basis projector augmented-wave approach50 as implemented in the Vienna Ab-initio Simulation Package (VASP)51,52. A plane-wave energy cutoff of 450 eV has been used. A 12 × 12 × 12 k-points mesh was used to execute self-consistency calculations, while the Fermi surface was calculated with a 20 × 20 × 20 k-points mesh. The original structure atomic positions were optimized until the energy change does not exceed 0.05 meV and forces 0.5 meV/\(\mathring{\rm A}\). The.cif file of the optimized crystal structure is provided in Supplementary Data 1. To build the Fermi surface, the modular code xsfconverter [https://github.com/jenskunstmann/xsfconvert] together with FermiSurfer53 and XCrySDen54 graphical packages.