Ferromagnetism and correlated insulating states in monolayer Mo₃₃Te₅₆

Abstract

Although the kagome model is fundamentally two-dimensional, the essential kagome physics, i.e., the kagome-bands-driven emergent electronic states, has yet to be explored in the monolayer limit. Here, we present the experimental realization of kagome physics in monolayer Mo₃₃Te₅₆, showcasing both ferromagnetic ordering and a correlated insulating state with an energy gap of up to 15 meV. Using a combination of scanning tunnelling microscopy and theoretical calculations, we find a structural phase of the monolayer Mo-Te compound, which forms a mirror-twin boundary loop superlattice exhibiting kagome geometry and multiple sets of kagome bands. The partial occupancy of these nearly flat bands results in Fermi surface instability, counteracted by the emergence of ferromagnetic order (with a coercive field ~0.1 T, as observed by spin-polarized STM) and the opening of a correlated hard gap. Our work establishes a robust framework featuring well-defined atomic and band structures, alongside the intrinsic two-dimensional nature, essential for the rigorous examination of kagome physics.

Introduction

Suppressing electron kinetic energy in solids by reducing band dispersion, ultimately forming flat bands (FBs), is a promising route to enhance many-body electron correlation effects¹. Unlike the mundane cases attributed to atomic orbital localizations, wavefunction overlapping and electron hopping processes are preserved in kagome flat bands², where the suppression of kinetic energy arises from destructive interference^2,3. Experimentally, several transition-metal compounds, such as Fe₃Sn₂, Co₃Sn₂S₂, and YMn₆Sn₆, are recognized as kagome magnets, given that transition metal atoms form corner-shared triangular sublattices^4,5,6,7,8,9. However, the non-negligible hopping along the direction out of the kagome plane complicates the electronic bands in these bulk materials, obscuring the distinct characteristics of FBs^8,9,10.

Given the two-dimensional (2D) nature of kagome physics and the enhanced e–e interactions due to reduced dimensionality, atomically thin van der Waals (vdW) materials that mitigate interlayer couplings offer an ideal platform for nearly pristine manifestations of flat-band physics. In artificially constructed multilayers like twisted graphene and transition metal dichalcogenides, nanometre-scale moiré supercells fold the original band structure to generate flat bands at or tuned to the Fermi level (E_F)^{11,12,13,14,15}, resulting in correlated insulating states^15,16,17, magnetism^17,18, and strong-coupling superconductivity^19,20. However, due to the large size of the moiré supercell in real space, the carrier densities in those moiré samples are usually low². In addition, the existence of flat bands and the corresponding quantum phases in twisted moiré layers have strict stacking-angle requirements, posing challenges to attain practical samples at the meso/macroscale, which is essential for future technological applications. A recent high-throughput calculation has sought to address these issues by predicting a large library of stoichiometric candidates². Experimentally, a colouring-triangular lattice has recently been visualized in a Mo₅Te₈ monolayer²¹, where two sets of kagome bands have been found, located tens of meV away from the Fermi level. Despite these efforts^10,22,23, the core characteristic of the kagome lattice, namely the correlated phenomena induced by kagome bands near the Fermi level, has yet to be realized in the single-atomic-layer limit.

In this study, we demonstrate the emergence of kagome-related electronic states, including itinerant ferromagnetism and correlated insulating behaviour, in a monolayer Mo₃₃Te₅₆ system. This previously unreported stoichiometric phase derives from a MoTe₂ monolayer and features a thermodynamically stable mirror twin boundary (MTB) loop superlattice. Employing scanning tunnelling microscopy (STM), noncontact atomic force microscopy (nc-AFM), and theoretical calculations, we reveal that the MTB loop supercell forms an electronic kagome lattice. Two sets of kagome bands are mixed across the E_F, which results in partially filled bands sitting at the E_F, where a hard gap of 15 meV was experimentally measured at 0.35 K. This gap was elucidated to be a correlated one by temperature-dependent spectroscopic measurements. Advanced spin-polarized STM measurements reveal distinct ferromagnetic hysteresis with an effective coercive field of 0.1 T at 0.35 K. Further analysis of the spin density distribution and Stoner criteria calculations supports that the spontaneous spin-polarization is primarily driven by the high density of states at the E_F, contributed by these partially filled kagome bands. This work illustrates a viable pathway to realize exotic flat-band-driven quantum phases in van der Waals monolayers through targeted structural engineering.

Using molecular beam epitaxy (MBE) technique, we first synthesized H-phase monolayer MoTe₂ on bilayer graphene (BLG)/SiC (0001) [see Methods]. Following synthesis, the sample underwent annealing at 450 °C, which induced the formation of a dominantly ordered superstructure phase within the monolayer (Fig. 1a and Supplementary Fig. 1)²⁴. The height difference between the Mo₃₃Te₅₆ phase and the underlying BLG substrate is approximately 0.81 nm, which is very similar to that of the ML-MoTe₂ phase, as detailed in Supplementary Fig. 2. Magnified images of this superstructure (Fig. 1b, c) show a wagon-wheel-like morphology, characterized by six lines (spokes of the wagon-wheel) encircled in a skewed hexagonal pattern. This superlattice exhibits a periodicity of 19.1 Å (red dashed lattices in Fig. 1b, c), with the spoke lines dividing the atomic layer into two types of triangular domains (as elucidated later), marked by green- and orange-dashed triangles. At a high sample bias (2.0 V; Fig. 1b), the line boundaries appear as bright stripes, while at a low sample bias (0.8 V; Fig. 1c), they manifest as dark trenches. These morphologic characteristics are consistent with the STM observations of MTBs-arranged wagon-wheel patterns in H-phase TMDs^{25,26,27,28,29}. The possibility of this pattern being a moiré superlattice³⁰, was also excluded, as it shows no dependence on the twist angle relative to the BLG substrate (Supplementary Fig. 2).

Fig. 1: The atomic structure and tunnelling differential conductance spectrum of the ML-Mo₃₃Te₅₆.

a Large-scale STM topographic image of the sample hosting the dominant portion of the MTB loop superlattice; the corresponding line profile is taken along the blue line. b, c Zoomed-in STM images of the marked region in (a). The supercell is labelled a red rhombus with a lattice constant of 1.91 nm. d Atomically resolved nc-AFM images acquired in repulsive regime (details in Supplementary Fig. 3). Only the top-layer Te atoms are shown. e The DFT simulation of the frequency shift image [“Methods”]. f Atomic model of the MTB loop superlattice. Grey and blue balls represent Te and Mo atoms, respectively. The boundary-formed triangular loops, TD_E, TD_U, and centre sites are marked as shown. g Experimental dI/dV spectra acquired on pristine H-MoTe₂ (light-blue) and a typical spectrum of the Mo₃₃Te₅₆ (blue). Five pronounced peaks are labelled P₋₁, P₀, P₁, P_2, and P₃. h Upper panel: dI/dV spectra taken at four high-symmetry sites (TD_E/U, centre, and boundary) and their average spectra are shown in the middle panel. Here, the dI/dV spectra are normalized by the total conductance (I/V) to mitigate the influence of bias-dependent tunnelling probability on the spectral weight. The calculated DOS with spin-polarization and SOC (corresponding to Supplementary Fig. 7b) is shown in the bottom panel. Scanning parameters: a bias voltage V_bias = 1.0 V, tunnelling current I_t = 10 pA, b 2.0 V, 50 pA, c 0.8 V, 50 pA. The stabilization parameters in g are V_bias = 1.0 V, I_t = 100 pA, and lock-in modulation V_mod = 8 mV; in h are V_bias = 0.7 V, I_t = 200 pA, V_mod = 6 mV.

Frequency-shift nc-AFM image (Fig. 1d, acquired in the repulsive regime), primarily capturing surface Te atoms^26,31, provides further information on the atomic structures. The triangular domains (green and orange) match those in the STM images (Fig. 1b, c), showing the same wagon-wheel morphology^{25,26,27,28,29}. There are discernible contrasts between the Te atoms at the boundary and those in the domain regions, which can be attributed to differences in their relative heights and/or local electron densities^26,31. Each triangular domain consists of six surface Te atoms (i.e., three Mo–Te units) with an interatomic distance of 3.65 Å, which is nearly identical to that in pristine H-MoTe₂²⁸, and there is one Te atom at the rotational centre. Based on these observations, we propose an atomic structure for the observed superlattice, as schematically shown in Fig. 1f³². The simulated frequency-shift image (Fig. 1e and the Methods section) based on this model shows good agreement with the experimental morphology. The validity of this model is further verified by its lowest formation energy among various candidate models in a specific range of the Te chemical potential (Supplementary Fig. 4). In this structure, Te atoms on the MTB form closed triangular loops (marked by red dashed triangles in Fig. 1f), previously referred to as MTB loops³³. By comparing experimental and theoretical bias-dependent dI/dV maps (Fig. 2 and Supplementary Fig. 5), we identify the green triangle domains in the STM and AFM images as enclosed by individual MTB loops, denoted as TD_E. The unenclosed orange triangular domains are referred to as TD_U. The corner joint region of three MTB loops is denoted as the centre site, highlighted by a black-dashed triangle.

Fig. 2: Characterizations of the kagome geometry: DFT analysis, tight binding simulation, and experimental mapping.

a Calculated band structures of the MTB supercell are plotted along high-symmetry momentum paths in the Brillouin zone near the E_F. The wine-red and blue solid lines mark states that are odd (KBs1) and even (KBs2), respectively, under mirror operation $\hat{M}$. The irreps of states at Γ and K momenta are labelled for each band in (a) (Supplementary Table 1). b is the corresponding results with spin polarization and SOC. The solid and dashed lines mark two spin components that are spin-1 and spin-2, respectively. c–e Simulated (upper panel) and experimental (lower panel) dI/dV maps for P₋₁, P₀, and P₁. Results for other peaks and simulation details refer to Supplementary Fig. 5. The white dashed lines in c serve as a visual guide, illustrating the kagome pattern. The boundary loop is depicted by the pink shadows in c. The green and orange triangles in c–e represent the TD_E and TD_U regions, respectively. The red rhombus in c–e indicates a supercell. The scanning parameters: c, V_bias = −150 mV, I_t = 200 pA, V_mod = 2 mV; d V_bias = -35 mV, I_t = 200 pA, V_mod = 1 mV; e V_bias = +190 mV, I_t = 100 pA, V_mod = 4 mV.

Uniformity was not observed in previous MTB-arranged wagon-wheel structures, leading to the belief that they were line defects^{25,26,27,28,29}. In contrast, the present pattern exhibits a high degree of uniformity across the extensively characterized regions of the sample. This uniformity leads to the formation of a new stoichiometric Mo–Te compound, akin to the recently discovered Mo₅Te₈ phase^21,34. Based on our proposed atomic model, the stoichiometry is determined to be Mo₃₃Te₅₆ (i.e., MoTe_2-x with x = 10/33), more discussion see Supplementary Fig. 4. Figure 1g plots the differential conductance spectra (dI/dV) of both Mo₃₃Te₅₆ and H-MoTe₂ monolayers for comparison. Differing from the large band gap observed in H-MoTe₂²⁸, the Mo₃₃Te₅₆ spectrum exhibits at least five pronounced peaks, labelled P₋₁, P₀, P₁, P₂, and P₃ in ascending order of energy. Notably, peak P₀ is centred around the E_F along with a zero-bias dip feature. The spatially dependent dI/dV spectra of four representative sites (upper panel of Fig. 1h) indicate that these primary spectroscopic features are global properties of the system, rather than being attributed to local defects as observed in previous studies on MTBs^25,29. This distinction also sets our system apart from the honeycomb domain-wall network proposed in the 1T-TaS₂³⁵ (Supplementary Note 1). In Supplementary Fig. 6, we thoroughly examine the transition of local boundary states into delocalized states as the interboundary separation decreases.

Our density-functional theory (DFT) calculations reveal the spin-non-polarized and spin-polarized (with spin-orbit coupling, SOC) band structures of the Mo₃₃Te₅₆ monolayer (Fig. 2a, b, and Supplementary Fig. 7b), in which an on-site Coulomb energy U = 0.5 eV (see detailed discussion on the choice of the U value in Supplementary Fig. 7) was considered. To establish the resemblance between our DFT results and STS measurements, we plot the density of states (DOS) profile extracted from the spin-polarized (with SOC) band structures (Supplementary Fig. 5) in the bottom panel of Fig. 1h, which yields a reasonably good agreement with the averaged experimental dI/dV curve (middle panel of Fig. 1h). Specifically, the theoretical DOS reproduces the spectroscopic peaks P₋₁ to P₃ and captures the dip/gap features observed between P₁ and P₀, and below P₋₁. The spatial distributions of P₋₁–P₃ states also show a notable consistency between our DFT results and experimental measurements (Fig. 2c–e and Supplementary Fig. 5).

These sharp maxima in the DOS profile correspond to a cascade of narrow-width bands, indicative of FBs. To elucidate their physical origins, we adopt a simplified approach by examining a spin-non-polarized band structure focused around P_-1 and P₀ (Fig. 2a), which provides an illustrative understanding of the overall physical scenario. In the spin-degenerate case, six bands near the E_F with a relatively simple structure are labelled B1–B6 in descending order of energy. Based on the symmetry analysis, we identify the origin of these narrow-width bands as two sets of kagome bands, as follows. The MTB loop lattice has the P$\bar{6}$ (No. 174) space group and the C_3h point group, which includes a threefold rotation ${\hat{C}}_{3z}$ around the out-of-plane $\hat{z}$ axis and a mirror symmetry $\hat{M}$ by the ${xy}$ plane. Bands B3, B4, and B5, referred to as kagome band set 1 (KBs1, in wine red), are odd under the $\hat{M}$ operation (see details in Supplementary Table 1), while bands B1, B2, and B6, referred to as kagome band set 2 (KBs2, in blue), are even. Bands B3, B4, and B5 (KBs1) exhibit prototypical energy dispersions of a kagome lattice without inversion symmetry, forming a breathing kagome lattice. Band B5 is nearly flat (with a width of 0.4 meV) and touches the B4 band at the $\Gamma$ point (Brillouin zone centre), and bands B3 and B4 are separated by an energy gap at the ${{{\rm{K}}}}$ point (Brillouin zone corner).

The formation of an effective breathing kagome lattice by these three bands can be theoretically established using the band representation theory^36,37. The calculated irreducible representations (irreps) of these three bands are ${\Gamma }_{2}+{\Gamma }_{4}+{\Gamma }_{6}$ and ${{{{\rm{K}}}}}_{2}+{{{{\rm{K}}}}}_{4}+{{{{\rm{K}}}}}_{6}$ at the $\Gamma$ and ${{{\rm{K}}}}$ points, respectively. These symmetry analysis results match well with those for bands formed by Wannier orbitals placed at the $3j$ Wyckoff positions, which correspond to three inequivalent sites per unit cell related by threefold rotation³⁸. Geometrically, the $3j$ Wyckoff positions indeed form an effective kagome lattice. In real space, the effective breathing kagome lattice is further verified. The P₋₁ state, whether spin-degenerate or spin-polarized, is exclusively contributed by KBs1 (B4&5), while the P₀ state is mainly contributed by bands B1–B3. The wave-function norm of P₋₁ states distributed around the MTBs (Fig. 2c). The centres of the MTBs naturally form a kagome lattice akin to the line graph of a honeycomb lattice³⁹ (outlined by the white dashed lines; for a more detailed discussion, see Supplementary Note 2). Using this schema, our tight-binding (TB) calculations, with fitted hopping parameters and on-site energies, well reproduce the electronic band structures obtained from our DFT calculations (Fig. 2a). Detailed methodology of the TB simulations is documented in Supplementary Fig. 8 and Supplementary Note 2.

The spin-degenerate band structures (Fig. 2a) reveal that the Fermi level intersects bands B1, B2, and B3, leading to partial fillings of these bands with a total filling factor of 2/3 in the free-standing monolayer. These bands host 4 electrons with bands B1 and B2 occupied by approximately 2.5 electrons and band B3 by approximately 1.5 electrons per unit cell. The partial filling results in a high density of states at the E_F [$D\left({E}_{{{{\rm{F}}}}}\right)=5.7$] destabilizing the Fermi surface. The Stoner criterion (ST), calculated as S$T=I{{{\rm{\cdot }}}}\rho \left({E}_{{{{\rm{F}}}}}\right)$ with the Stoner parameter I = 0.49, was determined to a value of 2.79, suggesting a spontaneously itinerant FM state in monolayer Mo₃₃Te₅₆. Although the I value is moderate compared to those of other itinerant magnets like Fe₃GeTe₂ (FGT) where I_FGT = 0.72⁴⁰, the significantly larger DOS compared to FGT (1.5) and FeTe (4.9)⁴¹ is the primary reason for the large ST value. This suggested FM state is corroborated by our DFT results, showing that the FM state is more stable than the non-magnetic state by 11.2 meV/supercell (with SOC) or 24.5 meV/supercell (without SOC). The spin-exchange interaction and SOC cause a spin splitting of approximately 0.1 eV, as observed in B6 at the Γ point (Fig. 2b) and detailed in Supplementary Fig. 9.

To confirm the itinerant FM properties of monolayer Mo₃₃Te₅₆, we performed spin-polarized STM (SP-STM) measurements with an out-of-plane magnetic field (see Fig. 3a). Antiferromagnetic Cr-coated W tips, with stable tip apex magnetization under varying external magnetic field, were utilized (tip calibrations and other technical details are discussed in Supplementary Fig. 10 and the Methods section)^42,43. Figure 3b presents the averaged normalized tunnelling conductance recorded on the four representative sites of an ML-Mo₃₃Te₅₆ sample under fields B = ± 0.5 T. The spectroscopic intensities varied noticeably across the energy range due to the magnetization flip of the sample moment under the reversal of the external magnetic field. The magnetic asymmetry A_magn is a widely used quantity to represent such intensity variation, thus the spin-polarization of samples⁴⁴. We extracted A_magn as a function of energy by dividing the subtraction of energy-dependent conductance values under positive and negative fields by the sum of them [Methods], as shown in Fig. 3c⁴⁴. This magnetic asymmetry is proportional to the difference between the spin-up and -down DOS, D_sp(E) = D_up(E) – D_down(E), which could be derived from the spin-polarized band structures (Supplementary Fig. 9c). The appearance of D_sp(E) reproduces major features of the experimental A_magn spectrum. Additionally, the pre-calibrate tip results indicate that the tip magnetization is along the +z direction (Supplementary Fig. 10). This orientation identification correlates the experimental and theoretical (spin-up) majority spins predominately exhibited in the P₋₁ state, whereas the P₀ and P₁ states are polarized non-negligibly in the minority spin.

Fig. 3: Ferromagnetism in the monolayer Mo₃₃Te₅₆.

a Schematic of SP-STM measurement on ML-Mo₃₃Te₅₆ using a Cr-coated W tip with a perpendicular magnetic field. The grey and blue arrows represent the magnetization directions of the tip and sample, respectively. b The spin-polarized tunnelling spectra under ±0.5 T magnetic fields. Here are displayed the average curves obtained from four sites (Supplementary Fig. 12). The detailed data processing steps can be found in Supplementary Fig. 13. c Magnetic asymmetry (A_magn) calculated from the experimental curves in b (upper panel) and the theoretical spin DOS (lower panel). d Tunnelling spectra zoomed-in on the P₋₁ state at B = ± 0.15 T and 0 T, with forward/backward magnetic field sweeps in the left/right panel. e dI/dV signals of the P₋₁ state as a function of the magnetic field at 0.35 K (upper panel) and 4.3 K (lower panel). The FM hysteresis loops are outlined with rectangular solid lines. Red and blue points/curves represent forward and backward sweeps. Insets in b and e illustrate the magnetization configurations between the tip and the sample. f DFT-calculated spin density plot, where the pink (blue) contours denote the spin-up (spin-down). Yellow and blue circles highlight Mo atoms with the largest and second-largest magnetic moments, respectively. Stabilization parameters: b V_bias = 700 mV, I_t = 200 pA, and V_mod = 6 mV; d, e V_bias = 400 mV, I_t = 200 pA, and V_mod = 4 mV. The experimental data presented in (b–d) are normalized by the total conductance (I/V).

With B continuously swept within −0.2 T to 0.2 T in 0.02 T increments, we conducted further spectroscopic measurements focusing on P_-1 due to its substantial A_magn magnitude (Fig. 3c). Data on other peaks are in the Supporting Information (Supplementary Fig. 11, 12). Selective spectra focusing on P₋₁ under B = 0, ±0.15 T are shown in Fig. 3d. The conductance value is plotted as a function of B in Fig. 3e. The magnetization curve is slightly shifted due to the residual stray field of the tip⁴³. A rectangular hysteresis loop is observable at either 0.35 or 4.3 K. The coercive field is approximately 0.10 T at 0.35 K (upper panel) and 0.06 T at 4.3 K (lower panel). The FM hysteresis behaviour was reliably reproduced using multiple spin-polarized tips, as presented in Supplementary Fig. 14. Moreover, we further reveal orbital diamagnetism coexisting with spin ferromagnetism through observing the magnetization-polarized-Zeeman effect for the P₁ state (Supplementary Fig. 15), akin to those found in 3D kagome magnets^6,7,45.

A spatial plot of the spin density for the FM state (Fig. 3f) indicates that spin-polarized electrons, contributing to a total magnetization of 1.66 μ_B, are distributed throughout the monolayer, mainly on Mo atoms around the MTBs and shared corners. The Mo atoms highlighted by yellow circles have the largest local magnetic moment of 0.20 μ_B, those by blue circles have a reduced moment of 0.13 μ_B, while other Mo sites have even smaller moments (<0.10 μ_B). These local magnetic moments are an order of magnitude smaller than those in typical classical magnets. Experimental observations align with this finding, showing delocalized magnetic moments, as evidenced by a similar appearance in the A_magn spectrum across the entire primitive cell (Supplementary Fig. 12). The quantitative differences observed at various sites are consistent with the spin density distribution illustrated in Fig. 3f. This behaviour differentiates flat bands with a destructive interference origin from those with suppressed wavefunction overlap, such as the Mott-like flat band in the 1T-TaSe₂ CDW monolayer, which is subject to a local moment picture^46,47 (Supplementary Fig. 12).

Alongside the itinerant ferromagnetism, another intriguing feature is the puzzling dip at the E_F. Figure 4a presents high-resolution tunnelling spectra within a zoomed-in energy range (±100 meV) taken at 0.35 K. A hard gap is clearly observed, with Δ₀ for the zero-conductance region determined as 15.23 ± 1.86 meV by statistical analysis (Supplementary Fig. 16). This gap-width magnitude persists across different locations, although minor differences in the line shapes occur. To identify the origin of this hard gap, we measured its evolution with temperature. As shown in Fig. 4b, the hard gap continuously narrows with increasing temperature and closes above 12 K. However, this gap closing cannot be explained solely by thermal broadening. In Fig. 4c, the 0.35 K spectrum (black curve) was used to simulate the thermal broadening at T = 20 K (red curve; see Supplementary Note 3 for details). The simulated 20 K data significantly deviates from the measured 20 K spectrum (grey curve). To quantify the gap size over a wide temperature range, we define Δ_Ν by the midpoints of the edge slopes. Figure 4d shows the simulated thermal broadening of Δ_Ν and the experimentally measured values as a function of temperature, where the experimental results exhibit a more rapid decay. This spectroscopic behaviour indicates that the observed gap at E_F is due to electron correlation effects^48,49, resulting in a correlated insulating state rather than a single-particle band insulator, as reported in monolayer Mo₅Te₈²¹.

Fig. 4: Experimental evidence for the correlated gap at the Fermi level.

a High-resolution dI/dV spectra recorded at the TD_E/U and centre sites at T = 0.35 K, showing a relatively uniform hard energy gap. Δ₀ denotes the size of the zero-conductance region (Supplementary Fig. 18). b Temperature-dependent dI/dV spectra (centre site) from 0.35 K – 77 K, plotted on a logarithmic scale. c Selective spectra taken at 0.35 K (black curve) and 20 K (grey curve). The red curve is a thermal broadening simulation (T = 20 K) of the 0.35 K spectrum. d Experimental Δ_Ν (black spots) at different sample temperatures. The grey line is a visual guide, and the red line represents the simulated Δ_Ν (T) from thermal broadening of the experimental 0.35 K spectrum (Supplementary Note 3). Δ_N is the gap defined by the midpoints of the edge slopes, i.e., the maximum point in the |d²I/dV²| spectra in (c). Stabilization parameters: V_bias = 100 mV, I_t = 200 pA, and V_mod = 2 mV.

In Supplementary Note 4, we elaborate that the observed gap is not a Coulomb gap, typically caused by a repulsive action during the electron tunnelling process. This assertion is based on the spectroscopic line shape (i.e., a hard gap) and the independence of gap size on the flake area (see Supplementary Fig. 16 and the detailed discussion on the Coulomb effect in Supplementary Note 4). A CDW bandgap can also be ruled out, as supplementary experimental results show no charge modulation in the bias-dependent STM imaging (Supplementary Fig. 17). Furthermore, the possibility of a superconducting gap is excluded, as the gap remains unsuppressed by applying magnetic fields up to 12 T (Supplementary Fig. 18). While these common scenarios have been ruled out, pinpointing a precise model to describe the states of matter arising from the partially filled flat bands with specific filling factors and interaction parameters remains challenging^50,51. This complexity is further exacerbated by the large unit cell and complex orbital composition in the present monolayer, necessitating substantial additional research efforts that integrate multiple experimental techniques^15,17 beyond STM and more advanced theoretical methods than DFT + U. Such investigation will be pursued in follow-up work, as it is beyond the scope of the present study that primarily introduces a new single-layer crystalline structure with FB-driven correlated physics.

In conclusion, we have demonstrated a monolayer kagome system exhibiting FB-driven correlated magnetism in Mo₃₃Te₅₆ through a precisely engineered MTB loop superlattice. In this 2D limit, the well-defined kagome bands emerge because the removal of interlayer hopping effects present in 3D cases minimizes the mixing across both energy and momentum scales. Within certain partially filled bands, strong electron correlations result in gap openings up to 15 meV. Spin-polarized STS measurements directly demonstrate the presence of spontaneous spin polarization and the formation of ferromagnetic order. Future studies are expected to explore the doping dependence of the correlated insulator by tuning the carrier density. This work introduces a strategy for designing monolayer vdW materials with pronounced correlation effects, unlocking versatile opportunities for engineering nontrivial emergent quantum phenomena.

Methods

Sample preparation

In this study, samples were grown on bilayer graphene formed by heating SiC (0001) in a home-built MBE system with a base pressure of ~ 1.2 × 10⁻¹⁰ Torr. Te (99.999%) was evaporated from Knudsen cells, and High-purity Mo (99.95%) was evaporated from an e-beam evaporator, respectively. The flux ratios of Mo/Te were 1:30. To obtain the ordered MTB superstructure, one follows the following procedure. The substrate was kept at ~250 °C during the deposition. After deposition, the sample was annealed for 10 min at the growth temperature with the Te flux maintained. Subsequently, the substrate temperature was increased to 450 °C and annealed for 1 h (with the Te flux off).

STM/ Q-plus AFM measurements

The STM/STS measurements were performed by a commercial Unisoku 1300 system (base pressure <1 × 10⁻¹⁰ Torr). Electrochemically etched W-tips were used in all measurements, which are calibrated spectroscopically against the Shockley surface states of cleaned Cu (111) surfaces before performing measurements on MoTe₂. STM topographic images are acquired in constant-current mode. The dI/dV spectra are measured using a standard lock-in amplifier with modulation at a frequency of 932 Hz; other parameters are specified in figure captions. All STM/S measurements were taken at 4.3 K unless otherwise specified.

The nc-AFM measurements were performed using a standard W tip. The nc-AFM imaging is performed by the frequency shift (Δf) of the qPlus resonator in constant-height mode with an oscillation amplitude of 100 pm. The resonance frequency of the AFM probe is f₀ = 29.10 kHz, and its quality factor Q is 57962.

For spin-polarized STM measurements, a W tip was coated with ~20 atomic layers of Cr (purity: 99.995%) after being flashed to 2200 K to remove oxides. The tip was then annealed at ~600 K for 10 min to recrystallize the Cr coating, stabilizing the structural phase at the tip apex. This process generated an antiferromagnetic tip with an out-of-plane net magnetic moment and high coercivity, making it stable against external magnetic fields^42,43. The spin-polarized properties of the Cr-coated W tips were verified using Co islands on Cu (111) following standard calibration procedures (Supplementary Fig. 10).

The magnetic asymmetry (A_magn) is calculated using the formula: A_magn = [(dI/dV)_+B – (dI/dV)_-B)]/[(dI/dV)_+B + (dI/dV)_-B]⁴⁴, where (dI/dV)_+B and (dI/dV)_-B correspond to the conductance for +B = +0.5 T and −B = −0.5 T, respectively, as shown in Fig. 3b.

DFT calculation

The theoretical calculations were performed using the Vienna Ab initio Simulation Package (VASP) based on DFT⁵². The electron-ion potential and exchange-correlation functional were described by projected augmented wave (PAW) and generalized gradient approximation (GGA)^53,54, respectively. A dispersion correction was made at the van der Waals density functional (vdW-DF) level, with the optB86b functional for the exchange potential (optB86b-vdW)⁵⁵. On-site Coulomb interactions were considered on the Mo d orbitals with an effective value U = 0.5 eV. A kinetic energy cut-off of 500 eV was used for the plane-wave basis set for geometric and electronic structure calculations. A k-mesh of 5 × 5 × 1 was adopted to sample the first Brillouin zone of the considered √28 × √28 supercell. The vacuum region of 20 Å was used to avoid the periodic interaction. The lattice of the supercell was fully relaxed, and all atoms in the supercell were allowed to relax until the residual force per atom was less than 0.01 eV Å⁻¹ and the energy convergence criteria were 1 × 10⁻⁵ eV. The lattice constant of the supercell is 19.13 Å. The band decomposed charge density was calculated by reading the converged wave-function file. The spin DOS was calculated using ${N}_{{{{\rm{spin}}}}}\left(E\right)=\,{N}_{{{{\rm{spin}}}}-{{{\rm{up}}}}}\left(E\right)-{N}_{{{{\rm{spin}}}}-{{{\rm{down}}}}}\left(E\right)$. The irreps calculations were performed using the IRVSP programme in conjunction figure with VASP⁵⁶.

The DFT simulation of the frequency shift AFM image was obtained using the following approach⁵⁷: a 3D field of the frequency shift⁵⁸ was captured with a resolution of 20 × 20 × 1 data points in a box of 19.13 Å × 19.13 Å × 25 Å above the Mo₃₃Te₅₆ sample. All the calculation results presented above do not include the graphene substrate (Supplementary Fig. 7).