Synthesis of incommensurate moiré structures with short-range-ordered charge density modulation

Abstract

Moiré structures have attracted great interest for tuning emergent quantum phenomena and enabling novel device functionalities. For the twisted bilayer graphene or semiconducting transition-metal dichalcogenides, studies have been focused on ordered systems, atomically and electronically. However, disordered/short-range-ordered moiré electronic structures, likewise involving emergence of novel physical properties and functional applications, remain elusive. Here, we report an incommensurate moiré structure with short-range-ordered charge density modulation state, which is formed between monolayer metallic NiTe₂ and superconductor NbSe₂. This state features intra-moiré-cell irregular charge orders with spontaneous on-site ordering at the moiré scale, while breaking the crystalline symmetries at the atomic scale. The short-range charge order likely originates from enhanced electron correlations driven by the cooperative effect of moiré-confined strain and localization of electron density. Furthermore, with the layers of NiTe₂, we have successfully tuned the short-range state. Our findings provide a promising platform for exploring novel electronic states beyond conventional Bloch-based frameworks and potential applications in quantum devices.

Introduction

Moiré structures, formed by stacking two-dimensional (2D) van-der-Waals materials with a twist angle or lattice mismatch, have emerged as a powerful platform for engineering electronic structures and exploring novel quantum phenomena^{1,2,3,4,5,6,7,8}. These structures introduce long-wavelength periodic potentials into otherwise uniform atomic lattices, fundamentally reshaping the low-energy electronic landscape and enabling precise control over the band topology, bandwidth, and electron correlations^9,10. This leads to a plethora of correlated and topological phases, including unconventional superconductivity¹¹, Mott insulating states¹², stripe phases^13,14, nematicity¹⁵, quantized anomalous Hall effect^16,17, fractional Chern insulators^18,19,20, and Wigner crystals^21,22,23. The ability to design and manipulate these emergent states through structural engineering makes moiré systems a fertile ground for both fundamental research and potential quantum device applications.

To date, twisted bilayer graphene and semiconducting transition-metal dichalcogenides (TMDs) have been the primary focus of moiré research. In these systems, the low-energy electronic structures can be effectively described by continuum Hamiltonians that preserve translational symmetry at the moiré scale^24,25,26,27, giving rise to well-defined minibands that underpin our understanding of the emergent ordered electronic states. Great efforts have thus far centered on moiré systems with long-range atomic and electronic orders. In contrast, electronic states featuring disorder or short-range order fall beyond the scope of conventional Bloch descriptions, and are of significance for achieving novel quantum phenomena and enabling potential applications in functional devices^28,29,30. However, such moiré structures of disordered or short-range-ordered electronic states have never been reported, to the best of our knowledge.

Here, we synthesize an incommensurate moiré structure between two metallic TMDs, and observe a short-range-ordered charge density modulation state by using scanning tunneling microscopy/spectroscopy (STM/S). This state features short-range irregular charge orders inside each moiré cell with spontaneous on-site ordering across the moiré scale. The short-range charge orders exhibit a √3×√3 quasi-periodicity and reflect local symmetry breaking. The coexistence of global periodicity and intra-moiré-cell irregularity defines this distinct many-body state. Combined with density functional theory (DFT) calculations, the short-range charge order likely originates from the enhanced electron correlation driven by the cooperative effect of moiré-confined strain and localization of electron density at the AB_Hollow regions. Furthermore, by controlling the thickness of the NiTe₂ layer, the short-range-ordered state has been effectively tuned.

Results

NiTe₂ has recently been identified as a Type-II Dirac semimetal hosting Dirac fermions and topological surface states^32,33. It exhibits pronounced thickness-dependent electronic tunability³⁴ and has been theoretically predicted to exhibit superconducting instability in the single-layer limit³⁵. Despite these intriguing properties, no charge density wave (CDW) transition has been experimentally observed in the NiTe₂ system to date. We start with constructing the moiré structure by direct epitaxial growth of metallic monolayer (1 L) NiTe₂ film on a superconducting NbSe₂ substrate (Fig. 1a), which enables the formation of a short-range-ordered charge density modulation state (Fig. 1b). Both the sharp diffraction spots of the low-energy electron diffraction (LEED) pattern (Fig. 1c) and large-scale STM image (Fig. 1d) indicate the ordered moiré superstructure of the NiTe₂/NbSe₂. Two sets of the LEED pattern of the NiTe₂ (red circles) and NbSe₂ (blue circles) reveal that monolayer NiTe₂ share the same orientation with NbSe₂, further indicating the formation of the moiré superstructure is due to lattice-mismatch. The measured in-plane lattice constants of monolayer NiTe₂ and NbSe₂ are approximately 3.69 Å and 3.46 Å, respectively. In the 1L-NiTe₂/NbSe₂ heterostructure, the moiré wavelength is measured to be about 3.78 nm. Thus, the resulting moiré structure is incommensurate. Long-range-ordered charge modulation at the moiré scale is observed with high-resolution STM (Fig. 1e), which features three inequivalent local regions within a moiré unit cell, i.e., the bright (AB_Hollow), dark (AA), and intermediate (AB_Se) regions, respectively (Supplementary Figs. S1, S2, see Methods). Remarkably, electronic structures at these three local regions exhibit strong intensity contrast, with the brightest AB_Hollow regions appearing as bubbles, suggesting a significant charge density modulation. The dI/dV spectra of the three regions in NiTe₂ further corroborate the charge density modulation (Supplementary Fig. S3). The dI/dV maps reveal distinct inversion of intensity contrast at +250 meV and −250 meV at the same spatial location (Supplementary Fig. S4), further confirming the large-periodic charge density modulation at the moiré scale^36,37.

Fig. 1: Synthesis of moiré structure between NiTe₂ and NbSe₂.

a Schematic showing the construction of a moiré structure by epitaxial growth of a monolayer metallic TMDs (NiTe₂) on a bulk metallic TMDs (NbSe₂) surface, illustrated by using VESTA³¹. b Formation of a distinct short-range-ordered charge density modulation state, featuring short-range irregular charge orders inside moiré unit cells with spontaneous on-site ordering across the moiré scale. c LEED pattern, showing the formation of large-scale moiré superstructure (yellow circles) with sharp diffraction spots from NiTe₂ (red circles) and NbSe₂ (blue circles). d A large-scale STM image (V_s = −2.0 V, I_t = 0.05 nA), showing a moiré pattern of the 1 L NiTe₂ on the NbSe₂. e A zoom-in STM image (V_s = −990 mV, I_t = 0.03 nA) taken at the black-box region in (d), showing a bubble-like large-periodic charge modulation at the moiré scale. The black rhombus marks a unit cell of the superstructure, highlighting the three inequivalent local regions AB_Hollow, AA and AB_Se inside the unit cells.

Next, we demonstrate the short-range charge orders inside each moiré unit cell. High-resolution STM images (Fig. 2a and Supplementary Fig. S5) reveal that the short-range charge orders emerge at each AB_Hollow regions of the moiré superstructure, with spontaneous on-site ordering across the moiré scale. The charge orders have a rough preference to localize on Te sites (Supplementary Fig. S6). In addition, the short-range charge orders are consistently observed across four NiTe₂/NbSe₂ samples using eight different STM tips, demonstrating their robustness and reproducibility (Supplementary Fig. S7). The variation in the number of charge orders within each AB_Hollow region arises from region-dependence across the sample, reflecting their intrinsic local inhomogeneity and short-range nature. The sharp spots of the fast Fourier transform (FFT) pattern (Fig. 2b) indicates the Bragg lattice of NiTe₂ and the ordered superstructure of NiTe₂/NbSe₂. The broad spots surrounding six hexagonal wavevectors at (√3×√3)R30° positions (Q_√3×√3) with respect to NiTe₂ Bragg lattice suggest the formation of short-range orders at approximately √3×√3 of the NiTe₂ unit cell. We further analyze the spatial ordering of the charge states across multiple monolayer NiTe₂ samples. Each bright state is marked as a center, from which we measure the spatial distance to its nearest neighbors and extract the angular deviation between the C₃ symmetry axes of adjacent states (inset in Fig. 2c, d). The short-range orders are irregular, varying between different AB_Hollow regions, as evident by the distribution statistics of rotation angle and periodicity (Fig. 2c, d). Thus, the overall charge density modulation has the character of long-range periodic modulation pinned by the moiré landscape, whereas the short-range orders within each moiré cell exhibit quasi-hexatic characteristics, with quasi-sixfold rotational symmetry and a quasi-periodicity of approximately √3 times of the NiTe₂ lattice constant.

Fig. 2: Demonstration of short-range charge orders inside moiré unit cells.

a An atomic-resolution STM image, showing the moiré superstructure, the atomic lattice of NiTe₂, and short-range charge orders within the AB_Hollow regions (V_s = −50 mV, I_t = 5 nA). b The symmetrized Fourier transform pattern, showing sharp diffraction spots from the NiTe₂ Bragg lattice (orange circles), the ordered moiré superstructure (green circles), and a set of broad spots (yellow arrows) around (√3×√3)R30° positions (blue arrow) associated with the short-range irregular orders (blue circles). Distribution statistics of the rotation angle φ (c) and distance l (d) of the short-range charge orders, revealing quasi-sixfold rotational symmetry and a quasi-periodicity of ~√3 times the NiTe₂ lattice constant. Insets define the rotation angle φ and distance l. Red solid circles in the insets represent the charge orders. The dI/dV maps (I_t = 1.0 nA, V_mod = 5 mV) at +100 meV (e) and −100 meV (f), respectively, showing an inversion in intensity contrast. Each map consists of 50 pixels × 50 pixels. The AB_Hollow regions and the internal short-range charge orders are highlighted by the red and yellow circles, respectively. Spatial distribution of the amplitude (g) and phase (h) corresponding to the short-range charge orders, showing that charge orders mainly occur within the AB_Hollow regions (white circles) with variation of the phases.

We further analyze the short-range charge order state inside the moiré unit cells. At the same spatial locations, the density of states (DOS) shows contrast inversion on opposite sides of E_F (+100 meV and −100 meV) (Fig. 2e, f and Supplementary Figs. S8, S9), with the same short-range √3×√3 orders. The spatial distribution of the amplitude and phase of the Q_√3×√3 reveals the short-range charge order within the AB_Hollow regions with phase variation (Fig. 2g, h). These results demonstrate the emergence of short-range √3×√3 charge orders inside the moiré unit cells, while it breaks the underlying crystalline symmetries and losses long-range phase coherence.

We next examine the possible origins of the observed short-range charge orders. Known defect configurations in NiTe₂, such as Ni interstitials and chalcogen substitutions, can exhibit C₃-symmetric orbital features, but they are generally random in distribution and lack layer dependence^38,39, or instead form long-range periodic structural reconstructions⁴⁰. In contrast, the localized states observed here are consistently confined within the AB_Hollow regions, forming a quasiperiodic pattern across extended areas. They are exclusive to monolayer NiTe₂ and disappear in thicker layers (as discussed below), in contrast to defect-induced features that typically persist regardless of film thickness. Moreover, we keep T-rich atmosphere during the growth, and the average density of the short-range charge order remains nearly unchanged with increasing Ni:Te flux ratio during growth (Supplementary Fig. S10), further ruling out Ni interstitials at the interface⁴⁰. These characteristics strongly suggest that the short-range charge modulation does not arise from atomic-scale defects, but rather reflects an alternative, electronically driven mechanism.

We also note that NbSe₂ hosts a 3×3 CDW below ~33 K. Although the short-range-ordered charge modulations in NiTe₂ exhibit local C₃ symmetry, they are distinct from the long-range 3×3 CDW of the underlying NbSe₂ substrate in terms of coherence length, orientation, and energy scale. Specifically, the NiTe₂ charge order is short-range and rotated by ~30° relative to the lattice, and persists above room temperature (Supplementary Fig. S11). These differences suggest only weak or negligible coupling between the two charge modulations.

To further study the possible mechanism of the short-range charge order within AB_Hollow regions, we analyzed the moiré-confined strain distribution^41,42, charge transfer, and electron density distribution in the 1 L NiTe₂ on NbSe₂ with DFT calculations. Although such calculations, performed within the Bloch framework, cannot fully capture the experimentally observed electronic states with broken translational symmetry, they provide useful insight into possible mechanisms for the emergence of short-range charge order. The charge order is absent in the pristine NiTe₂^32,33,34, and the phonon dispersion of monolayer NiTe₂ shows no phonon softening, indicating its dynamical stability (Supplementary Fig. S12). The on-site Coulomb interactions also show slight impact on the electronic band structure of the monolayer NiTe₂ (Supplementary Fig. S13). However, in the moiré structure of NiTe₂/NbSe₂, the monolayer NiTe₂ within AB_Hollow regions are predicted to experience significant compressive strain (about 3%, Fig. 3a). The strain field distribution in atomically-resolved STM image by using a computational framework of the atomic displacement analysis along both crystallographic axes (Method) estimate that about 3% compressive strains exist within AB_Hollow regions (Supplementary Fig. S14). In addition, the calculated charge distribution shows a strong localization of the electron density within AB_Hollow regions (Supplementary Fig. S15). While the calculation results based on a simplified model of a uniformly strained NiTe₂ monolayer cannot fully describe the emergent electronic states associated with short-range ordering (Fig. 2a), they nevertheless provide valuable insight into the possible driving forces that enhance electronic correlations. Although the system still remains dynamically stable under 3% strain (Supplementary Fig. S16), larger compressive strain can, in principle, induce electronic instabilities in monolayer NiTe₂. The band structure upon applying a 10% compressive strain (Fig. 3b) shows that the energy band around the K point anchored by the Dirac point is pulled closer to the Fermi level, leading to the formation of a small hole pocket located at K point precisely, and thereby enhancing its density of states. Moreover, the maximum electronic susceptibility is moved to K point, which indicates the nesting between K and K′ points, suggesting an enhanced electron correlation (Fig. 3c). As a result, the Fermi surface nesting vector gives rise to a √3×√3 charge order vector (Fig. 3d, e). The atomic structure model for the √3×√3 phase shows a lower symmetry (C_3v) compared to the high-symmetry of the pristine phase (D_3d) (Fig. 3f, g). Therefore, we speculate that the short-range charge order originates mainly from the enhanced electron correlations driven by the cooperative effect of moiré-confined strain and localization of electron density within the AB_Hollow regions.

Fig. 3: Formation mechanism of the short-range charge order.

a In-plane strain distribution in the monolayer NiTe₂ on NbSe₂ with DFT calculations, showing a large compressive strain in the AB_Hollow regions. b Band structures (left panel) of the pristine monolayer NiTe₂ (black) and 10% compressive-strained monolayer NiTe₂ (red), showing that the band around the K point anchored by the Dirac point is pulled down toward the Fermi level. Right panel: Projected density of states onto p orbital of Te and d orbital of Ni in the strained NiTe₂. c Calculated real part of the electronic susceptibility ${\chi }^{{\prime} }$(q) for the pristine (black) and 10% compressive-strained monolayer NiTe₂ (red), showing a movement of the maximum susceptibility from the Γ point to the K point after applying the strain. The nesting peak suggests an enhancement of electron correlations, corresponding to the emergent √3×√3 charge order (Q_CO = ΓK). Fermi surfaces of the pristine (d) and 10% compressive-strained monolayer NiTe₂ (e), showing formation of small hole pockets at the K point after applying the compressive strain (highlighted by black arrows in (e)). The Fermi surface nesting between the two hole-pockets at K and K′ points is attributed to the √3×√3 charge order in the AB_Hollow regions (Vector: Q_CO = ΓK). Atomic structure of the pristine monolayer NiTe₂ (f) and NiTe₂ with the √3×√3 charge order (g), showing the reduction of the symmetry from D_3d to C_3v. The arrows in (g) label an off-center shift of Ni atoms, forming distinct groups (shading triangles). Solid rhombuses label the unit cells.

Intriguingly, we show the tuning of the short-range charge orders by controlling the thickness of NiTe₂ (Supplementary Fig. S17). In comparison to the 1 L NiTe₂, in the bilayer (2 L), the moiré modulation remains while the √3×√3 short-range charge orders are suppressed (Fig. 4a). The density of states in equivalent local regions, e.g., AB_Hollow, exhibit evident inhomogeneity, suggesting that the electronic state still retains short-range nature. The dI/dV maps further reveal the distinct electronic states within different moiré regions and weaker charge modulation than that of the 1 L NiTe₂ at the moiré scale (Supplementary Fig. S18). The disappearance of the short-range charge order with increasing NiTe₂ layer is further supported by DFT calculations (Supplementary Fig. S19), which reveal substantially reduced strain and electron localization within the AB_Hollow regions, insufficient to induce electron correlations. As the thickness further increases to trilayer, the moiré modulation are further suppressed and eventually disappeared at the four-layer NiTe₂ (Fig. 4b, c). Furthermore, we construct a planar homojunction of 2L-NiTe₂/NbSe₂ and 1L-NiTe₂/NbSe₂ (Fig. 4d), with the interface defined by the sharp step edge of the top NiTe₂ layer. The corresponding dI/dV maps reveal long-range charge density modulation with inlaid short-range charge orders in the 1 L NiTe₂ region, whereas the 2 L exhibits much weaker modulation without charge orders (Fig. 4e and Supplementary Fig. S20), indicating the formation of a moiré junction with and without short-range charge order state. Notably, a pronounced charge edge state at about 45 meV emerges at the interface across the 1L-2L junction (Fig. 4f), but is absent at the 2L-3L junction (Supplementary Fig. S21).

Fig. 4: Tuning of the short-range-ordered charge density modulation state.

a Bilayer NiTe₂ on NbSe₂ (STM image, V_s = −10 mV, I_t = 5 nA), showing large-periodic charge modulation at the moiré scale without short-range charge order. b, c Atomically-resolved STM images of the trilayer (3 L) (V_s = −10 mV, I_t = 5 nA) and four-layer (4 L) NiTe₂ (V_s = −300 mV, I_t = 1 nA), respectively, showing that the moiré potential and short-range-ordered state are gradually suppressed with increasing layers. The lattice contrast inversion between the 3 L and 4 L topographies arises from the difference in sample bias. d Homojunction formation between 1 L and 2 L NiTe₂ (STM image, V_s = −100 mV, I_t = 80 pA), showing the boundary between the 1 L region with short-range √3×√3 charge order (yellow circles) and the 2 L region without short-range charge order. The corresponding dI/dV maps of (d) (I_t = 1.0 nA, V_mod = 1 mV) at 6 meV (e) and 45 meV (f), respectively, showing the difference of the electronic state in 1 L and 2 L regions and the formation of a charge edge state at the interface. Each map consists of 75 pixels × 75 pixels.

Finally, the short-range-ordered modulation state is also observed in the superconducting regime. The superconducting phase in NiTe₂ is induced by proximity effect (Fig. 5a, b). The coherence peak height, peak-to-peak distance, and consequently the superconducting gap size show strong modulation across the AB_Hollow regions (Fig. 5c). Within these regions, the coherence peak height is enhanced, while the superconducting gap size is reduced. This spatially-modulated pair density represents a secondary phase of the charge density modulation at the moiré scale, as a result of the correlation between the short-range charge order and the proximity-induced superconductivity. The periodicity of the pair density modulation is obviously variable and the dI/dV map at the energy of coherence peak shows inhomogeneous characteristics (Fig. 5d, e), suggesting a quasi-ordered pair density modulation and reinforcing the disordered nature of the electronic state in such moiré system.

Fig. 5: Short-range modulation in the superconducting state of NiTe₂.

a The spatially-averaged dI/dV spectra (V_s = −3 mV, I_t = 1 nA, V_mod = 0.05 meV) at NbSe₂ substrate, monolayer (1 L), bilayer (2 L), and trilayer (3 L) NiTe₂, respectively, showing broadening two coherence peaks and increasing in-gap conductance with the layer of NiTe₂. b The SC gap size subtracted from Dyen function fitting from dI/dV spectra as a function of the layer of NiTe₂ (bare NbSe₂, L = 0), showing a monotonic decrease with increasing layers, indicating the proximity effect. c The waterfall and intensity plot of the dI/dV linecut (V_s = −3 mV, I_t = 1 nA, V_mod = 0.05 meV) across AB_Hollow regions (dash circles) along the green arrow in the topography, showing strong modulation of superconducting coherence peak and gap size, indicating a pair density modulation. d, e STM topography of the monolayer NiTe₂ and corresponding dI/dV map at −0.64 meV (100 pixels × 100 pixels), showing inhomogeneous superconductivity and the enhancement of in-gap states in AB_Hollow regions (yellow dotted circles) (V_s = −2 mV, I_t = 1 nA, V_mod = 0.05 meV), suggesting a formation of quasi-ordered pair density modulation state.

Discussion

In summary, we have successfully synthesized a moiré structure formed between monolayer metallic NiTe₂ and superconducting NbSe₂. This moiré structure is found to host a short-range-ordered charge density modulation state with short-range charge orders inside moiré unit cells. The characteristics of the spontaneous on-site ordering and irregularity across the superlattice are analogue to the procrystal^{43,44,45,46,47}, based on which we dub an electronic procrystalline (EPC) state. Such unique EPC state represents an emergent electronic correlation and can be parallel to the electronic liquid-crystalline (ELC) state, serving as a valuable platform for exploring novel quantum phenomena and potential applications in devices. Our findings open the door towards a promising class of disordered quantum phase, which generally exists in other moiré systems formed by metallic bilayers with competing incommensurate lattice potentials⁴⁸.

Methods

Sample preparation

NiTe₂ film with various thickness were epitaxially grown on freshly-cleaved NbSe₂ substrates by molecular beam epitaxy approach under ultra-high vacuum conditions (UHV, base pressure ~5 × 10⁻¹⁰mbar). High-quality NbSe₂ single crystals used as substrates were fabricated by chemical vapor transport approach. The clean NbSe₂ substrates were obtained by cleaving in vacuum and subsequently annealing in UHV at 600 K for several hours. The NiTe₂ films were grown by e-beam evaporation of Ni (99.9%, Goodfellow Cambridge Ltd.) and simultaneous deposition of atomic Te (99.99%, Sigma-Aldrich) from a Knudsen cell at a substrate temperature of 520 K. During growth, the Te flux approximately an order of magnitude greater Ni flux (Te-rich conditions) is used. The layers of the epitaxial NiTe₂ film can be precisely controlled by tuning the deposition time and flux, which allows the construction of well-defined planar homojunctions between regions of different NiTe₂ layers on the same NbSe₂ substrate (Supplementary Fig. S17).

STM/STS and LEED

STM/STS measurements were performed in an ultrahigh vacuum (1 × 10⁻¹⁰mbar) ultra-low temperature condition equipped with 11 T magnetic field. The stable sample temperature can be kept at a base temperature of 0.4 K and 4.2 K, respectively. The electronic temperature is 620 mK at a base temperature of 400 mK. All the scanning parameters (setpoint voltage and current) of the STM topographic images are listed in the captions of the figures. Unless otherwise noted, the differential conductance (dI/dV) spectra were acquired by a standard lock-in amplifier at a modulation frequency of 973.1 Hz. Tungsten tip was fabricated via electrochemical etching and calibrated on a clean Au(111) surface prepared by repeated cycles of sputtering with argon ions and annealing at 770 K. LEED was employed with a 4-grid detector (Omicron Spectra LEED) in the UHV chamber at room temperature.

Two-dimensional lock-in technique and strain analysis

To explore the √3×√3 charge modulations, we employ a two-dimensional lock-in technique⁴⁹ to determine the amplitude and phase of the modulations. For any arbitrary real space image:

$$A({{{\bf{r}}}})={\sum }_{{{{\bf{Q}}}}}{a}_{{{{\bf{Q}}}}}({{{\bf{r}}}}){{{{\rm{e}}}}}^{-i{{{\bf{Q}}}}\cdot {{{\bf{r}}}}}$$

(1)

where ${a}_{{{{\bf{Q}}}}}({{{\bf{r}}}})$ is the complex amplitude at wavevector Q and position r. If Q is the wavevector of interest, it can be extracted from the Fourier transform $A\left({{{\bf{q}}}}\right)$ by shifting it back to the center and multiplying a Gaussian window with a cut-off length σ in q-space. The approximate complex amplitude in real space ${A}_{{{{\bf{Q}}}}}({{{\bf{r}}}})$ can be obtained by inverse Fourier transform as following:

$${A}_{{{{\bf{Q}}}}}({{{\bf{r}}}})={F}^{-1}\left[{A}_{{{{\bf{Q}}}}}({{{\bf{q}}}})\right]=\int {{{\rm{d}}}}{{{\bf{R}}}}A({{{\bf{R}}}}){{{{\rm{e}}}}}^{i{{{\bf{Q}}}}\cdot {{{\bf{R}}}}}{{{{\rm{e}}}}}^{-\frac{{(r-{{{\bf{R}}}})}^{2}}{2{\sigma }^{2}}}$$

(2)

Thus, using this technique, the amplitude $\left|{A}_{{{{\bf{Q}}}}}({{{\bf{r}}}})\right|$ and spatial phase ${\Phi }_{{{{\bf{Q}}}}}^{A}\left({{{\bf{r}}}}\right)$ of the modulation at Q can be written as:

$$\left|{A}_{{{{\bf{Q}}}}}({{{\bf{r}}}})\right|=\sqrt{{{{\mathrm{Re}}}} \, {A}_{{{{\bf{Q}}}}}{({{{\bf{r}}}})}^{2}+{{{\rm{Im}}}} \, {A}_{{{{\bf{Q}}}}}{({{{\bf{r}}}})}^{2}}$$

(3)

$${\Phi }_{{{{\bf{Q}}}}}^{A}({{{\bf{r}}}})={\tan }^{-1}\frac{{{{\rm{Im}}}} \, {A}_{{{{\bf{Q}}}}}({{{\bf{r}}}})}{{{{\mathrm{Re}}}} \, {A}_{{{{\bf{Q}}}}}({{{\bf{r}}}})}$$

(4)

Since the phase calculated by the algorithm may have a 2π jump at certain pixels, it is necessary to perform unwrap function on the obtained amplitude and phase information. When we select the wave vector Q to perform 2D-lockin, the signal after low-pass filtering can be approximated as:

$$A\left({{{\rm{r}}}}\right)\sim {e}^{-i({{{{\rm{q}}}}}_{0}-{{{\rm{Q}}}})\cdot {{{\rm{r}}}}}={e}^{-i\varDelta {{{\rm{q}}}}\cdot {{{\rm{r}}}}}$$

(5)

The phase can be written as:

$${\varPhi }_{Q}\left(r\right)=\varDelta {{{\rm{q}}}}\cdot {{{\bf{r}}}}+{\varPhi }_{{local}}\left({{{\boldsymbol{r}}}}\right)$$

(6)

If q₀ = Q, then only the local phase information ${\varPhi }_{Q}\left({{{\boldsymbol{r}}}}\right)={\varPhi }_{{local}}\left(r\right)$ remains. However, if q₀ ≠ Q, In the phase, there will be an additional linear term $\varDelta {{{\rm{q}}}}\cdot {{{\rm{r}}}}$. Here, our short-range order is associated with a set of broad spots in blue circles (Fig. 2b), so the occurrence of a reference phase mismatch is reasonable. The linear background corresponds to a very strong and low-frequency phase gradient, which will not be eliminated by Gaussian filtering. Unwrapping restores the high-frequency components that are weakened by Gaussian filtering, thus making the linear background more prominent after unwrapping. To avoid confusion, we use the least squares method to subtract these linear backgrounds, thereby making the obtained phase information more closely resemble the actual phase information in Fig. 2h. To obtain the amplitude and phase information in Fig. 2h, the selecting wavevector Q is at (√3×√3)30° position marked in Fig. 2b. The Gaussian filter order is 16 and the filter length scale is 28 pixels × 28 pixels (1.9 nm × 1.9 nm).

Based on 2D lock-in technique, the strain filed can be extracted with Lawler-Fujita algorithm⁵⁰. For a hexagonal lattice NiTe₂, considering the total displacement field $u\left({{{\bf{r}}}}\right)$, the real space image can be written as:

$$A\left({{{\bf{r}}}}\right)={a}_{1}{e}^{i{{{{\bf{Q}}}}}_{1}\cdot \left({{{\rm{r}}}}-u\left({{{\bf{r}}}}\right)\right)}+{A}_{2}{e}^{i{{{{\bf{Q}}}}}_{2}\cdot \left({{{\bf{r}}}}-u\left({{{\bf{r}}}}\right)\right)}+{A}_{3}{e}^{i{{{{\bf{Q}}}}}_{3}\cdot \left({{{\bf{r}}}}-u\left({{{\bf{r}}}}\right)\right)}+{{{\rm{c}}}}. \, {{{\rm{c}}}}$$

(7)

where ${{{{\rm{Q}}}}}_{1}$ and ${{{{\rm{Q}}}}}_{2}$ are the reciprocal lattice basis and ${{{{\rm{Q}}}}}_{3}={{{{\rm{Q}}}}}_{1}+{{{{\rm{Q}}}}}_{2}$. The fourth term c.c. represent possible displacement due to picometer scale piezoelectric drift and thermal drift. $u\left({{{\bf{r}}}}\right)$ is defined as displacement filed. By applying local Gaussian filtering in the momentum space near the Bragg wave vector (20 × 20 pixels window), high-frequency oscillations can be filtered out. We can get the two displacement field components ${u}_{1}\left({{{\bf{r}}}}\right)$ and ${u}_{2}\left({{{\bf{r}}}}\right)$:

$${u}_{i}\left({{{\bf{r}}}}\right)={Q}_{i}\cdot u\left({{{\bf{r}}}}\right)=-{{{\rm{arg }}}}\left({a}_{i}{e}^{i{{{{\boldsymbol{Q}}}}}_{i}\cdot u\left({{{\boldsymbol{r}}}}\right)}\right)(i=1,2)$$

(8)

The components along the lattice ${u}_{1}\left({{{\bf{r}}}}\right)$ and ${u}_{2}\left({{{\bf{r}}}}\right)$ can be decomposed into x and y axes ${u}_{x}\left({{{\bf{r}}}}\right)$ and ${u}_{y}\left({{{\bf{r}}}}\right)$.Total strain field $u\left({{{\bf{r}}}}\right)$ consists of strain field $s\left({{{\bf{r}}}}\right)$ and drift field $u\left({{{\bf{r}}}}\right)$. Drift field $d\left({{{\bf{r}}}}\right)$ typically includes piezo drift and thermal drift, and it can generally be fitted using a 3-order polynomial. By fitting with the STM images, $d\left({{{\bf{r}}}}\right)$ can be obtained, and thus the strain field can be determined as:

$$s\left({{{\bf{r}}}}\right)=u\left({{{\bf{r}}}}\right)-d\left({{{\bf{r}}}}\right)$$

(9)

Next, as described in the article (Supplementary Fig. S14), the strain tensor can be expressed as:

$${s}_{{ii}}\equiv \frac{\partial {s}_{{ii}}\left(r\right)}{\partial i}(i=x,y)$$

(10)

Definition of AB_Hollow, AB_Se, and AA regions in the moiré structure of NiTe₂/NbSe₂

To determine the local atomic structure of the bright, dark, and intermediate regions within the moiré superstructure, we construct a structural model of monolayer NiTe₂ on NbSe₂. After structural optimization, the moiré superstructure shows three distinct local regions, denoted as AB_Hollow, AB_Se, and AA based on different atomic alignments (Supplementary Fig. S1). In the AB_Hollow region, Ni atoms are positioned on top of hollow sites in NbSe₂ substrate, while in the AB_Se and AA regions, Ni atoms are aligned above Se and Nb atoms, respectively. The DFT simulated STM image based on the optimized atomic model shows excellent agreement with the experimental data (Supplementary Fig. S2). Consequently, we attribute the bright, dark, and intermediate regions within the moiré unit cells to AB_Hollow, AA, and AB_Se regions, respectively.

DFT calculations

The DFT calculations were carried out using Vienna ab initio simulation package (VASP)⁵¹. The projected augmented wave (PAW) method⁵² was used to describe the core-valence interactions. The generalized gradient approximation (GGA)⁵³ in the form of Perdew-Burke-Ernzerhof (PBE) is adopted for the exchange-correlation functional. In the calculations of free-standing NiTe₂, a 15 Å vacuum layer is used, and all atoms are fully relaxed until the residual forces on each atom are smaller than 0.01 eV/Å. The k-points sampling is 27 × 27 × 1 with the Gamma scheme. Wave functions were expanded on a plane-wave basis set up to 500 eV energy cutoff. The fully relaxed lattice constants of monolayer NiTe₂, bilayer NiTe₂ and NbSe₂ are 3.715 Å, 3.814 Å and 3.459 Å, respectively. A slab model is used for the moiré superlattice. For the monolayer NiTe₂, the heterostructure is constructed using a 10 × 10 monolayer NiTe₂ on an 11 × 11 monolayer NbSe₂, resulting in a lattice mismatch of ~2.4%. The corresponding moiré lattice periodicity of this configuration is 3.81 nm. For the bilayer case, the heterostructure is composed of a 9 × 9 bilayer NiTe₂ on a 10 × 10 monolayer NbSe₂, with a reduced lattice mismatch of ~0.8%. The resulting periodicity of the moiré lattice is 3.46 nm. Wave functions were expanded on a plane-wave basis set up to 400 eV energy cutoff. Van-der-Waals interactions were considered at the DFT-D3 level. The vacuum layer is 15 Å. A Γ-only k-point sampling in the first Brillouin zone is used for the moiré superlattice. All atoms except the NbSe₂ substrate are fully relaxed until the net force is smaller than 0.01 eV/Å. The in-plane strain is introduced by changing the lattice constant of NiTe₂. Moreover, PBE + U calculations were also used to study the effects of on-site Coulomb interaction on the Ni 3d-orbitals. Here, we used the method proposed by Dudarev et al. where only the effective Coulomb interaction U_eff defined by the difference between the correlation energy and the exchange energy was meaningful⁵⁴. A Hubbard U correction of U_eff = 2 eV and 4 eV were applied to Ni 3d-orbitals. The strain is defined by ΔL/L, where L and ΔL are the lattice constant and compressed length of pristine NiTe₂. The real part of susceptibility is calculated as the following formula,

$${x}_{0}^{{\prime} }\left({{{\boldsymbol{q}}}}\right)={\sum}_{{{{\boldsymbol{k}}}}} \, \frac{f\left({\epsilon }_{{{{\boldsymbol{k}}}}{{{\boldsymbol{+}}}}{{{\boldsymbol{q}}}}}\right)-f\left({\epsilon }_{{{{\boldsymbol{k}}}}}\right)}{{\epsilon }_{{{{\boldsymbol{k}}}}}-{\epsilon }_{{{{\boldsymbol{k}}}}{{{\boldsymbol{+}}}}{{{\boldsymbol{q}}}}}}$$

(11)

where the f denotes the electron’s Fermi distribution function.