Introduction

The role of stacking order in vdW layered materials is pivotal and has been empirically substantiated across diverse domains, such as sliding ferroelectricity, low-dimensional magnetism and moire superlattices1,2,3,4,5. The extensively studied twisted bilayer graphene showcase the emergence of flat bands around the magic angle of approximately 1.1 degrees6. These flat bands arise from the interaction between electronic states originating from different local stacking configurations in real space, offering a fertile ground for exploring novel quantum states and phenomena, including moire physics7,8, superconductivity9,10, and quantum anomalous Hall (QAH) effects11. Another typical example is bilayer CrI3 film, in which interlayer sliding can induce distinct interlayer magnetic interaction from ferromagnetic to antiferromagnetic states3,12,13,14.

Meanwhile, the recent theoretical prediction15,16,17 and experimental fabrication of MnBi2Te4 (MBT)18,19,20,21,22,23, the long-sought antiferromagnetic topological insulator24, have sparked intensive research interests25,26,27, such as axion electrodynamics28,29, and layer Hall effect30,31. As a typical van der Waals (vdW) layered material, bulk MBT is characterized by its ABC-stacked Te-Bi-Te-Mn-Te-Bi-Te septuple-layer (SL) building blocks. The antiferromagnetic interlayer interaction between adjecent ferromagnetic SLs culminate in A-type antiferromagnetism. The interplay between topology and intrinsic magnetism in MBT provide a ideal material platform for layer-dependent topological properties, with odd SLs exhibiting quantum anomalous Hall (QAH) insulating behaviors, and even SLs manifesting axion insulating characteristics15,16,32,33,34. Although stacking-dependent topological phases have been recognized in MBT35,36,37,38 and chemically similar materials, including Bi2Te339 and GeBi2Te440, the exploration of various stacking patterns in MBT multilayer films remains significantly underexplored, and a unified model Hamiltonian to describe distinct stacking orders is still absent.

In this work, we theoretically predict that the manipulation of stacking orders in MBT thin films can significantly alter their electronic and topological characteristics. Although all stacking orders can maintain interlayer antiferromagnetism, there emerge topological phase transitions between C = 1 and C = 0 topological phases only through lateral shift of the topmost SL. The “incorrect" stacking order unexpectedly exhibits quantum anomalous Hall (QAH) insulators with C = 1 in even SLs with compensated AFM phases and trivial magnetic insulators with C = 0 in odd SLs with uncompensated AFM phases. In addition, a tight-binding model is developed to offer a comprehensive and detailed description of electronic and topological properties exhibited by various stacking configurations, and provide a valuable tool in the exploration of other relevant properties of MBT multi-layer films.

Results

Lateral shift and energy landscape in MnBi2Te4 few-SL films

Taking a representative example, the crystal structure of 3-SL MBT is plotted in Fig. 1a, where the lateral shift of the bottom SLs relative to the top SL is denoted by the vector d. The potential energy surface of 3-SL MBT with different stacking orders in Fig. 1b is evaluated with reference to the ground magnetic state. The ABC stacking order is at the global minimum, and the CAC-stacking is energetically comparable and also located at local minimum. Here, the ABC-stacking is the usual stacking order in MBT bulk forms and thin films, and the CAC-stacking refers to the “incorrect" stacking order that can be obtained by shifting the remaining bottom SLs by [1/3, -1/3] relative to the top SL (Fig. 1a). This above notation also applies to other MBT films, such as CABC- and BCAC-stacking in 4-SL MBT. Our further phonon dispersions (see supplementary material) also validate that CAC-stacking MBT is dynamically stable, implying that 3-SL MBT thin films can be trapped in the CAC stacking order. Contrary to ABC- and CAC-stacking, the BBC-stacking is energetically unfavorable, and will spontaneously evolute into ABC- and CAC-stacking under small perturbation.

Fig. 1: Atomic structure and potential energy map of three septuple layers (SL) MnBi2Te4 (MBT).
figure 1

a Crystal structure of 3-SL MBT, with Mn, Bi and Te atoms denoted as purple, blue and gray balls, respectively. Lateral shift of the bottom SLs relative to the top SL is denoted by the vector d. b Stacking energy of 3-SL MBT as a function of lateral shift vector d, compared to the ABC-stacking MBT.

Full size image

First-principles calculation on electronic and topological properties in MBT few-SL films

In contrast to the stacking-order dependent interlayer magnetic interaction observed in bilayer CrI33,4, interlayer exchange interaction in 3-SL MBT remains antiferromagnetic for all stacking configurations(see supplementary material). However, the electronic band gaps are dramatically altered by lateral shift, as depicted in Fig. 2a, which even leads to a topological phase transition between quantum anomalous Hall (QAH) insulators and trivial magnetic insulators. The QAH phase with Chern number C = 1 is predominantly found in the vicinity of ABC-stacking, which is consistent with previous studies15,16. Other stacking orders, such as CAC-stacking, result in topologically trivial magnetic insulators with C = 0.

Fig. 2: Electronic and topological properties of 3-SL MBT thin films.
figure 2

a Calculated band gap as a function of lateral shift. The positive (negative) sign of band gaps indicates quantum anomalous Hall insulators (C = 1) and trivial magnetic insulators (C = 0). b The evolution of band gaps and stacking energy between ABC- and CAC-stacking order along the high-symmetry [1-10]. The grey shaded area indicates the topological transition point. Anomalous Hall conductance σxy as a function of Fermi energy (c) and the distribution of Berry curvature in the momentum space (d, e) in the ABC- and CAC-stacking order, respectively.

Full size image

Similar to sliding ferroelectricity experimentally observed in 2D van der Waals materials, the nudged elastic band (NEB) method is commonly employed to accurately calculate the transition barrier between two metastable phases. In this study, the NEB method is utilized to determine the transition barrier by constructing an interpolated path that precisely follows the high-symmetry line [1-10]. As illustrated in Fig. 2b, the energy barrier for the interlayer shift is measured to be 56 meV, close to the theoretically predicted lateral sliding energy barrier observed in CrI33. The moderate energy barrier for lateral displacement suggests that in-plane sliding can be employed to induce topological phase transitions in multi-SL MBT films, providing an additional degree of manipulation beyond strain, pressure and external electromagnetic field. The strong coupling between stacking order and electronic properties in MBT films provides additional degree of freedom in materials engineering, enables the manipulation of real-space patterns with chiral edge states and facilitates the exploration of the interplay between orbital and intrinsic magnetism in the moire patterns formed by twisted MBT multi-SL films41.

Furthermore, the quantized anomalous Hall conductance displayed in Fig. 2c, where σxy = − e2/h (σxy = 0) is clearly observed within the band gap for ABC (CAC)-stacking, indicates a Chern number C = 1 (C = 0). Fig. 2d, e illustrates the distribution of Berry curvature (Ω) in the momentum space for ABC- and CAC-stacking, respectively. In both stacking configurations, the Berry curvature is predominantly concentrated around the Brillouin-zone center. However, due to the coexistence of spatial inversion symmetry \({\mathcal{P}}\) and threefold rotation C3z in ABC-stacking, the Berry curvature exhibits a sixfold rotational symmetry around Γ. Conversely, CAC-stacking breaks \({\mathcal{P}}\), resulting in the Berry curvature only displaying a threefold rotational symmetry around Γ. The alternating positive and negative Berry curvatures in CAC-stacking renders it topologically trivial.

Let us turn to the even-SL case. The potential energy surface of 4-SL MBT is almost the same as in 3-SL, and the zero Hall plateau QAH state (C = 0) only exist close to the energetically favored CABC-stacking (see supplementary material). Unlike the odd-SL ABC stacking orders that preserve \({\mathcal{P}}\), antiferromagnetic order in the even-SL ABC stacking orders break both \({\mathcal{P}}\) and time-reversal (\({\mathcal{T}}\)) symmetries simultaneously. However, the combined symmetry operator \({\mathcal{PT}}\) is still respected (Fig. 3a). The existence of \({\mathcal{PT}}\) ensures double degeneracy at each band and vanishing Berry curvature throughout the Brillouin zone, resulting in a constraint of C = 0. Contrarily, the local energy minimum BCAC-stacking breaks \({\mathcal{PT}}\), leading to the non-degenerate band structures (Fig. 3b) and non-vanishing Berry curvature in the whole momentum space (Fig. 3c). The calculated Berry curvature in BCAC stacking is still concentrated at Γ and exhibit threefold rotational symmetry. Anomalous Hall conductance shown in Fig. 3d is well quantized into σxy = − e2/h inside the bulk gap of 24 meV, also validating its nontrivial topological properties. We further calculate the interpolated path by the NEB method, found that topological phase transition and energy barrier in the 4-SL case is similar to the 3-SL case.

Fig. 3: Electronic and topological properties of 4-SL MBT.
figure 3

a Schematic illustration of the presence (absence) of \({\mathcal{PT}}\) symmetry in CABC-(BCAC-) stacking order. Electronic band structures (b), Berry curvature distribution in the momentum space (c) and anomalous Hall conductance σxy as a function of Fermi energy (d) in the BCAC-stacking order.

Full size image

Our further calculations on other multi-SL MBT films, ranging from 5 to 7 SLs, yield similar results (Table 1), indicating that stack-dependent electronic and topological properties discussed above are universal. A significant band gap (> 20 meV) is present in all these high-symmetry stacking orders, and topological phase transitions consistently occur in the transition path between the ABC- and CAC-stacking orders. The ABC-stacking order, which is commonly observed in multi-SL MBT films, can exhibit a QAH insulating phase with C = 1 in odd-SL and a zero plateau QAH phase with C = 0 in even-SL, consistent with previous theoretical and experimental studies15,16,32. However, the thickness-dependent topology shows an opposite trend in the CAC-stacking order: even-SL exhibits a C = 1 phase with compensated antiferromagnetic (AFM) phases, while odd-SL exhibits a C = 0 phase with uncompensated AFM phases.

Table 1 Band gaps (in unit of meV) and Chern numbers (C) in multi-SL MBT thin films of distinct stacking orders
Full size table

Model hamiltonian and analysis

By taking each SL as a building blocking, and assuming only nearest SLs have interlayer coupling, the minimal tight-binding hamiltonians can be

$$\begin{array}{lll}H({{\bf{k}}}_{\parallel })\,=\mathop{\sum}\limits _{{\bf{k}},ij}\left[({H}_{0}({\bf{k}})+{H}_{Z}){\delta }_{i,j}\right.\\\qquad\quad \,\,+\left.{H}_{+}({\bf{k}}){\delta }_{i,j-1}+{H}_{-}({\bf{k}}){\delta }_{i,j+1}\right],\end{array}$$
(1)

where i denote the ith-SL, k∥ is the in-plane momenta (kx, ky), H0, H+ (H−), and HZ represent intralayer, interlayer and Zeeman hamiltonians, respectively. Only the intralayer term H0 is assumed to be momentum-dependent, and the latter two terms only dependent on relative lateral shift, shown as in Fig. 4a. Following the effective \(\vec{{\bf{k}}}\cdot \vec{{\bf{p}}}\) hamiltonians based on four low-lying state \(\left\vert P{1}_{z}^{+},\uparrow (\downarrow )\right\rangle\) and \(\left\vert P{2}_{z}^{-},\uparrow (\downarrow )\right\rangle\)42,43, the intralayer and interlayer hamiltonians can be written as

$$\begin{array}{lll}{H}_{0}\,=\,\left[\begin{array}{cccc}\bar{C}+\bar{M}&0&0&{\bar{A}}_{-}\\ 0&\bar{C}+\bar{M}&{\bar{A}}_{+}&0\\ 0&{\bar{A}}_{-}&\bar{C}-\bar{M}&0\\ {\bar{A}}_{+}&0&0&\bar{C}-\bar{M}\end{array}\right],\\ {H}_{+}\,=\,\left[\begin{array}{cccc}-{\bar{C}}_{1}-{\bar{M}}_{1}&0&i{\bar{B}}_{0}&0\\ 0&-{\bar{C}}_{1}-{\bar{M}}_{1}&0&-i{\bar{B}}_{0}\\ i{\bar{B}}_{0}&0&-{\bar{C}}_{1}+{\bar{M}}_{1}&0\\ 0&-i{\bar{B}}_{0}&0&-{\bar{C}}_{1}+{\bar{M}}_{1}\end{array}\right],\\ {H}_{Z}\,=\,\left[\begin{array}{cccc}{Z}_{1}+{Z}_{2}&0&0&0\\ 0&-{Z}_{1}-{Z}_{2}&0&0\\ 0&0&{Z}_{1}-{Z}_{2}&0\\ 0&0&0&-{Z}_{1}+{Z}_{2}\end{array}\right],\end{array}$$
(2)

where \(\bar{C}={C}_{0}+{C}_{2}({k}_{x}^{2}+{k}_{y}^{2})\), \(\bar{M}={M}_{0}+{M}_{2}({k}_{x}^{2}+{k}_{y}^{2})\), \({H}_{-}={H}_{+}^{\dagger }\), \({\bar{A}}_{\pm }={A}_{0}({k}_{x}\pm i{k}_{y})\), and higher-order terms are ignored. All the parameters in the tight-binding model hamiltonians can be determined by fitting with first-principles band structures of ABC- and ACB-stacking ferromagnetic bulk MBT, whose numerical values are shown in the supplementary material. Compared to ABC stacking, the interlayer hopping strength is smaller in ACB-stacking, which originates from the enlarged interlayer Te-Te distance. Multi-layer slabs with varied thickness and distinct stacking orders are built based on the above tight-binding Hamiltonians, and the corresponding electronic band gaps and topological properties are obtained, which is well in accordance with our previous first-principles calculations (Fig. 4b), confirming the validity of the aforementioned four-band model.

Fig. 4: Underlying mechanism of stacking-dependent topology in multi-SL MBT films.
figure 4

a The intralayer and interlayer hopping matrix in the four-band model hamiltonians. b Comparison of topological invariants (C) and band gaps between first-principles calculations and model hamiltonians. c Isosurface of charge density difference (Δρ) in the CAC- and ABC-stacking orders, respectively. d, e Schematic illustrations of topological phases in distinct stacking orders of 3-SL and 4-SL MBT films, respectively.

Full size image

Given that most physical properties in density functional theory are based on charge distribution (ρ) in the ground state, we further performed a systemic study on the impact of lateral shift on charge density difference (Δρ). The charge density difference (Δρ) is defined as the net charge distribution that arises from the interaction between two subsystems, given by the formula: Δρ = ρtotal − ρtop − ρbot, where ρtotal is the charge density of the combined system and ρtop and ρbot are the charge densities of the non-interacting individual subsystems, the top 1-SL and remaining part of multi-SL thin films, respectively. Remarkably, Δρ is negligible in the CAC-stacking, but it becomes pronounced in the ABC-stacking (Fig. 4c), which means lateral shift significantly alters Te-Te interaction at the vdW interface. Compared to the consistently strong interlayer interaction in the ABC stacking, the CAC-stacking can be regarded as the superposition of two loosely connected components — the top SL and other SLs. where interlayer interaction between these two components is relatively weaker. Thus, the CAC-stacking films exhibit a completely opposite oscillating pattern in topological properties: QAH insulating phase with C = 1 in even SLs and zero-plateau QAH phase with C = 0 in odd SLs (Fig. 4d).

Lateral displacement in multi-layer MBT films induce distinct types of magnetic space groups in the high-symmetry structure of odd- and even-SL MBT films, and we proposed two feasible and nondestructive optical measurement setups to determine its stacking order before conducting transports experiments. Second harmonic generation (SHG) is forbidden by inversion \({\mathcal{P}}\) in the ABC-stacking odd-SL MBT, but become allowed in other stacking orders, exhibiting sixfold symmetry under incident linear polarized laser44. Magneto-optical Kerr effect is forbidden by \({\mathcal{PT}}\) in the ABC-stacking even-SL MBT, and the angle-independent MOKE can be expected in other stacking orders45. The breaking of \({\mathcal{P}}\) and \({\mathcal{PT}}\) symmetries in other low-symmetry structures result in the coexistence of these two optical signals. Therefore, these two optical setups, both sensitive to material symmetries, can serve as a powerful tool to detect and distinguish local structures formed by distinct stacking orders.

Discussion

In the work, we make theoretical predictions the stack-order dependent electronic and topological properties in multi-SL MnBi2Te4 (MBT) thin films. Lateral displacement of the topmost layer in multi-layer MBT thin films can lead to significant modifications in electronic band structures, and even leading to topological phase transitions. When the top SL is shifted from ABC stacking to CAC stacking, there emerges the opposite thickness-dependent topology contrary to the usual ABC-stacking order: uncompensated AFM trivial insulators with C = 0 in odd SLs and compensated AFM QAH insulating insulators with C = 1 in even SLs. Our effective tight-binding hamiltonians provide a thorough and detailed description of both ABC and CAC stacking configurations, highlighting the crucial role played by the varying interlayer Te-Te interaction. Finally, two complementary experimental setups are proposed to distinguishing different local structures. Our theoretical predictions on strong coupling between stacking order and topology emphasizes the profound effects of interlayer sliding in magnetic topological materials, opening up new possibilities for discovering a rich landscape of opportunities for tailoring and optimizing topological quantum materials.

Methods

Computational details

First-principles calculations are performed within the framework of density functional theory (DFT) through Vienna abinitio Simulation Package (VASP)46. The projector augmented wave method and the Perdew-Burke-Ernzerhof (PBE) functional47 are adopted to simulate ionic core environment. Structural relaxations are performed with a force criterion of 0.01 eV/Å. The van der Waals correction is included by the DFT-D3 method48 to tackle with interlayer interaction in MnBi2Te4 (MBT) few-layer films. The localized d-orbitals on Mn atoms are treated by the DFT + U method with U = 4 eV, which is consistent with the previous papers15,49. The plane-wave basis with energy cutoff of 350 eV and Monkhorst-pack k-point mesh grid of 9 × 9 × 1 are adopted in the calculation. Phonon dispersions are calculated with the frozen phonon method implemented in the Phonopy package50, where a 3 × 3 × 1 supercell together with a 3 × 3 × 1 uniform k-mesh grid is employed.

The topological properties of MBT few-layer films are indeed characterized by topological invariants, specifically the Chern numbers (C), whose can be evaluated by the sum of Berry curvature in the whole Brillouin zone:

$$C=\frac{1}{2\pi }{\int}_{{\rm{BZ}}}B(\vec{k})d\vec{k}.$$
(3)

The Berry curvature of a single band can be written as

$${B}_{n}(\vec{k})=-{\rm{Im}}\sum _{{n}^{{\prime} }\ne n}\frac{\left\langle n\left\vert \nabla H\right\vert {n}^{{\prime} }\right\rangle \times \left\langle {n}^{{\prime} }\left\vert \nabla H\right\vert n\right\rangle }{{\left({E}_{n}-{E}_{{n}^{{\prime} }}\right)}^{2}},$$
(4)

and the Berry curvature of all the occupied bands can be obtained by summing the Berry curvature of all the valence bands,

$$B(\vec{k})=\sum _{n\in {\rm{occ}}}{B}_{n}(\vec{k}).$$
(5)

Based on the above equation, the momentum-dependent Berry curvature and Chern number (C) are calculated in the WannierTools package51, based on the tight-binding Hamiltonians constructed from maximally localized Wannier functions52, with the initial projectors of p-orbitals on Bi and Te atoms.