Half-quantized Hall metal and marginal metal in disordered magnetic topological insulators

Abstract

A semimagnetic topological insulator—a heterostructure combining a topological insulator with a ferromagnet—exhibits a half-quantized Hall effect, characterized by a quantized Hall conductance of $\frac{1}{2}\frac{{e}^{2}}{h}$ (where e is the elementary charge and h is the Planck constant), which reinforces the established understanding of topological phenomena in condensed matter physics. However, its stability in realistic, disordered systems remains poorly understood. Here, we demonstrate the robustness of the half-quantized Hall effect in weakly disordered systems, stemming from a single gapless Dirac cone of fermions and coexisting with weak antilocalization due to the π Berry phase that suppresses backscattering. Furthermore, we uncover a marginal metallic phase emerging between weak antilocalization and Anderson insulation—a transition that defies conventional metal-insulator transitions by lacking an isolated critical point—where both conductance and normalized localization length exhibit scale invariance, independent of system size. The half-quantized Hall metal and the marginal metallic phase challenge existing localization theories and provide insights into disorder-driven topological phase transitions in magnetic topological insulators, opening avenues for exploring quantum materials and next-generation electronic devices.

Introduction

The quantum anomalous Hall effect (QAHE) in ferromagnetic insulators is a cornerstone of modern condensed matter physics, characterized by quantized Hall conductivity in the absence of an external magnetic field, arising from the topological properties of electronic states^1,2,3. In two-dimensional (2D) metallic ferromagnets, the Hall conductivity is typically non-quantized and can be expressed as an integral of Berry connection over the Fermi surface⁴. Recent theoretical and experimental advances have revealed that systems featuring a single gapless Dirac cone of electrons in the first Brillouin zone can exhibit a half-quantized Hall conductivity^5,6,7. This phenomenon bears a striking resemblance to the parity anomaly of massless Dirac fermions in quantum field theory^8,9,10,11. Unlike the QAHE observed in insulating phases, which is characterized by integer Chern numbers and the emergence of chiral edge states^{12,13,14,15,16,17,18,19,20}, the half-quantized Hall effect (HQHE) occurs in systems with a finite Fermi surface and non-zero longitudinal conductivity. The effect has been experimentally observed in a semi-magnetic structure of Cr-doped topological insulator (TI) (Bi, Sb)₂Te₃²¹, sparking significant interest in its realization, robustness, and dissipative properties^{22,23,24,25,26,27}.

Despite these advances, the HQHE presents a fundamental challenge to the theory of localization. In 2D systems with broken time-reversal symmetry, disorder-induced localization of electronic states is commonly anticipated, leading to integer-quantized Hall conductivity²⁸. This raises a critical question: Can the HQHE in semimagnetic TIs survive or remain robust in the presence of disorder? The localization behavior of disordered systems depends on their dimensionality and the symmetry class of the Hamiltonian²⁹. Recent insights have highlighted the profound influence of topological wave functions on localization properties^{30,31,32,33,34,35,36}. Specifically, relativistic Dirac electrons, su ch as those in a single-flavor gapless Dirac cone, are theoretically resistant to localization even under strong disorder^32,33,34, provided there is no coupling to other bands. This suggests that a system exhibiting the HQHE should retain its metallic character under moderate disorder. In condensed matter systems, a single-flavor Dirac cone typically appears on one surface of a three-dimensional ${{\mathbb{Z}}}_{2}$ TI, remaining robust until strong disorder drives the system into a topologically trivial insulating state with zero Hall conductivity³⁷. Given that a semimagnetic TI hosts only a single gapless Dirac cone, the transition between the HQHE and the Anderson insulating state warrants thorough exploration.

In this work, we employ well-established numerical methods (see “Methods”) to investigate disorder-driven topological phase transitions in semimagnetic TI thin films modeled on a real-space lattice. We reveal the robustness of the disordered half-quantized Hall metal (HQHM) and the emergence of a marginal metallic phase, shedding light on topological materials, disorder physics, and quantum transport research. Moreover, since the half-quantized quantum Hall effect has been experimentally observed²¹, the phenomena we describe offer insights for future experimental design and observation of related effects.

Results

The phase diagram

The main results of this study are summarized in Fig. 1, which depicts the phase diagram of a disordered semimagnetic TI in the E_F–W plane, where E_F is the chemical potential and W is the disorder strength. The phase diagram comprises three distinct regions: (1) the HQHM at weak disorder, (2) the marginal metal (MM) at moderate disorder, and (3) the Anderson insulator (AI) at strong disorder. In the absence of disorder, a semimagnetic TI hosts a single gapless Dirac cone of fermions on one surface and a massive Dirac cone of fermions on the other surface. The gapless Dirac cone arises from the surface electrons of the underlying strong TI, localized on the bottom layer, which is spatially separated from the top magnetic-doped layers. The presence of this single gapless Dirac cone results in a half-quantized Hall conductance of $\frac{1}{2}\frac{{e}^{2}}{h}$^7,38. In the weak disorder regime, the half-quantized Hall conductance remains robust against disorder, defining the HQHM phase. In this region, the suppression of backscattering for gapless Dirac fermions leads to weak antilocalization (WAL) behavior in the longitudinal conductivity σ. Specifically, the quantum correction to the conductivity follows $\delta \sigma \simeq \frac{{e}^{2}}{\pi h}\ln (L/{l}_{e})$, where L is the sample size and l_e is the elastic mean free path. This logarithmic increase in conductivity with sample size indicates a metallic phase. By introducing the scaling function $\alpha =\pi \frac{h}{{e}^{2}}\frac{\partial \sigma }{\partial \ln L}$, the HQHM phase is characterized by α = 1 when the Fermi energy only intersects with the gapless Dirac cone. As the disorder strength increases, the system transitions into the MM phase^{39,40,41,42,43,44,45,46} in which interband scattering among Dirac bands plays a significant role. In this regime, the conductivity becomes scale-invariant, exhibiting no dependence on system size (α = 0). Simultaneously, the Hall conductivity decays gradually to vanish. Crucially, the MM phase persists over a finite range of disorder strengths, in contrast to conventional metal–insulator transitions, which occur at a single critical point. This extended phase highlights the unique nature of the disorder-driven transition in semimagnetic TIs. At an even stronger disorder, the system enters the AI phase, where all electronic states become fully localized.

Fig. 1: The phase diagram in disordered semimagnetic topological insulator films.

a Schematic of a semimagnetic topological insulator film. The acronyms HQHM (half-quantized Hall metal), MM (marginal metal), and AI (Anderson insulator) label distinct phases in the diagram. The red solid line indicates that the clean system is a metal rather than MM. b Evolution of the spectral function A(ϵ, k) with the disorder strength W. c The phase diagram of the semimagnetic topological insulator in the W–E_F plane: The metallic phases of the scaling function α = 1 with the half-quantized anomalous Hall conductivity; the marginal metallic phase of α = 0 with non-quantized Hall conductivity, and the Anderson insulating phase. Here we take the system size L large enough.

This phase diagram reveals a disorder-driven transition pathway in semimagnetic TIs, distinct from conventional metal–insulator transitions. The robustness of the HQHM phase, the emergence of the scale-invariant MM phase, and the eventual transition to the AI phase underscore the intricate interplay between topology, disorder, and electronic localization in these systems.

HQHE in weak disorder

To demonstrate the robustness of the HQHE, we first calculate the Hall conductivity σ_xy for a tight-binding model of the semimagnetic TI on an L_x × L_y × L_z lattice using the real-space Kubo formula⁴⁷ (see “Methods: The non-commutative Kubo formula for the Hall conductivity”). Periodic boundary conditions are applied along the x- and y-directions, while open boundary conditions are used along the z-direction to model the finite thickness of a thin film. The calculated σ_xy in the E_F–W plane is shown in Fig. 2a. A pronounced Hall plateau, indicated by the bright yellow region, is observed in the lower-left corner of the phase diagram. This plateau demonstrates that the Hall conductivity remains robust under moderate disorder even when the chemical potential deviates from the Dirac point, but is still located with the gap of the top surface states induced by magnetic doping. Adjacent to this region, a non-quantized anomalous Hall effect, indicated by the chartreuse region, is observed, and gradually decays to vanish with increasing disorder strength, as indicated by the dark purple region. To investigate the behavior of σ_xy in the thermodynamic limit, we perform a finite-size analysis of the Hall conductivity (see Supplementary Note 1) along a vertical line at a fixed disorder strength W = 1 eV as shown in Fig. 2c-1. The Fermi energy E_F is varied from 0 to 0.2 eV, with the color gradient transitioning from red to blue in increments of 0.02 eV. For E_F < 0.06 eV, where the Fermi level E_F intersects a single gapless Dirac cone, σ_xy quickly converges to $\frac{1}{2}\frac{{e}^{2}}{h}$ as the system size L varies from 12 to 28. In this region, the σ_xy nearly collapses onto a single curve (dashed line) and follows a power-law behavior:

$$\begin{array}{r}{\sigma }_{xy}={\sigma }_{xy}^{0}\left[1-{\left(\frac{{l}_{e}}{L}\right)}^{4}\right]\end{array}$$

(1)

in units of $\frac{{e}^{2}}{h}$, where l_e is a characteristic length, and ${\sigma }_{xy}^{0}$ denotes the Hall conductivity in the thermodynamic limit. As exhibited in Fig. 2c-2, the extracted value of ${\sigma }_{xy}^{0}$ is 0.49997 ± 0.00021, closely approaching the half-integer quantization with high precision. The scaling behavior of σ_xy differs fundamentally from that of the integer quantum Hall effect, which is strictly quantized to integer values (0 or 1)²⁸. The shift of σ_xy by 1/2 from the conventional integer quantum Hall effect can be interpreted as a manifestation of the parity anomaly, whose robustness against disorder highlights the exotic and unique topological nature of the HQHE. When E_F intersects a single gapless Dirac cone, the random gapless Dirac Hamiltonian belongs to the symplectic symmetry class with time reversal symmetry. However, the bottom gapless surface states are connected to the top gapped surface states through the side surface, effectively making the gapped surface states act as a time-reversal symmetry-breaking boundary condition for the gapless Dirac cone^6,32,48. As a consequence, the gapless Dirac cone on the surface of a semimagnetic TI film preserves time-reversal symmetry in the bulk but breaks time-reversal symmetry on the boundary. This results in nontrivial topology, distinguishing it from the ordinary symplectic symmetry class. As shown in ref. ⁶, a half-quantized Hall current circulates around the boundary between the gapped and gapless Dirac cone regions. This corresponds to a HQHE and reflects the parity anomaly of gapless Dirac fermions in condensed matter systems. In the framework of effective field theory³², implementing an infinite-mass boundary condition as a regularization scheme introduces a topological θ term into the low energy effective action of the nonlinear sigma model for the disordered single gapless Dirac cone, with the topological angle fixed at θ = π (with the sign in our case depending on the sign of the Zeeman field, ${{\rm{sgn}}}({V}_{0})$). The topological term exerts minimal influence in the metallic regime with a large longitudinal conductivity σ_xx, but should crucially alter the renormalization group flow in the strong disorder regime with small σ. For small σ, the theory of a gapless single-node Dirac fermion is analogous to the critical line of the integer quantum Hall transition. There exists an attractive fixed point σ^*, such that as the temperature decreases the system is driven into a metallic state exhibiting a HQHE, characterized by $({\sigma }_{xx},{\sigma }_{xy})=({\sigma }^{* },\pm \frac{{e}^{2}}{2h})$. This behavior may explain the temperature-dependent flow of longitudinal and Hall conductance observed in semi-magnetic TIs at zero magnetic field²¹. For large σ, the system exhibits a metal–insulator transition at a critical point σ_sp, similar to the behavior observed in the symplectic class^49,50. Several numerical investigations indicate that a single gapless Dirac cone with nonmagnetic disorder potential is always metallic with a positive beta function^33,34. However, these studies are typically conducted in momentum space by imposing a hard cutoff at a sufficiently large momentum. This approach overlooks the influence of boundary effects with symmetry breaking, which can play a significant role in the system’s transport behavior. Despite differing in certain aspects, these studies consistently conclude that the isolated single gapless Dirac cone state remains metallic in the presence of disorder when the Fermi energy E_F is near the Dirac point.

Fig. 2: Robustness of the Half-quantized Hall effect in weak disorder.

a The phase diagram of the Hall conductivity in the W–E_F plane. We set the lattice size L_x = L_y = L = 20 in the simulation. The red solid line indicates that the clean system is metallic. The bright yellow areas highlight the half-quantized Hall metal (HQHM) phase, whereas the chartreuse region indicates marginal metal (MM) with non-quantized Hall conductivity. The white solid line marks the phase boundary as determined by the effective medium theory. b The calculated Hall conductivity and disorder renormalized Dirac mass ${\widetilde{m}}_{0}$ and energy broadening η_top at E_F = 0.01 eV as a function of W in the effective medium theory. The black dashed line denotes the critical threshold W_c = 2.6 eV. The finite-size scaling analysis of the quantization error, ${\sigma }_{xy}^{0}-{\sigma }_{xy}$, is presented as the function of 1/L in c-1 along the vertical line of W = 1.0 eV for varying E_F, and d-1 along the horizontal line of E_F = 0.01 eV for varying W. The corresponding ${\sigma }_{xy}^{0}$ and l_e are displayed in (c-2, d-2), respectively. The error bars reflect the uncertainties arising from the numerical fitting. We have used the set of parameters L_z = 10, ${L}_{z}^{{{\rm{Mag}}}}=3$, λ_∥ = 0.41 eV, λ_z = 0.44 eV, t_∥ = 0.566 eV, t_z = 0.40 eV, V₀ = 0.1 eV, and lattice constants a = b = 1 nm and c = 0.5 nm unless otherwise specified. The raw data points are averaged over 50 random samples.

When E_F is larger than 0.06 eV, the Fermi surface intersects both gapless and gapped surface Dirac cones. In the absence of disorder, the Hall conductivity is non-quantized due to the breaking of time-reversal symmetry in the two Fermi surface loops³⁸. When disorder is introduced, the Hall conductivity still shows an increasing trend with growing system size L as shown in yellow and blue lines in Fig. 2c-1. Our numerical analysis reveals that in this regime σ_xy does not seem to converge to a quantized value. This observation contradicts the expected physical picture, where disorder is expected to localize states associated with the gapped Dirac cone, and the Hall conductivity should solely arise from the delocalized gapless Dirac cone, scaling the system to $\frac{1}{2}\frac{{e}^{2}}{h}$. However, this simplified picture overlooks the potential impact of inter-surface scattering, which could significantly modify the underlying physics. As E_F increases further toward the high-energy band edge, the system transitions into an Anderson insulating state, where σ_xy scales to 0. Our numerical results reveal that the crossover region between the metallic phase with ${\sigma }_{xy}=\frac{1}{2}\frac{{e}^{2}}{h}$ and AI state with σ_xy = 0 actually constitutes a marginal (or critical) phase exhibiting non-quantized Hall conductivity, instead of shrinking to a single critical point. Similarly, a finite-size analysis of σ_xy along the horizontal axis at E_F = 0.01eV is shown in Fig. 2d-1. Here, as L increases, σ_xy exhibits a pronounced saturation to the half-quantized value up to a critical disorder strength W_c ≈ 2.0 eV. The determined ${\sigma }_{xy}^{0}$ value fluctuates within a narrow margin around the half-integer plateau, as depicted in Fig. 2d-2, with extracted bounds spanning between 0.4993 ± 3.735 × 10⁻⁴ and 0.5002 ± 1.929 × 10⁻⁴. Beyond this critical strength, the Hall conductivity begins to deviate significantly from the quantized value.

The lower-left corner of the phase diagram, corresponding to weak disorder and small Fermi energy E_F represents the HQHM characterized by a HQHE and metallic behavior. In 2D or quasi-2D systems, the Hall effect of an insulating state must be integer-quantized, and this result remains robust even in the presence of disorder due to the topological nature of the quantum Hall effect. In the phase diagram, regions outside the HQHM state that exhibit a non-zero but non-quantized Hall conductivity must correspond to a metallic state.

To understand the origin of the transition from HQHE to non-quantized AHE, we can employ the effective medium theory in conjunction with the Kubo formula for electrical conductivities^51,52 (see “Model and material realization” and “Methods: Effective medium theory”), and the results are presented in Fig. 2b. The self-consistent Born approximation is a powerful tool and is applied extensively to investigate the physics in topological AI^{53,54,55,56,57,58,59,60,61,62}. We extend the self-consistent Born approximation to a layered structure. After averaging over disorder, the Green’s function recovers the translational symmetry: ${G}^{R}=\langle {[{E}_{{{\rm{F}}}}+{{\rm{i}}}{0}^{+}-{H}_{0}-{H}_{{{\rm{imp}}}}]}^{-1}\rangle ={\left({E}_{{{\rm{F}}}}-{H}_{0}-{\Sigma }^{R}\right)}^{-1}$. Both the mass, magnetic gap, and the chemical potential are renormalized by the real part of the self-energy, with ${m}_{0}\to {\widetilde{m}}_{0}$, ${V}_{0}\to {\widetilde{V}}_{0}$ and ${E}_{{{\rm{F}}}}\to {\widetilde{E}}_{{{\rm{F}}}}$. As depicted in Fig. 2b, the renormalized Dirac mass ${\widetilde{m}}_{0}$ (orange □ line) is remarkably enhanced with increasing disorder strength, directly accounting for the persistent gapless Dirac cone observed in Fig. 1b. The imaginary part of the self-energy, η, is crucial for understanding the non-quantization of σ_xy. η(E_F) depends on the energy E_F and determines the band broadening for each band. In the absence of disorder, when chemical potential lies within the magnetic gap ∣E_F∣ < ∣V₀∣, half-quantized Hall currents propagate on the gapless surface without dissipation⁶. However, disorder renormalizes the magnetic gap and abruptly triggers an energy level broadening η_top (orange △ line) for the top surface states at a critical value W_c ≈ 2.6 eV. Beyond W_c, this broadening η_top becomes more pronounced and rapidly exceeds multiples of the magnetic gap, as evidently demonstrated in Fig. 2b. The effective coupling between two chiral currents, arising from the impurity scattering-induced η_top, leads to quantization errors and ultimately completely destroying the conducting edge currents. Hence, when accounting for disorder effects, the condition for quantization of Hall conductivity should be modified as $| {\widetilde{E}}_{{{\rm{F}}}}| < | {\widetilde{V}}_{0}| -{\eta }_{{{\rm{top}}}}$. Especially, for E_F = 0, when the disorder exceeds the threshold value W_c (where $| {\widetilde{V}}_{0}| ={\eta }_{{{\rm{top}}}}({E}_{{{\rm{F}}}}=0)$), the band broadening induced by impurity scattering fills the magnetic gap, causing the Hall conductivity to deviate from its quantized value and ultimately decay to zero. We further compute σ_xy by integrating the Kubo–Bastin formula with the effective medium theory. Below the threshold W_c, η_top diminishes, yielding a distinct half-quantized plateau in σ_xy (blue ∘ line), as illustrated in Fig. 2b. The black dashed line marks the onset of η_top. As η_top further increases, σ_xy rapidly deviates from half-quantization simultaneously, thereby substantiating our preceding analysis. Figure 2a demonstrates the phase boundary (denoted by the white circle line), derived from the effective medium theory, which aligns quantitatively with numerical results. Ultimately, the effective medium theory elucidates the mechanism underlying the breakdown of HQHE.

MM and its transition to Anderson localization

To explore the quantum criticality in semimagnetic TI induced by disorder, we conduct a comprehensive investigation of localization physics by employing the transfer matrix method (see “Methods: Normalized localization length”). The transfer matrix method is used to calculate the localization length in a quasi-1D geometry, which consists of a bar with a cross-section of L_y × L_z in the transverse direction and infinite length L_x in the longitudinal direction. In the calculations, the thickness of the film is fixed at L_z = 10 and the width L_y is varied from 10 to 18. The normalized localization length Λ_x along the x-direction encodes the information of localization and delocalization. If Λ_x grows with the system size L_y, then in the thermodynamic limit L_y → + ∞, Λ_x becomes divergent, indicative of a metallic state. Conversely, it would become vanishing and describes a localized state. At the quantum critical point, Λ_x does not change with L_y and exhibits scale-invariant behavior as a marginal or critical metal. As depicted in Fig. 3, our analysis reveals that, for extremely strong disorder, the normalized localization length exhibits a decreasing trend with respect to system width L_y, thereby providing a definitive manifestation of Anderson localization. The unexpected discovery elicited significant surprise when it was observed that within the moderate disorder regime prior to the onset of the Anderson localization, the normalized localization length converges onto a single line, defying expectations of an isolated critical point, thus underscoring the unconventional scale-invariant behavior of the system in this regime. This finite scale-invariant region suggests the emergence of a MM⁴⁵ (also named critical metal⁴³) phase rather than a single critical point, delineating a unique and unconventional form of quantum criticality reminiscent of the Berezinskii–Kosterlitz–Thouless (BKT) phase transition^39,40,44,46. Further reducing the disorder strength (approximately below 5 eV), the normalized localization lengths increase drastically up to one or two orders, which far exceeds the system sizes of the cross-section and might indicate the potential limitations or unreliability of the approach. Instead, we shall use an alternative technique to tackle the problem in the weak disorder regime.

Fig. 3: The normalized localization length and the emergence of the marginal metal region in moderate disorder.

a The normalized localization length Λ_x along the x direction, where the error bars show standard deviations of Λ_x⁸⁵. The used longitudinal size is L_x = 2 × 10⁶. The thickness is L_z = 10, and we set the Fermi energy at E_F = 0.01 eV. In the marginal metallic phase, the normalized localization length exhibits a unique behavior by collapsing onto a single line and showing no size dependence, demonstrating its exotic criticality. b The normalized delocalization length near the transition point. c The fit of the numerical data near the transition point via a universal scaling function ${\Lambda }_{x}=F\left({L}_{y}/\xi \right)$. The transition points and the critical exponents are enumerated in the figure.

In order to delve deeper into the enigmatic disorder-induced quantum phase transition, we adhere to the traditional one-parameter scaling hypothesis and conduct finite-size scaling analysis in the vicinity of the transition point. The data for the normalized localization length in proximity to the transition point conform to a universal function ${\Lambda }_{x}=F\left({L}_{y}/\xi \right)$, which can be expressed through a Taylor series expansion up to the second order

$$\begin{array}{r}{\Lambda }_{x}={\Lambda }_{{{\rm{c}}}}+{c}_{1}{\left(\frac{{L}_{y}}{\xi }\right)}^{\nu }+{c}_{2}{\left(\frac{{L}_{y}}{\xi }\right)}^{2\nu },\end{array}$$

(2)

where ξ represents the correlation length and exhibits a divergent behavior within the quantum critical phase. Consequently, we interpret ξ as embodying the form $\xi =\exp \left[b/\sqrt{\frac{W-{W}_{{{\rm{c}}}}}{{W}_{{{\rm{c}}}}}}\right]$, which is highly evocative of the BKT criticality arising in the 2D XY spin model⁶³. b is a non-universal number that depends on the system parameters. Numerical investigations of BKT criticality in disorder-driven quantum phase transitions have been carried out in various systems, including 2D electron gas with random magnetic flux³⁹, Graphene with long-range impurities⁴⁰, ${C}_{4z}{{\mathcal{T}}}$ network model⁴³, 2D random SU(2) electron gas in a constant magnetic field⁴⁴, 2D Rashba electron gas with a Zeeman split⁴⁵, and spin Chern insulator with structure inversion asymmetric impurity potential⁴⁶, among others. Our finite-size scaling analysis yields the exponent ν = 0.176 ± 0.005 for the MM-to-AI transition as shown in Fig. 3, which raises an open question regarding its universality. Further investigation into the application of the renormalization group calculations would be merited in order to systematically derive the exponents, thereby shedding light on the underlying mechanism governing this phenomenon.

Quantum interference effect and phase transition

We now investigate the metallicity of the system through the lens of longitudinal conductivity σ_xx. Unlike the Hall conductivity, σ_xx is primarily determined by the scattering of electrons near the Fermi surface, making it a sensitive probe of localization behavior at E_F. The Ioffe–Regel criterion, k_Fl ≲ 1, defines a strong scattering regime where the nature of wave transport remains ambiguous⁶⁴. Here k_F = E_F/ℏλ_∥ is the Fermi wave number, and l is the mean free path. The condition k_Fl ~ 1 is often associated with the onset of Anderson localization, while k_Fl > 1 corresponds to a weak scattering regime. In the latter, quantum interference between time-reversal symmetric counter-propagating paths can lead to quantum corrections in σ_xx, resulting in weak localization (WL) or WAL, depending on the system’s symmetry^65,66,67.

To explore the quantum transport properties, we employ the quantum interference theory to evaluate the lowest-order quantum correction δσ_qi to the conductivity, which is valid in the weak scattering regime (see Supplementary Note 4 for detailed calculations).

(i) When E_F lies within the magnetic gap of the top (t) surface states and only intersects the bottom (b) surface states (E_F < V₀), the problem simplifies to the localization of a single gapless Dirac Hamiltonian. The quantum correction is found to scale with the system size L as: $\delta {\sigma }_{qi}\simeq \frac{{e}^{2}}{\pi h}\ln (L/{l}_{e})$. The degeneracy point in momentum space acts as a Dirac monopole, causing the wave functions to acquire a π phase shift when transported around the Fermi surface in the Berry connection. This results in destructive quantum interference, leading to WAL. In the WAL regime, σ_xx exhibits a logarithmic divergence with system size L with α_L = +1 (Fig. 4a, b).

Fig. 4: Results of quantum interference corrections.

Dependence of ${\alpha }_{L}=\frac{\pi h}{{e}^{2}}\frac{\partial \sigma }{\partial \ln L}$ on E_F/V₀ for different inter-surface scattering strength a ζ = 0 and b ζ = 0.1 with L/l_m varied and ${l}_{m}=\frac{{\lambda }_{\parallel }^{2}}{{V}_{0}{U}_{0}}$. The characteristic length scales ${({l}_{e}^{s}/{l}^{s})}^{2}$ for two Cooperon channels (s = t, b) as a function of V₀/E_F for c ζ = 0 and d ζ = 0.1. Here, ${l}_{e}^{s}$ represents the mean free paths for the respective channels.

(ii) When E_F lies outside the magnetic gap, it intersects two bands of the surface states. As shown in Supplementary Note 4, the full expression for α_L can be expressed as a sum over two Cooperon channels: ${\alpha }_{L}={\sum}_{s=t,b}\frac{{\alpha }^{s}}{{(L/{l}^{s})}^{2}+1}$ where α^s and l^s are the weighting factors and characteristic lengths associated with the two channels, respectively. Here, l^s measures the suppression of quantum interference effects due to dephasing or symmetry-breaking mechanisms. It determines the decay length of the Cooperon mode: a smaller l^s corresponds to stronger suppression of quantum interference, while a larger l^s allows for more pronounced interference effects. In the absence of inter-surface scattering, the two surface states are independent, and the total quantum correction is simply the sum of the individual contributions from the two surface states. The bottom surface exhibits genuine WAL with α^b = 1 and l^t → ∞, shown as the red line in Fig. 4c. In contrast, the top surface states show a crossover from WL (α^t = − 1, l^t → ∞) to WAL (α^t = + 1, l^t → ∞) as E_F varies from band edge (E_F ~ V₀) to high energy (∣E_F∣ ≫ ∣V₀∣). This occurs because, in these two limiting regimes, an orthogonal type and a symplectic type of time-reversal symmetry are recovered on the Fermi surface, even though the magnetic gap explicitly breaks time-reversal symmetry. In other cases, the bottom surface channel exhibits suppressed WL or WAL with finite l^t. As L increases, the contribution from the top surface channel is suppressed following $\sim 1/[{(L/{l}^{t})}^{2}+1]$. In the thermodynamic limit, only the top surface channel survives, and α_L approaches  + 1 (except at E_F/V₀ = 1). In the presence of inter-surface scattering, the scaling behavior of α_L becomes qualitatively different. Inter-surface scattering introduces a finite Cooperon gap to the bottom surface’s Cooperon channel, resulting in a finite l^b, as shown in Fig. 4d. Consequently, the contribution from the bottom channel is also suppressed, following $\sim 1/[{(L/{l}^{b})}^{2}+1]$. In this case, for E_F/V₀ > 1, α_L approaches ${\sum}_{i=t,b}{\alpha }^{i}/[{(L/{l}^{i})}^{2}+1]\to 0$ instead of 1 in the limit of large L, as illustrated by the dashed line. We conclude that this crossover behavior in the scaling of α_L around V₀ arises due to inter-surface scattering.

When E_F < ∣V₀∣, the π Berry phase, inherent to a single gapless Dirac cone, serves as the foundation for the interconnected phenomena of WAL and the HQHE. Arising from the Dirac cone’s topological structure, the π Berry phase drives WAL by enhancing conductivity through quantum interference in the presence of spin–orbit coupling^66,67. Simultaneously, the anomalous Hall conductivity in metallic systems can be linked to the Berry phase accumulated over the Fermi surface loop, enabling the HQHE under broken time-reversal symmetry³⁸. Together, these concepts underscore the profound relationship between topology, quantum interference, and electronic transport in topological metallic systems, setting them apart from the integer quantum Hall effect, which is exclusively associated with insulating phases. When E_F intersects two surface state bands, we confirm that inter-surface scattering events introduce finite Cooperon gaps across all channels, leading to a strong suppression of quantum interference corrections. To conclusively determine the phase in the thermodynamic limit, high-loop quantum correction calculations are necessary. Recent numerical studies have shown that the MM can emerge in 2D ferromagnetic electron gases with random scalar potentials and Rashba spin–orbit interactions, where time-reversal symmetry breaking, spin-orbit coupling, and interband scattering are essential ingredients⁴⁵. In contrast, we demonstrate a distinct scenario where the MM phase is mediated by residual Berry curvature and weak inter-surface scattering, highlighting a different mechanism for realizing this phase in disordered semimagnetic TIs. For a finite system of size L, the correlation length ξ is bounded by L. Near a critical point, temperature T and sample size L can be related through scaling arguments: ${{\rm{d}}}\ln L/{{\rm{d}}}\ln T=-{\nu }_{T}$ with ν_T as the critical exponent, which reflects how the temperature scales relative to the sample size. In the MM phase, where α = 0, the longitudinal conductivity σ_xx becomes temperature-independent. This distinctive feature can serve as a key experimental signature for identifying the MM phase. Finally, we emphasize that while reducing the film thickness can enhance inter-surface scattering, thereby making the observation of the MM phase in ultrathin films more promising, the HQHM phase tends to be suppressed, as the surface Dirac cones are more likely to develop a finite-size energy gap⁶⁸.

The arithmetic and geometric mean density of states (DOS)

The emergence of the metallic phases in semimagnetic TI can be characterized by analyzing the two types of means of the local DOS: the arithmetic mean ρ_a and the geometric mean ρ_t (see “Methods: Density of states”). The local DOS, ${\rho }_{{{\bf{r}}},\alpha }(E)=\left\langle {{\bf{r}}},\alpha \right\vert \delta \left(E-H\right)\left\vert {{\bf{r}}},\alpha \right\rangle$, quantifies the amplitude of the wave function at site r for a given energy E, where $\left\vert {{\bf{r}}},\alpha \right\rangle$ denotes an α-orbital electron wave function at that site. The spatial distribution of ρ_r,α(E) contains direct information about the localization properties, which are closely intertwined with the topology of the quantum system^28,69,70. The arithmetic and geometric mean DOS for a disordered system are defined as ${\rho }_{{{\rm{a}}}}(E)={\left\langle \frac{1}{V}\mathop{\sum }\limits_{i = 1}^{V}\mathop{\sum }\limits_{\alpha = 1}^{4}{\rho }_{{{{\bf{r}}}}_{i},\alpha }(E)\right\rangle }_{{{\rm{imp}}}}$ and ${\rho }_{{{\rm{t}}}}(E)=\exp \left[\frac{1}{{N}_{s}}\mathop{\sum }\limits_{i=1}^{{N}_{s}}{\left\langle \ln \mathop{\sum }\limits_{\alpha = 1}^{4}{\rho }_{{{{\bf{r}}}}_{i},\alpha }(E)\right\rangle }_{{{\rm{imp}}}}\right]$, respectively^71,72. A zero value of arithmetic mean DOS ρ_a(E) at a finite interval defines an energy gap, such as a band insulator, while a nonzero value of ρ_a(E) means that there exist states at the energy E, which can be either localized or delocalized. The geometric mean DOS ρ_t(E) may reveal more information for the localization of the states. For energy E where all states are extended, local DOS are uniformly distributed throughout the system, and we have ρ_t ≃ ρ_a. However, for the states whose wave-functions are localized in real space, there is no contribution to ρ_t⁷³. Generally, comparison of ρ_a(E) to ρ_t(E) can investigate the localization properties of particle states. In the case of quasi-2D TI films featuring surface states, the analysis of these properties becomes more complex (see Supplementary Note 2 for detailed analysis). Further study shows that since the surface states are localized near the top or bottom surface in z direction and extended in x and y direction, these states do not contribute to the value of ρ_t, but do to ρ_a. For a non-vanishing ρ_a, if the ratio ρ_t/ρ_a is 0, it can define two different phases: one is that all states are localized in real space in all directions, i.e., AI phase, and another one is that the states are only localized at the top or bottom surface, i.e., the surface states. For a finite ratio of ρ_t/ρ_a, it indicates the existence of delocalized states, defining a metallic phase.

Figure 5 exhibits the arithmetic and geometric DOS as a function of disorder at a fixed chemical potential and as a function of the chemical potential at a fixed disorder, respectively. In Fig. 5a, we set E_F = 0.01eV, which is close to the Dirac point. The vanishing ρ_a(E) to ρ_t(E) at weak disorder indicates that the surface states are localized near one surface in HQHM. The disorder-induced finite DOS appears in a moderate disorder, serving as critical evidence for the MM phase. Both ρ_a(E) and ρ_t(E) tend to vanish for a strong disorder, demonstrating the localization of electrons. Specifically, we find that ρ_t(E) exhibits a nearly linearly decay behavior when approaching the MM–AI phase boundary. Therefore, we linearly fit the ρ_t(E) data for W from 7 eV to 12 eV and extrapolate to the intersection with the horizontal axis at W ≈ 13.47 eV. We expect that in the thermodynamic limit, ρ_t(E) is finite for the MM phase, and vanishes for the AI phase. In Fig. 5b, we take a weak disorder W = 1.0 eV. The geometric mean DOS ρ_t(E) is almost zero for a chemical potential residing around the Dirac point (E < 0.06 eV), consistent with that in Fig. 5a. The DOS ρ_t(E) becomes finite when the chemical potential intersects with the bands of the massive Dirac fermions, indicating the metallicity of the system even in the presence of disorder in this quasi-2D system. The DOS ρ_t(E) approaches zero again when the Fermi level is close to the top of the band (about 5.91 eV), indicating the localization of the electrons. The calculated DOSs provide alternative and substantial evidence to support the phase diagram in Fig. 1. Some key features are summarized in Table 1.

Fig. 5: The arithmetic and geometric mean DOS ρ_a(E) and ρ_t(E).

a The DOS as the function of disorder strength W at the Fermi energy E_F = 0.01 eV, revealing the phase transitions between half-quantized Hall metal (HQHM), marginal metal (MM), and Anderson insulator (AI). The cyan dashed line fitted to ρ_t intersects the horizontal axis at the hexagonal marker (W = 13.47 eV), thereby demarcating the phase boundary between MM and AI. b The DOS as a function of the Fermi energy E_F at W = 1.0 eV. The insets show an enlarged view of the band center and band edge.

Table 1 Summary of the key characteristics of the distinct phases in semimagnetic topological insulator films

Model and material realization

We adopt the tight-binding model on a cubic lattice for the semi-magnetic TI,

$${H}_{0}=\mathop{\sum}\limits_{{{{\bf{r}}}}_{i}}{\Psi }_{{{{\bf{r}}}}_{i}}^{{\dagger} }({M}_{0}+{V}_{z}){\Psi }_{{{{\bf{r}}}}_{i}}+\mathop{\sum}\limits_{{{{\bf{r}}}}_{i},\alpha =x,y,z}({\Psi }_{{{{\bf{r}}}}_{i}}^{{\dagger} }{{{\mathcal{T}}}}_{\alpha }{\Psi }_{{{{\bf{r}}}}_{i}+{{{\bf{e}}}}_{\alpha }}+{{\rm{H.c.}}}),$$

(3)

where ${{{\mathcal{T}}}}_{\alpha }={t}_{\alpha }{\tau }_{z}{\sigma }_{0}-\frac{{{\rm{i}}}{\lambda }_{\alpha }}{2}{\tau }_{x}{\sigma }_{\alpha }$ and ${M}_{0}=\left({m}_{0}-4{t}_{\parallel }-2{t}_{z}\right){\tau }_{z}{\sigma }_{0}$ with the lattice size L_x × L_y × L_z^15,74,75. ${\Psi }_{{{{\bf{r}}}}_{i}}^{{\dagger} }$ and ${\Psi }_{{{{\bf{r}}}}_{i}}$ are creation and annihilation operators of an electron at site i. τ_α and σ_α’s are Pauli matrices acting on the orbital and spin spaces, respectively. The magnetic doping is modeled by introducing a Zeeman potential V_z = V_z(i_z)τ₀σ_z. V_z(i_z) = V₀ for the top layers ${i}_{z}\leqslant {L}_{z}^{{{\rm{Mag}}}}$, and 0 otherwise. The model in Eq. (1) incorporating four orbitals, $\left\vert P{1}_{-}^{+},\pm \frac{1}{2}\right\rangle$ and $\left\vert P{2}_{+}^{-},\pm \frac{1}{2}\right\rangle$, was proposed to describe the topological nature of strong TI Bi₂Se₃ and Bi₂Te₃ by taking the bulk band gap m₀ = 0.28eV. The model has been extensively used to explore the physics of strong topological materials. Finally, we follow the common practice in the study of Anderson localization and introduce the Anderson disorder ${H}_{{{\rm{imp}}}}={\sum}_{{{{\bf{r}}}}_{i}}{\Psi }_{{{{\bf{r}}}}_{i}}^{{\dagger} }{u}_{{{{\bf{r}}}}_{i}}{\tau }_{0}{\sigma }_{0}{\Psi }_{{{{\bf{r}}}}_{i}}$ through random on-site energy ${u}_{{{{\bf{r}}}}_{i}}$ that is uniformly distributed in $\left[-W/2,+W/2\right]$.

The semimagnetic TI has already been realized experimentally in Cr-doped (Bi, Sb)₂Te₃, in which the half-quantized Hall conductance was measured²¹.

Discussion

Our primary analysis focuses on short-ranged non-magnetic impurities, and we have also examined the robustness of the HQHE against other disorder types. The robustness of the HQHE is intrinsically linked to the protection of the gapless Dirac cone in the system. For magnetic impurities, the impact depends critically on the spin symmetry of the disorder: (i) Spin-flip in-plane magnetic disorders (e.g., σ_x and σ_y-type) effectively shift the Dirac cone position while preserving its gapless nature, allowing the HQHE to remain robust. (ii) In contrast, σ_z-type out-of-plane magnetic disorder opens a gap in the Dirac cone, introducing fluctuations in the Hall response. However, since the energy gap induced by σ_z-type impurities vanishes under disorder averaging, the disorder-averaged Hall conductivity still maintains its half-quantized value⁷⁶. This demonstrates that while local fluctuations may occur, the topological protection of the HQHE survives when disorder preserves the global-averaged time-reversal symmetry. These results highlight the remarkable stability of the HQHE against various disorder types, provided they maintain the essential gapless nature of the Dirac spectrum either locally or on average.

We employ a combination of numerical and analytical methods to determine the phase diagram of a disordered semimagnetic TI, each with its own merits and limitations. (i) The calculation of the Hall conductivity, combined with effective medium theory, establishes the full phase boundary between the HQHM and MM phase in the (W, E_F) plane. These phases exhibit distinct size-scaling behaviors: the HQHE scales to $\frac{1}{2}\frac{{e}^{2}}{h}$ in the large system size limit (L → ∞), while the MM phase converges to a finite value below $\frac{1}{2}\frac{{e}^{2}}{h}$. Effective medium theory further supports this transition, showing that the Hall conductivity deviates from $\frac{1}{2}\frac{{e}^{2}}{h}$ as the system enters the MM phase due to the emergence of an imaginary part in the self-energy. It is also further supported by the calculation of the DOS. (ii) Quantum interference calculations, based on a quasiparticle picture, are valid for weak disorder and applicable to few-band systems. They confirm the phase transition between the HQHE metallic phase, where E_F intersects a single gapless Dirac cone, and the MM phase, where E_F intersects two surface state bands at weak disorder. The transition point is consistent with that of the Hall conductivity. (iii) Localization length calculations using the transfer matrix method are particularly effective in strongly disordered regimes, where the localization length is small. This method accurately captures the transition from the scale-invariant MM phase to the AI phase. (iv) To probe the localization nature of bulk and surface states, we compute the arithmetic mean (ρ_a) and geometric mean (ρ_t) DOS. For weak disorder, surface states localized on the surface of the quasi-2D film exhibit a vanishingly small ρ_t, distinct from conventional extended states in purely 2D systems. In the MM phase, bulk-extended states lead to a significant increase in ρ_t. By integrating all these results, we believe that a comprehensive and reliable phase diagram has been established for the disordered semimagnetic TI.

In purely 2D systems, Anderson localization depends critically on the symmetry class of the system. These symmetries are broadly categorized into three main classes: orthogonal, unitary, and symplectic²⁹. In the orthogonal and unitary symmetry classes, all states in 2D are localized for any amount of disorder^77,78. However, in the symplectic class, which preserves time-reversal symmetry and includes strong spin-orbit coupling, the system exhibits a metal–insulator transition as a function of disorder strength^50,79. This transition arises due to WAL effects, which counteract localization and allow for extended states at critical disorder strengths. The integer quantum Hall transition further enriches our understanding of localization by introducing the interplay between Landau level quantization, disorder, and topology⁸⁰. In the integer quantum Hall regime, extended states emerge at critical energies within Landau levels, while all other states remain localized. This transition is characterized by a change in the topological invariant (Chern number) and is fundamentally distinct from conventional Anderson localization. The integer quantum Hall transition exemplifies how topology can stabilize extended states even in the presence of disorder. The behavior of semimagnetic TI films, however, presents a unique and complex scenario that defies simple classification. At first glance, one might assume that magnetic doping on the surface breaks the global time-reversal symmetry, placing the system in the unitary symmetry class, where all states would be localized for any amount of disorder. However, this is not the case. The quasi-2D nature of TI films, combined with their nontrivial topological properties, leads to distinct features that differentiate them from purely 2D systems. The surface state on the undoped side of the film retains higher symmetry (symplectic time-reversal symmetry), protecting it from complete localization in weak disorder. For high-energy states that extend into the bulk and interact with the magnetic surface, the symmetry is effectively lowered, causing these states to become localized. In the intermediate energy regime, instead of a single critical point, there exists a finite region where critical extended states can emerge. As a result, the localization behavior in semimagnetic TI films depends strongly on the position of the chemical potential. Different energy regimes exhibit distinct local symmetries, leading to varied localization behaviors. The interplay between magnetic doping, topological protection, and spatial symmetry breaking leads to a rich variety of localization behaviors, distinct from those in conventional 2D systems.

Methods

The non-commutative Kubo formula for the Hall conductivity

The Hall conductivity of a disordered system on a lattice can be calculated using the non-commutative Kubo formula⁴⁷

$${\sigma }_{xy}=\frac{{e}^{2}}{h}{\left\langle 2\pi {{\rm{i}}}{{\rm{Tr}}}\left\{P\left[-{{\rm{i}}}\left[x,P\right],-{{\rm{i}}}\left[y,P\right]\right]\right\}\right\rangle }_{{{\rm{imp}}}},$$

(4)

where P represents the projector onto the occupied states, and x and y are the coordinate operators on the lattice, respectively. ${\left\langle \cdots \right\rangle }_{{{\rm{imp}}}}$ denotes the average over the disorder sampling. We apply the periodic boundary condition in both the x and y directions in order to minimize the boundary effect while adopting an open boundary condition in the z direction for a thin film.

Density of states

The spectral function is defined as

$$\begin{array}{r}A\left(\epsilon ,{{\bf{k}}}\right)=\mathop{\sum}\limits_{n}\left\langle n{{\bf{k}}}\right\vert \delta \left(\epsilon -H\right)\left\vert n{{\bf{k}}}\right\rangle ,\end{array}$$

(5)

where H = H₀ + H_imp is the total Hamiltonian including the impurity potential H_imp in real space. $\left\vert n{{\bf{k}}}\right\rangle$ is the eigenstate of the Hamiltonian H₀. To overcome the computational complexity for a large system, we implement the Chebyshev polynomial expansion of $A\left(\epsilon ,{{\bf{k}}}\right)$ employing the kernel polynomial method^73,81:

$$\begin{array}{r}A\left(\epsilon ,{{\bf{k}}}\right)=\frac{1}{\pi {\epsilon }_{\max }\sqrt{1-{\widetilde{\epsilon }}^{2}}}\mathop{\sum}\limits_{n}\mathop{\sum}\limits_{m=0}^{M-1}\left(2-{\delta }_{m,0}\right){g}_{m}^{{{\rm{J}}}}{T}_{m}\left(\widetilde{\epsilon }\right)\left\langle n{{\bf{k}}}\right\vert {T}_{m}\left(\widetilde{H}\right)\left\vert n{{\bf{k}}}\right\rangle ,\end{array}$$

(6)

in which ${\epsilon }_{\max }$ is chosen as an adequately large energy scale ensuring that $\widetilde{\epsilon }=\epsilon /{\epsilon }_{\max }$ and eigenvalues of $\widetilde{H}=H/{\epsilon }_{\max }$ lie within the domain of Chebyshev polynomials of the first kind ${T}_{m}\left(x\right)=\cos \left(m\arccos x\right)$. The Jackson damping kernel

$$\begin{array}{r}{g}_{m}^{{{\rm{J}}}}=\left(1-\frac{m}{M+1}\right)\cos \frac{m\pi }{M+1}+\frac{1}{M+1}\sin \frac{m\pi }{M+1}\cot \frac{\pi }{M+1}\end{array}$$

(7)

serves to mitigate the Gibbs oscillation arising from truncating the first M term. Besides, the Jackson damping kernel has been identified as the optimal choice for expanding the quantum resolution operator, which is essentially a series of Dirac δ peaks, as it minimizes broadening for a given value of M. More specifically, away from $\widetilde{\epsilon }=\pm 1$ it approximates the Dirac δ distribution with a Gaussian distribution with a width of $\delta \epsilon =\pi {\epsilon }_{\max }/M$. In this work, we take a large lattice size L_x = L_y = 400, and M = 6000 to achieve high resolution. Other parameters are the same as those in Fig. 2.

The localization properties can be characterized by analyzing the two types of means of the local DOS: the arithmetic mean ρ_a and the geometric mean ρ_t. The local DOS, ${\rho }_{{{\bf{r}}},\alpha }(\epsilon )=\left\langle {{\bf{r}}},\alpha \right\vert \delta \left(\epsilon -H\right)\left\vert {{\bf{r}}},\alpha \right\rangle$, quantifies the amplitude of the wave function at site r for a given energy ϵ, where $\left\vert {{\bf{r}}},\alpha \right\rangle$ denotes an α-orbital electron wave function at that site. We derive the local DOS ρ_r,α(ϵ) via substitution into the aforementioned Chebyshev polynomial expansion, replacing the Bloch plane wave state $\left\vert n{{\bf{k}}}\right\rangle$ with a localized site state $\left\vert {{\bf{r}}},\alpha \right\rangle$. The arithmetic and geometric mean DOS for a disordered system are defined as ${\rho }_{{{\rm{a}}}}(\epsilon )={\left\langle \frac{1}{V}\mathop{\sum }\limits_{i = 1}^{V}\mathop{\sum }\limits_{\alpha = 1}^{4}{\rho }_{{{{\bf{r}}}}_{i},\alpha }(\epsilon )\right\rangle }_{{{\rm{imp}}}}$ and ${\rho }_{{{\rm{t}}}}(\epsilon )=\exp \left[\frac{1}{{N}_{s}}\mathop{\sum }\limits_{i=1}^{{N}_{s}}{\left\langle \ln \mathop{\sum }\limits_{\alpha = 1}^{4}{\rho }_{{{{\bf{r}}}}_{i},\alpha }(\epsilon )\right\rangle }_{{{\rm{imp}}}}\right]$, respectively^71,72. Here we randomly choose a finite number of lattice sites N_s ≪ V = L_xL_yL_z to improve the statistics of ρ_t⁸². Other parameters are the same as the calculation of spectral functions.

Normalized localization length

The normalized localization length serves as a fundamental metric that underlies research on the physics of Anderson localization^83,84,85. In this approach, the disordered sample is fabricated into a lengthy quasi-1D geometry along the x direction with a finite cross-section L_y × L_z, and the Schrödinger equation is rearranged into the following form:

$$\begin{array}{r}\left[\begin{array}{c}{\psi }_{n+1}\\ {\psi }_{n}\end{array}\right]={T}_{n}\left[\begin{array}{c}{\psi }_{n}\\ {\psi }_{n-1}\end{array}\right],\end{array}$$

(8)

where ψ_n is the wavefunction of the n-th slice along the longitudinal direction, and T_n is the transfer matrix defined as:

$$\begin{array}{r}{T}_{n}=\left[\begin{array}{cc}{H}_{n,n+1}^{-1}\left({E}_{{{\rm{F}}}}-{H}_{n,n}\right)&-{H}_{n,n+1}^{-1}{H}_{n-1,n}^{{\dagger} }\\ {I}_{n}&0\end{array}\right].\end{array}$$

(9)

H_n,n is the Hamiltonian of the n-th slice, and H_n,n+1 is the hopping matrix between the n-th and $\left(n+1\right)$-th slice. I_n is an identity matrix. Owing to the disorder, the wave function decays exponentially as ${\psi }_{n,i} \sim {{{\rm{e}}}}^{\pm n/{\lambda }_{i}}$, and the localization length λ_i can be extracted as follows. First, we define a consecutive product ${O}_{{L}_{x}}={\prod }_{n = 1}^{{L}_{x}}{T}_{n}$ that transforms the wave function of the first slice to the L_x-th slice. According to the Oseledec’s theorem, the limit $O=\mathop{\lim }\limits_{{L}_{x}\to +\infty }{\left({O}_{{L}_{x}}^{{\dagger} }{O}_{{L}_{x}}\right)}^{\frac{1}{2{L}_{x}}}$ exists and its eigenvalues take on the form of $\left\{{{{\rm{e}}}}^{\pm {\gamma }_{i}}\right\}$. γ_i’s are called Lyapunov exponents. The smallest positive Lyapunov exponent is found to be inversely related to the largest localization length through the expression λ_x = 1/γ. Ultimately, the normalized localization length is defined as Λ_x = λ_x/L_y(z), depending on which quantity is changed. Furthermore, it is crucial to introduce numerical stabilization after every few iterations of transfer matrix multiplication, as accumulated round-off errors can corrupt the calculation of Lyapunov exponents⁸⁵.

Effective medium theory

To understand the origin and robustness of the HQHM in disorder, we can employ the effective medium theory in conjunction with the Kubo formula for electric conductivities^51,52. The self-consistent Born approximation is a very useful tool to investigate the physics in topological AI^{53,54,55,56,57,58,59,60,61,62}. In the semi-magnetic structure with multiple layers, the electron wave function has a layer degree of freedom, which leads to a matrix structure of the Green’s function and self-energy. We extend the self-consistent Born approximation to the layered structure (see Supplementary Note 3 for details), within which the retarded self-energy is diagonal in the layer degree of freedom subspace and can be expressed as: ${\Sigma }_{{i}_{z}{i}_{z}}^{R}({E}_{{{\rm{F}}}})=\frac{{W}^{2}}{12S}{\sum}_{{{\bf{k}}}^{\prime} }{G}_{{i}_{z}{i}_{z}}^{R}({{\bf{k}}}^{\prime} ,{E}_{{{\rm{F}}}}),$ where i_z is the layer number, ${G}^{R}({{\bf{k}}}^{\prime} ,{E}_{{{\rm{F}}}})={\left[{E}_{{{\rm{F}}}}-{H}_{{{\rm{0}}}}({{\bf{k}}}^{\prime} )-{\Sigma }^{R}\right]}^{-1}$ is the dressed Green’s function. Despite the translational symmetry breaking in the z direction, our scheme is still applicable. The self-consistent equations for self-energy can be solved numerically⁶¹. With the help of the self-energy, the Hall conductivity can be calculated by means of the Kubo–Bastin formula:

$$\begin{array}{rl}{\sigma }_{xy}&=\int_{-\infty }^{+\infty }{{\rm{d}}}\epsilon {n}_{{{\rm{F}}}}\int\frac{{{{\rm{d}}}}^{2}{{\bf{k}}}}{{(2\pi )}^{2}}{{\rm{Tr}}}\left[\left({v}_{x}\frac{{{\rm{d}}}{G}^{R}}{{{\rm{d}}}\epsilon }{v}_{y}-{v}_{y}\frac{{{\rm{d}}}{G}^{A}}{{{\rm{d}}}\epsilon }{v}_{x}\right)\left({G}^{R}-{G}^{A}\right)\right],\end{array}$$

(10)

where ${G}^{R}({{\bf{k}}},\epsilon )={\left[\epsilon -H({{\bf{k}}})-{\Sigma }^{R}({{\bf{k}}},\epsilon )\right]}^{-1}$. Here, the self-energy Σ^R(k, ϵ) is approximated as Σ^R(k, E_F) that is numerically obtained in the effective medium theory, and ${v}_{x/y}=\frac{\partial H}{\partial {k}_{x/y}}$ are the velocity operators. ${n}_{{{\rm{F}}}}=1/({{{\rm{e}}}}^{(\epsilon -{E}_{{{\rm{F}}}})/T}+1)$ is the Fermi-Dirac distribution. In the effective medium theory, the translational invariance in each layer is restored, allowing the momentum k to be well-defined and conserved. ${{\rm{Tr}}}$ denotes the trace, which is performed over the degrees of freedom associated with layers, spin, and orbital.

Quantum interference theory: quantum corrections to conductivity

The quantum interference correction to conductivity arises from the coherent backscattering of electrons due to impurity scattering, leading to either WL or WAL, depending on the system’s symmetries and the nature of the scattering processes. To calculate the correction, the diagrammatic approach has been developed to incorporate disorder-averaged Green’s functions, vertex corrections, and Cooperon modes. Consider the low-energy effective Hamiltonian H = H_t ⊕ H_b, which describes the top (H_t) and bottom (H_b) surface states of the system. A disorder potential including both intra-surface scattering U₀ and inter-surface scattering U₁ is taken into account. The disorder is modeled as short-range and characterized by the correlation functions: ${\langle {U}_{i}({{\bf{r}}}){U}_{j}({{{\bf{r}}}}^{{\prime} })\rangle }_{dis}={U}_{i}^{2}{\delta }_{ij}\delta ({{\bf{r}}}-{{{\bf{r}}}}^{{\prime} })$ where i, j = 0, 1 represent intra-surface and inter-surface scattering, respectively, and 〈…〉_dis denotes the disorder averaging. The total quantum correction to conductivity, σ_qi, is contributed by both intra-surface and inter-surface Cooperon channels.

The intra-surface contribution is evaluated by means of the dressed Hikami box formalism, which accounts for both bare and dressed vertex corrections. The bare Hikami box represents the leading-order quantum correction (Fig. 6d), while the dressed ones (Fig. 6e) include higher-order corrections arising from multiple scattering events. In systems with strong spin–orbit coupling, such as TIs, the dressed Hikami box contribution is significant and often carries an opposite sign relative to the bare contribution, leading to a suppression or enhancement of the quantum interference effect. The inter-surface contribution, on the other hand, is influenced by the inter-surface scattering potential U₁, which couples the top and bottom surface states. The contribution is particularly important in thin films where the two surfaces are in close proximity, allowing for significant inter-surface scattering.

Fig. 6: Feynman diagrams for evaluating quantum corrections to the longitudinal conductivity.

a Diagrammatic representation of the bare scattering vertex for intra- and inter-surface Cooperon channels. The blue and red circles represent the Fermi surfaces for the bottom and top surface states, respectively. Yellow and black arrows denote the incoming and outgoing momenta. Diagrammatic representation of the self-consistent equation for b the dressed vertex and c for the Cooperon. Leading-order quantum correction to conductivity from d the bare Hikami box and e the dressed Hikami box.

The Cooperon functions are solved self-consistently via the Bethe–Salpeter equation as illustrated in Fig. 6c:

$${\Gamma }_{s{s}^{{\prime} }}^{s{s}^{{\prime} }}({\theta }_{{{\bf{k}}}},{\theta }_{{{{\bf{k}}}}^{{\prime} }},{{\bf{q}}})= \, {\gamma }_{s{s}^{{\prime} }}^{s{s}^{{\prime} }}({\theta }_{{{\bf{k}}}},{\theta }_{{{{\bf{k}}}}^{{\prime} }})+\int\frac{{d}^{2}{{{\bf{k}}}}^{{\prime}\prime}}{{(2\pi )}^{2}}\mathop{\sum}\limits_{{s}^{{\prime}\prime}=t,b}{\gamma }_{s{s}^{{\prime}\prime}}^{s{s}^{{\prime}\prime}}({\theta }_{{{\bf{k}}}},{\theta }_{{{{\bf{k}}}}^{{\prime}\prime}})\\ \times {G}_{{s}^{{\prime}\prime}}^{R}({{{\bf{k}}}}^{{\prime}\prime}){G}_{{s}^{{\prime}\prime}}^{A}({{\bf{q}}}-{{{\bf{k}}}}^{{\prime}\prime}){\Gamma }_{{s}^{{\prime}\prime}{s}^{{\prime} }}^{{s}^{{\prime}\prime}{s}^{{\prime} }}({\theta }_{{{{\bf{k}}}}^{{\prime}\prime}},{\theta }_{{{{\bf{k}}}}^{{\prime} }},{{\bf{q}}}),$$

(11)

where $s,{s}^{{\prime} }=t,b$ represent the top and bottom surfaces, θ_k and ${\theta }_{{{{\bf{k}}}}^{{\prime} }}$ label the incoming and outgoing momenta on Fermi surfaces s and ${s}^{{\prime} }$, respectively, and ${\gamma }_{s{s}^{{\prime} }}^{s{s}^{{\prime} }}$ is the bare scattering vertex as illustrated diagrammatically in Fig. 6a. The Cooperon functions ${\Gamma }_{s{s}^{{\prime} }}^{s{s}^{{\prime} }}$ are categorized into intra-surface modes (e.g., ${\Gamma }_{tt}^{tt}$ and ${\Gamma }_{bb}^{bb}$) and inter-surface modes (e.g., ${\Gamma }_{tb}^{tb}$ and ${\Gamma }_{bt}^{bt}$). The intra-surface Cooperon modes dominate when the scattering is confined to one surface, while the inter-surface modes become relevant when the inter-surface scattering is significant. The functions ${\Gamma }_{tb}^{bt}$ and ${\Gamma }_{bt}^{tb}$ describe the processes where the two outgoing momenta lie on distinct Fermi surfaces. As a result, their total momentum summation cannot vanish, making them irrelevant for the low-energy Cooperon modes. After performing tedious calculations, as outlined in Supplementary Note 4, the total quantum correction to conductivity is expressed as:

$$\begin{array}{r}{\sigma }_{qi}=\frac{{e}^{2}}{2\pi h}\left({\alpha }^{t}\ln \frac{{({l}_{e}^{t})}^{-2}+{({l}^{t})}^{-2}}{{L}^{-2}+{({l}^{t})}^{-2}}+{\alpha }^{b}\ln \frac{{({l}_{e}^{b})}^{-2}+{({l}^{b})}^{-2}}{{L}^{-2}+{({l}^{b})}^{-2}}\right),\end{array}$$

(12)

where the coefficients α^t and α^b are defined as

$${\alpha }^{t}= \, [({w}_{0}^{t}+{w}_{2}^{t})(-1+{\varpi }_{tra,0}^{t})+{w}_{1}^{t}(1-{\varpi }_{tra,1}^{t})]{\eta }_{t}^{2}\\ +[{w}_{1}^{t}{\varpi }_{ter,1}^{t}-({w}_{2}^{t}{\varpi }_{ter,2}^{t}+{w}_{0}^{t}{\varpi }_{ter,0}^{t})]{\eta }_{t}{\eta }_{b},\\ {\alpha }^{b}= \, {w}_{1}^{b}(1-{\varpi }_{tra}^{b}){\eta }_{b}^{2}+{w}_{1}^{b}{\varpi }_{ter}^{b}{\eta }_{t}{\eta }_{b}.$$

(13)

The total quantum interference correction can be organized into two distinct contributions, corresponding to the top (t) and bottom (b) surface states. Each of these contributions includes both intra-surface and inter-surface Cooperon channel effects. Here, ${w}_{i}^{s}$ are the weight factors for each Cooperon function with angular momentum i = 0, 1, 2 and the surface s = t, b. ${\varpi }_{tra,i}^{s}$ and ${\varpi }_{ter,i}^{s}$ represent renormalization factors from the dressed Hikami box corrections for intra- and inter-surface Cooperon channels, respectively. The factors η_s account for the vertex corrections, and l^s is the characteristic length associated with each contribution, quantifying the suppression of quantum interference due to the dephasing or symmetry-breaking mechanism. The lengths play a crucial role in determining the strength of the quantum correction, in which larger values of l^s indicate less suppression of quantum interference effects.