Introduction

A 3-dimensional (3D) topological insulator (TI) is characterized by insulating bulk and surface Dirac cone states1,2. The surface Dirac cone states have been extensively studied3,4,5,6, but the bulk bands have received little attention. The inverted bulk bands in a TI are nontrivial7,8, and can undergo a band topological phase transition (TPT) with a gap closing point as the quantum critical point. The transport properties near the band TPT remain largely unexplored9,10, unlike the non-Fermi-liquid behaviors near the quantum critical point in strongly correlated materials11.

The band TPT in nonmagnetic TIs can be triggered by changes in chemical composition, pressure, strain or external magnetic field12. When the band gap is small and a magnetic field is present, a unique band TPT occurs on Landau bands (LBs). Spectroscopic measurements have mainly explored these band TPTs13,14,15,16,17,18, but they cannot reveal the exotic physical properties near the band closing point. To our knowledge, no distinct transport signatures have been uncovered on the transition point. However, electrical transport under a strong magnetic field provides a unique opportunity to explore the physics near the gap closing point by continuously going through the band TPT. In this study, we report a significant transport signal in β-Ag2Se, a band TPT-induced spinor-dominated magnetoresistance (MR) anomaly. During a magnetic-field-driven band TPT, the presence of two intrinsically backscattering forbidden 1-dimensional (1D) conducting channels in a 3D TI, mimics the ballistic helical edge modes in 2-dimensional (2D) TIs.

Results and Discussion

Mechanism of spinor-prohibited backscattering in topological insulators

Before delving into the experimental results, let us first discuss the unique physics behind the band TPT of LBs in TI. In a TI, the bulk band splits into a series of LBs with strong orbital quantization under a magnetic field. The energies of the LBs are determined by two key scales, Zeeman splitting (g*μBB) and orbital cyclotron energy (ℏωc), where ℏ is the Planck constant divided by 2π, ωc = eB/m* is the cyclotron frequency, e is the elementary charge of electron, m* is the effective mass, g* is the effective g-factor, and μB is the Bohr magneton. For a free electron system, ℏωc is equal to g*μBB, while ℏωc dominates in topological materials with ultra-light m* ( ~ 0.1m0), here m0 is the bare electron mass. Due to the band inversion of TI, the orbital cyclotron and Zeeman effects will disentangle the two inverted bands, leading to a magnetic-field-driven band TPT as illustrated in Fig. 1a. The Zeeman effect can be treated as a modified orbital cyclotron effect due to the spin-polarized nature of LLBs, sharing the same linear-in-B relation with orbital cyclotron effect. We can describe this process in details by using a typical k ⋅ p model of TI, written as \(H(\overrightarrow{k})={M}_{{{{{\bf{k}}}}}}{\tau }_{z}+\hslash ({\sum}_{a=x,y,z}{v}_{a}{k}_{a}{{{\Gamma }}}_{a})\), where τz labels for the orbitals while the second term is the massless Dirac equation based on the Γ matrices. This model is characterized by a mass control term \({M}_{{{{{\bf{k}}}}}}={M}_{0}+{M}_{\perp }({k}_{y}^{2}+{k}_{z}^{2})+{M}_{x}{k}_{x}^{2}\), where M0, Mx and M⊥ are band parameters for band gap at kx = 0, in- and out-of-plane parabolic energy dispersions, respectively7. In the ultraquantum limit with B ∥x (a)-axis, there are only one pair of spin-polarized last LBs (LLBs) on the Fermi level. The energy dispersion of the two LLBs is \({E}_{{0}_{\pm }}\) = \(\sqrt{{m}^{2}+{\hslash }^{2}{k}_{x}^{2}{v}_{x}^{2}}\), where the mass term (or equivalently, the band gap at kx = 0) of the two LLBs is m = M0 + \({M}_{x}{k}_{x}^{2}\) + \({M}_{\perp }/{l}_{B}^{2}\), where \({l}_{B}=\sqrt{\hslash /eB}\) is the magnetic length, controlling the cyclotron energy. At kx = 0, the inverted band is directly controlled by the mass term m = M0 + \({M}_{\perp }/{l}_{B}^{2}\), where signs of M0 and M⊥ are opposite due to the inverted dispersion. Then, linear-in-B cyclotron energy will lever the inverted gap M0 and finally dominates by crossing a gapless intermediate state with m = 0, underpinning the process of the band TPT as illustrated in Fig. 1a. Furthermore, we can see that the critical field μ0Hc for m = 0 is determined by M0/M⊥, causing μ0Hc proportional to 1/M⊥, which can be reflected by the effective mass m* in transport measurements. Then, anisotropic m* measured from an ellipsoid-like Fermi surface can roughly track μ0Hc, namely m* ∝ μ0Hc.

Fig. 1: Topological-phase-transition-induced forbidden backscattering and band properties of β-Ag2Se.
figure 1

a The two last Landau bands (LLBs) of typical strong topological insulator (TI) go across a phase transition driven by external magnetic field. Only two states, denoted as black dots \(\left\vert \pm {k}_{x}\right\rangle\), are involved in this 1D channel. b \(\left\vert \pm {k}_{x}\right\rangle\) are orthogonal to each other when mass term m is zero, which causes the spinor-prohibited backscattering. c As a result, a dip appears on the longitudinal magneto-resistance (MR) at critical magnetic field (Bc). d A photo of single crystalline β-Ag2Se. e The orthorhombic crystal structure of β-Ag2Se. f High symmetric k point in a unit cell. g Band calculations of β-Ag2Se based on PBE and HSE approxiamations. h Angle-dependent Shubnikov-de Haas (SdH) oscillations in ab (θ) and ac (ϕ) planes. i The extracted frequencies of SdH along the two tilted angles, θ and ϕ, respectively. Inset shows the mapped Fermi surface with a Fermi level determined by experimental carrier concentration (n). The slight mismatch of frequency around zero angle was probably caused by sample bending, in the different rounds of measurement.

Full size image

The consequence of m = 0 can be envisioned by looking at the highly nontrivial properties of LLB whose wave function uniquely contains a spinor eigenvector, inherited from the 4 × 4 matrices-based Dirac Hamiltonian and a normal part from the harmonic oscillator. As shown in Fig. 1b, the two spinors of +kx and -kx become orthogonal when m = 0 at the band TPT, here the \({M}_{x}{k}_{x}^{2}\) in the mass term m is treated to be negligible in the ultraquantum limit due to the field suppression of Fermi level. Consequently, as shown in Fig. 1b, the LLBs become gapless at a critical field (μ0Hc), causing the transport form factor Is which is proportional to the transition matrix between the two spinors at +kx and -kx to vanish identically19. The vanished Is leads to an extremely long transport relaxation time (τ ∝ 1/Is), making the backscattering forbidden (Fig. 1b) and leading to a pronounced dip in longitudinal MR at μ0Hc (Fig. 1c). Because the 1D dispersion of LLB is always determined by the direction of magnetic field, this additional transport channel occurs only when the magnetic field is along the current direction, making the longitudinal MR a unique probe to detect this anomaly. The longitudinal MR anomalies here differ from the forbidden backscattering in graphene20, where electrons on the zero-field Dirac band, instead of LBs, are forbidden from backscattering due to the natural orthogonality in Pauli matrices. The anomaly also differs from the chiral anomaly, a 1+1D chiral fermion effect with unbalanced left and right movers on any 1D band dispersions (LLBs) across the Fermi level21, related to the recent observation of a broad negative MR and bending over behaviors in topological crystalline insulator Pb1−xSnxSe, where the LLBs are gapped out by alloy-mixing disorders at critical field22,23. Our theory is based on Z2 topological insulators, the forbidden backscattering takes place at a critical field with level crossing, inducing a sharp field-symmetric resistance dip, which may occur twice depends on the relative signs between M0,x,⊥19. Evidence of level crossing at critical magnetic field have been revealed by spectroscopic measurements in 3D topological insulators17,18.

Ultraquantum-limit longitudinal magnetoresistance in β-Ag2Se

To observe this unique longitudinal MR, we require a TI with only one pair of inverted bands around the Fermi level and a narrow gap accessible by a magnetic field. Unfortunately, narrow-gapped strong TIs are rare, and most with no isolated topological bands near the Fermi level23,24. In this work, we have chosen β-Ag2Se25,26, as a candidate of TI, an platform to achieve the spinor-dominated MR. As shown in Fig. 1d, e, β-Ag2Se, appears as long ribbons and crystallizes in an orthorhombic unit cell. The electronic band calculation by Heyd-Scuseria-Ernzerhof (HSE) approximation indicates that β-Ag2Se is a strong TI with Z2 = (1; 0 0 0) and an indirect band gap around 4–10 meV (Fig. 1f, g), which is consistent with the previous calculations and experiments on the strong TI nature of β-Ag2Se27,28. We note here that there is no conclusive spectroscopic results from angle-resolved photoemission spectroscopy (ARPES) experiment in β-Ag2Se, which should be explored in the future for nailing down the existence of topological surface states compatible with the Z2 index. The HSE-based results compared with the Perdew-Burke-Ernzerhof (PBE) approximation, which usually underestimates the gap and shows that β-Ag2Se is a semimetal. This issue is also addressed by transport and optical spectroscopic measurements29,30, our calculations are also consistent with the previous band structure calculation based on HSE.

We conducted low-field electrical transport measurements on sample S1 with carrier concentration n = 1.1 × 1018 cm−3, and as shown in Fig. 1h, i, we observed clear single-frequency Shubnikov-de Haas (SdH) oscillations with a quantum limit around 3 T when the magnetic field is tilted towards b(θ) and c(ϕ) axes. We found no new frequencies in all the mapped ranges of angles. The angular dependence of extremal cross-sectional areas, with frequencies Fθ,ϕ, depicts a 3D anisotropic Fermi pocket of the bulk state consistent with the calculation (see Supplementary Note 1 for details). The small electron pocket and narrow band gap make β-Ag2Se as an ideal platform for investigating the physics of pure inter-1D channels (LLBs) scattering in the ultra-quantum limit under a modestly strong magnetic field. We then conducted electrical transport measurements on another sample S2 in a strong magnetic field because it has a lower quantum limit  ~ 2 T. Figure 2a shows the complicated dependence of longitudinal MR (Δρxx(H)/ρ0) at 1.5 K characterized by two anomalous dips at 1.5 K, denoted as B1 at 5 T and B2 at 51 T, with the current and magnetic field applied along the same crystallographic a direction.

Fig. 2: Ultraquantum-limit longitudinal magnetoresistance of β-Ag2Se.
figure 2

a The longitudinal magnetoresistance (MR), at T = 1.5 K, exhibits two anomalies denoted as B1 and B2. Inset is a sketch shows the experimental configuration of magnetic field and current. b Rescaled temperature-dependent magnetoresistance, Δρxx/ρ0, is plotted against the magnetic field up to 56 T. c Rescaled angle-dependent magnetoresistance, Δρxx/ρ0, is plotted against the magnetic field up to 56 T. Inset shows the setup of sample rotation.

Full size image

To investigate the physical mechanism underlying B1 and B2, we tracked the two anomalies by performing temperature and angle dependence of Δρxx(H)/ρ0. Figure 2b shows B1 and B2 gradually vanish as temperature increases, results in a smooth background that persists even at 70 K. The critical field of B2 slightly shifts to a lower field when temperature rises, which is against many-body effects such as a charge density wave (CDW) transition in a magnetic field31. Moreover, the angular dependence of Δρxx(H)/ρ0, as shown in Fig. 2c, shows that the B1 and B2 shift towards higher fields when the direction of the magnetic field is tilted away from the current direction, indicating that B2 is affected by the anisotropy of band dispersion. The temperature dependence of B2 suggests a specific band effect that causes the MR dips, rather than a many-body effect.

We attempted to understand the anomalies B1 and B2 in the ultraquantum limit by examining the relationship between the smooth background of Δρxx(H)/ρ0 and B1, B2. Figure 3a shows conductance Gxx at each tilted angle (θ), defined as 1/ρxx(H), against the out-of-plane magnetic field component. We observed that the backgrounds of Gxx align on a single curve at each tilted angle, indicating a normal orbital MR effect. Additionally, the peak values of Gxx replotted against θ in Fig. 3b can be fitted by a cosnθ (n = 4.6) empirical relation. According to the Shockley-Chambers formula32, the angle-dependent conductivity of a cylinder-shaped Fermi surface yields a  ~ cos2θ relation due to the anisotropy of m*. If the transport relaxation time τ is isotropic, the angle-dependent conductivity dampens slower than a  ~ cos2θ relation for an anisotropic Fermi surface. The obvious deviation on exponent n (Fig. 3b) shows that the isotropic transport relaxation time τ assumption must break down, indicating the existence of an additional conducting channel only when the magnetic field is tilted along the direction of the current.

Fig. 3: Existence of an additional conducting channel originates from a pure band effect in β-Ag2Se.
figure 3

a The longitudinal conductance Gxx, rescaled versus out-of-plane component of magnetic field. Inset displays the definition of angle θ. b The amplitudes of anomaly B2 are plotted against tilted angle θ, and fitted by cosnθ function with index n = 4.6, indicates the fast damping of B2 when the magnetic field is tilted away from a-axis. c The temperature-dependent critical fields of B2 shows linear T behaviors, as indicated by the linearly fitted bold red curve. d The angle-dependent critical magnetic fields of anomalies B1 and B2. e The rescaled angle dependence of B1 and B2, as shown in (d), coincides with the angle-dependent frequency of SdH oscillations, which points to the m* in cyclotron energy ℏωc.

Full size image

We further explored this additional conducting channel by examining the related control parameters. As shown in Fig. 3c, the fields of B2, denoted as \({\mu }_{0}{H}_{B2}^{c}\), linearly increases with rising temperature, indicating that the thermal energy (kBT) competes with cyclotron/Zeeman energy which is proportional to the magnetic field. On the other hand, \({\mu }_{0}{H}_{B1}^{c}\) and \({\mu }_{0}{H}_{B2}^{c}\) follow the same angular dependence of SdH oscillations at 4.2 K as shown in Fig. 1i. This angular dependence is obvious when we scaled \({\mu }_{0}{H}_{B1,2}^{c}(\theta )/{\mu }_{0}{H}_{B1,2}^{c}({0}^{o})\) versus Fθ/\({F}_{\theta = {0}^{o}}\) in Fig. 3e. The frequency of SdH oscillations is expressed by the quantization rule \(F=\frac{\hslash }{2\pi e}{{{{{\rm{A}}}}}}_{{{{{\rm{F}}}}}}\), where e is the elementary charge, and AF is the extremal cross-sectional area, which is roughly proportional to the \({k}_{F}^{2}\). Then, F reflects the anisotropy of effective mass (m*) if energy dispersion \(\epsilon =\frac{{\hslash }^{2}{k}_{F}^{2}}{2{m}^{* }}\) is adopted, while m* directly determines the cyclotron energy ℏωc. As we already pointed out in the introductory part, the m* is proportional to the critical field \({\mu }_{0}{H}_{B2}^{c}\) of the band TPT on the LLBs, our experimental observation of the coincidence between \({\mu }_{0}{H}_{B2}^{c}\) and m* confirms the longitudinal MR anomaly is caused by a band TPT-induced mechanism.

Simulations based on the band calculations of β-Ag2Se

We ascribe the anomaly in longitudinal MR induced by the band TPT to intrinsic forbidden backscattering on LLBs caused by spinor orthogonality around the phase transition, as illustrated in Fig. 1. To support our interpretation, we conducted a detailed simulation on β-Ag2Se. Before quantitative simulation, we thoroughly inspect the mass term of LLBs m = M0 + \({M}_{x}{k}_{x}^{2}\) + \({M}_{\perp }/{l}_{B}^{2}\). The magnetic dependence of this term is important, because it controls the form factor Is ( ∝ m2). In this well-established TI model, the relative signs of M0, Mx and M⊥ determine the band inversions along different high-symmetric k paths. M0Mx < 0 and M0M⊥ < 0 represent a strong TI, while only one of them satisfied represents a weak TI, and none of them satisfied represents a normal insulator19. We then simulated the magnetic field dependence of m in different topological phases. As shown in Fig. 4a–c, m (Is) shows no zero crossing (no dip), one zero crossing (one dip) and two zero crossings (two dips) in the quantum limit of normal insulator, weak TI and strong TI, respectively. The appearance of zero crossing at small field in strong TI is caused by the field-dependent Fermi level on LLB19, which approaches the forbidden momenta kx = \(\scriptstyle\sqrt{-{M}_{0}/{M}_{x}}\) at small field. This mechanism in low field is subtle and depends on the precise estimation of \({M}_{x}{k}_{x}^{2}\) term. We found that the field dependence of m exhibits distinct behaviors in the three phases based on the detailed mathematical structures of m, which depends on the relative signs of M0, Mx and M⊥. Therefore, we can expect different numbers of longitudinal MR dips in different topological phases when m vanishes across the band TPT.

Fig. 4: Simulations of the spinor-dominated longitudinal MR of in β-Ag2Se based on band structure calculations.
figure 4

a–c Mass term m (form factor Is) shows no zero crossing (no dip), one zero crossings (one dip) and two zero crossings (two dips) in the quantum limit for normal insulator, weak TI and strong TI, respectively. d Color map of critical magnetic fields, \({\mu }_{0}{H}_{B2}^{c}\), versus coefficients Mx and M⊥ of a typical strong TI k ⋅ p model. Here M0 is fixed to be -0.036 eV, while the dashed red line indicates the fitted value of M⊥ based on band structure. e With considering the Zeeman effect, \({\mu }_{0}{H}_{B2}^{c}\) locates in a range, indicated by pink thick curve, which is consistent with our experimental values.

Full size image

By adopting a k ⋅ p model with specific parameters from band calculations (refer to Supplementary Note 3), our aim is to determine the values of μ0Hc where anomalies occur. The main parameter is the mass control term m = M0 + \({M}_{x}{k}_{x}^{2}\) + \({M}_{\perp }/{l}_{B}^{2}\) with M0 = -0.036 eV fixed by direct energy difference around Γ point, part of the band parameters summarized in Table S1 of SI are also constrained by experimental ones derived from the temperature-dependent resistivity and quantum oscillations of β-Ag2Se25,29. If we ignore the Zeeman energy, we map out the zero crossings at the critical magnetic field, \({\mu }_{0}{H}_{B2}^{c}\) versus Mx and M⊥ in Fig. 4d. As mentioned before, the \({M}_{x}{k}_{x}^{2}\) term contracts into a negligible value in a strong magnetic field. Therefore, we can see that \({\mu }_{0}{H}_{B2}^{c}\) is sensitively dependent on M⊥, and the value from band fitting (red dotted curve) indicates \({\mu }_{0}{H}_{B2}^{c}\) around 180 T, which contradicts our experimental B2 anomaly around 51 T. While the above estimation is solely based on the cyclotron energy (\({M}_{\perp }/{l}_{B}^{2}\)) of LLBs without including the Zeeman effect. As we discussed before, the Zeeman effect on LLBs can be easily included as a modified cyclotron energy, then the mass term can be written as m = M0 + \({M}_{x}{k}_{x}^{2}\) + \({\tilde{M}}_{\perp }/{l}_{B}^{2}\), where \({\tilde{M}}_{\perp }={M}_{\perp }+{g}^{* }{\mu }_{B}\hslash /4e\) is the modified in-plane mass parameter. As shown in Fig. 4e, \({\mu }_{0}{H}_{B2}^{c}\) decreases when g* increases. The bold pink curve indicates the range of \({\mu }_{0}{H}_{B2}^{c}\) based on g* from Zeeman splitting, where a range of g* is used for a reasonable comparison with the experimental B2 anomaly due to the anisotropy of β-Ag2Se (see Supplementary Note 1 for details on experimental extraction of g*-factors). We can now see that the \({\mu }_{0}{H}_{B2}^{c}\) of anomaly B2 falls within the colored range, which supports the spinor-dominated mechanism driven by band TPT. The precise determination of the related band parameters and g* might need further spectroscopic experiments. The remaining anomaly \({\mu }_{0}{H}_{B1}^{c}\) cannot be precisely determined due to large uncertainty in the estimation of \({M}_{x}{k}_{x}^{2}\) term as explained before.

In conclusion, we have observed large longitudinal MR dips in the ultraquantum limit in β-Ag2Se. The ultraquantum-limit anomaly points to an underlying mechanism based on spinor-dominated forbidden backscattering on LLBs. The magnetic field acts as a tunable knob that controls the access of distinct topological phases. Therefore, the newly discovered MR effect presents a rare MR effect, with a possible topological origin, that can be adopted to design logic devices with on/off functions. Our study also signifies the physics comes from bulk bands of TI, mainly connected with band TPT, where a plethora of unexplored physics should be targeted in the future.

Methods

Sample preparation and characterization

Single crystals of β-Ag2Se can be grown by modified vapor transfer method as described in ref. 25. Polycrystalline β-Ag2Se was sealed in a tube silica ampoule, and then placed in a tube furnace subjected to a temperature gradient from 500 °C to around room temperature for several days. The shape of as-grown samples are ribbon-like, and the crystalline axes were determined by single crystalline X-Ray diffraction measurements as described in ref. 25.

Transport measurements

Magneto-transport measurements (56 T) were performed at the Wuhan National High Magnetic Field Center (WHMFC). A Digital lock-in technique was employed with f = 100 kHz and i = 5  mA by using a non-destructive pulse magnet with a pulsed duration of 60 ms. Resistance was measured by a standard 4-probe method. Measurements with both positive and negative field polarities were performed to eliminate the effects of contact asymmetries. Data for the up-sweeping and down-sweeping of the pulse field were in good agreement, thus the self-heating effect of the sample, in the pulsed high magnetic fields, can be excluded.

First-principles calculations

First-principles calculations of β-Ag2Se were performed using density functional theory implemented in the Vienna ab initio simulation package (VASP) code33. The energy cutoff for plane wave basis was set at 400 eV. The Brillouin zone was sampled within the Monkhorst-Pack scheme and the k mesh was set as 8 × 5 × 4. The generalized gradient approximation (GGA) expressed by the Perdew–Burke–Ernzerhof (PBE) functional was implemented for the exchange-correlation energy34. To achieve a better description of electronic properties and band gaps, screened hybrid functional of Heyd–Scuseria–Ernzerhof (HSE06) was used in band structure calculations35 with 25% of the nonlocal Fock exchange combining with 75% of the PBE exchange. Hartree-Fock screening was set to be 0.2 to achieve a good balance between accuracy and computational cost.