Introduction

Searching for yet topological phases has garnered immense interest, not only in the research community but also for low-energy-low-cost technology and quantum information science due to the existence of topologically protected conduction channels. The topological band theory proposed by Kane and Mele uses the concept of the topological invariant to classify the topological quantum materials with robust boundary states protected by time-reversal-symmetry1,2,3,4, where the topological phases are achieved by band inversion with varied eigenvalues of wavefunction parities of the occupied states after turning on the spin-orbit coupling (SOC). The prototypical examples of these materials, the strong (weak) topological insulators, can be realized by such band inversion occurring at odd (even) numbers of time-reversal-invariant (TRI) momenta, which has been experimentally confirmed in recent decades under the condition that the band inversion only involves one conduction band and one valence band at each TRI momentum5,6,7,8,9,10,11,12,13,14. The situation becomes complicated when multiple band inversions (MBI) take place in TRI momentum, leading to versatile topological phases due to the additional freedom in the modulation of band inversion. In fact, the MBI has been investigated under the condition of various characteristics15,16,17,18,19,20,21,22,23,24. For instance, the multiple Dirac cones are derived from the combined effect of Rashba splitting and MBI, which have the potential to unveil the intertwined physical phase by Rashba spin physics and topological nontriviality15,16,17,23. The strain effect can realize the MBI and the consequent weak topological insulating phase or semi-Dirac semimetal phase through the topological phase transition (TPT)19,20. The topological compensated semimetal phase and topological excitonic insulating phase are identified in the MBI systems under the strong SOC strength17,22. Moreover, the quasi-one-dimensionality is benefit to the emerge of MBI at different TRI momenta, dedicating to the unusual topological phase such as high-order topological insulator (HOTI)21. Therefore, searching for the appropriate materials with multiple conduction band and valence band pairs involved in the SOC-induced band inversions can extend the experimental study of the unusual topological phases correlated with the topological band theory.

Results and discussion

Crystal structures of Bi4(Br1-xIx)4

Three-dimensional (3D) α’-Bi4Br4, which is constructed from one-dimensional (1D) chains, features double band inversions around Fermi level at two TRI momenta, at the M point and the L point of the 3D Brillouin zone (BZ), as displayed in Fig. 1a–c. For the L point, the opposite signs (+−) of the parity eigenvalues near the Fermi level for the bulk valence bands (BVBs) are retained after the inclusion of SOC, leading to a trivial state (ℤ2 = 0). In contrast, the parity eigenvalues of BVBs vary from (−−) to (++) at the M point, resulting in a HOTI state (ℤ4 = 2) with gapped surface states but 1D helical gapless hinge states21,25,26,27. Thus, α’-Bi4Br4 is an excellent parent phase to use for designating the special topological phases by varying the eigenvalues at different TRI momenta through importing perturbations. Leveraging the combined techniques of angle-resolved photoemission spectroscopy (ARPES) and scanning transmission microscopy (STEM), we identify the unusual dual topology modes, strong topological insulator (STI) and HOTI, which simultaneously coexist in this material system after applying the isocharge doping of I atoms on Br sites. Moreover, the interplane stacking orders are also modulated varying the ratio between the halide elements, engendering abundant topological phases in this quasi-1D system. Accordingly, a topological phase diagram of Bi4(Br1-xIx) as a function of I concentration was established to provide a route to precise control of a single crystal quantum system.

Fig. 1: Crystal structures of Bi4(Br1-xIx)4 and calculated band structures of Bi4Br4.
figure 1

a Crystal structure with the unit cell labeled by red lines of α’ phase. The Bi and Br/I atoms are marked by blue and gray balls, respectively. The two sites of Bi atoms are denoted as Biin and Biout, respectively. The chain direction is along the crystal b axis direction and highlighted in red. b Three-dimensional (3D) Brillouin zone (BZ) with (001) and (100) projected surface BZs. Schematics represent two nondegenerate bands near the Fermi level at \(M\) and \(L\). c Calculated bulk bands of α’-Bi4Br4 without (W/O) spin-orbit coupling (SOC) (left panel) and with SOC (right panel), respectively. Parity eigenvalues of each band are labeled. The red and blue dots represent the contributions of the p-orbitals of Biin and Biout atoms, respectively. d, e Left: 3D topography of the (001) surface and the (100) surface of α’-Bi4Br4 with step height of 0.96 nm and 0.66 nm, respectively. Scale bar, 5 nm. Right: Atomic-resolved topography of the red squares in the left panels. Scale bar, 1 nm. The blue rectangle and yellow rectangle denote the (001) plane and (100) plane of the unit cell with corresponding atoms, respectively. Lattice constants of a, b, and c are ~ 1.33 nm, ~ 0.44 nm, and ~ 2.01 nm, respectively. f Atomic-resolved images of the (010) planes of the samples with different I content. Scale bar, 1 nm. The blue, yellow, and pink colors present different layers in each stacking mode. The red, yellow, and blue rhombuses stand for the (010) planes of the unit cells for α’ phase, γ phase, and α phase, respectively. g Crystal parameters summarized from our data as well as the previous references21,25,29,30,31,32,33. Error bars are determined by average variation of X-ray diffraction (XRD), transmission electron microscopy (TEM), and scanning tunneling microscopy (STM) results, respectively. All the lattice constants increase with I content.

Full size image

A structural model of α’-Bi4X4 with the C2m symmetry group is shown in Fig. 1a, where the structural building block is the quasi-1D bismuth chain passivated by halide atoms, which are connected with each other through the van der Waals (vdW) force along two directions (the lattice a direction and the lattice c direction). Therefore, there are two cleavable planes, the (001) surface and the (100) surface, which were suitable for the surface characterizations in this system. Figures 1d, e displays the scanning tunneling microscope (STM) topographies of the cleaved (001) and (100) surfaces of Bi4Br4, respectively, where large flat planes and lattice constants can be identified. High-angle annular dark-field ‒ scanning transmission electron microscopy (HAADF-STEM) combined with the focused ion beam (FIB) technique was applied to characterize the structure of the uncleavable (010) plane. The antiparallel nature of two adjacent layers is revealed in Fig. 1f, forming the A-B stacking mode and consistent with the cross-sectional view of the structural model in Fig. 1a as well as previous reports27,28. The high quality of these single crystals is indicated by the sharp diffraction peaks in X-ray diffraction (XRD) measurements (see Supplementary Fig. 1 for details). For the single crystals doped with I atoms, the actual I concentration was measured by the energy-dispersive spectroscopy (EDS), which shows a monotonic change with the nominal I concentration in the raw materials (see Supplementary Fig. 2 for details). We refer to the actual I concertation obtained from the EDS measurements below as the I concentrations. There is a redshift of the positions of (00 l) peaks in the XRD spectra with increasing I concentrations, implying the possible increase of the c-axis lattice constant. HAADF-STEM measurements on the (010) planes of the different samples were performed to determine the positions of the I atoms and their influence on the stacking arrangement along the lattice c axis direction. The high-resolution STEM images in Fig. 1f show that there are no atoms in the intercalation positions, indicating that the Br atoms were successfully replaced by I atoms. Moreover, most of the samples possess the same A-B stacking mode (α’ phase) expected for the samples with I concentrations of 0.5 and 1, in line with the results on powder samples in a previous work29. We categorize the structures of Bi4(Br0.5I0.5)4 and Bi4I4 as γ phase and α phase, respectively, and the corresponding STM topography of the (001) planes with crystal parameters can be seen in Supplementary Fig. 3. Figure 1g displays the lattice constants extracted from our XRD curves, STM images, STEM results, and other references21,25,29,30,31,32,33 as functions of I concentration, where all three values increase with increasing I content, which is due to the larger ionic radius of I compared to Br. It should be noted that the increased amplitudes of the lattice constants for a axis and c axis are much larger than that of b axis, implying that the I doping mainly affects the interchain space rather than the intrachain space. All these results demonstrate that the lattice constants of Bi4Br4 are modulated by I doping and that the α’ phase can be retained over a large concentration range.

ARPES spectra of the cleaved (001) plane

We first performed ARPES measurements on the in-situ cleaved (001) surfaces of Bi4(Br1-xIx)4 crystals to reveal the bulk-structure evolution with the I content due to the fact that there is no surface state in the (001) plane34. 3D plots of the ARPES spectra on the (001) surface in reciprocal (kx, ky, E) space, where kx and ky are wave vectors and E is the energy, for samples with different I content are shown in Fig. 2a. An elliptical electron pocket emerges at the \(\bar{{{{\rm{M}}}}}\) point of the (001) surface BZ, and no band can be observed at the \(\bar{\Gamma }\) point of the (001) surface BZ near the Fermi level (see Supplementary Fig. 4 for details). These ARPES spectra are well consistent with the calculation results in Fig. 1c, as both the M point and L point of the 3D BZ, where the near-Fermi-level bands are located, are projected into the \(\bar{{{{\rm{M}}}}}\) point for the two-dimensional (2D) BZ of the (001) plane. Figure 2b shows the energy-momentum dispersion at the \(\bar{{{{\rm{M}}}}}\) point cut along the chain direction (ky) of the samples with different I content. A bulk energy gap \({\varDelta }_{B}\) can be observed, and the gap size initially decreases, reaching a minimum value, and then increases with the increasing I content. The value of \({\varDelta }_{B}\), which can be estimated through energy differentiation between the edges of the full width at half maximum of the top valence bands and bottom conduction bands in the energy distribution curves (EDCs), as displayed in Fig. 2c, is 135 meV, 77 meV, 48 meV, 17 meV, 29 meV, and 137 meV for the crystals with I contents of 0%, 25%, 50%, 62%, 80%, and 100%, respectively (see Supplementary Fig. 5 for details). This nonmonotonic trend in the gap value implies possible gap closing and band inversion at the critical point near 70% I concentration. In fact, our temperature-dependent resistance measurements of different I content samples show that the 73% I concentration sample is the only one with metallic behavior (see Supplementary Fig. 6 for details), which indicates that the bulk gap is possibly closed and agrees well with the gap size trend in the APRES spectra. Additionally, the density functional theory (DFT)-calculated band structures of 3D Bi4(Br1-xIx)4 show a band inversion at x = 0.75 (see Supplementary Fig. 7 for details), in strong agreement with the experimental observations.

Fig. 2: ARPES spectra of the cleaved (001) planes of Bi4(Br1-xIx)4 crystals.
figure 2

a E-kx-ky dispersion spectra measured at the in-situ-cleaved (001) surfaces of Bi4(Br1-xIx)4 crystals with x ranging from 0 to 1(from left to right). The (001) projected time-reversal-invariant (TRI) momenta at the \(\bar{{{{\rm{M}}}}}\) point and the \(\bar{\Gamma }\) point are labeled by a blue dot and a red dot, respectively. The inset shows a schematic diagram of the experimental setup for the (001) surface. b The E-ky dispersion at \(\bar{{{{\rm{M}}}}}\) along the chain direction. The edge-to-edge gap of bulk bands is indicated by the white dashed lines, where the bulk gap firstly shrinks, and then reopens with a transition point around x ~ 0.7. c Energy distribution curves (EDCs) of the corresponding spectra in panel b. The EDCs of ky = 0 are marked in red. The blue and red dots are obtained by the peak fitting of EDCs according to the Lorentz-type distribution equation (see Supplementary Note 4 for details), indicating the half-maximum edge locations of the valence band and the conduction band, respectively. The bottom arrow is used to indicate the trend for the I content in different crystals.

Full size image

Topological phase transitions evoked by band inversions

Since both the M point and the L point of the 3D BZ are projected into the \(\bar{{{{\rm{M}}}}}\) point of the 2D BZ of the (001) plane in which the band inversion occurs, it is hard to distinguish the evolution of the 3D band structure between these two points from the ARPES spectra of the (001) plane. Nevertheless, the M point and the L point of the 3D BZ are separately projected into the \(\bar{\Gamma }\) point and the \(\bar{{{{\rm{Z}}}}}\) point of the 2D BZ of the (100) plane, as indicated in Fig. 1b. Thus, the (100) surface characterizations are expected to provide more information on the band structure. The cleaved (100) surface is rough compared to the cleaved (001) surface due to the stronger interchain coupling strength along the lattice a direction than along the lattice c direction35,36. We applied the laser-based ARPES measurements with micrometer light size to characterize the electronic structure of the in-situ cleaved (100) surface of these single crystals to acquire high-resolution spectra. Unlike the (001) surface, the ARPES spectra of the (100) surface exhibit surface states (see Supplementary Fig. 8 for details). 3D plots of ARPES spectra on (100) surface in (ky, kz, E) space for different crystals are shown in Fig. 3a, where the highly anisotropic electronic structures of 2D surface states are identified because of the 1D nature of their building blocks. The E-ky dispersion spectra of different crystals at the \(\bar{\Gamma }\) point and \(\bar{{{{\rm{Z}}}}}\) point are presented in Fig. 3b, c respectively. For α’-Bi4Br4, the surface states at both the \(\bar{\Gamma }\) point and \(\bar{{{{\rm{Z}}}}}\) point are constructed from two nondegenerate Dirac bands with surface energy gaps \({\varDelta }_{S}\) that have opened at the band crossing points away from the TRI momenta. The values of \({\varDelta }_{S}\) for Bi4Br4 are ~38 meV and ~35 meV around the \(\bar{\Gamma }\) point and \(\bar{{{{\rm{Z}}}}}\) point, respectively (see Supplementary Figs. 9, 10 for details of the gap analysis). After introducing the doped I atoms, the surface bands of the (100) plane show subtle changes, such as varied Fermi velocities and decreased \({\varDelta }_{S}\) at both the \(\bar{\Gamma }\) point and the \(\bar{{{{\rm{Z}}}}}\) point, in the condition of the fixed α’ phase. Interestingly, there is transition from the double Dirac band structure to the single Dirac band structure at the \(\bar{{{{\rm{Z}}}}}\) point of the (100) BZ of α’-Bi4(Br0.2I0.8)4, which has been confirmed by the second derivative results in Supplementary Fig. 11. It is notable that Bi4(Br0.2I0.8)4 belongs to a different phase from the HOTI region, as its I content (0.8) is just a step over the critical point (~0.73) (Fig. 2b). Hence, the spectra of the (100) surface states coincide with the bulk structures obtained in ARPES results measured in the (001) plane, confirming the occurrence of a phase transition.

Fig. 3: Electronic structures of the (100) planes of Bi4(Br1-xIx)4 crystals.
figure 3

a E-ky-kz dispersion spectra measured at the in-situ-cleaved (100) surfaces of Bi4(Br1-xIx)4 crystals with x ranging from 0 to 1(from left to right). The inset indicates the experimental setup for the (100) surface characterization. The red dot and pink dot represent the (100) projected \(\bar{\Gamma }\) point and \(\bar{{{{\rm{Z}}}}}\) point, respectively. b, c High-resolution E-ky dispersions at the \(\bar{\Gamma }\) point and \(\bar{{{{\rm{Z}}}}}\) point, respectively. The yellow dashed lines are guides to the eye for the surface state dispersion. The coupling gap \({\varDelta }_{S}\) is indicated by the white dashed line. There are two nondegenerate Dirac points (DPs) at both the \(\bar{\Gamma }\) point and the \(\bar{{{{\rm{Z}}}}}\) points for I concentrations of 0%, 25%, 62%, and 100%. The Bi4(B0.5I0.5)4 crystal exhibits a triple-layer unit cell and is identified as a weak topological insulator (WTI)36. The crystal with 80% I content, however, shows double DPs at the \(\bar{\Gamma }\) point but a single DP at the \(\bar{{{{\rm{Z}}}}}\) point (labeled by a yellow arrow). The bottom orange arrow denotes the I concentration.

Full size image

Composite topological phase

The (001) monolayer Bi4Br4 has been predicted and subsequently confirmed by experiment to be a large gap (~0.2 eV) quantum spin Hall insulator (QSHI)25,34,35. The 3D α’-Bi4Br4 can also be regarded as an alternating arrangement of two antiparallel (001) monolayers with only in-plane inversion symmetry (see Supplementary Fig. 12 for details). Our first-principle calculations of the band structure of two adjacent layers (A layer and B layer) in Supplementary Fig. 12 imply nondegenerate electronic structures, which has been experimentally observed from scanning tunneling spectroscopy curves (see Supplementary Fig. 13 for details). The structural discrepancy lifts the degeneracy of two neighboring (001) monolayers (see Supplementary TABLE 3 for details), and the two pairs of BVBs and bulk conduction bands (BCBs) at either the M point or the L point of the 3D BZ near the Fermi level in Fig. 1c are contributed by the A layer and the B layer, respectively. Therefore, the 1D edge states of the two QSHI layers, the A layer and the B layer, are also nondegenerate with the different Fermi velocities and energy positions of the Dirac point. These nondegenerate edge states are arranged along the lattice c direction to form the (100) surface states with two Dirac band structures and a surface gap opened at their crossing points by the finite interlayer interaction (see Supplementary Fig. 12 for details). It should be noted that multiple Dirac bands have been resolved in the MBI system with different surface terminations, which supports the observation of two sets of Dirac bands in α’-Bi4Br4 since both of the two edges are exposed to form the (100) plane15,16,17. Consequently, these two nondegenerate Dirac bands can be directly treated as the result of the double band inversions at both the M point and the L point of the 3D BZ under the action of SOC, and the single Dirac band at the \(\bar{{{{\rm{Z}}}}}\) point of the 2D (100) surface BZ of Bi4(Br0.2I0.8)4 is derived from the transition from double band inversion to single band inversion at the L point of the 3D BZ, which is confirmed by the surface band calculations of (100) plane of the heavily I doped crystal (see Supplementary Fig. 7 for details).

The HOTI nature of α’-Bi4Br4 is correlated with the change of the parity eigenvalues from (−−) to (++) at the M point of the 3D BZ after switching on SOC. The double band inversion at the L point makes no contribution to the symmetry-indicator topological nontrivial invariants due to the invariable parity eigenvalues (−+), as displayed in Fig. 4a, b. The transition from double band inversion to the single band inversion at the L point engenders the typical topological transition to the STI as the only inverted parity eigenvalue, as shown in Fig. 4c. Additionally, the double band inversion at the M point responsible for the HOTI characteristic is unchanged, giving rise to the unusual dual topological phase of HOTI and STI in α’-Bi4(Br0.2I0.8)4.

Fig. 4: Topological phase diagram of the Bi4(Br1-xIx)4 system.
figure 4

Schematic illustration of the band structure of α′-Bi4Br4 at the M (\(\bar{\Gamma }\)) point and L (\(\bar{{{{\rm{Z}}}}}\)) point of the 3D (two-dimensional (2D) (100)) BZ a without SOC and b with SOC. The parity eigenvalues are labeled on each band. The blue bands and red bands are correlated with the bulk structures contributed by the Biin component and Biout component, respectively. The green bands and yellow bands stand for the 2D surface states (SS) evoked by the double bulk band inversions at these two TRI momenta. The pink solid lines denote the one-dimensional (1D) hinge states (HS) of the high-order topological insulator (HOTI) state residing in the energy gap induced by the coupling of two nondegenerate SS. The parity eigenvalues switch from (--, ++) to (++, --) at the M point, corresponding to the trivial ℤ2 but nontrivial invariant ℤ4 = 2. BCB, bulk conduction bands. BVB, bulk valence bands. c Dual topological phase with the coexistence of HOTI and strong topological insulator (STI) states generated by the transition of double band inversions to a single band inversion at the L point of the 3D BZ. d Topological phase diagram of Bi4(Br1-xIx)4 and the bulk gap ΔB (brown squares) as a function of I content. The sizes of the SS gap \({\varDelta }_{S}\) around \(\bar{{{{\rm{Z}}}}}\) for samples with different I content are also marked. The solid (hollow) red squares and blue squares denote the energy positions of the bottom surface conduction band (BSCB) and top surface conduction band (TSCB) at \(\bar{{{{\rm{Z}}}}}\) (\(\bar{\Gamma }\)), respectively. NI, normal insulator. Error bars of I content and energy are determined by elemental variation measured by EDS and fitting uncertainty of EDCs, respectively.

Full size image

According to the above results, several TPTs are revealed in this quasi-1D bismuth halide system, and the electronic phase diagram of Bi4(Br1-xIx) is plotted in Fig. 4d. α’-Bi4Br4 is treated as the parent phase in this system. The \({\varDelta }_{S}\) resulting from the interlayer interaction is correlated with the HOTI gap, where the 1D hinge states reside. With increasing I concentration, both the \({\varDelta }_{B}\) and \({\varDelta }_{S}\) are diminished, which may be caused by the enlarged interchain space identified by our structural results in Fig. 1 and the reduced interchain coupling. Thus, the strain effect can be the main reason for the reduction of gap values (See Supplementary Fig. 15). With further increased I content, \({\varDelta }_{B}\) continually decreases and reaches zero at the critical point of I content around 0.73 (labeled by the green ball). After this TPT, the bulk gap reopens and the double band inversions at the \(\bar{Z}\) point turn into a single band inversion. The composite topological phase, STI plus HOTI, is generated with the coexistence of 2D surface states and 1D hinge states. The existence of the hinge states in this composite topological phase is supported by the calculation results of 20*a × 20*c α’-Bi4(Br0.25I0.75)4 model (see Supplementary Note 6 for details). Although these band inversions should induce a Z4 = 3 state and STI order according to the original theory37, the recent work indicates that the similar composite topological order can be realized in elemental As film38. It should be noted that the crystal structures are altered when the I content are 50% and 100%, forming the γ phase and α (β) phase, respectively. The γ phase is a structure including three alternately arranged layers, corresponding to a non-degenerate weak topological insulator (WTI), while the α phase and β phase are ascribed to a topological trivial insulator and a WTI according to topological band theory, respectively39. It should be noted that two Dirac bands similar to α’-Bi4Br4 can be identified in α-Bi4I4 at both the \(\bar{\Gamma }\) point and the \(\bar{{{{\rm{Z}}}}}\) point (see Fig. 3b, c). α-Bi4I4 also possesses the double-layer structure and the feature of double band inversions after considering SOC (see Supplementary Fig. 16 for details). Thus, such surface states of (100) plane also originate from the double band inversions regardless of their different symmetry-indicator topological invariants.

The first-order topological insulator and second-order topological insulator, which possess the varied bulk-boundary correspondence with 2D surface states and 1D hinge states, respectively, are intuitively regarded to be incompatible in one material due to their order sequence in topological theory and their dimensional discrepancy in real space. Our work providing compelling evidence for the realization of this composite topological phase, the coexistence of HOTI and STI, in a quasi-1D system with the parent phase of HOTI by varying the halide element ratio and the subsequent topological indices assigned at different TRI momenta in BZ. Moreover, a rich phase diagram containing abundant topological phases: high-order topology, weak topology, the unusual dual topology, and trivial/weak topology, is established following the multiple TPTs in this quasi-1D system, which provides an ideal material platform for developing topological materials and quantum devices that could have immense benefits for the fields of topological physics and materials science.

Methods

Single crystal synthesis and elemental identification

Solid-state reaction and the chemical vapor transport method were applied to grow the Bi4(Br1-xIx)4 single crystals. Highly pure Bi, BiBr3, and BiI3 powders were mixed under Ar atmosphere in a glove box and sealed in a quartz tube under vacuum, where the mole ratio of BiI3 to BiBr3 were adjustable from 0% to 100%. The mixture was placed in a two-heating-zone furnace with the temperature gradient from 558 K to 461 K for 72 h. Monocrystal Bi4(Br1-xIx)4 nucleated at the high-temperature side of the quartz after cooling down. The nominal and actual I content of each sample were identified by the mole ratio between the raw materials BiBr3/BiI3 and the energy dispersive spectroscopy (EDS) results for the synthesized single crystals, respectively.

XRD and HAADF-STEM characterizations

XRD measurements were performed in an AERIS PANalytical X-ray diffractometer at room temperature. The cleaved (001) surfaces were carefully set to be parallel with the sample stage, and the divergence slit was set at 1/8° with beta-filter Ni accompanied by Cu radiation. For STEM characterization, the transmission electron microscopy (TEM) lamella was prepared by using a Zeiss Crossbeam 550 focused-ion-beam (FIB)-scanning electron microscopy (SEM). Then, the characterization was conducted on a probe and image corrected FEI Titan Themis Z microscope equipped with a hot-field emission gun working at 300 kV.

STM measurement

The STM measurements were carried out using a low-temperature ultrahigh vacuum (UHV) STM/scanning near-field optical microscopy system (SNOM1400, Unisoku Co.), where the bias voltages were applied to the substrate. The step size of the (001) plane is consistent with monoclinic structure as c×sin107°/2 = 0.96 nm, and the step size for the (100) plane equals a/2 = 0.66 nm. With the projected crystal structure, the atomic-resolved topographies of the crystal structures of the (001) and (100) surfaces are shown in Fig. 1c, e respectively. The crystal parameters a, b, and c of α’-Bi4Br4 obtained by STM measurements were 1.33 nm, 0.44 nm, and 2.01 nm, respectively. The differential conductance spectra, dI/dV, were acquired by using a standard lock-in technique with a modulation at 973 Hz. All the STM measurements were performed at 77 K.

ARPES characterization

ARPES measurements were performed on in-situ-cleaved thick and homogeneous crystals, attached on the sample holder with torr seal glue with the exposed (001) or (100) surface on the sample holder. The Helium light (with photon energy ~21.2 eV) ARPES characterizations were performed at T = 6 K at the Photoelectron Spectroscopy Station in the Beijing Synchrotron Radiation Facility using a SCIENTA R4000 analyzer and in our laboratory using a DA30L analyzer. The total energy resolution was better than 15 meV, and the angular resolution was set to ∼0.3°, which gave a momentum resolution of ∼0.01 π/a. Laser-based ARPES measurements were performed using DA30L analyzers and vacuum ultraviolet 7 eV lasers at 77 K in Tsinghua University. The overall energy and angle resolutions were set to 3 meV and 0.2°, respectively.

Electric transport measurements

The standard four-wire method was used to perform the electric transport measurements. The geometry electrodes were fabricated on the freshly cleaved surfaces of Bi4(Br1-xIx)4 single crystals, and silver epoxy was applied to connect the samples and electrodes. The results were collected with a physical property measurement system (Quantum Design, Dynacool 9T) in our home laboratory.

First-principles calculations

The first-principles calculations were performed using the Vienna ab initio simulation package40 within the projector augmented wave method41 and the generalized gradient approximation of the Perdew-Burke-Ernzerhof42 exchange-correlation functional. The energy cut-off of 300 eV was used, and 9 × 9 × 3 and 11 × 11 × 4 Γ-centered k-grid meshes were adopted for structural relaxation and electronic structure calculations, respectively. 11 × 11 × 1 Γ-centered k-grid meshes were used for the Bi4Br4 monolayer electronic structure. Employing and fixing the experimental lattice symmetries, the crystal structure of Bi4(Br1-xIx)4 was fully relaxed with van der Waals correction until the residual forces on each atom were less than 0.01 eV/Å. The converging criterion on total energy is 1 × 10−6 eV in all calculations. The SOC effect has been considered in our calculations. A tight-binding (TB) Hamiltonian based on the maximally localized Wannier functions (MLWF)43 was constructed to further calculate the surface states with the WannierTools package44. The Bi4(Br1-xIx)4 models were generated by substituting different number of Br with I in a periodic cell containing eight Bi atoms and eight Br atoms.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.