Introduction

Topological insulators have attracted a tremendous amount of attention because of their peculiar surface states with spin chirality1,2. The surface states of topological insulators arise from the non-trivial topology of the bulk electronic structure, whose existence and chiral nature (with spin-momentum locking), are protected by the topology. However, most topological insulators, such as binary compounds based on Bi and Sb, have exhibited metallic conductivity dominated by bulk carriers that originates from doping or formation of an impurity band by unintentional defects3,4,5,6,7. This hinders the observation of novel physics from the topological surface states, because many theoretical proposals with respect to topological insulators are based on the assumption that the surface states, Fermi energy and Dirac point are all located inside an energy gap without any bulk carriers8,9. The realization of topological surface states in a bulk-insulating sample is therefore a critical step towards both fundamental studies and possible applications of topological insulators, such as realization of topological quantum computing through the generation of Majorana fermions1,2,7.

Recently, a large bulk resistivity was realized in the Bi-based ternary compound, Bi2Te2Se10. Clear signatures of the chiral surface states were observed, including spin polarization found by spin-polarized angle-resolved photoemission spectroscopy (ARPES)11, the non-trivial Berry phase in quantum oscillations10 and unique scattering properties of electrons observed by scanning tunnelling microscopy (STM)12,13. Further improvement on the bulk-insulating properties was achieved in the quaternary compound, Bi2−xSbxTe3−ySey (BSTS), where the Fermi energy was reportedly tunable by adjusting the stoichiometry of the four elements14,15,16,17,18. Even though the amount of alloying is substantial, crystallinity of BSTS has been confirmed for a wide range of compositions16 and the electronic properties (of BSTS) that stem from topological surface states have been observed by various methods, such as ARPES and electronic transport measurements14,15,16,17,18. However, in contrast to Bi2Te2Se, where the ordering of chalcogens within a quintuple layer is considered to be important for reducing defects, the crystallographic order of BSTS is expected to be highly suppressed by alloying of chalcogens (Se and Te) and pnictogens (Bi and Sb). In this respect it is crucial to clarify whether the reduced structural order degrades the topological properties of the surface states by gaining local information on the atomic scale, something that has not yet been attained in BSTS12,19.

Here we report on the first observations of the BSTS surface with STM and scanning tunnelling spectroscopy (STS). STM is a powerful technique for investigating surface states because of its extremely high spatial resolution and surface sensitivity. Many essential characteristics of topological insulators have been studied in binary compounds by STM and STS, such as the spin texture and non-trivial Berry phase20,21, quantization of massless Dirac fermions in a magnetic field22,23,24 and interaction of the topological surface state with various types of defects and adatoms19,25,26,27. Using STM we examined the x = 0.5, y = 1.3 composition of BSTS, Bi1.5Sb0.5Te1.7Se1.3, as it has the most insulating bulk among the possible reported stoichiometries16. Although each layer is a mixture of two elements, a well-ordered hexagonal lattice is observed on the atomically flat surface. Scattering of electrons in the surface state was revealed by STS results and it clearly demonstrated the two important characteristics of the topological surface state: a linear dispersion and a suppression of back-scattering1,2. STS also revealed that the Fermi energy is located inside the bulk energy gap. All the evidence presented here is consistent with the existence of topological surface states on the surface of BSTS with a fully insulating bulk.

Results

Figure 1a shows the crystal structure of BSTS: a hexagonal crystal structure with a rhombohedral unit cell consisting of five atomic layers along the crystallographic c axis (otherwise known as quintuple layers). Each layer forms a binary alloy Bi2−xSbx and Te3−ySey with two-dimensional hexagonal lattices perpendicular to the c axis. Two adjacent layers are strongly coupled within one quintuple layer, but are weakly coupled between two quintuple layers16.

Figure 1
figure 1

Atomic structure and resistivity of the BSTS samples.

(a), Crystal structure of Bi2−xSbxTe3−ySey and optical image of the single crystal. (b), Temperature dependence of the resistivity of Bi1.5Sb0.5Te1.7Se1.3 samples, normalized by their room temperature values. In total, 5 crystals (S1-100 μm, S1-90 μm, S1-80 μm, S2-80 μm and S2-140 μm) were cut and measured, where they were labelled with the sample batch (S1 and S2) and their thickness in μm. The inset shows the Arrhenius plot for the normalized resistivity; the solid lines are the fits at high temperatures, yielding an activation energy of ~30 meV.

Full size image

The typical resistivity of BSTS crystals as a function of temperature exhibited semiconducting behaviour down to 2 K, as shown in Fig. 1b. This measurement was repeated for several samples taken from two batches (S1 and S2); most of which showed a negative temperature dependence (dρ/dT < 0) at temperatures below 100 K. That the resistivity ratio, ρ(2 K)/ρ(300 K), exceeds ~100 and the resistivity is as high as 2–7 Ω·cm at 2 K, clearly indicates the bulk-insulating character of our BSTS crystals. The Arrhenius plot for the resistivity in the inset of Fig. 1b shows a rapid increase of the bulk resistivity due to the reduction of the thermally excited bulk carriers. The activation energy, estimated from the data at high temperatures, is ~30 meV. A rough estimate of the bulk conductivity determined by extrapolating from the high temperature data is at least two orders or magnitude smaller than the experimentally measured value at ~20 K, as shown in the inset of the Fig. 1. At low temperatures, the resistivity becomes saturated as the contribution from surface conduction becomes significant. Systematic studies of the thickness dependence of the resistivity on our samples recently demonstrated that the surface sheet conductivity Gs ~ 3.5 × 10−4 Ω−1 and the bulk conductivity Gb ~ 0.94 Ω−1cm−1 at low temperatures28. For our crystals, the surface contribution remains ~4% of the bulk contribution, which is relatively large considering the typical thickness of the crystal ~100 μm16.

The weak bond between quintuple layers breaks easily, so that the BSTS crystals can be cleaved along the interface between these layers; the revealed surfaces are composed of Te and Se. The crystalline structure of the cleaved surfaces was confirmed by STM topographs. The freshly cleaved surfaces showed atomically flat terraces as large as a few hundred nanometres wide (Fig. 2a) and individual atoms in the top surface layer that could be clearly resolved by magnifying the flat region (Fig. 2b). The surface atoms showed a well-ordered hexagonal structure with a lattice constant of 4.2 ± 0.1 Å, in agreement with the previously reported value16. The step height between terraces was 10 ± 1 Å, corresponding to the height of one quintuple layer (inset of Fig. 2a). Even though the surface is atomically flat some corrugations still appear (such as small dents appearing as dark coloured dots), as shown in Fig. 2b. We suggest that these corrugations were not defects due to several characteristics that they possessed: firstly, the hexagonal lattice continued without disruption over the corrugations (see supplementary information, Fig. S1); secondly, the peak-to-peak variation in the height, 1.0 Å, was much smaller than the lattice constant; and finally, none of the corrugations showed a scattering wave emanating from them, which is typical for defects in other materials (Fig. 3a, c–i)19,26. These corrugations may be the result of small structural or electronic distortions caused by alloying. Actual defects on the surface were extremely rare: only a couple or so of them were observed in a 50 nm × 50 nm area. When they were observed, such defects typically appeared as atomic size protrusions or triangular shaped features, such as with other Bi and Sb based compounds19,26,29 (see supplementary information, Fig. S2).

Figure 2
figure 2

STM topographs.

(a) and (b), STM topographs on a flat area where the step edges and atomic lattices were visible. In the inset of (a), a cross section of the topograph along the blue line marked ‘A’ is plotted. For (a), the scan size was 250 nm × 250 nm with a bias voltage, VB = 0.5 V and a set point current, I = 1 nA. For (b), the scan size was 10 nm × 10 nm with VB = −0.4 V and I = 0.2 nA.

Full size image
Figure 3
figure 3

Conductance spectroscopy and maps.

(a), Topograph of the area where the conductance spectra were taken (50 nm × 50 nm, VB = 0.6 V, I = 0.5 nA). (b), Plot of the average conductance spectrum over 256 × 256 regularly spaced points in (a). (c–i) and (j–p), Conductance maps generated from the spectra and their FTs, respectively (VB = −300, −140, 0, 140, 280, 440 and 600 mV). For the FT maps, the centre corresponds to the Γ point of the surface Brillouin zone; the horizontal line indicates the Γ-M direction. All the FT maps were fully symmetrised according to the symmetry of the lattice and then low-pass filtered to reduce the speckle noise. The modulation voltage and frequency for the lock-in amplifier was VM = 5 mV and fM = 1,317 Hz, respectively.

Full size image

The electronic structure of the sample can be unveiled through differential conductance g(r,E), measured by STS, where r is the position vector and E is the energy from the Fermi level inferred from the sample bias voltage, VB, as E = eVB (where e is the electron charge). This quantity is proportional to the electron local density of states (LDOS) and the spatial map of g(r,E) at a certain energy reflects the variation of the LDOS. We obtained the conductance spectra at 256 × 256 regularly spaced points covering the area shown in Fig. 3a; Fig. 3b shows the average of the spectra taken. The averaged conductance spectrum is similar to typical trends for Bi2Se3 and Bi2Te318,19,22,26, which implies that they share a similar band structure with the topological surface state. The positions of the valence band (VB), conduction band (CB) and surface state (SS) were determined by comparing the STS data with previously reported ARPES data14 and substantiated by analyzing the Fourier transform (FT) of the conductance maps shown later in this paper (Fig. 4g). We interpret this spectrum as showing an energy gap of 300 meV that embraces the SS, as denoted in Fig. 3b. The boundaries between the bands are estimated to be located at −40 meV for the VB and SS band and at 260 meV for the CB and SS band. In contrast to typical semiconductors, there was a non-zero conductance inside the energy gap, with a dip that fits to the SSs with a cone shaped band observed by ARPES14. The minimum point of the conductance spectrum corresponds to the Dirac point, which was estimated to be at −3 ± 19 meV. The amount of variation, 38 meV, is comparable to that measured in doped Bi2Te3 and Bi2Se326 and indicates that the topological surface states are well-preserved against a substantial amount of alloying.

Figure 4
figure 4

Band structure and FT of the STS.

(a), (c) and (e) are schematics of constant energy contours in k space and (b), (d) and (f) are corresponding FT maps in q space, at VB = 440, 280 and 140 mV, respectively. Prominent scattering vectors, such as the ones from backscattering (qb) and the warping effect (qwM and qwK), are plotted as dotted lines and solid lines, respectively. (g), A line-cut of the FT maps along the Γ-K and Γ-M directions stacked along the bias voltage, VB. The colour scale indicates the amplitude of the FT, the x axis corresponds to the scattering vector q (≡|q|) and the y axis corresponds to VB. A schematic of the band structure is drawn on the right hand side, where the bulk band is coloured in grey and the surface band is coloured in blue and red for spins up and down, respectively.

Full size image

To see the spatial dependence of the LDOS at different energies, maps of the differential conductance were taken for a bias voltage range of −300 mV to +600 mV in 20 mV steps. To enhance the observed features we plotted conductance maps at a normalized conductance, (where and σg are the average and standard deviation of g in each map, respectively), so that the average of gn was 0 and the standard deviation was 1. According to scattering theory, electrons on the surface of materials can be scattered by defects and make an interference pattern as a standing wave. This is often called quasiparticle interference (QPI)31,32,33, where scattering vectors, q, of QPI have been measured in various materials20,21,30,34,35 and are directly related to the band structure of the materials. While the bulk states do not usually develop well distinguished QPI patterns, because of the surface sensitive nature of the STM and possible scattering vectors having broad and continuous values, QPI by surface state electrons often appears as well-distinguished standing waves. Since topological surface states exist on the BSTS surface, a standing wave pattern from QPI is expected, as shown in Fig. 3e–i. That the wavelength of the standing wave becomes shorter as the bias increases reflects the dispersion of the topological surface states. This dispersion starts at around −40 mV and continues up to the maximum measured bias of 600 mV (which implies the existence of surface states up to that energy). However, it should be noted that conductance maps below −40 mV showed a drastic change where the corrugations shows no periodicity and no sign of QPI, indicating the disappearance of the surface state and a transition to the bulk valence band.

Detailed information of the QPI, such as the magnitude and direction of the scattering vectors, can be extracted from the Fourier transforms (FTs) of the conductance maps (or FT maps in short, Fig. 3j–p)32. The intensity in the FT map corresponds to the amount of QPI at a certain scattering vector q20,21. The FT maps may be divided into two different types: one above and the other below −40 mV. At a bias larger than −40 mV, a high intensity region around the centre is observed that increases in size as the energy increases (Fig. 3l–p and 4g). At a bias below −40 mV, on the other hand, the FT amplitude becomes extremely large (compared to FT maps for a bias above 40 mV), without the appearance of any prominent scattering vectors (Fig. 3j, k; also, see the normalized FT maps in supplement Fig. S3). This observation agrees with the band structure of the bulk valence band under −40 mV, with topological surface states starting from −40 mV and dispersing all the way up to 600 mV. For energies larger than 400 mV, another high intensity region appears at the centre of the map (Fig. 3o, p and Fig. 4g). Recent studies have shown that another type of surface state can be induced inside the energy of the bulk conduction band of chalcogenides due to band bending at the surface36. We suspect that the band starting at 400 mV is such a type of surface state, which indicates that the band edge of the conduction band should occur at some value below 400 mV.

Discussion

Topological surface states are usually characterized by two unique features: the linear dispersion of the band with a degenerate energy where all bands converge to one point; and a spin texture where an electron with momentum k has a unique spin direction perpendicular to k1,2. To confirm these features, we compared the expected constant energy contours of the surface states in BSTS (Fig. 4a, c, e) to our FT maps (Fig. 4b, d, f). The constant energy contours were drawn with the assumption that the Fermi velocity, vF = 4.0 × 105 m/s and the Dirac point, ED = 0 mV, based on previously measured transport and ARPES results14,16,17. Even though the FT maps showed high intensity spots at non-zero q, we found that none of them corresponded to the back-scattering vector qb. Because back-scattering in the topological surface state is strictly prohibited (due to spin texture)37, the absence of back-scattering is a strong indicator of the topological nature of the surface state.

If the surface state band had a perfect cone shape, there would be no high intensity point in the FT maps except at the Γ point37. Indeed, the FT maps taken at the energy near the Dirac point (i.e., −40–160 mV), showed no pronounced peak except at the origin (Fig. 3l, m). For energies higher than 180 meV, however, high intensity peaks were observed at certain values of q, such as qwM and qwK. This phenomena has been observed in various chalcoginides11,12,26,30,38,39 and is well-described by the distortion of the topological surface state band from a perfect cone to a warped shape (usually called the “warping effect”)40. As the surface state approaches the conduction band, the shape of the constant energy contours changes from a circle to a rounded hexagon and then to a star shape (Fig. 4a, c, e). Warping of the constant energy contours results in two important effects: the distortion in spin texture with the out-of-plane component and a nesting effect from the parallel part of the constant energy contours. Oblique scattering (that is not strictly prohibited by spin texture), is enhanced by these effects and gives a strong QPI that appears as high amplitude points in the FT maps (Fig. 4b, d, f).

The energy dependence of the electron scattering can be emphasized by plotting cross sections of the FT maps along the Γ-K and Γ-M directions stacked in the vertical direction for all measured energies (Fig. 4g). The high amplitude region shows a linear dispersion that crosses the y-axis at E = 0 meV, indicating a linear dispersion of the surface state with a Dirac point at 0 meV. All the band structure information (including the energy range and dispersion of bulk and surface bands), as summarised in the schematic given on the right side of Fig. 4g, matches well with previous transport and ARPES results14,16,17.

In summary, we have studied the electronic structure of a single crystal of the topological insulator, BSTS, which exhibited improved bulk-insulating properties due to alloying. STM measurements revealed an atomically flat surface with a hexagonal lattice. From STS measurements (including the resultant conductance maps and FTs), we confirmed the presence of a bulk energy gap and showed that it encompassed the Fermi energy and the surface state. QPI measured by FT maps indicated a linear dispersion and suppression of the back-scattering of electrons in the surface state. Our results clearly demonstrate that the topological nature of the surface state is robust against the alloying of the constituent elements. Chemical alloying, therefore, is a promising route to achieve full suppression of bulk conduction in topological insulators whilst keeping the topological surface state intact. These results suggest that BSTS could be a highly promising material as it exhibits both the Fermi energy within the bulk gap and a suppression of bulk carriers: requirements for many theoretical proposals for topological insulators.

Methods

Single crystals of Bi1.5Sb0.5Te1.7Se1.3 were grown by the self-flux method, in which high-purity starting materials (Bi, Sb, Te and Se) were mixed and annealed according to a previously reported recipe27. Samples were checked by energy dispersive spectroscopy and X-ray diffraction to verify the stoichiometry and high crystallinity, respectively. The mobility of a thick flake cleaved from the sample reached 4,000 cm2/V·s at 4.2 K in a separate Hall measurement28, which is comparable to other results reported on binary compounds5,28. For STM measurements, a Unisoku low-temperature STM was used. The sample was cleaved in an ultrahigh vacuum chamber (~10−10 Torr) at room temperature and then transferred to the low-temperature STM sample stage, where the temperature was kept at 2.8 K. Spectroscopy of the differential conductance was performed by a conventional lock-in technique.