Introduction

Topological matters have attracted extensive interest for their distinct and robust bulk-boundary correspondence1,2,3,4,5. In particular, the interplay between the crystal symmetry and electronic bands gives rise to a variety of topological states, such as the topological crystalline insulator6,7,8,9 and higher-order topological insulator (TI)10,11,12,13,14,15,16. Moreover, when the magnetism and topological bands are entangled together, it will greatly enrich more exotic topological states17,18,19,20,21,22,23, such as the quantum anomalous Hall effect20,21, axion TIs24,25, and magnetic topological semimetals26,27. More recently, a distinct type of topological state, namely the fragile topology28,29, has been proposed. Recent advances in magnetic space groups (MSGs) have also enriched the classification of magnetic fragile topology30,31. Different from the stable topology, the fragile topology generally cannot exhibit robust edge states. Meanwhile, it hosts the Wannier obstruction, which means that it should undergo a gap closing to evolve into a trivial insulator. Fragile topology shows many intriguing properties32,33,34,35,36,37. For example, the fragile topologically flat band contributes a nontrivial superfluid weight in 2D superconductor34,35, and hence enhances the superconducting transition temperature. Under a varying magnetic field, the corner states of the fragile Hofstadter topological system can pump into bulk by magnetic flux36. Besides the conceptual breakthrough, searching for more natural fragile topological materials is highly desirable.

As a wonderful platform to study the emergent phenomena among superconductivity, magnetism, and topological bands, FeSe has generated much attentions recent years, in which high-Tc superconductivity38,39,40, topological superconductivity41,42,43, topological states44,45,46,47 have been reported. In particular, Wang et al. have reported that the monolayer FeSe in the checkerboard antiferromagnetic (cb-AFM) phase is a 2D AFM TI46. After that, edge states are observed at the nematic domain boundaries of monolayer FeSe47. A bound state near the Fermi level is further observed at the end of boundaries. However, the topological origin of the bound state has not been discussed, and the topological nature in the cb-AFM monolayer FeSe remains elusive until now.

In this work, by using the first-principles calculations and magnetic topological quantum chemistry22,23,48,49,50,51, we systematically study the topological properties of the cb-AFM monolayer FeSe. We demonstrate that their low-energy physics can be well captured by a double-degenerate nearly flat band with fragile topology just below the Fermi level, which gives rise to a 2D second-order topological insulator (SOTI) that supports the bound state with fractional charge e/2 at the sample corner. The winding of square Wilson loop confirms that such fragile topology is protected by the S4z symmetry, which may lead to intriguing response to disorder and magnetic field. Further analyses find that cb-AFM monolayer FeSe is very close to a well-defined AFM TI with nontrivial spin-Chern number characterized by the Sz symmetry. These results can well explain the previous observations of topological edge states and bound state near the Fermi level46,47, and also provide a platform to study the intriguing properties of the magnetic fragile topology.

Results

Crystal and magnetic structure

As illustrated in Fig. 1a, b, both bulk and monolayer FeSe adopt the tetragonal lattice structure with the inversion symmetric space group P4/nmm (No. 129), in which Fe atoms occupy the Wyckoff position 2a (0, 0.5, 0), and Se atoms occupy the Wyckoff position \(2c^{\prime}\) (0, 0, z). In this work, we mainly focus on the band structures and topological properties in the cb-AFM phase of monolayer FeSe as shown in Fig. 1a, c, d. In this magnetic configuration, the nearest neighboring Fe ions are antiferromagnetically coupled to each other and the inversion symmetry is broken, leading to a MSG \(P4^{\prime}\)/\(n^{\prime} m^{\prime} m\) (No. 129.41652). \(P4^{\prime}\)/\(n^{\prime} m^{\prime} m\) is generated from a space-time inversion \({{{\mathcal{PT}}}}\), a twofold screw C2x (\(\{{\hat{C}}_{2x}| \frac{1}{2},0,0\}\)), and an improper fourfold rotation S4z (\(\{{\hat{C}}_{4z}{\hat{M}}_{z}| 0,\frac{1}{2},0\}\)).

Fig. 1: Illustration of the unit cell and three boundary cuts for cb-AFM monolayer FeSe.
figure 1

a The crystal and magnetic structure of monolayer FeSe. b Maximal symmetry Wyckoff positions in MSG \(P4^{\prime}\)/\(n^{\prime} m^{\prime} m\). Since monolayer FeSe is 2D, only the z = 0 Wyckoff positions are manifested. c, d Three distinct boundary cuts. Cut #1 represents the boundary of AFM edge. Cuts #2 and #3 represent the boundary of FM edge with spin-up and spin-down polarization along c-direction.

Full size image

Electronic structures and fragile topology

First, we carry out the calculations without spin-orbit coupling (SOC) of the cb-AFM monolayer FeSe, and plot the corresponding projected density of states (DOS) of the 3d-orbitals on Fe2 ion (see Fig. 1a) in Fig. 2a. It gives rise to a Zeeman splitting about of 2.5 eV between the majority spin states and minority spin states. As a result, most majority spin states are pushed below −0.25 eV, while the states between −0.25 ~ 2.25 eV are mainly contributed by the minority spin 3d-orbitals. Further analysis demonstrates that the low-energy bands around the Fermi level (0 eV) are dominated by the minority spin of \({d}_{{z}^{2}}\)- and dyz/xz-orbital, which have very weak hybridization with the majority spin components as shown in Fig. 2a. These features can also be represented by the spin-polarized band structures as shown in Fig. 2b, which indicates that five majority spin bands are fully occupied and only one minority band is occupied. Such occupancy actually coincide with the 3d6 configuration of the Fe2+ ion. Further detailed orbital components analysis demonstrates that the one occupied minority band is mainly contributed by the \({d}_{{z}^{2}}\)-orbital, except a band inversion with the dyz/xz-orbital at the M point near the Fermi level (see Fig. 2c). The band inversion leads to a semimetallic feature with a parabolic touching at the M point, as shown in Fig. 2b. When the SOC is included, the gapless touching of dyz- and dxz-orbital at M point is opened with an insulating gap about of 50 meV. Since the dispersion of the \({d}_{{z}^{2}}\) band is very weak, the gap opening also leads to an isolated nearly flat band just below the Fermi level, as shown in Fig. 2c. We further note that each band is double-degenerate in the presence of \({{{\mathcal{PT}}}}\) symmetry.

Fig. 2: The electronic structures of cb-AFM monolayer FeSe.
figure 2

a Projected DOS of the 3d-orbitals on Fe2 for the calculations without SOC. b Spin-polarized band structures with the red (blue) color indicating the minority (majority) spin states. c Band structures with SOC, the red and blue color indicate the weights of \({d}_{{z}^{2}}\)-orbital and dyz/zx-orbital, respectively. d The Wilson loop spectra of the isolated nearly flat band. The red (blue) lines represent the square (rectangle) Wilson loop spectra with loops patch in the upper (lower) right inset panel, where the arrow indicate the direction of loops.

Full size image

To investigate the topological properties of the cb-AFM monolayer FeSe, we employ the symmetry analysis under the theory of magnetic topological quantum chemistry22,23,48,49,50,51, in which the magnetic elementary band representations (MEBRs) as the basis of magnetic atomic insulators (AIs) are defined. Within this theory, if the irreducible co-representations (co-irreps) of the occupied bands at high symmetry momenta points can be decomposed into the linear combination of MEBRs with positive integer, the system is equivalent to a magnetic AI. Otherwise, the system must be topologically nontrivial. We have calculated the co-irreps of all occupied bands with SOC, which indicate that cb-AFM monolayer FeSe is a magnetic AI, but not the AFM TI reported in refs. 46,47. As pointed out in ref. 28, the band structures of AI have two combination of manners: (i) AI = AI + AI, (ii) AI = AI + fragile topological bands. When we focus on the isolated nearly flat band just below the Fermi level, the calculated co-irreps give rise to {\({\overline{{{\Gamma }}}}_{7}\), \(\,{\overline{X}}_{2}\,{\overline{X}}_{4}\), \(\,{\overline{M}}_{6}\)} as shown in Fig. 2c, and its fragile topological characteristic will be discussed below. Comparing to the MEBRs of the MSG \(P4^{\prime}\)/\(n^{\prime} m^{\prime} m\) tabulate in Table 1, one can find that the co-irreps of the isolated nearly flat band can only be expressed as a combination of MEBRs with negative integer:

$$\{{\overline{{{\Gamma }}}}_{7},\,{\overline{X}}_{2}\,{\overline{X}}_{4},\,{\overline{M}}_{6}\}={(\overline{E})}_{2c}\uparrow G\ominus {({\!}^{2}{\overline{E}}_{2})}_{2a}\uparrow G,$$
(1)

the negative coefficient ⊖ manifests that it is a fragile topological band28,29.

Table 1 The MEBRs induced by maximal symmetry Wyckoff positions in \(P4^{\prime}\)/\(n^{\prime} m^{\prime} m\)22,52.
Full size table

Such fragile topological characteristic can be usually revealed by the nontrivial winding of Wilson loop along some symmetric path over the whole Brillouin zone (BZ)30,53,54. Here, we design a series of square loops li that satisfy S4z symmetry, as shown in the upper right of Fig. 2d, with l0 at the M point and lend corresponding to the boundary of the first BZ. The calculated Wilson loop spectra of the fragile topological band are plotted as red lines in Fig. 2d, which give rise to a conjugate pairs {eiθ, e−iθ} for each li yielding to the \({{{\mathcal{PT}}}}\) symmetry55. More importantly, each branch of the Wilson loop spectra evolves a 2π phase from the M point to the whole BZ, leading to a nontrivial winding number 1. We also design the other series of rectangle loops that break S4z symmetry, as shown in the lower right of Fig. 2d. The corresponding Wilson loop spectra are plotted as blue lines in Fig. 2d, which obviously show that the crossing at θ = ±π is opened, resulting in a trivial winding number 0. Therefore, we conclude that the fragile topological band in cb-AFM monolayer FeSe is protected by the S4z symmetry. It may lead to intriguing response to disorder, comparing to the inversion symmetry-protected fragile topological band56,57.

In addition to the fragile topology, we would like to point out that the cb-AFM monolayer FeSe is very close to a stable AFM TI. Such stable AFM TI phase is protected by the spin-conserving symmetry Sz, which decouples the interactions between spin-up and spin-down flavor. In general, the Sz symmetry is broken in realistic electron systems58,59. However, we demonstrate that the Sz-breaking interaction in cb-AFM monolayer FeSe is very tiny, which can be verified by two bodies of evidence. First, we have symmetrized the Wannier Hamiltonian \({\hat{H}}_{0}\) of cb-AFM monolayer FeSe by an additional Sz symmetry, i.e. \({\hat{H}}_{S}=({H}_{0}+{S}_{z}{\hat{H}}_{0}{S}_{z}^{-1})/2\), and plot the band structures of \({\hat{H}}_{S}\) (red dash line) as a comparison to the bands of \({\hat{H}}_{0}\) (blue solid line) in Fig. 3a. The red bands in Fig. 3a match the blue bands very well, which strongly implies that the \({\hat{H}}_{0}\) is approximate to having Sz symmetry. The approximately Sz symmetry is also consistent with our first-principles calculations, which display the weak hybridization between majority spin states and minority spin states as shown in Fig. 2a, b. Second, one can quantitatively estimate the Sz-breaking process as a competition between the SOC interaction λsoc and the onsite magnetic polarization splitting Δ, where λsoc regards the spin mixing ability and Δ preserves spin conserving30. Reminding that the magnetic splitting Δ in cb-AFM monolayer FeSe is estimated as ~2.5 eV from our first-principles calculations, and the λsoc of Fe is usually considered as ~0.03 eV. Such a huge energy difference makes the Sz symmetry be a good approximation in cb-AFM monolayer FeSe.

Fig. 3: The electronic structures of Wannier Hamiltonian \({\hat{H}}_{0}\) and those with additional Sz symmetry \({\hat{H}}_{S}\).
figure 3

a Band structures of \({\hat{H}}_{S}\) (red dash line) and \({\hat{H}}_{0}\) (blue solid line). b Edge states of the \({\hat{H}}_{S}\) (red) and \({\hat{H}}_{0}\) (blue), with a surface onsite energy −0.1 eV. c, d The Wilson loop spectra of all occupied bands in \({\hat{H}}_{S}\) and \({\hat{H}}_{0}\), respectively.

Full size image

When Sz symmetry is restored, the system can be considered as two copies of spin-Chern insulators that are connected by the \({{{\mathcal{PT}}}}\) symmetry. In Fig. 3b, we calculate and plot the edge states of \({\hat{H}}_{S}\) on the AFM boundary (cut #1 in Fig. 1c) as red lines, which shows that two edge states cross each other exactly and form a massless Dirac cone at the Γ point. As a result, we confirm that the \({\hat{H}}_{S}\) describes an AFM TI protected by the Sz symmetry. As a comparison, we also plot the calculated edge spectra of \({\hat{H}}_{0}\) on #1 boundary as the blue lines in Fig. 3b. As expected, the edge spectra of \({\hat{H}}_{0}\) are very similar to those of \({\hat{H}}_{S}\), except a very tiny massive gap about of 0.5 meV at the Γ point yielding to the Sz symmetry breaking. These results confirm again that the Sz-breaking is very weak in the cb-AFM monolayer FeSe, which is very close to a well-defined AFM TI. This is why the quantum spin Hall-like edge states have been reported and observed in refs. 46,47. Apart from the edge states, the topological properties can also be revealed by the Wilson loop spectra of all occupied bands60. In Fig. 3c, d, we plot the Wilson loop spectra along the Γ − M path of \({\hat{H}}_{S}\) and \({\hat{H}}_{0}\), respectively. The spectra in Fig. 3c obviously show gapless crossing at the time-reversal momenta points, reflecting a well-defined spin-Chern number ∣Cs∣ = 1. However, the Wilson loop spectra of \({\hat{H}}_{0}\) are generally gaped, indicating that cb-AFM monolayer FeSe is not a well-defined AFM TI.

Corner states

The fragile topology usually relates to 2D SOTI and leads to filling anomaly15. Such filling anomaly could give rise to partially occupied corner states at the sample corner13,15. In the system with S4z symmetry, one can identify 2D SOTI phase by the S4z eigenvalues of occupied bands12,16. We note that for nonsymmorphic MSG \(P4^{\prime}\)/\(n^{\prime} m^{\prime} m\), the S4z eigenvalues at Γ point have to come in complex-conjugate pairs \(\{\xi {\exp }^{i\pi /4},\xi {\exp }^{-i\pi /4}\}\), whilst come in \(\{\xi {\exp }^{i\pi /4},-\xi {\exp }^{-i\pi /4}\}\) at M point and C2z eigenvalues (C2z = \({S}_{4z}^{2}\)) at X point come in \(\{\xi {\exp }^{i\pi /2},\xi {\exp }^{i\pi /2}\}\) with ξ = +1 or ξ = −1. By counting the numbers of ξ = +1 and ξ = −1 states at Γ, M and X point, we find the following Fu-Kane-like formula could mapping the invariant indicator Z2,

$${(-1)}^{{Z}_{2}}=\mathop{\prod }\limits_{n=1}^{N/2}{\xi }_{n}({{\Gamma }}){\xi }_{n}(M){\xi }_{n}(X),$$
(2)

where N is the number of occupied bands. Comparing with the MEBRs as tabulated in Table 1, we find the MEBRs induced from the atomic site 2a are fall into the Z2 = 0 phase, while Z2 = 1 indicates a 2D SOTI that would give rise to corner states. The calculated results of all occupied bands are ξ(Γ) = −1, ξ(M) = 1 and ξ(X) = 1, which certify that cb-AFM monolayer FeSe falls into the 2D SOTI phase with Z2 = 1.

The 2D SOTI property can also be understood from the opposite mass term of the edge states on two different FM boundary (cut #2 and cut #3 in Fig. 1d)61. We calculate and plot the spin-polarized edge states along the #2 and #3 boundary in Fig. 4a and b, respectively. In both of which, a massive gap can be observed between two opposite spin-polarized edge states. Further analysis shows that the gap opening on the spin-up boundary (#2) is induced by a mass term m#2 ∝ σy, while the gap opening on the spin-down boundary (#3) is induced by an opposite mass term m#3 ∝ − σy, as shown in the inset of Fig. 4a and b, respectively. The opposite mass term is consistent with previous work that two boundaries are related by the \({C}_{4z}{{{\mathcal{T}}}}\) symmetry12. Therefore, one can expect that the mass terms could cancel each other at the intersection of #2 and #3 boundary, which can induce corner states at the 2c position.

Fig. 4: The edge states and corner states of cb-AFM monolayer FeSe.
figure 4

a, b Spin-polarized edge states along the cuts #2 and #3 boundary, respectively. The weights of positive (negative) σz eigenvalue is characterized as red (blue). The inset pictures show the calculated mass term σy near the massive Dirac cone. c Energy spectra of the 20 × 20 finite square structure. d Spatial distribution of the two occupied corner states.

Full size image

In Fig. 4c, we plot the low-energy spectra of a square sample with the center at 2c position31 constructed by 20 × 20 unit cells by using the the Wannier Hamiltonian \(\hat{{H}_{0}}\). As expected, four corner states including two occupied and two unoccupied (blue dots in Fig. 4c) appear near the Fermi level. However, because \(\hat{{H}_{0}}\) has no chiral symmetry, the four corner states are not pinned at zero energy exactly. In Fig. 4d, we plot the spatial distribution of the two occupied corner states, which are localized at four corners and each corner has e/2 charge, corresponding to previous conclusions of the 2D SOTI protected by \({C}_{4z}{{{\mathcal{T}}}}\) symmetry very well16. Finally, we emphasize that such calculated corner states are strongly supported by the previous experimental observations, in which bound state near the Fermi level has been observed at the crossing point of four domain walls by scanning tunneling microscopy47. We would like to suggest the experimenters to measure the charge of each bound state, which will be very helpful to confirm its topological origin.

Discussion

At last, we would like to discuss that the 2D SOTI phase and corner states can only exist in the cb-AFM phase, but cannot be realized in other magnetic phases such as the nonmagnetic or stripe AFM phase. As we all know, the corner states are associated with the mismatch between the atomic sites and the charge centers13. As shown in Fig. 1c, d, the electrons on the nearest neighboring Fe ions are oppositely polarized in the cb-AFM configuration, which could strongly suppress the nearest neighbor hopping t1(see Fig. 1c), and makes the next nearest neighbor hopping t2 dominate. The dominant t2 will lead to the charge centers move to the 2c site, and give rise to corner states at the sample corner. However, t1 is always dominant in the NM or stripe AFM configuration. This makes the charge centers locate at the 4d site (see Fig. 1b), which cannot support the corner states appearing at the sample corner. Therefore, our calculations, combining with the observation of the corner states47, strongly suggest that the cb-AFM phase is realized in the few layers of the tensile FeSe.

Methods

First-principles calculations

Our first-principles calculations are performed by the Vienna ab initio simulation package62,63. Similar as in ref. 47, Perdew–Burke–Ernzerhof type of the generalized gradient approximation + Hubbard U (GGA + U)64,65 is used as the exchange-correlation potential with U = 1.0 eV and J = 0.2 eV for the Fe 3d-orbitals. The crystal parameters a = 3.85 Å and Se atomic coordinates z = 0.255 are used in all calculations. The cut-off energy for the wave function expansion is set to 400 eV, and a 9 × 9 × 1 k-mesh in the first BZ is used for self-consistent calculations. The SOC is considered self-consistently. We then construct the Wannier functions without performing maximally localized procedure for the Fe 3d-orbitals using WANNIER9066.