Introduction

With the rapid progress in classification of topological electronic states1,2,3,4,5, different kinds of topological materials are predicted theoretically and confirmed experimentally6,7,8. However, the prior studies are mainly focused on the first-order topological materials9,10,11, where the topological boundary state only appears at dimension one lower than that of the bulk state. When this conventional bulk boundary correspondence is extended to the higher-order form12, a new class of topological materials called higher-order topological insulator emerges13,14,15,16. The m-dimensional nth-order topological insulator holds gapless state at (m-n)- dimensional boundary but gapped state otherwise. Therefore, the two-dimensional (2D) second-order topological insulator (SOTI) is characterized by the gapped topological edge state and in-gap topological corner state. Different to the symmetry-protected bulk topological index, physically, the emergence of in-gap topological corner state can also be understood by the well-known mass-inversion mechanism in Jackiw-Rebbi model17, which can support a localized domain wall state, as shown schematically in Fig. 1a.

Fig. 1: Fractional mass-kink induced topological corner state with a fractional charge of e/4.
figure 1

Schematic mass distribution along the edge (red line) and localized domain wall state (blue line) for a mass-inversion, b fractional mass-kink. The mass M is defined in edge state space, m is mass-intensity, θ is mass-angle, Δθ is phase shift across the domain wall, τx and τy are Pauli matrix of the edge state. c Schematic checkerboard AFM order in monolayer FeSe cluster and the gapless topological edge states. d Schematic real-space distribution of gapless topological edge state (QSH edge state), showing the propagating direction (arrow) and spin (red and blue color). e Schematic canted checkerboard AFM order in monolayer FeSe cluster and the gapped topological edge states. Each corner is connected by two ferromagnetic edges with the same spin. The adjacent edges have a different mass-angle of θ and θ + π/2. f Schematic real-space distribution of in-gap topological corner state, holding a fractional charge of e/4.

Full size image

Currently, although a variety of theoretical models are proposed for the 2D SOTI18,19,20,21,22,23,24,25,26,27,28,29,30,31,32, its experimental measurement is very difficult. Besides some artificial structures33,34,35,36,37,38,39, few realistic electronic materials have been reported. What’s more, the limited SOTI materials, such as graphdiyne40,41, γ-graphyne42 and twisted-bilayer graphene43,44,45, are all captured by the mass-inversion mechanism. Recently, this physical mechanism is further extended to a more universal form for generating the domain wall state46. In the helical Dirac edge state space, the edge state Hamiltonian is written as \(H = k\tau _z\), where \(\tau _z\) is Pauli matrix. In order to gap out this edge state, a general mass Hamiltonian can be written as \(M(\theta ) = m(\tau _x\cos \theta + \tau _y\sin \theta )\), where m is mass-intensity and θ is mass-angle that controls the mass projection on Pauli matrix \(\tau _x\) and \(\tau _y\). As shown schematically in Fig. 1b, the mass distribution along the edge is characterized by mass-angle, which is θ and \(\theta + {\Delta}\theta\) for left- and right-part, respectively. Based on model calculations, the localized domain wall state is achieved in this fractional mass-kink47 with any non-zero phase shift (\({\Delta}\theta \ne 0\)). Obviously, the mass-inversion in Fig. 1a just corresponds to one special case of Δθ = π in Fig. 1b. The fractional mass-kink further lowers the physical requirement to realize the domain wall state, providing more freedoms to design SOTI. However, the material realization of this peculiar non-π phase shift associated topological corner state is still a challenging task.

Due to the tunable topological phase transition between first-order and higher-order topological states48,49, the quantum spin Hall (QSH) materials provide a good platform to explore the 2D SOTI. As the parent compound of Fe-based superconductor, the monolayer FeSe has been intensively studied50. Recently, the QSH state with checkerboard antiferromagnetic (AFM) order (Fig. 1c) is reported experimentally in monolayer FeSe51, demonstrating a new topological phase in this high-temperature superconductor52,53,54,55,56. Moreover, the AFM order is reported experimentally in non-superconducting monolayer FeSe through the magnetic exchange bias effect measurement57. After electron doping, the monolayer FeSe will change from non-superconducting to superconducting phase. The AFM order can be suppressed in this process, but it is still a very possible spin configuration in the superconducting phase. In this work, starting from the AFM QSH state, we further report a 2D SOTI in monolayer FeSe with canted checkerboard AFM order (Fig. 1e). An intriguing fractional mass-kink induced topological corner state is identified by both analytic model and first-principles calculations, which is robust to perturbation and edge orientation. The main discoveries of our work are summarized in Fig. 1. Since the AFM QSH state has a pair of helical Dirac edge states (Fig. 1c, d), the coupling between them will break the band degeneracy at the Dirac point. Physically, this can be achieved by the canted spin induced in-plane Zeeman field. Meanwhile, the fractional mass-kink with phase shift of π/2 is formed at the rectangular corner (Fig. 1e), resulting in an in-gap topological corner state with fractional charge of e/4 (Fig. 1f).

Results

Quadratic Bulk Hamiltonian

The nontrivial band topology of monolayer FeSe with checkerboard AFM order is captured by band structures around M point in the first Brillouin zone51. In the basis of dxz and dyz orbitals, its low-energy effective Hamiltonian is written as58

$${\it{H}} = {\it{a}}_0({\it{k}}_{\it{x}}^2 + {\it{k}}_{\it{y}}^2){\it{s}}_0\sigma _0 + {\it{a}}_1{\it{k}}_{\it{x}}{\it{k}}_{\it{y}}{\it{s}}_0\sigma _{\it{x}} + \lambda {\it{s}}_{\it{z}}\sigma _{\it{y}} + {\it{a}}_2({\it{k}}_{\it{x}}^2 - {\it{k}}_{\it{y}}^2){\it{s}}_0\sigma _z$$
(1)

where σ0 and s0 are identity matrixes, \(\sigma _{{\it{x}},{\it{y}},{\it{z}}}\) is orbital Pauli matrix, \({\it{s}}_{\it{z}}\) is spin Pauli matrix, \({\it{a}}_{0,1,2}\) is fitting constant, and λ is intrinsic spin-orbital coupling (SOC). Since the spin-up and -down bands are approximately decoupled in first-principles calculations51,58, we do not need to use the three-orbital model59 to describe it. The characterized band structures of Eq. (1) are shown in Fig. 2, exhibiting the fourfold rotational symmetry (\({\it{C}}_4 = {\it{i}}\sigma _{\it{y}}{\it{e}}^{{\it{i}}\pi {\it{s}}_{\it{z}}/4}\)). Without SOC, there is a quadratic band degeneracy at M point (Fig. 2a). With SOC, the degeneracy is lifted and an AFM QSH state is realized (Fig. 2b). One can see that this effective Hamiltonian is dramatically different to Kane-Mele60 and Bernevig–Hughes–Zhang61 model described QSH state that only includes the linear-term of momentum. The helical Dirac edge state for such a quadratic QSH Hamiltonian62,63,64 has not been analytically derived yet.

Fig. 2: Quadratic bulk Hamiltonian and edge orientation.
figure 2

a and b Characterized band structures of the low-energy effective Hamiltonian without and with SOC, showing the quadratic touching point and quadratic QSH. c Reciprocal space rotated coordinate system for describing the edge along k2 direction that has an angle φ with kx axis. d Real space rectangular cluster with four edges, corresponding to φ = 0, π/2, π and 3π/2 in (c). The arrow denotes the positive direction of the edge.

Full size image

Helical Dirac Edge State

The first term in Eq. (1) is an identity matrix that can modify the group velocity of the Dirac edge state, but does not affect our final conclusion. To simplify the derivation, we will omit it in the following part. In order to derive the helical Dirac edge state along different edges, we define a rotated coordinate system \({\it{k}}_1 - {\it{k}}_2\), where the edge along k2 direction has an angle φ with kx axis, as shown in Fig. 2c. For the rectangular cluster with 90° corners, the four regular edges are along the direction of φ = 0, π/2, π and 3π/2, as shown in Fig. 2d. In the \({\it{k}}_1 - {\it{k}}_2\) plane, Eq. (1) can be rewritten as

$${\it{H}}_ \pm = \mp {\it{a}}_1{\it{k}}_1{\it{k}}_2{\it{s}}_0\sigma _{\it{x}} + \lambda {\it{s}}_{\it{z}}\sigma _{\it{y}} \pm {\it{a}}_2({\it{k}}_2^2 - {\it{k}}_1^2){\it{s}}_0\sigma _z$$
(2)

where \({\it{H}}_ +\) for φ = 0, π and \({\it{H}}_ -\) for φ = π/2, 3π/2. Assuming Eq. (2) is defined in half-space (\({\it{x}}_1 \,>\, 0\)) of the \({\it{x}}_1 - {\it{x}}_2\) plane, we replace \({\it{k}}_1 \to - {\it{i}}\partial _{{\it{x}}_1}\), \({\it{k}}_2 \to 0\) and substitute a trial edge state solution \(\psi={\it{e}}^{\eta {\it{x}}_1}\phi\) into it, where η is a decay constant. Then, Eq. (2) is reduced to the form

$${\tilde {H}}_ \pm = \lambda {\it{s}}_{\it{z}}\sigma _{\it{y}} \pm {\it{a}}_2\eta ^2{\it{s}}_0\sigma _z$$
(3)

The spin is decoupled in this bulk Hamiltonian, so the edge state solution will also be spin-decoupled, which has a form of \(\left| {\psi _{ \uparrow \pm }} \right\rangle = \left( {\left| {\phi _{ \uparrow \pm }} \right\rangle ,0} \right)^{\it{T}}\) and \(\left| {\psi _{ \downarrow \pm }} \right\rangle = \left( {0,\left| {\phi _{ \downarrow \pm }} \right\rangle } \right)^{\it{T}}\). Substituting \(\left| {\psi _{ \uparrow + }} \right\rangle\) and \(\left| {\psi _{ \downarrow + }} \right\rangle\) into \({\tilde {H}}_ +\), considering the solution divergence (\({\it{x}}_1 \to + \infty\)) and time-reversal symmetry (\({\it{T}} = {\it{is}}_{\it{y}}{\it{K}}\), K is complex conjugate), we obtain the edge state solution of \({\tilde {H}}_ +\) as

$$\begin{array}{l}\left| {\phi _{ \uparrow + }} \right\rangle = \alpha {\it{e}}^{\eta {\it{x}}_1}\left| {\xi _ + } \right\rangle + \beta {\it{e}}^{\eta ^ \ast {\it{x}}_1}\left| {\xi _ - } \right\rangle \\ \left| {\phi _{ \downarrow + }} \right\rangle = - \left| {\phi _{ \uparrow + }^ \ast } \right\rangle \end{array}$$
(4)

where \(\eta ^2 = {\it{i}}\lambda /{\it{a}}_2\), α and β are coefficients, and \(\left| {\xi _ \pm } \right\rangle = \sqrt 2 /2\left( {1, \pm 1} \right)^{\it{T}}\) are eigenstates of \(\sigma _{\it{x}}\). Since \({\tilde {H}}_ \pm\) are connected by C4 symmetry, that is \({\tilde {H}}_ - = {\it{C}}_4^{ - 1}{\tilde {H}}_ + {\it{C}}_4\), the edge state solution of \({\tilde {H}}_ -\) can be obtained from the relation

$$\begin{array}{l}\left| {\psi _{ \uparrow - }} \right\rangle = {\it{C}}_4^{ - 1}\left| {\psi _{ \uparrow + }} \right\rangle \\ \left| {\psi _{ \downarrow - }} \right\rangle = {\it{C}}_4^{ - 1}\left| {\psi _{ \downarrow + }} \right\rangle \end{array}$$
(5)

In the edge state space \(\left( {\left| {\psi _{ \uparrow \pm }} \right\rangle ,\;\left| {\psi _{ \downarrow \pm }} \right\rangle } \right)\), to the leading order of k2, Eq. (2) can be transformed into the standard Dirac equation as

$${\it{h}}_ \pm = - {\it{a}}_1{\it{k}}_2 \,{\mathrm{Im}} \,(\eta )\tau _z$$
(6)

where τz is Pauli matrix of the edge state, showing a general form of the helical Dirac edge state for the quadratic QSH Hamiltonian.

Fractional mass-kink

In order to gap out the helical Dirac edge state in Eq. (6), we consider an in-plane Zeeman field as48

$${\it{H^{\prime}}} = {\it{us}}_{\it{x}}\sigma _0 + {\it{vs}}_{\it{y}}\sigma _0$$
(7)

where sx, y is spin Pauli matrix, u and v are coefficients that control the intensity and direction of in-plane Zeeman field. This new term breaks the spin-degeneracy and mixes the spin-up and -down in Eq. (1). Similar to the above derivation, Eq. (7) can be written in the edge state space as46

$$\begin{array}{l}{\it{h^{\prime}}}_ + = {\it{m}}\left( {\tau _x\cos \theta + \tau _y\sin \theta } \right)\\ {\it{h^{\prime}}}_ - = {\it{m}}\left[ {\tau _x\cos (\theta + \pi /2) + \tau _y\sin (\theta + \pi /2)} \right]\end{array}$$
(8)

where \({\it{h^{\prime}}}_ +\) for φ = 0, π and \({\it{h^{\prime}}}_ -\) for φ = π/2, 3π/2. \(\tau _{{\it{x}},{\it{y}}}\) is Pauli matrix of the edge state, \({\it{m}}_1 = {\mathrm{Re}} \left( {\frac{{\alpha ^2}}{{2\eta }} + \frac{{\beta ^2}}{{2\eta ^ \ast }}} \right)\), \({\it{m}}_2 = {\mathrm{Im}} \left( {\frac{{\alpha ^2}}{{2\eta }} + \frac{{\beta ^2}}{{2\eta ^ \ast }}} \right)\), \({\it{m}} = [({\it{u}}^2 + {\it{v}}^2)({\it{m}}_1^2 + {\it{m}}_2^2)]^{1/2}\), \({\it{mcos}}\,\theta {\it{ = um}}_{\it{1}} - {\it{vm}}_{\it{2}}\) and \({\it{msin}}\,\theta {\it{ = um}}_2{\it{ + vm}}_1\). Combining Eq. (6) and Eq. (8), the helical Dirac edge state is gapped out with a band gap of \({\it{E}}_{\it{g}} = 2{\it{m}}\). From the definition of m, one can see that the band gap only depends on the intensity of in-plane Zeeman field, but does not depend on its direction. Most remarkably, one notices that the mass-angle is θ and θ + π/2 for edge along φ = 0, π and φ = π/2, 3π/2, respectively, forming a fractional mass-kink with phase shift of Δθ = π/2 at each 90° corner. According to Moore’s theory46, such a phase shift will support a topological corner state with fractional charge of Ns = e|Δθ/2π| = e/4. Therefore, a 2D SOTI is identified in the monolayer FeSe, originated from the in-plane Zeeman field induced fractional mass-kink with non-π phase shift. More details about the analytic derivations can be found in the Supplementary Material.

In-plane Zeeman field

In order to introduce the effect of in-plane Zeeman field in monolayer FeSe, a special canted checkerboard AFM order can be considered (Fig. 1e)65. The direction of the canted spin is determined by zenith angle (γ) and azimuth angle (δ), as shown schematically in inset of Fig. 3b. Experimentally, this spin configuration can be realized by applying an in-plane magnetic field, making the spin canted along the field direction, where zenith and azimuth angle is tunable by field intensity and direction66. The in-plane magnetic field vs canted zenith angle can be roughly estimated as B ~ (Ecanted − Enon-canted)/μcanted (see Supplementary Fig. 1), where Ecanted and Enon-canted are the energy of canted and non-canted spin configuration, and μcanted is the net magnetic moment in canted spin configuration.

Fig. 3: Robust topological edge and corner states.
figure 3

a The gapless QSH edge state along ferromagnetic edge of monolayer FeSe with checkerboard AFM order. b–f Zoom-in gapped topological edge state by canting the spin direction with δ = 45°, (b) γ = 0.25°, (c) γ = 0.5°, (d) γ = 0.75°, (e) γ = 1.0° and (f) γ = 1.25°. The canted spin-up and -down (red arrows) directions are determined by angle γ and δ, as shown schematically in inset of (b). g Discrete energy levels of rectangular monolayer FeSe cluster with four naturally cleaved ferromagnetic edges of the same spin (see also Fig. 1e). The four in-gap topological corner states are marked in red-dot around the Fermi-level. The inset is spatial charge density distribution of the corner states. The radii of circle on each atom denotes the absolute value of charge density. h is the same as (g), but for a rotated rectangular cluster with non-naturally cleaved ferromagnetic edges of the same spin.

Full size image

Bulk topological Index

To further support our analytic results, the first-principles calculations are performed to directly identify the 2D SOTI in monolayer FeSe from bulk, edge and corner three aspects. If the direction of in-plane spin component is perpendicular to the diagonal mirror plane, the mirror symmetry is conserved in monolayer FeSe with canted checkerboard AFM order. Therefore, we can use the mirror-graded winding number to show its nontrivial bulk topology. In the first Brillouin zone, the high-symmetric k-path along M-Γ-M direction is invariant under mirror operation. Along this k-path, the wavefunctions (χ) can be divided into two separated sets with opposite mirror eigenvalues ±i. The mirror-graded winding number (also called the Zak phase) is calculated along M-Γ-M loop for each set as67 \(\varphi _ \pm = i{\oint} {\left\langle {\chi _{ \pm i}\left( k \right)} \right|\partial k\left| {\chi _{ \pm i}\left( k \right)} \right\rangle dk}\). The quantized Zak phase is φ± = π, showing a topological invariant to characterize the 2D SOTI68 in monolayer FeSe.

Topological edge state

For the monolayer FeSe with checkerboard AFM order, the gapless QSH edge state along ferromagnetic edge of [100] direction is shown in Fig. 3a, where one Dirac point is sitting along Γ-X direction51. By canting the spin direction, interestingly, we found that the spectra of topological edge state is insensitive to azimuth angle (δ) (see Supplementary Fig. 2). That is, it does not depend on the direction of in-plane Zeeman field, which is consistent with our analytic derivation. Without losing the generality, we will fix azimuth angle (δ) and discuss the effect of zenith angle (γ) in the following part. For azimuth angle δ = 45° along [110] direction, the zoom-in topological edge state with small zenith angle ranging from γ = 0.25° to 1.25° is shown in Fig. 3b, f, respectively. Comparing to the gapless spectra shown in Fig. 3a, there are two significant features can be observed for the canted spin. First, the spin-degenerate bulk bands are split and bulk gap around X point is reduced with the increasing of γ. This feature can be attributed to the in-plane Zeeman field induced spin-splitting that results in upshift and downshift of the opposite spin bands (see Supplementary Fig. 3). With the increasing of γ, the intensity of in-plane Zeeman field is enhanced, so the band splitting is increased and bulk gap between opposite spins is reduced. Second, the topological edge state is gapped out and Dirac gap exhibits a non-monotonic behavior with the increasing of γ. This feature can be attributed to the in-plane Zeeman field-induced mass-term and band-reshaping. According to our analytic results, the Dirac gap is proportional to the intensity of in-plane Zeeman field. In principles, it will increase with the increasing of γ. However, the reduced bulk gap moves the bottom branch of topological edge state gradually close to the top bulk band, making the global Dirac gap decreased with further increasing of γ. Therefore, based on canted checkerboard AFM order, we confirm the gapped topological edge state in monolayer FeSe, identifying the first unique character of 2D SOTI.

Topological corner state

Since the band topology is same in the canted spin opened Dirac gap, we will focus on one gapped topological edge state with γ = 0.75° and δ = 45° (Fig. 3d) to illustrate its corner state (see Supplementary Fig. 4a–c). The rectangular cluster with four 90° corners is constructed by cutting four ferromagnetic edges with the same spin along naturally cleaved [100] and [010] directions69. The discrete energy levels of the cluster are shown in Fig. 3g. Clearly, there are four nearly degenerate corner states around the Fermi-level, as labeled by the red dot. Here, the slight energy splitting of corner state is due to the finite size effect-induced coupling between different corners (see Supplementary Fig. 5). The spatial distribution of them is shown in the inset of Fig. 3g, which is localized at four corners. Therefore, we confirm the in-gap topological edge state in monolayer FeSe, identifying the second unique character of 2D SOTI. Additionally, if only spin-down is canted in the checkerboard AFM order, we found that there are still four corner states in the gapped topological edge states (see Supplementary Fig. 6), so the 2D SOTI is also realizable by applying out-of-plane magnetic field. Furthermore, by accounting the total electrons in the system40, there is only one electron is left after filling all energy levels below the four corner states. Consequently, each corner state will hold a fractional charge of e/4, which is consistent with our analytic derivation.

Robustness of corner state

In order to facilitate the possible experimental measurement, the robustness of topological corner state against perturbation and edge-cutting orientation are further investigated. The local perturbation is simulated by adding a finite random on-site energy to atoms around the corner, and the global perturbation is simulated by breaking the C4 symmetry slightly. In both cases, the topological corner states are still localized at four corners, but the relative intensity becomes different among them (see Supplementary Fig. 4d and Supplementary Fig. 7). Moreover, we found that the rectangular cluster constructed from non-naturally cleaved ferromagnetic edge along arbitrary orientation can also support the existence of topological corner state. The discrete energy levels and spatial distribution of corner states are shown in Fig. 3h and its inset, which are comparable to those shown in Fig. 3g and its inset. The same results are obtained for the rectangular cluster with four AFM edges (see Supplementary Fig. 4e). Consequently, the topological corner state in monolayer FeSe can be revealed by cutting the edge along irregular orientation, providing more convenience for scanning tunneling microscopy detection. If the monolayer FeSe is stacked vertically into a three-dimensional (3D) structure of (LiFe)OHFeSe70, the angle-resolved photoemission spectroscopy measured band structures71 are similar to the monolayer FeSe studied in this work51. Considering the weak coupling between the neighboring FeSe layers in (LiFe)OHFeSe72, a 3D weak SOTI73 is also realizable by the canted checkerboard AFM order, where the topological corner states are stacked into a one-dimensional topological hinge state with little dispersion along the vertical direction (see Supplementary Fig. 8).

Discussion

Recently, the 2D SOTI in monolayer FeSe with checkerboard AFM order is also studied by Xu et al.74. In their work, the topological edge state is gapped by Sz symmetry breaking, and the topological corner state with fractional charge of e/2 is realized at 90° corner connected by two ferromagnetic edges with opposite spin. While in our work, the topological edge state is gapped by spin canting, and the topological corner state with fractional charge of e/4 is realized at 90° corner connected by two ferromagnetic edges with the same spin. Previously, the domain wall state near the Fermi-level has been observed experimentally in FeSe without nematicity75. Within the framework of our theory, it may be attributed to the spin canting induced mass-angle shift in different domains.

In conclusion, based on analytic model and first-principles calculations, we identify an intriguing fractional mass-kink induced topological corner state in monolayer FeSe with canted checkerboard AFM order. Our results greatly extend the topological physics for mass-inversion induced domain wall state and provide a way to design the higher-order topological materials with arbitrary fractional charge.

Methods

First-principles calculations

The first-principles calculations are carried out in the framework of generalized gradient approximation with Perdew-Burke-Ernzerhof functionals using the Vienna Ab initio Simulaiton Package (VASP)76. All calculations are performed with a plane-wave cutoff of 400 eV on the 11 × 11 × 1 Monkhorst-Pack k-point mesh, and the convergence criterion of energy is 10−5 eV. The vacuum layer of 15 Å thick is used to ensure decoupling between neighboring slabs. During the structural relaxation, all atoms are relaxed until the forces are smaller than 0.01 eV/ Å. The lattice constant of monolayer FeSe is fixed to the bulk lattice constant of SrTiO3 at 3.901 Å51. In order to simulate the canted checkerboard AFM order under the in-plane magnetic field, the direction of magnetic moment for all atoms in monolayer FeSe are constrained in our calculations.

First-principles tight-binding Hamiltonian

The first-principles tight-binding Hamiltonian is obtained from the band fitting by using the Wannier90 package77. The five d orbitals of Fe and three p orbitals of Se are used in band fitting to the first-principles bands with spin-orbital coupling. The unit cell of monolayer FeSe includes 32 fitted Wannier orbitals. By extending this unit cell first-principles tight-binding Hamiltonian to ribbon and cluster configurations, the spectrum of topological edge and corner states for monolayer FeSe can be calculated. The low-energy effective Hamiltonian in Eq. (1) is only used to analytically derive the fractional mass-kink in monolayer FeSe, which is not used to calculate the spectrum of topological edge and corner states. The detailed derivation of Eq. (1) can be found in ref. 58, which is also supported by the first-principles calculations.