Introduction

Research on topological insulators has advanced significantly in recent decades1,2,3,4,5. The Zak phase is a key concept in topological insulators which represents the Wannier center’s displacement and relates to polarization, leading to charge accumulation at sample boundaries6,7,8,9. It can be used to predict edge states and is quantized to either \(0\) or \(\pi\) in parity-time (PT) symmetric systems. In conventional bulk-boundary theory, the Zak phase difference \(\triangle v\) across a boundary (either 0 or π) of a band gap predicts the number of boundary modes, given by \(\triangle v/\pi\). However, this criterion is inadequate in multi-band systems where the number of bands \(n\ge 3\)10,11. In this case, it is useful to use non-Abelian band topology theory12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42 to understand the relationship between topological invariants and edge states in multi-band PT-symmetric systems. Unlike conventional band theory that focuses on single band or band gaps, the non-Abelian band topology theory considers the eigenstates of all bands globally, drawing analogy of their eigenframe rotation in momentum space to the rotation of liquid crystal molecules around a loop encircling nematic liquid crystal disclinations, with topological charges represented by quaternions (\(n=3\)) or generalized quaternions (\(n > 3\)). Here, PT symmetry constrains the Hamiltonian to remain real across the Brillouin zone, which facilitates topological characterization within a Clifford algebra framework. Currently, this theory has been applied in some multi-band systems, including those with band degeneracy and those with fully gapped bands. The former, known as non-Abelian semimetal, is primarily used to investigate trajectory-dependent nodal point collisions13,19, admissible nodal line configurations14,16,20,21,22,23,24, and the Euler class26,27,28,29,36,37. The latter, known as non-Abelian topological insulator, has unconventional bulk-boundary correspondence, which extends beyond the conventional description based on integers18,38. In contrast to conventional theories where domain-wall states are determined by the difference in topological charges, non-Abelian topology theory predicts them based on the quotient of charges which are matrix-like entities18. Up to now, non-Abelian band topological systems have been experimentally realized in various platforms including metamaterials17,20,21,22, transmission line networks18,38 and acoustic systems19,24,40. Advancing the field into technological areas, such as photonics, is of great importance, as the non-Abelian features and their associated matrix-form operations are useful for various applications, including optical computing and topological quantum computing. However, a model for realizing non-Abelian band topological systems at optical frequencies remains elusive, and corresponding experimental demonstrations have yet to be achieved.

In this study, we propose a minimal model for non-Abelian topological insulators with time-reversal symmetry and experimentally realize a six-band system in photonic waveguide arrays at optical frequencies. We first show theoretically that the non-Abelian topological charges in the six-band system describe eigenvector frame rotations as Bloch momentum changes and are characterized by generalized quaternions, including \({\pm q}_{123456}\), which have no counterparts in systems with \(n < 6\) and exhibit unique edge/domain-wall state distributions beyond the single-band Zak phase prediction. Subsequently, we experimentally realize the proposed model at optical frequencies in a laser-written photonic waveguide array. We design and implement an experimental configuration in which light could encounter structures with different non-Abelian topological phases along the propagation direction. In this way, different non-Abelian topological charges can be revealed in experiments. The non-Abelian quotient relation is also experimentally substantiated by constructing an interface between two samples with different non-Abelian topological charges.

Results

Minimal non-Abelian topological insulator model with six bands

We first consider a quasi-1D tight-binding model as shown in Fig. 1a, which contains three coupled SSH chains. Each unit cell (dashed rectangle) contains six sites labeled “1” to “6”. The onsite energies and hopping strengths are chosen to ensure PT symmetry. The Hamiltonian is expressed as:

$$H\left(k\right)=\left(\begin{array}{cccccc}{E}_{a} & {h}_{12} & s & 0 & 0 & 0\\ {h}_{12}^{*} & {E}_{a} & 0 & s & 0 & 0\\ s & 0 & {E}_{b} & {h}_{34} & t & 0\\ 0 & s & {h}_{34}^{*} & {E}_{b} & 0 & t\\ 0 & 0 & t & 0 & {E}_{c} & {h}_{56}\\ 0 & 0 & 0 & t & {h}_{56}^{*} & {E}_{c}\end{array}\right)$$
(1)
Fig. 1: Minimal model of the six-band non-Abelian topological insulator.
Fig. 1: Minimal model of the six-band non-Abelian topological insulator.
Full size image

a The tight-binding model. The Bloch bands and energy spectra of a 20-unit-cell lattice, corresponding to charges \({q}_{123456}\) (b) and \({q}_{1256}\) (c), respectively. d Some representative group multiplication relations of the generalized quaternion group \({\bar{P}}_{6}\). The colored unidirectional arrow represents the multiplication of the element with the same color, while its reverse direction means multiplying by the opposite partner of this element. For example, a gray unidirectional arrow starts from the element \({-q}_{3456}\) (blue sphere) and points to the element \({q}_{12}\) (red sphere), representing the multiplication relations \({-q}_{3456}\bullet \left({-q}_{123456}\right)={q}_{12}\) and \({q}_{12}\bullet {q}_{123456}={-q}_{3456}\), with the gray spheres denoting the element \({-q}_{123456}\). The colored bidirectional arrow represents the multiplication in both directions with the same operation, without changing the sign of the multiplied group elements. For example, a green bidirectional arrow connects the elements \({q}_{12}\) (red sphere) and \({q}_{56}\) (orange sphere), representing the multiplication relations \({q}_{12}\bullet \left({-q}_{1256}\right)={q}_{56}\) and \({q}_{56}\bullet \left({-q}_{1256}\right)={q}_{12}\), with the green sphere denoting the element \({-q}_{1256}\). The eigenstate trajectories in the \({u}_{1}{u}_{2}{u}_{3}\), \({u}_{3}{u}_{4}{u}_{6}\) and \({u}_{4}{u}_{5}{u}_{6}\) subspaces for charges \({q}_{123456}\) (e) and \({q}_{1256}\) (f), respectively. The translucent gray spheres represent unit spheres containing all trajectories. g The evolution of the energy spectra with parameter tuning for a 20-unit-cell lattice, featuring five distinct phase regions separated by degenerate points: \({q}_{123456}\), \({q}_{1256}\), \({q}_{123456}\), \({q}_{16}\) and \({q}_{123456}\). h The Hamiltonian loop expands and passes through topological charges on the parameter plane during the evolution. i The representative field patterns of left/right edge states (I/II).

Here, \({h}_{12}={w}_{1}+{v}_{1}\exp (-ik)\) \({h}_{34}={w}_{2}+{v}_{2}\exp (-ik)\) and \({h}_{56}={w}_{3}+{v}_{3}\exp (-ik)\), where \(k\) is the Bloch vector.

Before discussing the non-Abelian band topology, we first examine the correspondence between the single-band Zak phase and the existence of edge states. In Fig. 1b, c, we show the band structures (left panels) and energy spectra of finite samples with 20 unit cells (right panels) for two different parameter sets (see Supplementary Note 1). All the coupling strengths are set to be positive here. The Zak phases are labeled in the band structures. In the left panel of Fig. 1b, the Zak phases for the individual bands from first to sixth are all \(\pi\), so if we sum up the Zak phases for the bands below a gap, the Zak phases for the first, third, and fifth gaps are \(\pi\) (dark gray), and the second and fourth gaps are \(0\) (light gray). According to the conventional bulk-boundary correspondence, the number of edge states in the first, third, and fifth gaps should be odd, while they should be zero in the second and fourth gaps. However, in the right panel of Fig. 1b, we find only three pairs of edge states in the third gap and none in the first and fifth gaps, which disagrees with the prediction from the Zak phase perspective. A similar analysis in Fig. 1c also shows deviations from conventional theory. Before explaining the failure of the Zak phase in predicting edge states, we first explain why this model can be considered a non-Abelian topological insulator.

According to the non-Abelian band topology theory, the PT-symmetric Hamiltonian \(H({{\bf{k}}})\) can always be transformed into a real Hamiltonian \(\widetilde{H}({{\bf{k}}})\) through a unitary transformation \(\widetilde{H}({{\bf{k}}})=V\bullet H({{\bf{k}}})\bullet {V}^{{\dagger} }\). The unitary matrix \(V\) can be derived from Takagi’s decomposition \(U=V{V}^{T}\), where \(U\) is a symmetric matrix representing the parity inversion operation. In our model, the unitary matrix \(V\) is chosen as

$$V=\frac{1}{\sqrt{2}}\bullet \left(\begin{array}{cccccc}1 & 1 & 0 & 0 & 0 & 0\\ i & -i & 0 & 0 & 0 & 0\\ 0 & 0 & 1 & 1 & 0 & 0\\ 0 & 0 & 0 & 0 & i & -i\\ 0 & 0 & 0 & 0 & 1 & 1\\ 0 & 0 & i & -i & 0 & 0\end{array}\right)$$
(2)

After the unitary transformation, the obtained Hamiltonian is

$$\widetilde{H}\left(k\right)=\left(\begin{array}{cccccc}{E}_{a}+{\widetilde{h}}_{11} & {\widetilde{h}}_{12} & s & 0 & 0 & 0\\ {\widetilde{h}}_{12} & {E}_{a}-{\widetilde{h}}_{11} & 0 & 0 & 0 & s\\ s & 0 & {E}_{b}+{\widetilde{h}}_{33} & 0 & t & {\widetilde{h}}_{36}\\ 0 & 0 & 0 & {E}_{c}-{\widetilde{h}}_{55} & {\widetilde{h}}_{45} & t\\ 0 & 0 & t & {\widetilde{h}}_{45} & {E}_{c}+{\widetilde{h}}_{55} & 0\\ 0 & s & {\widetilde{h}}_{36} & t & 0 & {E}_{b}-{\widetilde{h}}_{33}\end{array}\right)$$
(3)

Here, \({\widetilde{h}}_{11}={w}_{1}+{v}_{1}{{\rm{c}}}{{\rm{os}}}(k)\), \({\widetilde{h}}_{33}={w}_{2}+{v}_{2}{{\rm{c}}}{{\rm{os}}}(k)\), \({\widetilde{h}}_{55}={w}_{3}+{v}_{3}{{\rm{c}}}{{\rm{os}}}(k)\), \({\widetilde{h}}_{12}={v}_{1}\sin (k)\), \({\widetilde{h}}_{36}={v}_{2}\sin (k)\) and \({\widetilde{h}}_{45}={v}_{3}\sin (k)\). In this new Hamiltonian, all matrix elements are real, so the eigenvectors are real vectors denoted as \({\left({u}_{1},{u}_{2},{u}_{3},{u}_{4},{u}_{5},{u}_{6}\right)}^{T}\). These six real components define a 6D parameter space, and the evolution of the eigenvectors maps to trajectories within this parameter space as Bloch \(k\) varies. Considering all the bandgaps between adjacent bands, the order-parameter space of the Hamiltonian is \({M}_{6}=O(6)/{O\left(1\right)}^{6}\), where \(O(6)\) is the 6D orthogonal group and \({O\left(1\right)}^{6}\cong {{\mathbb{Z}}}_{2}^{6}\) arises from the \(\pm 1\) gauge freedom of the eigenvectors. The corresponding fundamental homotopy group is given by the Salingaros vee group12 that \({\pi }_{1}\left({M}_{6}\right)={\bar{P}}_{6}={{{\rm{U}}}}_{{n}_{i}\in \left\{{\mathrm{0,1}}\right\}}\left\{\pm {e}_{1}^{{n}_{1}}{e}_{2}^{{n}_{2}}\cdots {e}_{5}^{{n}_{5}}\right\}\), which is formed by products of the basis elements of the real Clifford algebra \(C{{{\mathcal{l}}}}_{{\mathrm{0,5}}}\): \(\left\{{e}_{1},\,{e}_{2},\,{e}_{3},\,{e}_{4},\,{e}_{5}\right\}\), where \(\left\{{e}_{i},{e}_{j}\right\}=-2{\delta }_{{ij}}\). The visual interpretation of the basis \({e}_{j-1}\) is a π rotation in the plane spanned by the eigenvectors of the 1st and \(j\)-th bands. We rename the elements of \({\bar{P}}_{6}\) for convenience in band structure analysis:

$${e}_{j-1}\to {q}_{1j},$$
$${e}_{i-1}{e}_{j-1}\to {q}_{{ij}}\,\left(i < j\right),\,$$
$${e}_{i-1}{e}_{j-1}{e}_{k-1}\to {q}_{1{ijk}}\,\left(i < j < k\right),\,$$
$${e}_{i-1}{e}_{j-1}{e}_{k-1}{e}_{l-1}\to {q}_{{ijkl}}\,\left(i < j < k < l\right),$$
$${e}_{1}{e}_{2}{e}_{3}{e}_{4}{e}_{5}\to {q}_{123456}$$
(4)

The topological phase classification is determined by this non-Abelian fundamental group \({\bar{P}}_{6}\), and the group elements are referred to as the non-Abelian topological charges characterizing the topological phases. The group \({\bar{P}}_{6}\) has 64 elements grouped into 34 conjugacy classes: \(\left\{+1\right\}\), \(\left\{-1\right\}\), \({\bigcup }_{1\le i < j\le 6}\left\{\pm {q}_{{ij}}\right\}\), \({\bigcup }_{1\le i < j < k < l\le 6}\left\{\pm {q}_{{ijkl}}\right\}\), \(\left\{{+q}_{123456}\right\}\) and \(\left\{{-q}_{123456}\right\}\). The class \(\left\{+1\right\}\) denotes trivial rotation, and \(\left\{-1\right\}\) refers to a \(2\pi\) rotation in any rotation plane. The \(\left\{\pm {q}_{{ij}}\right\}\) represent \(\pm \pi\) rotations in the rotation plane spanned by the eigenvectors of the i-th and j-th bands. The \(\left\{\pm {q}_{{ijkl}}\right\}\) correspond to isoclinic double rotations of angle \(\pi\) in the 4D subspace of the i-th, j-th, k-th and l-th bands. The \(\left\{{+q}_{123456}\right\}\) and \(\left\{{-q}_{123456}\right\}\) correspond to triple rotations in the 6D space with opposite chirality. Figure 1d illustrates part of the group multiplication structure, showing elements as colored spheres and their multiplications with colored arrows. More group multiplication relationships are shown in Supplementary Tables 1-8.

Based on the above theoretical framework, the non-Abelian topological charges of the Hamiltonians in Fig. 1b, c are calculated to be \({q}_{123456}\) and \({q}_{1256}\), respectively (see Supplementary Note 2 for the calculation details). Their corresponding eigenvector trajectories within the \({u}_{1}{u}_{2}{u}_{3}\), \({u}_{3}{u}_{4}{u}_{6}\) and \({u}_{4}{u}_{5}{u}_{6}\) subspaces are plotted in Fig. 1e, f, respectively, where the red, green, blue, purple, cyan, and yellow trajectories represent the tips of the eigenstates of the bands from first to sixth, with increasing line thickness showing \(k\) from \(-\pi\) to \(\pi\). From Fig. 1e, all band eigenstates show a \(\pi\) rotation as the Hamiltonian evolves across 1D Brillouin zone, consistent with the charge \({q}_{123456}\). In Fig. 1f, the first, second, fifth, and sixth bands rotate by \(\pi\), while the third and fourth bands show trivial rotations, where their trajectory can be continuously contracted to a point, indicating a charge \({q}_{1256}\).

Edge/domain-wall states of six-band non-Abelian topological insulators

We now discuss the edge states of the proposed six-band system and the inadequacy of the single-band Zak phase in predicting the existence of edge modes. Figure 1g shows the eigenvalue spectrum and the topological phase diagram obtained by varying a tunable parameter \(\triangle=s=t\) in the minimal model of Fig. 1a. It displays the eigenvalue spectrum of a 20-unit-cell lattice, where edge and bulk states are distinguished by their inverse participation ratio (IPR). The IPR is defined as \({\mbox{IPR}}={\sum }_{i}{\left|{\varphi }_{i}\right|}^{4}\), where \({\varphi }_{i}\) is the normalized wave function at lattice i. The states with a large IPR (appear in red) are considered edge states, while those with a small IPR (appear in gray) are considered bulk states. The corresponding IPR threshold is approximately 0.15. The diagram shows five phase regions separated by gap-closing critical points: P1 (\({q}_{123456}\)), P2 (\({q}_{1256}\)), P3 (\({q}_{123456}\)), P4 (\({q}_{16}\)), and P5 (\({q}_{123456}\)). The band dispersions shown in Fig. 1b, c correspond respectively to \(\triangle=0.5\) and \(\triangle=2.5\). To provide an intuitive understanding of this phase diagram, Fig. 1h illustrates a 2D parameter plane in momentum space, with each loop representing the Brillouin zone of a one-dimensional Hamiltonian. If the frame rotation of the 1D Hamiltonian along the loop is characterized by a specific generalized quaternion charge, then the loop encloses singularity points in the 2D plane, corresponding to gap-closing points of the Hamiltonian. A phase transition, marked by gap-closing of the Hamiltonian, occurs if the loop crosses these singularity points. We can label these singularity points with the generalized quaternion, indicating that loops that encircle those points are characterized by the same topological charges from a frame rotation perspective. Figure 1i shows representative field patterns of edge states, namely I and II, which are marked in Fig. 1g.

Since non-Abelian topological insulators need to consider multiple band gaps, their edge state distribution in the band structure becomes complicated. We first discuss the case of charges \(\left\{{\pm q}_{{ij}}\right\}\) (\(i < j\)). In an ideal model with charge \({\pm q}_{{ij}}\), where the rotational trivial bands are decoupled from the nontrivial (i-th and jth) bands (with “nontrivial” referring to eigenvectors that rotate to their antipodal points in parameter space), and therefore the ith and jth bands form a two-band subsystem. The gap of this subsystem exhibits a Zak phase of \(\pi\), while the trivial band has a Zak phase of zero. As a result, the system hosts an odd number of edge states. When perturbations couple the trivial and nontrivial bands, the charge \({\pm q}_{{ij}}\) can be factorized into a series product of charges, and all gaps between the i-th and j-th bands carry a Zak phase of \(\pi\) with an odd number of edge states. For instance, in the topological phase of \({q}_{16}\) in Fig. 1g, each gap has one edge state, as \({q}_{16}\) can be factorized into \({q}_{16}={q}_{12}{q}_{23}{q}_{34}{q}_{45}{q}_{56}\).

Compared to the quaternion \({\bar{P}}_{3}\) in a three-band system corresponding to 3D rotation, the generalized quaternion \({\bar{P}}_{6}\) in a six-band system exhibits two kinds of unique topological charges: \(\pm {q}_{{ijkl}}\) and \({\pm q}_{123456}\), which have no counterpart in \({\bar{P}}_{3}\). These charges correspond to double and triple rotations involving two and three orthogonal planes, respectively, thus cannot be described using the single band Zak phase description. The Zak phase of \(\pi\) only indicates the rotation angle of \(\pi\) but not the rotation plane, which becomes indeterminate in four or higher-dimensional spaces. Thus, while the Zak phase of \(\pi\) is useful in two-band and three-band systems, it does not provide sufficient information in higher-dimensional systems. Taking the charge \({q}_{1256}\) as an example, it can be factorized in special cases into three forms: \({q}_{12}{q}_{56}\), \({q}_{16}{q}_{25}\), and \({-q}_{15}{q}_{26}\). These factorizations can transform continuously without closing gaps, each representing a distinct combination of rotation planes and edge state distributions. In the general case, it is a superposition of these three factorizations and cannot be decomposed into the product of two simple topological charges. The edge state distribution also lies between these three factorization cases. For the charges \({\pm q}_{123456}\), there are 15 distinct factorizations (\({C}_{6}^{2}{C}_{4}^{2}/3!\)), leading to more complex edge state distributions. In the phase diagram of Fig. 1g, the edge states lie between various gaps depending on the factorization.

In non-Abelian band topology, the distribution of domain-wall states between two samples is related to the quotient of their topological charges, \(\triangle Q={Q}_{L}/{Q}_{R}\), known as the quotient relation18. Here, \({Q}_{L}\) and \({Q}_{R}\) are the topological charges of the left and right samples. The distribution of domain-wall states is determined by \(\triangle Q\). Note that the quotient relation is valid only if the two samples share common gaps, allowing discussion of domain-wall states within the gaps in the original non-Abelian topological charge framework. We apply this principle to six-band non-Abelian topological insulators. In Fig. 2a, two samples with charges \({q}_{123456}\) and \({q}_{1256}\) are placed together, each with 20 unit cells. The couplings at the interface are averaged from the two samples (Fig. 2b). The Bloch bands for each sample are shown in Fig. 2c, and the energy spectrum in Fig. 2d reveals a single domain-wall state in the gap between the third and fourth bands, consistent with the quotient relation \({q}_{123456}/{q}_{1256}={q}_{34}\). Representative field patterns of edge and domain-wall states are shown in Fig. 2e. Similarly, for the \({q}_{123456}\) and \({q}_{16}\) samples (Fig. 2f–j), the energy spectrum (Fig. 2i) shows two domain-wall states in the second-third and fourth-fifth gaps, consistent with \({q}_{123456}/{q}_{16}={q}_{2345}\).

Fig. 2: The quotient relation of domain-wall states in the six-band non-Abelian topological insulator.
Fig. 2: The quotient relation of domain-wall states in the six-band non-Abelian topological insulator.
Full size image

a The interface between the \({q}_{123456}\) and \({q}_{1256}\) samples. b Detailed couplings at the interface. c The Bloch bands of \({q}_{123456}\) and \({q}_{1256}\), respectively. d The energy spectrum of the configuration in (a). e The representative field patterns of left/right edge states (I/II) and domain-wall states (III), with the corresponding states highlighted in (d). f The interface between the \({q}_{123456}\) and \({q}_{1256}\) samples. g Intuitive understanding of the quotient relation. h The Bloch bands of \({q}_{123456}\) and \({q}_{16}\), respectively. i, The energy spectrum of the configuration in (f). j The representative field patterns of left/right edge states (I/II) and domain-wall states (III or IV), with the corresponding states highlighted in (i).

The quotient relation can be intuitively understood as follows. Similar to Fig. 1h, Fig. 2g displays a parameter plane, where each loop corresponds to a one-dimensional Hamiltonian. As we transition from the left domain \({q}_{123456}\) to the right region \({q}_{16}\) in real space across the boundary, this process can be represented in parameter space by contracting the loop from a bigger \({q}_{123456}\) loop (solid yellow) to a smaller loop \({q}_{16}\) (dashed yellow). The \({q}_{123456}\) loop encloses charges \({q}_{2345}\) and \({q}_{16}\), while the \({q}_{16}\) loop encloses only \({q}_{16}\). As the loop contracts, it passes through \({q}_{2345}\), causing the domain-wall state distribution to align with the edge state of this charge. It is important to note that the quotient relation has its own limitations. It is only capable of predicting the types of singularities that the closed loop will pass through, as well as the associated band gap closures, during the continuous transition between two Hamiltonians. Based on the Jackiw-Rebbi-type argument43, domain-wall states appear at gap-closing points, but this applies primarily to points near the gap closing. Therefore, if two samples, together with their domain wall, can be described by a parameter perturbation model around the phase transition point, the quotient relation can accurately predict the distribution of domain-wall states. However, when the parameter differences between the two samples are large, the distribution of domain-wall states becomes highly complex, making the quotient relation unable to directly predict their distribution. In our study, since the parameters of the two samples do not differ significantly, the prediction based on the quotient relation remains effective.

Experimental demonstration of edge states associated with different non-Abelian topological charges

To show that the model is realizable in photonic platforms, we fabricated photonic waveguides featuring the non-Abelian model in Fig. 1g by employing the femtosecond laser direct writing technique44,45,46,47 (see Methods). Figure 3a depicts a schematic diagram of triple-layered photonic waveguides, with a microscopic photograph of the fabricated sample given in Fig. 3b. The gap distance between the upper (or lower) and middle layer is denoted by \(g(z)\), which is particularly designed to vary linearly along the waveguiding direction (z-axis) so that light could experience multiple position dependent topological phases in one single sample. We will show that this experimental setting allows us to reveal different topological phases through probing the adiabatic evolution of edge states.

Fig. 3: Experimental measurement on the edge states associated with different non-Abelian topological charges.
Fig. 3: Experimental measurement on the edge states associated with different non-Abelian topological charges.
Full size image

a Schematic diagram of a six-band non-Abelian topological insulator consisted of triple-layered photonic waveguides, where each unit cell contains six waveguides. b Microscopic photograph of the cross-section of the fabricated waveguide arrays. c The calculated propagation constants as a function of \(g\) in a system with \({g}_{{{\rm{v}}}1}\) = 13.7 μm, \({g}_{{{\rm{w}}}1}\) = 7.3 μm, \({g}_{{{\rm{v}}}2}\) = 12.3 μm, \({g}_{{{\rm{w}}}2}\) = 8.7 μm, \({g}_{{{\rm{v}}}3}\) = 10.9 μm, and \({g}_{{{\rm{w}}}3}\) = 10.1 μm, where the red curves represent the topological edge modes, and the three green dashed lines mark the phase transition points. The simulated light intensity distribution at the output port of the waveguide arrays as a function of \({g}_{{{\rm{out}}}}\), where the edge mode I (d) or II (e) is excited at the input port with \({g}_{{{\rm{in}}}}\) = 1.8 μm (marked by the star). Experimentally measured light diffraction patterns at the output facet of samples by exciting the edge mode I (f), II (g) and III (h). In each figure, the left panels are experimental results, while the right panels are the simulated eigenfield distribution of the corresponding edge mode. The annotations P1, P2, P3, and P4 correspond to samples with \({g}_{{{\rm{out}}}}\) = 1.0 μm, 1.8 μm, 2.8 μm, and 4.4 μm, respectively, while they share the same \({g}_{{{\rm{in}}}}\) = 1.8 μm. There is only one exception that the left-lower panel in (h) is the experimental result of a sample with \({g}_{{{\rm{in}}}}={g}_{{{\rm{out}}}}\) = 4.4 μm. The right panels in (f–h) were created with MATLAB.

Figure 3c plots the calculated propagation constant \(\beta\) (which is the eigenvalue of our Hamiltonian) of the waveguide array as a function of \(g\), where \({\beta }_{0}\) denotes that of each single-mode waveguide. The waveguide system can well reproduce the feature of the model in Fig. 1g, as it supports a variety of topological edge modes marked by red curves, which exist in the band gaps, with some labeled as I, II and III. The light signal transport is confined to the edge in these edge modes. The entire parameter space of interest can be divided into four regions by three phase transition points at \(g\) = 1.42 μm, 2.31 μm, and 3.06 μm (see the green dashed lines). We name the phase regions as P1, P2, P3, and P4, each associated with a non-Abelian topological charge \({q}_{123456}\), \({q}_{16}\), \({q}_{123456}\) and \({q}_{1256}\), respectively. By defining the gap distance at the input and output as \({g}_{{{\rm{in}}}}\) and \({g}_{{{\rm{out}}}}\), respectively, we first numerically probe the phase transition point between P3 and P4 by exciting the edge mode I. Figure 3d shows the calculated output light intensity distribution in samples with different \({g}_{{{\rm{out}}}}\) by fixing \({g}_{{{\rm{in}}}}\) = 1.8 μm, where the edge mode I located at the left boundary is excited as marked by the star (see the upper-right panel of Fig. 3f for the eigenfield distribution which is used as the input, also see Methods for simulation details). We find that when \({g}_{{{\rm{out}}}}\) falls within the topological phases P1, P2, and P3, the excited edge mode I evolves along the flat band adiabatically, maintaining a localized feature at the boundary waveguides. In contrast, when the system goes into the topological phase P4, the flat band disappears, and therefore the edge mode I gradually diffracts into the bulk. Following the same idea, we can use the evolution of edge mode II to numerically probe the phase transition point between P1 (or P3) and P2 (Fig. 3e).

In experiments, the phase transition cannot be directly observed, as it occurs during the light evolution process. Instead, this setting can be employed to distinguish different topological phases, as crossing the phase transition point could affect the features of the excited edge states. We fabricated four samples with \({g}_{{{\rm{out}}}}\) = 1.0 μm, 1.8 μm, 2.8 μm and 4.4 μm, corresponding to the phase P1, P2, P3 and P4, respectively, where they share the same \({g}_{{{\rm{in}}}}\) = 1.8 μm. The edge mode I is first excited at the input by a laser light at 808 nm (see Methods), and the measured light intensity distributions at the output facet of the sample are shown in the left panels of Fig. 3f. We find that light is mainly localized in the boundary waveguides A1 and C1 in the two samples located in P2 (upper-left panel) and P3 (middle-left panel). The corresponding simulated eigenfield distributions of edge mode I are also given in the right panels of Fig. 3f for reference. In the sample with its output region in P4 (lower-left panel), light propagating in the sample crosses over the phase transition point between P3 and P4, resulting in the vanishing of the topological edge mode I and thus the diffraction of light into the bulk.

The other two phase transition points can affect the light dynamics when exciting the edge mode II. The experimental results for the three samples associated with P1, P2, and P3 are given in the left panels of Fig. 3g. The localization feature is maintained only in the sample associated with P2 (middle-left panel), while the edge mode II is diffracted into the bulk when it enters the phase P1 (upper-left panel) or P3 (lower-left panel). Different from the edge modes I and II, the edge mode III can be supported in all four topological phases (Fig. 3h). These experimental results serve as evidence for identifying different topological phases in the proposed non-Abelian topological insulator.

Experimental demonstration of the non-Abelian bulk-boundary correspondence

We introduce a detuning Δ to all the gap distances. The phase diagrams of two lattices, namely lattice I with Δ = −0.4 μm and lattice II with Δ = 0.4 μm, are shown on top of Fig. 4b, where the phase transition points are found to shift to a smaller (larger) \(g\) with a negative (positive) Δ. Such tunability allows us to combine the two lattices to demonstrate the non-Abelian bulk-boundary correspondence. Figure 4a shows the schematic diagram of a coupled system in which the light grey and dark grey waveguide arrays represent the defined lattices I and II, respectively. By setting the interface gap distance \({g}_{{{\rm{b}}}}\) = 4.8 μm, we show the calculated eigenvalues of the system as a function of \(g\) in Fig. 4b, where the interface modes are marked by red. We calculate the associated non-Abelian topological charge of the combined system by using the non-Abelian quotient relation. The corresponding phase diagram is given on top of Fig. 4b, which is consistent with the distribution of all the interface modes. For example, in the defined phase region P2, the two interface modes are located in the second-third and fourth-fifth gaps, which resulted from the charge \({q}_{123456}/{q}_{16}={q}_{2345}\).

Fig. 4: Experimental measurement on the non-Abelian bulk-boundary correspondence.
Fig. 4: Experimental measurement on the non-Abelian bulk-boundary correspondence.
Full size image

a Schematic diagram of two non-Abelian topological insulators merged at an interface. Both the two lattices share parameters \({g}_{{{\rm{v}}}1}\) = 13.7 μm + Δ, \({g}_{{{\rm{w}}}1}\) = 7.3 μm + Δ, \({g}_{{{\rm{v}}}2}\) = 12.3 μm + Δ, \({g}_{{{\rm{w}}}2}\) = 8.7 μm + Δ, \({g}_{{{\rm{v}}}3}\) = 10.9 μm + Δ, and \({g}_{{{\rm{w}}}3}\) = 10.1 μm + Δ, except that Δ = −0.4 μm for the lattice I (light grey waveguide array) and Δ = 0.4 μm for the lattice II (dark grey waveguide array). b The calculated propagation constants as a function of \(g\) for the coupled lattice with \({g}_{{{\rm{b}}}}\) = 4.8 μm, where the red curves represent the interface modes. The phase diagrams, including the non-Abelian topological charge as a function of \(g\) for the lattice I (upper), lattice II (middle), and the coupled lattice (lower), are depicted on top of the figure. c-e, Experimentally measured light diffraction patterns at the output facet of samples with \(g\) = 1.5 μm (c), 2.4 μm (d), and 3.2 μm (e). In each figure, the upper panel shows the simulated eigenfield distribution of the interface mode. The upper panels in c-e were created with MATLAB.

In experiments, we fabricated three samples with constant interlayer gap distance \(g\) = 1.5 μm, 2.4 μm, and 3.2 μm, corresponding to non-Abelian topological insulator in phases P1, P2 and P3, respectively. The calculated eigenfield distribution and experimental measurement of the interface modes I, II and III are shown in Fig. 4c–e, respectively. The experimental results capture the key characteristic of the corresponding interface mode, thus serving as direct evidence for the non-Abelian bulk-boundary correspondence. Our model can be employed to design non-Abelian topological insulators with an arbitrary number of energy bands. Additional numerical and experimental results in a four-band system are provided in Supplementary Notes 3–7, Supplementary Figs. 1–10, and Supplementary Table 9.

Discussion

In summary, we have proposed a minimal model for realizing non-Abelian topological insulators in high dimensions with time-reversal symmetry. Through unitary transformation, we have proved that the evolution of the eigenstates of a six-band system corresponds to 6D rotations and can be described using the generalized quaternion \({Q}_{64}\), which, instead of the single-band Zak phase framework, can be used to predict the edge state distributions. We have experimentally realized the non-Abelian topological insulator at optical frequencies using photonic waveguides, in which different non-Abelian topological phases as well as the non-Abelian quotient relation have been demonstrated. The idea can be extended to photonic systems with more energy bands, providing a novel perspective for the future design of multi-band photonic topological devices.

Methods

Numerical simulations of light propagation in the non-Abelian topological insulator

We explain in detail how Fig. 3d, e is obtained from numerical simulations. We take Fig. 3e for example, in which we have considered a total of 205 samples with \({g}_{{{\rm{out}}}}\) ranging between 0.2 μm and 4.5 μm, while all the samples share the same \({g}_{{{\rm{in}}}}\) = 1.8 μm. In the simulation of each sample, we model the system by a Hamiltonian \(H(z)\) and solve the Schrödinger-like equation \(H(z)|\varphi \left(z\right)\rangle=-{i\partial }_{z}|\varphi \left(z\right)\rangle\), where \(|\varphi \left(z\right)\rangle\) is the state vector to be solved. To excite the edge mode II efficiently at the starting point, we choose \(|\varphi \left(z=0\right)\rangle\) as the solved eigenvector of the edge mode II, which is given in the right panel of Fig. 3g. Since Fig. 3e only shows the field distributions in the lattice C, we only mark a star at site C2. The coordinates of the star, \(\left(1.8{{\rm{\mu }}}{{\rm{m}}},2\right)\), mark the two key excitation parameters: \({g}_{{in}}=1.8{{\rm{\mu }}}{{\rm{m}}}\) and the site index \(n=2\) corresponding to excitation site C2 on lattice C. After performing simulations for all the samples, we only extract the output result \(|\varphi \left(z=L\right)\rangle\) for each case. By combining \(|\varphi \left(z=L\right)\rangle\) of all the 205 simulations, we then draw \(|\varphi \left(z=L\right)\rangle\) as functions of \({g}_{{{\rm{out}}}}\), which is finally shown in Fig. 3e. This is the procedure for obtaining Fig. 3d, e in numerical simulations.

Sample fabrication and measurement

The photonic waveguide arrays were fabricated using the femtosecond laser direct writing techniques. In our experiment, we focused a laser (Light Conversion CARBIDE 5 W) inside a boroaluminosilicate glass chip (with a refractive index of ~1.504) by using a microscope with a numerical aperture of 0.75. To write photonic waveguides following a predesigned trajectory, the movement of the glass chip was controlled by using an Aerotech motion system with a moving speed of 40 mm/s. In the focused region of the laser, a refractive index increment of ~0.0025 can be induced, which then acts as the waveguide core, while the unaffected region acts as the waveguide cladding. The fabricated waveguide exhibits a cross-section of ~6 μm × 6 μm. In such a configuration, the waveguide supports only one vertical-polarized fundamental mode, which is used to probe the physics of the non-Abelian topological insulator. The length L of all the samples is 25 mm.

To effectively excite the topological edge mode and interface mode, we prepared a case-by-case excitation with a light field distribution similar to the eigenfield distribution of the desired mode. For the edge mode I in Fig. 3f, the light energy is mainly localized in waveguides A1 and C1, but the electric fields in the two waveguides are out of phase. To excite this anti-symmetric mode, we first used a Y-branch to split the injected laser into two beams. We then employed a three-waveguide system featuring a stimulated Raman adiabatic passage to generate a geometric phase π for one beam. Meanwhile, we applied another straight waveguide for transporting the other beam without this geometric phase. By connecting the two waveguides carrying these two beams with a phase difference π to waveguides A1 and C1, the edge mode I can be excited effectively. For the edge mode II in Fig. 3g, light is mainly localized in waveguides B2 and C2, while the corresponding electric fields in the two waveguides are in phase. By using a Y-branch beam splitter ahead of the sample, the edge mode II can be excited effectively. The edge mode III was excited using a similar method. For the interface modes I, II, and III, the distributions of light intensity and phase are even more complex, but our excitation method is still effective. Due to the difficulty in ideally exciting these interface modes, our experimental results are not exactly the same as the numerical results, but they still capture the key characteristic of the corresponding interface mode. At the output facet of the sample, we employed a charge-coupled device (XG500, XWJG) to collect the light intensity distributions.