Introduction

Modern photonic integrated circuits (PICs) are undergoing a paradigm shift toward dense integration, which is essential for scaling bandwidth, minimizing energy per bit, and supporting on-chip convergence of diverse optical functionalities1. Achieving such advancements requires photonic platforms that offer not only high index contrast but also wide spectral transparency, strong nonlinear and electro-optic properties2. Conventional silicon, silicon nitride and III–V platforms, while mature in CMOS-compatible processes, face inherent limitations in nonlinear performance, electro-optic response, or material transparency. In this context, thin-film lithium niobate on insulator (TFLNOI) has emerged as a promising alternative that combines a wide optical transparency window, strong nonlinear optical performance and high-speed electro-optic modulation3,4. Recent advances in TFLNOI-based electro-optic modulators have demonstrated unprecedented 100 GHz bandwidths and near-1V driving voltages at different wavelength ranges5,6, enabling more than a tenfold footprint reduction compared to bulk lithium niobate counterparts. These breakthroughs, driven by applications such as terabit per second optical communications7, precision optical metrology8 and quantum photonic processors9, position TFLNOI as a versatile platform for multifunctional photonic systems. However, the relatively modest refractive index contrast in TFLNOI waveguides, especially those fabricated with partial etching, inherently limits the mode confinement10. Thus, device dimensions such as waveguide bending radius face limitations, which constrain the achievable chip integration density and challenge the scalability demands of large-scale PICs. To bridge the gap between TFLNOI’s exceptional material properties and dense chip integration, advanced photonic design methods are emerging as powerful tools toward compact footprints while maintaining compatibility with fabrication processes.

Inverse design, which leverages intelligent optimization algorithms to explore a vast parameter space, can significantly improve multistage efficiency in device design, simulation and fabrication by enabling fast creation of complex optical structures, e.g., metasurface, with ultra-compact size, unprecedented precision and functionality11,12. Significant breakthroughs have been achieved in silicon photonics during the last few years, with the device footprint scaled down to the micron level, paving the way to densely integrated on-chip optical systems13,14,15,16. However, to obtain inverse-designed PIC components and complex optical systems in TFLNOI, the inherent barriers of material, method and fabrication process are particularly intractable. First, an X-cut TFLNOI platform experiences refractive index anisotropy in the y/z plane, and the traditional scalar-based inverse design methods are inappropriate to address the scenarios of anisotropic material. Although a Z-cut TFLNOI platform can avoid the material anisotropy in the plane of lithium niobate thin film17, the access to the strong electro-optic coefficient of lithium niobate (r33) requires laying out electrodes vertically, which is inconvenient for chip integration. Second, a tensor-based inverse design method can be extended to the X-cut TFLNOI platform by considering the impact of material anisotropy of lithium niobate18. However, a computational complexity increase is inevitable compared to that of traditional scalar-based inverse design methods, which brings higher requirements for computing resources and simulation time. Third, the generally used physical direct etching method for TFLNOI can induce a slanted waveguide sidewall, which limits the device feature size19, resulting in significant degradation and variability in device performance. Thus, the inverse designed lithium niobate photonics requires an overall optimization to overcome the material anisotropy and limited device feature size, while reserving the access to excellent material properties of lithium niobate.

In this work, we demonstrate inverse-designed high-density multimode photonic circuits for high-speed and large-capacity data communications in a silicon nitride-TFLNOI (X-cut) hybrid platform. As shown in Fig. 1a, integrated multimode photonics can enable large-scale signal parallelism and enhance the information capacity of PICs by introducing higher-order waveguide modes as data carriers20, which has also attracted attention in optical computing21, sensing22 and spectroscopy23. Unfortunately, the arbitrary routing of multimode signals is usually built in a large chip area, even in a high-indexed material platform, due to the radiation leakage and inter-mode coupling14. As shown in Fig. 1b–d, we design, fabricate and experimentally demonstrate an optical mode-division (de)multiplexer (19 × 25 μm2), a multimode waveguide crossing (15 × 15 μm2) and a multimode waveguide bend (30 μm bending radius) for the first three quasi-transverse electrical modes (TE0, TE1 and TE2). The measured insertion losses are lower than 2.5 dB at the wavelength of 1550 nm, while the mode crosstalk stays below −15 dB at a wavelength range from 1530 nm to 1570 nm. These components are used to construct ultra-compact photonic circuits, enabling arbitrary routing of multimode signals through dense integration of more than 10 waveguide elements within a 0.06 mm2 (200 × 300 μm²) chip area. To demonstrate the advancement in combination of high integration density with excellent material properties, a high-speed data modulation and transmission system is implemented through the monolithic integration of electro-optical modulators and multimode photonic circuits. The single-channel data modulation and transmission rate for on–off key signal reaches up to 120 Gbps.

Fig. 1: Inverse-designed lithium niobate multimode photonic circuits.
Fig. 1: Inverse-designed lithium niobate multimode photonic circuits.
Full size image

a Schematic diagram of an on-chip data modulation and multimode transmission system, where the chip consists of an array of micro-ring-based electro-optical modulators and the inverse-designed multimode photonic circuit (PC Polarization Controller, O/E Optical/Electrical, CPU Central Processing Unit, GPU Graphics Processing Unit). b–d Schematic diagrams of the inverse-designed mode (de)multiplexer, multimode waveguide crossing and waveguide bend (SiN Silicon Nitride, LN Lithium Niobate, SiO2 Silicon Dioxide, Si Silicon).

Results

Waveguide platform

Silicon nitride is selected as an isotropic optical loading material. Thus, the optical structures in the loading layer can be optimized using a scalar-based inverse design method and fabricated by mature silicon nitride etching processes, enabling improved design efficiency and well-controlled device feature size. There are also other available loading materials, such as silicon, polymers and chalcogenide glasses. Among them, silicon is naturally an attractive loading material for dense integration since its high refractive index (~3.45 at 1550 nm) allows strong mode confinement in the loading rib, which can help to dramatically reduce the device footprint. However, the propagation loss of silicon-TFLNOI hybrid waveguide is relatively high24, the lower mode confinement in lithium niobate thin film may lead to reduced electro-optical modulation and optical nonlinear efficiency25, and the initial optical transparency window of silicon limits the waveguide’s usefulness for nonlinear optical applications across a large wavelength range26. By contrast, silicon nitride can be deposited by different methods with low optical loss, which offers a similar but slightly lower refractive index (~2.0 at 1550 nm) than lithium niobate (ne ~ 2.14, no ~ 2.22 at 1550 nm), and similar optical transparency window spanning from visible to mid-infrared wavelengths compared with lithium niobate27. The silicon nitride-TFLNOI (X-cut) hybrid platform is designed to avoid the direct etching of lithium niobate thin film, while providing a high mode confinement factor (~60%) in lithium niobate. The layers from top to bottom are air cladding, silicon nitride loading layer, lithium niobate slab, buried oxide layer and silicon substrate, respectively. A hybrid silicon nitride-TFLNOI ridge waveguide is formed by the deposited and etched silicon nitride loading rib and the unetched lithium niobate slab. Light is coupled to the hybrid ridge waveguide modes without evanescent coupling between different layers (see Supplementary Note 1 for details). The thicknesses of the silicon nitride and lithium niobate layers are both 300 nm, which avoids the mode lateral leakage at second-harmonic wavelengths28. The thickness of the buried oxide layer is selected as 4.7 μm to ensure index match and a relatively low radio-frequency loss for electro-optic modulation.

Mode (de)multiplexer

Mode (de)multiplexers (MUXs) serve as critical PIC components to enable parallel propagation of multimodal signals in a single waveguide. Previous works in TFLNOI platform are generally based on directional coupler structures by forward design29,30,31,32. In this case, the low-loss and low-crosstalk mode coupling relies on strict control of phase and mode field distribution, requiring a large device footprint. In contrast, in this contribution, we employ the inverse design method to implement a three-mode MUX with an ultra-compact effective size of only 19 × 25 μm2.

Figure 2a, b shows an illustration and a scanning electron microscope (SEM) image of the device structure, which is designed and fabricated by patterning the silicon nitride layer. By considering the anisotropy of lithium niobate, the device is designed along the crystallographic Z direction to avoid the mode hybridization between TE1 and TM0 modes33. The widths of the input single-mode waveguides and output multimode waveguide are designed as 1 μm and 4.3 μm, while the gaps between the adjacent input waveguides are 2 μm. Compared with the directly etched TFLNOI waveguide, a near-vertical waveguide sidewall is available when etching silicon nitride, which means a smaller device feature size is allowed. The achievable smaller feature size enables creation of a refined waveguide structure, which is demonstrated to significantly improve the device performance, including insertion losses and mode crosstalk (see Supplementary Note 2 for details). To ensure fabrication reproducibility, we set the minimum feature size to 100 nm in this work.

Fig. 2: Design and experimental demonstration of MUX.
Fig. 2: Design and experimental demonstration of MUX.
Full size image

a Illustration of the designed MUX with an effective size of 19 × 25 μm2. b SEM image of the fabricated MUX with a minimum feature size of 100 nm. c Microscope image of the cascaded devices for signal multiplexing and demultiplexing in the experiment. d Simulated electric field profiles for the TE0 mode input at different ports of the MUX. e–g Simulated transmission spectra of different mode channels at a wavelength range from 1530 nm to 1570 nm. Black, red and blue lines represent the detected TE0, TE1 and TE2 modes, respectively, when light is injected into different channels. h–j Measured transmission spectra of different mode channels at a wavelength range from 1530 nm to 1570 nm. Black, red and blue lines represent the output signals at O0, O1 and O2, respectively, when light is injected into I0, I1 and I2 ports.

We use two cascaded devices with the same structural parameters for signal multiplexing and demultiplexing in the experiment, as shown in Fig. 2c. Grating couplers are used as optical interfaces, which are designed with a period of 920 nm (940 nm for Y propagating) and a filling factor of 0.434. The simulated electric field profiles of a single device are monitored by using the three-dimensional finite-difference time-domain (3D-FDTD) method35, at a wavelength of 1550 nm. As shown in Fig. 2d, the input TE0 modes in the middle (I0), upper (I1) and lower (I2) waveguides are coupled to TE0, TE1 and TE2 modes in the output waveguide, respectively. The corresponding transmission spectra at a wavelength range from 1530 nm to 1570 nm are depicted in Fig. 2e–g. The simulated insertion losses of the TE0, TE1 and TE2 modes are 0.52 dB, 0.63 dB, and 0.87 dB at the wavelength of 1550 nm, while the mode crosstalk stays below −20.1 dB across the whole wavelength range. To further characterize the device performance, the transmission spectra of different modes are measured by using the device shown in Fig. 2c. As shown in Fig. 2h–j, the measured insertion losses of TE0, TE1, and TE2 mode channels are 1.05 dB, 1.42 dB, and 2.50 dB at the wavelength of 1550 nm, with the mode crosstalk lower than −15.9 dB across the whole wavelength range, which are in good agreement with the simulation results. The demonstrated MUX exhibits comparable performance with the conventional counterparts based on directional couplers (see Supplementary Note 3 for details), while representing a more than tenfold footprint reduction. Moreover, we have fabricated two other devices in which the widths of the etched waveguide patterns deviate from the optimal design by ±20 nm. The measured results indicate that the fabricated device maintains low insertion loss, low mode crosstalk and wide bandwidth across fabrication variations, demonstrating good fabrication tolerance (see Supplementary Note 4 for details).

Multimode waveguide crossing

Waveguide crossing is another important PIC component for dealing with the cross connect in waveguide routing36. The discontinuities at the intersection can induce mode coupling and interference, presenting inherent challenges for device design.

As shown in Fig. 3a, an ultra-compact three-mode waveguide crossing is designed with a device footprint of only 15 × 15 μm2. The widths of the input and output multimode waveguides are 4.3 μm, which are consistent with those of the MUX. As mentioned earlier, the minimum feature size is set to 100 nm, considering the trade-off between device performance and fabrication reproducibility. Figure 3b shows an SEM image of the fabricated device. To ensure the device performance, the electric field profiles of different modes are simulated at a wavelength of 1550 nm by using the 3D-FDTD method. As shown in Fig. 3c, the multimode signals pass through the waveguide crossing with negligible intermodal coupling and radiation loss. We characterize the waveguide crossing by using a pair of auxiliary MUXs based on directional couplers. The microscope image of fabricated devices is shown in Fig. 3d. The input TE0 modes are coupled to different mode channels at the input ports, and then pass through the device. The multimode signals are demultiplexed at the output ports for transmission spectra measurements. The insertion losses of the fabricated waveguide crossing are measured through the cut-back method. As shown in Fig. 3e, the measured and fitted insertion losses of TE0, TE1, and TE2 modes at the wavelength of 1550 nm are 0.16 dB, 0.33 dB, and 0.55 dB, respectively. In addition to the ultra-low insertion losses, the fabricated waveguide crossing is also demonstrated to have low inter-modal crosstalk at a wide wavelength range from 1530 nm to 1570 nm. As shown in Fig. 3f–h, the simulated transmission spectra predict that the inter-modal crosstalk for TE0, TE1, and TE2 mode channels stays below −19.4 dB, −46.2 dB and −18.9 dB. The corresponding measured results are shown in Fig. 3i-k, where the inter-modal crosstalk is lower than −17.7 dB, −15.0 dB and −16.1 dB for TE0, TE1, and TE2 mode channels, respectively. We have noted that the crosstalk for the TE1 mode channel is limited, due to the unexpected mode coupling in the auxiliary MUXs (see Supplementary Note 3 for details).

Fig. 3: Design and experimental demonstration of multimode waveguide crossing.
Fig. 3: Design and experimental demonstration of multimode waveguide crossing.
Full size image

a Illustration of the designed waveguide crossing with an effective size of 15 × 15 μm2. b SEM image of the fabricated device. c Simulated electric field profiles for different input modes: TE0, TE1 and TE2. d Microscope image of the cascaded devices connected to auxiliary MUXs for insertion loss measurement. e Measured and fitted insertion losses of TE0, TE1, and TE2 modes at the wavelength of 1550 nm. Black, red and blue lines represent the fitted results of TE0, TE1, and TE2 modes, respectively. f–h Simulated transmission spectra of different mode channels at a wavelength range from 1530 nm to 1570 nm. i–k Measured transmission spectra of different mode channels at a wavelength range from 1530 nm to 1570 nm. Black, red and blue lines represent the detected TE0, TE1 and TE2 modes, respectively, when light is injected into different channels.

It should also be noted that the anisotropy of lithium niobate has been included in the device optimization. Thus, the designed and fabricated device has similar performance when light propagates along the crystallographic Z direction of lithium niobate (see Supplementary Note 5 for details).

Multimode waveguide bend

The waveguide bending radius is strongly limited in the TFLNOI platform due to the relatively low refractive index. The single-mode waveguide bending radius in a micro-ring resonator typically reaches up to 80 ~ 100 μm, to reduce the mode mismatch and radiation losses37,38. The multimode waveguide bends require a larger size, inevitably due to the weaker confinement for higher-order modes. Though shape optimization like Euler bends31, B-spline39 or air grooves40 can significantly reduce the multimode waveguide bending radius to 100 ~ 160 μm, there is still a gap between the size required for dense integration. Here, we demonstrate an ultra-compact multimode waveguide bend with an effective bending radius of only 30 μm, which is equivalent to that of the sharpest single-mode waveguide bend with a partial-etched waveguide cross-section in X-cut TFLNOI platforms41,42, to the best of our knowledge.

The schematic diagram and SEM image of the multimode waveguide bend are shown in Fig. 4a, b. The widths of the input and output multimode waveguides are 4.3 μm. Figure 4c shows the electric field profiles of different modes, simulated by using the 3D-FDTD method at a wavelength of 1550 nm. After passing through the topology optimization area, the propagation directions of multimodal signals are turned by 90° with negligible inter-modal coupling. Similarly, we characterize the device by using the directional coupler-based auxiliary MUXs. The microscope image of fabricated devices is shown in Fig. 4d. The insertion losses of the fabricated waveguide bends are measured through the cut-back method. As shown in Fig. 4e, the measured and fitted insertion losses for TE0, TE1, and TE2 modes at a wavelength of 1550 nm are 1.02 dB, 1.96 dB, and 2.17 dB, respectively.

Fig. 4: Design and experimental demonstration of multimode waveguide bend.
Fig. 4: Design and experimental demonstration of multimode waveguide bend.
Full size image

a Illustration of the designed waveguide bend with an effective bending radius of 30 μm. b SEM image of the designed and fabricated device. c Simulated electric field profiles of different input modes. d Microscope image of the cascaded devices connected to auxiliary MUXs for insertion loss measurement. e Measured and fitted insertion losses of TE0, TE1, and TE2 modes at the wavelength of 1550 nm. Black, red and blue lines represent the fitted results of TE0, TE1, and TE2 modes, respectively. f–h Simulated transmission spectra of different mode channels at a wavelength range from 1530 nm to 1570 nm. i–k Measured transmission spectra of different mode channels at a wavelength range from 1530 nm to 1570 nm. Black, red and blue lines represent the detected TE0, TE1 and TE2 modes, respectively, when light is injected into different channels.

The waveguide bend has low inter-modal crosstalk over a wide wavelength range. As shown in Fig. 4f–h, the simulation transmission spectra predict that the inter-modal crosstalk for TE0, TE1, and TE2 mode channels can stay below −20.4 dB, −16.2 dB and −19.8 dB at a wavelength range from 1530 nm to 1570 nm. For the convenience of measurement, the propagation directions of the multimode signals are actually turned by 180° in the experiment (two 90° bends). The corresponding measured results are shown in Fig. 4i–k, where the inter-modal crosstalk for TE0, TE1, and TE2 mode channels is lower than −16.2 dB, −15.1 dB and −15.9 dB, agreeing well with the simulated results. Moreover, we have performed an additional design for a single-mode 90° waveguide bend by using the same method. It is demonstrated that the wide input and output waveguides offering strong mode confinement for the TE0 mode can help to reduce the waveguide bending radius to about 14 μm (see Supplementary Note 6 for details).

Multimode photonic circuits

To further demonstrate the device performance in dense integration, we design and fabricate two types of multimode photonic circuits. The first PIC shown in Fig. 5a consists of two independent circuits with path crossing of the two circuits, while the second one shown in Fig. 5b is a single circuit with multipath crossings of the same circuit. The input fundamental modes are first coupled into different mode channels by the mode multiplexer. The multiplexed multimode signals are then routed in the chip plane by using the waveguide crossings and bends, which are finally demultiplexed by another mode demultiplexer. An only a chip area of 200 × 300 μm² is occupied to implement these operations.

Fig. 5: Experimental demonstration of multimode arbitrary routing system.
Fig. 5: Experimental demonstration of multimode arbitrary routing system.
Full size image

a, b Microscope images of the fabricated multimode photonic circuits. c–e Measured transmission spectra of different mode channels for the first PIC, shown in (a). f–h Measured transmission spectra of different mode channels for the second PIC shown in (b). Black, red and blue lines represent the output signals at O0, O1 and O2, respectively, when light is injected into I0, I1 and I2 ports.

The measured transmission spectra of the first PIC are shown in Fig. 5c–e. The insertion losses for TE0, TE1 and TE2 mode channels are 4.2 dB, 7.8 dB and 7.1 dB, while the inter-modal crosstalk stays below −20.3 dB at the wavelength of 1550 nm. Similarly, the measured transmission spectra of the second PIC are shown in Fig. 5f–h. It can be seen that the insertion losses for TE0, TE1 and TE2 mode channels are 6.5 dB, 11.5 dB and 11.9 dB, while the inter-modal crosstalk still stays below −12.5 dB at the wavelength of 1550 nm. The measured insertion losses of PICs are very close to the sums of individual device insertion losses, with the deviations lower than 1 dB, 2.13 dB and 0.74 dB for TE0, TE1 and TE2 modes (see Supplementary Note 7 for details). The deviations can be attributed to the following two aspects. First, the reported insertion losses of waveguide crossing and bends are measured and fitted by using the cut-back method. It can be seen from Figs. 3e, 4e that the measured insertion losses actually vary for different components in the experiment due to the fabrication variations. Second, the PICs and individual components are fabricated in different regions on the same chip. Thus, the fabricated structural parameters for the same device can also be slightly different. Overall, despite a significant increase in the number of waveguide elements, the measured results still indicate good performance, presenting the good system scalability and dense integration capability of the inverse-designed devices.

To demonstrate the advancement in the combination of dense integration capability and excellent material properties, an on-chip data modulation and transmission system is implemented. We design and fabricate three electro-optical modulators based on the micro-ring resonator structure, corresponding to the employed three-mode channels. Light emitted by the lasers is modulated by high-speed radio-frequency signals, then routed in the chip plane by the multimode photonic circuits, and finally detected by photodetectors. A microscope image of the fabricated electro-optical modulator is shown in Fig. 6a. The designed waveguide width and the coupling gap are 1 μm and 750 nm, respectively. The radius of the ring waveguide is 300 μm, while the length of the straight track is 1.5 mm. The measured transmission spectrum of the micro-ring resonator is shown in Fig. 6b. The insertion loss at a non-resonant wavelength is lower than 0.54 dB, while the extinction ratio is larger than 12.4 dB. A ground-signal-ground (G-S-G) traveling-wave electrode is employed. The width of the signal electrode is designed as 23 μm, and the gap between signal and ground electrodes is 6 μm. Figure 6c shows the measured eye diagrams for different mode channels with the modulated and transmitted on-off key signal rates of 100 Gbps and 120 Gbps. The measured signal-to-noise ratios are larger than 5.4 dB. The clear and open eyes indicate good system performance.

Fig. 6: Experimental demonstration of on-chip data modulation and multimode signal transmission.
Fig. 6: Experimental demonstration of on-chip data modulation and multimode signal transmission.
Full size image

a Microscope image of the fabricated micro-ring-based electro-optical modulator. b Measured transmission spectra of the micro-ring resonator. c measured eye diagrams for different mode channels with the modulated and transmitted on-off key signal rates of 100 Gbps and 120 Gbps.

Discussion

We compare the overall device performance, including effective size, insertion loss, mode crosstalk and data modulation rate, with previously reported works, as shown in Table 1. It can be seen that the measured insertion losses and inter-modal crosstalk are comparable with those of the forward-designed counterparts, while the device footprint is significantly reduced by an order of magnitude. Even compared with the previous single- or dual-mode devices, the demonstrated multimode devices exhibit relatively compact size, low insertion loss and low mode crosstalk. Moreover, the demonstrated PICs achieve similar high performance in electro-optical modulation when compared with the directly etched platforms, due to the high mode confinement in lithium niobate thin film.

Table 1 Performance comparison of some previously reported TFLN components
Full size table

We also compare the key performance metrics of recently reported inverse-designed devices on different material platforms, including silicon on insulator (SOI)15,43,44,45,46, silicon nitride on insulator (SNOI)47 and directly etched TFLNOI17. As shown in Table 2, the inverse-designed MUX serves as an illustrative example. Despite the moderate intrinsic refractive indices of both lithium niobate and silicon nitride, the demonstrated device achieves a footprint comparable to those in silicon photonics. Furthermore, it delivers similarly low loss and crosstalk as pure SNOI and TFLNOI devices, while uniquely combining the inverse design capabilities of silicon nitride with the strong electro-optic and nonlinear optical response of TFLN, offering a functionality set unavailable before. The device performance can be further improved through increasing the thickness of the deposited silicon nitride thin film, which can be attributed to enhanced interaction between light and the etched nanostructures due to the stronger mode confinement in the silicon nitride loading rib (see Supplementary Note 8 for details). However, the reduced mode confinement factor in lithium niobate thin film also means lower electro-optical modulation and optical nonlinear efficiency.

Table 2 Performance comparison of some inverse-designed MUXs
Full size table

In summary, we have demonstrated ultra-compact multimode components and photonic circuits based on a silicon nitride-lithium niobate hybrid platform by using the photonic inverse design method. This hybrid platform shows a good combination of the excellent material properties of silicon nitride and lithium niobate. The optical structures in the isotropic silicon nitride loading layer are optimized by using a scalar-based inverse design method with improved design efficiency when compared with that using a tensor-based method for anisotropic materials. Moreover, the silicon nitride loading material owns mature fabrication processes, enabling improved device performance by means of small and well-controlled device feature size. The reduction in device footprint supports complex optical path design within an ultra-compact chip area, while more chip space is reserved for other important active functions such as data modulation, wavelength conversion, and spectral control. Our approach is scalable, which can be extended to a wide range of passive components, which are expected to be important building blocks for realizing densely integrated and large-scale PICs in the TFLNOI platform.

Methods

Inverse design method

Gradient optimization based on the adjoint method is used to implement topology optimization through iterative material distribution updates. The dielectric permittivity of the design area is initially projected onto a [0,1] space for parametric normalization (Eq. (1)). The material distribution is subsequently smoothed via a convolutional filtering (Eq. (2)) to eliminate subwavelength defects. Finally, an adaptive binarization projection function (Eq. (3)) is used to increase filter sharpness β and obtain a binary distribution of dielectric constants. We only need to optimize the distribution of isotropic silicon nitride loading material. Thus, the scalar-based inverse design method is applicable.

$${\rho }_{0}=\frac{{\varepsilon }_{r}(\rho )-{\varepsilon }_{{min }}}{{\varepsilon }_{{max }}-{\varepsilon }_{{min }}}$$
(1)
$${\rho }_{1}=w(x)\cdot {\rho }_{0}$$
(2)
$${\rho }_{2}=\frac{tanh (\beta \cdot \eta )+\, tanh (\beta \cdot ({\rho }_{1}-\eta ))}{tanh (\beta \cdot \eta )+\, tanh (\beta \cdot (1-\eta ))}$$
(3)

where ρ0, ρ1 and ρ2 denote the normalized design parameters of the initial, filtered and binary design area, while εr(ρ), εmin and εmax represent the material dielectric constants of the design area, air cladding and silicon nitride load. In addition, w(x) is the convolution kernel, β is the filter sharpness parameter, and η is the design threshold.

Device performance is quantified through a broadband figure of merit (FOM, Eq. 4).

$$FOM={\left(\frac{1}{{\lambda }_{2}-{\lambda }_{1}}{\int }_{{\lambda }_{1}}^{{\lambda }_{2}}{|{T}_{0}(\lambda )|}^{p}d\lambda \right)}^{1/p}-{\left(\frac{1}{{\lambda }_{2}-{\lambda }_{1}}{\int }_{{\lambda }_{1}}^{{\lambda }_{2}}{|T(\lambda )-{T}_{0}(\lambda )|}^{p}d\lambda \right)}^{1/p}$$
(4)

where T0(λ), T(λ) is the target and actual forward transmission at a wavelength of λ, while p is the generalized norm value which can control the optimization preference between transmission and bandwidth (see Supplementary Note 9 for details). In this instance, the effects of lithium niobate birefringence on device performance are included in the simulation.

Fabrication processes

We use an X-cut TFLNOI wafer from NanoLN, China. The silicon nitride thin film is deposited onto the chip by using a reactive sputtering process48. The grating coupler and waveguide structures are patterned by using electron-beam lithography, and then formed by using single-step inductively coupled plasma etching. Finally, the electrodes (500-nm-thick Au) are fabricated by using laser direct writing, e-beam evaporation deposition and lift-off processes. The propagation losses of the fabricated waveguides are measured to be about 0.25 dB/cm by using the cut-back method. The coupling losses of the fabricated grating couplers are measured to be 6 dB/facet, at a peak coupling wavelength of 1560 nm.

Experimental setup

Light is emitted by a tunable laser, followed by a polarization controller used to maintain TE polarization before coupling light into the chip. For static measurements, signals coupled out of the chip are directed to an optical spectrum analyzer for spectral analysis.

For a high-speed data modulation and transmission experiment, the laser light is modulated by the on-chip electro-optical modulator using a 210 − 1 pseudo-random binary sequence generated by an arbitrary waveform generator. After coupling into and out of the chip, the signals are amplified by an erbium-doped fiber amplifier, followed by an optical tunable filter for reducing background noise. Finally, the signals are received by a digital communication analyzer for eye diagram measurements (see Supplementary Note 10 for details).