Introduction

The escalating demand for higher capacity data traffic in commercial communication systems, driven by the global deployment and rapid expansion of Internet services, has prompted to the exploration of multi-band transmission systems and innovative multiplexing technologies.

Ultra-wideband wavelength division multiplexing (WDM) technology holds the potential to significantly enhance the throughput of fiber communication networks. Extending ultra-wideband WDM systems beyond the commonly utilized C (1530−1565 nm) or L (1565−1625 nm) bands to additional bands, such as O (1260−1360 nm), E (1360−1460 nm), S (1460−1530 nm), and U (1625−1675 nm) is currently under investigation to augment the channel capacity of single-mode fiber (SMF) communication systems1,2,3,4. The advancement of ultra-wideband modulators, transceivers, and amplifiers has paved the way for the realization of ultra-wideband WDM applications5,6. In recent years, significant milestones have been achieved in this domain. Demonstrations of fiber throughput exceeding 100 Tbps have been conducted in triple-band configurations encompassing the S, C, and L bands7,8. Furthermore, the transmission of five-band WDM signals (O, E, S, C, and L bands) employing high-order modulation formats has been successfully presented9. The U band shows promise as the next transmission band for multi-band networks, primarily due to its advantage of lower transmission loss. The National Institute of Information and Communications Technology constructed the world’s first O to U band system capable of dense WDM (DWDM) transmission in a commercially available standard optical fiber achieved with custom designed amplifier technology10. It is anticipated that spectrum expansion will emerge as a prominent trend in augmenting channel capacity in the future.

Space division multiplexing (SDM) extends the spatial dimensions of electromagnetic waves and is recognized as a next-generation fiber communication technology with immense potential11,12,13. Compared to the Shannon’s law for SMF optical communication systems, the capacity of SDM systems that can be reliably communicated incorporates a factor of the number of spatial channels. SDM systems achieve continuous breakthroughs in channel capacity by leveraging spatially orthogonal modes in few-mode fiber (FMF)/multi-mode fiber (MMF)14,15,16 or multi-core multiplexing based on spatially separated cores in multi-core fiber (MCF)17,18,19. Various modes of FMF/MMF have been reported, including 2-mode20, 4-mode21, 6-mode22, 9-mode23, 15-mode24, and even 55-mode25 configurations. Through a multi-dimensional hybrid multiplexing system integrating wavelength, space, polarization, and higher-order signal modulation formats, the transmission capacity per fiber for SDM can reach the order of Pb/s. Demonstrations of FMF/MMF signal transmission have showcased significant capacities, such as over 1 Pb/s for 15-mode and up to 1.53 Pb/s for 55-mode configurations, achieved by Georg Rademacher et al. in 202126 and 202227 respectively. These transmissions have also achieved notable bandwidths, reaching up to 82 nm26. With the development of few-mode MCF (FM-MCF), SDM communication systems, through a combination of mode channels and multi-core channels, exhibit considerable potential to increase total capacity28,29,30. Experimental SDM-WDM hybrid multiplexing communications have reached transmission capacities of up to 10.66 Pb/s in a 38-core-3-mode FM-MCF configuration31. While existing SDM studies have predominantly focused on traditional C + L bands or single-span bands (e.g., O band), the deployment of ultra-wideband strategies and SDM technologies may occur at different times due to cost considerations. However, they exhibit synergistic effects. Looking ahead, ultra-wideband SDM transmission represents a forefront research area in optical communication systems, promising further improvements in capacity and performance.

Over the past few years, a plethora of mode multiplexers has become integral to SDM systems, sparking notable research interest. Investigations have delved into mode multiplexers/demultiplexers employing various optical components, such as free-space combining28, phase diffraction gratings32, all-optical fibers33, planar lightwave circuits34, photonic integration35, optical geometrical transformations36, metasurfaces37, and multi-layer optical networks38,39,40,41,42. Among these devices, the maximum number of multiplexed modes up to 1025 Hermite-Gaussian modes in free space43 and 55 linearly polarized (LP) modes in fiber communication systems27, respectively. Multiple methods seamlessly integrate with WDM for wavelength-mode hybrid transmission across the C + L bands or O band26,44. Metasurfaces and multi-layer optical networks mark significant progress in multi-wavelength device technology due to their precise beam control, versatility in both application and manufacturing, and high efficiency. However, they are susceptible to considerable chromatic aberrations due to their operation as diffractive optical elements. Several pioneering studies have demonstrated that multi-layer diffraction structures capable of operating over a broad wavelength range can be achieved through dispersion control. Notable examples include achromatic multi-layer metasurface lenses45, wavelength-multiplexed diffractive neural networks46, broadband optical computing systems47 and ultra-wideband 90-degree optical hybrid48. As demand surges for higher bandwidth in SDM transmission applications, ultra-wideband multiplexer remains a major challenge. Wavelength dispersion plays an important role in designing ultra-wideband multiplexer. Designing ultra-wideband multiplexers with high performance and low complexity presents significant challenges, particularly in ensuring low wavelength- and mode-dependent characteristics, high mode purity, and low insertion loss (IL).

In this work, we utilize a wide-spectrum, dispersion-compensated, multi-layer diffractive network within an end-to-end inverse design framework to create ultra-wideband optical components. As a proof of concept, we demonstrate an ultra-wideband, 4-LP mode multiplexer that achieves high efficiency over a broad bandwidth ranging from 1260 to 1680 nm. We present a general model for an ultra-wideband mode multiplexing transmission system with a bandwidth exceeding 400 nm, significantly enhancing transmission capacity. To our knowledge, there are no reports of ultra-wideband mode division multiplexing (MDM) covering the entire O + E + S + C + L + U bands. Strategies to increase capacity per fiber mode involve combining data signals and using higher-order modulation formats across ultra-wideband wavelength and mode channels spanning the O + E + S + C + L + U bands, as shown in Fig. 1. By integrating our ultra-wideband mode multiplexing system with existing commercial WDM communication systems, we achieved a series of hybrid multiplexed communication transmissions. Our approach to ultra-wideband multi-layer diffractive network introduces significant innovations in full-color light manipulation, imaging, optical computing and high-capacity optical communications, advancing these designs toward practical applications.

Fig. 1: Schematic diagram of the ultra-wideband mode multiplexing communication transmission system.
Fig. 1: Schematic diagram of the ultra-wideband mode multiplexing communication transmission system.
Full size image

The system covers the wavelength channel of the entire communication bands of O + E + S + C + L + U.

Results

Ultra-wideband optical diffraction network and chromatic dispersion compensation architecture

Typically, light waves of different wavelengths exhibit varying diffraction effects, impacting phase control. The phase distribution resulting from an N-layer diffraction network is as follows:

$${\varphi }_{N}(\lambda )=\varphi ({\lambda }_{r})+{\delta }_{\varphi }(\lambda,{\mbox{Multilayers}})$$
(1)

where φ(λr) represents the basic phase profile related to the reference wavelength λr, and δφ (λ, Multilayers) denotes the additional phase shift between λ and λr. This phase shift, crucial for system sensitivity and accuracy, is influenced by the phase response of each unit element in the multi-layer network. It is vital for achieving stable, high modulation efficiency over an ultra-wideband range. Designing the multi-layer network to compensate for phase shifts is therefore crucial.

Here, a wide-spectrum dispersion-compensated multi-layer optical diffraction network is depicted in Fig. 2a. The N cascaded diffractive layers, positioned between the input and output apertures, handle the manipulation and dispersion compensation of complex light across different wavelengths. Each layer is coded with spatial features and a trainable thickness that allows full phase modulation from 0 to 2π. It is worth noting that all layers in the described framework are transmissive, although reflective layers can also be employed in practical experiments. The phase profiles of all the layers can be printed on a single reflective phase plate. In this work, the subsequently fabricated device is experimentally realized by utilizing a multi-pass reflective cavity formed by the phase plate and a mirror. The input, diffractive layers and output apertures are connected to each other through free space. The free space propagation can be modeled using the scalable angular spectrum method or the Fresnel diffraction approach. The phase shift introduced by the wavelength can be compensated by modulation, the effect of which can be evaluated and optimized by the modulation efficiency of the system. In our design, we propose a synergistic modeling mechanism with multi-objective and multi-physics coordination through a wide-spectrum modulation phase function ψ(λ) for compensation of ultra-wideband chromatic dispersion and spatial phase mismatch:

$$\psi (\lambda )=\frac{{\lambda }_{r}}{\lambda }\cdot {\psi }_{r}\cdot f\left(\frac{B}{W}\right)$$
(2)

where W denotes the spectral width of training and B is the working bandwidth. The reference wavelength λr is used to convert the pixel matrix of the optical diffraction layer to the physical structure. ψr is the basic phase profile at λr. The phase function ψ(λ) establishes a nonlinear mapping from the coupled space into the global modulation efficiency η, which jointly evaluates mode purity, IL, and intermodal crosstalk:

$$\eta \propto f(K\cdot \psi (\lambda ))$$
(3)
Fig. 2: Design of ultra-wideband mode multiplexer based on multi-layer optical diffraction network.
Fig. 2: Design of ultra-wideband mode multiplexer based on multi-layer optical diffraction network.
Full size image

a End-to-end inverse design framework for ultra-wideband mode multiplexer. \(O_{\lambda^c}\) and \(T_{\lambda^c}\) represent the generated and target modes light fields of wavelength λ at the c-th mode channel on the output plane. The symbols \(U^{k}_{\lambda^{c}_{min}} < /Emphasis > < /Subscript > \), \(U^{k}_{\lambda^{c}_{r}} < /Emphasis > < /Subscript > \), \(U^{k}_{\lambda^{c}_{max}} < /Emphasis > < /Subscript > \) denote the forward optical field at the minimum, reference, and maximum wavelengths, respectively, of the c-th mode channel. Similarly, \(U^{\prime k}_{\lambda^{c}_{min}} < /Emphasis > < /Subscript > \), \(U^{\prime k}_{\lambda^{c}_{r}} < /Emphasis > < /Subscript > \), \(U^{\prime k}_{\lambda^{c}_{max}} < /Emphasis > < /Subscript > \) denote the corresponding backward optical field. b Simulated phase distributions of ultra-wideband four linearly polarized (LP)-mode multiplexer. c Simulated wavelength-dependent average mode purity, maximum mode crosstalk, average insertion loss (IL) and mode dependent loss (MDL).

This mapping defines a dual-parameter dispersion balancing strategy based on reference wavelength (λr) and a wide spectral window (W), which balances spectral performance across short and long wavelengths across the entire target bandwidth B. A normalization factor (λ/λr) is applied to align the compensation phase across different wavelengths, effectively eliminating absolute phase offsets. Additionally, the coefficient K, associated with the number of diffractive layers, governs overall efficiency. Excluding free space propagation effects, the mean value μ of ψ(λ) across B determines λr. Setting λr where μ = ψ(λr) equalizes efficiency at the bandwidth boundaries:

$${\lambda }_{r}=\frac{2\cdot {\lambda }_{\min }^{{\prime}}\cdot {\lambda }_{\max }^{{\prime}}}{{\lambda }_{\min }^{{\prime}}+{\lambda }_{\max }^{{\prime}}}$$
(4)

Where \({\lambda }_{\min }^{\prime}\) and \({\lambda }_{\max }^{\prime}\) are the maximum and minimum wavelengths of the working bandwidth, respectively. Due to free space propagation, shorter wavelengths experience stronger diffraction. Therefore, as the number of diffracting layers or the propagation distance increases, λr should be adjusted to a shorter wavelength to compensate for this effect. It is noted that the standard deviation of efficiency across the working bandwidth B is positively correlated with the dispersion compensation effect. Thus, we use this parameter to characterize the wavelength dependence of the system.

Free-space architecture design of cascaded optical layers constructed by the multi-focal optical system

The concept of an ultra-wideband mode multiplexing is constructed. Arrays of Gaussian spots covering multiple wavelengths across an ultra-wideband spectrum are simultaneously directed into the cascaded optical layers. After these cascaded optical layers process the propagation of light waves within the Fourier planes of the multi-focal optical system, coaxial orthogonal multi-mode light with multi-spectral features is ultimately detected at the output plane.

Figure 2a illustrates the flowchart of the proposed inverse design algorithm. To convert the physical design problem into a mathematical optimization problem, a matrix of pixels with amplitude and phase is employed as the solution space. Initially, the transmission of all diffraction units in each layer at the reference wavelength is set to 1. These diffraction units in each layer exhibit varying phase responses \(\psi^{k}_{\lambda^{c}} < /Emphasis > < /Subscript > \) (xi, yi, zi) for different wavelength channels λ and mode channels c of light. To execute both forward and backward design simultaneously, the outputs of each layer are remapped as inputs to the subsequent layer, except for the final one. Each layer is constructed through wavelength-mode joint end-to-end phase matching to achieve synchronous extraction of global multi-parameter errors and coordinated correction of local features, thereby enabling high mode purity, low mode crosstalk and low IL performance across the entire working bandwidth. The local error field at each spatial coordinate (xi, yi) is computed by comparing the forward optical field \(u^{k}_{\lambda^{c}} < /Emphasis > < /Subscript > \)(xi, yi, zk) and the backward optical field \(g^{k}_{\lambda^{c}} < /Emphasis > < /Subscript > \) (xi, yi, zk):

$${{{\rm{Error}}}}_{{\lambda }^{c}}^{k}({x}_{i},{y}_{i})={u}_{{\lambda }^{c}}^{k}({x}_{i},{y}_{i},{z}_{k})^{\ast} \cdot {g}_{{\lambda }^{c}}^{k}({x}_{i},{y}_{i},{z}_{k})$$
(5)

the wavelength-mode-dependent global phase error at each layer is then quantified by summing these local phase errors across all spatial coordinates as follows:

$${\delta }^{k}(\lambda,c)={\mbox{Arg}}\left[\mathop{\sum }\limits_{x}\mathop{\sum }\limits_{y}{{{\rm{Error}}}}_{{\lambda }^{c}}^{k}\cdot \exp \left(-j{\psi }_{{\lambda }^{c}}^{k}\cdot \frac{{\lambda }_{r}}{\lambda }\right)\right]$$
(6)

local correction phase of each layer is iteratively refined through the following gradient-based optimization of global phase errors across all wavelength channels and mode channels:

$${L}^{k}({x}_{i},{y}_{i},{z}_{k})=\mathop{\sum }\limits_{\lambda }\mathop{\sum }\limits_{c}{{{\rm{Error}}}}_{{\lambda }^{c}}^{k}({x}_{i},{y}_{i},{z}_{k})\cdot \exp (-j{\delta }^{k}(\lambda,c))$$
(7)

Through the establishment of the relationship between the local correction phase and the coupled space of wavelength, mode, coordinates, working bandwidth, spectral width of training, free space transfer function and the sequence of diffraction layers, each layer is then updated accordingly. All layers within the diffraction network collaborate to handle focusing, mode conversion and dispersion compensation for ultra-wideband complex optical fields. The multi-layer device is determined after a series of iterative optimizations. The correlation function between the generated modes and target modes is defined as a criterion for iteration termination. (See Supplementary Note 1 for algorithm-related details).

Based on the wide-spectrum dispersion-compensated design framework, a four LP-mode multiplexer consists of six-layer optical diffraction network with a bandwidth 420 nm was developed. When the training spectral width W is set within the range of 600 to 800 nm, the mode purity within the bandwidth ranges of 1260 nm−1680 nm remains above 90% with low standard deviations (see Supplementary Fig. S3). Figure 2b shows that the design of W = 700 nm (1100−1800 nm). Six diffraction layers exhibit smooth phase distributions. Figure 2c illustrates the simulated wavelength-dependent average mode purity, maximum mode crosstalk, average IL and mode dependent loss (MDL) for all mode channels of the multiplexer. The mode purity exceeds 90% across the entire 420 nm bandwidth, with the highest mode purity exceeding 94%. The crosstalk remains below −15 dB, while the IL and MDL are below 0.53 dB and 0.22 dB, respectively. Additionally, the IL and mode-dependent loss exhibit low wavelength dependence, with fluctuations of less than 0.21 dB and 0.12 dB, respectively. With respect to its ultra-wideband characteristics, the device demonstrates a certain tolerance for alignment and propagation distance errors, remaining within the practically acceptable range (see Supplementary Note 2). Notably, the performance of the device varies significantly in the short wavelength range, which is an effect of the enhanced short-wavelength attenuation due to free space propagation. To ensure balanced performance across the band, the reference wavelength was set to the shorter wavelength (1380 nm) rather than 1440 nm as calculated from Eq. 4 (see Supplementary Fig. S4). This design simultaneously achieves performance balancing and comprehensive performance enhancement across the entire telecommunication range, advancing MDM technology from fixed-wavelength/band or narrowband dedicated devices toward an ultra-wideband universal platform (see Supplementary Fig. S5).

Experimental characterization for ultra-wideband mode multiplexing

In the experiments, a reflective ultra-wideband mode multiplexing system was implemented, as depicted in Fig. 3a. The experimental setup comprises a 2×2 fiber-microlens array with a standard 250 μm pitch and a reflective cavity, which includes a silicon-based gold-plated phase plate and a reflector. Six phase planes were printed on the same reflective phase plate and fabricated using a combination of ultraviolet exposure and dry etch etching in a four-cycle process (see Supplementary Fig. S6). A deep-to-shallow etching sequence was adopted, resulting in etching depths of 345, 173, 86, and 43 nm in each cycle. The spatial resolution was 8 μm. The distance between the fiber-microlens array and the reflective cavity was approximately 15 mm. Ultra-wideband fundamental mode Gaussian beams were incident at an angle of approximately 9.8° and reflected back and forth 11 times within the reflective cavity before being output as ultra-wideband LP01, LP11a, LP11b and LP21 mode light.

Fig. 3: Experiment of ultra-wideband four LP-mode multiplexing.
Fig. 3: Experiment of ultra-wideband four LP-mode multiplexing.
Full size image

a Experimental setup for mode multiplexing. FMF, Few-mode fiber. G-lens, Gradient-index lens. b Measured LP mode intensity distribution generated by the multiplexer in free space. c Measured maximum-mode crosstalk of the multiplexer. d Measure fiber-to-fiber losses after coupling into four-mode fiber and six-mode fiber.

The performance of the multiplexer was experimentally characterized in the O + E + S + C + L + U bands. To cover the entire bandwidth range, measurements were conducted separately using four tunable laser light sources, each operating in a different spectral band. Figure 3b illustrates the measured intensity distributions of the generated LP modes in these six bands. It can be observed that the mode fields exhibit concentrated intensity with minimal scattering. Figure 3c depicts the wavelength-dependent maximum mode crosstalk among the four modes, calculated from the reconstructed mode fields obtained through interferometric measurements captured by a digital holography experiment38. The maximum mode crosstalk measured in the 1260−1675 nm band is less than −17 dB, with the minimum crosstalk occurring near the center wavelength. As shown in Fig. 3d, the measured fiber-to-fiber losses in the 1260−1675 nm wavelength range were found to be less than 4.9 and 4.4 dB when coupling the LP modes generated in free space into a four-mode fiber (FM SI-4, YOFC) and a six-mode fiber (FM SI-6, YOFC), respectively. These fiber-to-fiber losses encompass the device loss of the optical field from the incident side before the SMF array to the free space side of the multiplexer, as well as the coupling losses of the mode field from free space to FMF. These results demonstrate that the designed ultra-wideband mode multiplexer is compatible with commercially available FMF standards, maintaining fiber-to-fiber losses at a low-loss level over the ultra-wideband range. This characteristic significantly reduces power consumption in the transmission system26,49,50,51. When employing a six-mode fiber, the fiber-to-fiber losses of the different modes are lower. Furthermore, it was observed in the experiments that the mode degradation of the four-mode fiber becomes more severe in the short wavelength range as the transmission distance increases. This finding is particularly crucial for subsequent communication transmission experiments.

Ultra-wideband mode multiplexing communication transmission experiments

Figure 4a illustrates a fundamental framework for ultra-wideband mode multiplexing communication systems, incorporating critical components, such as optical transceivers (Transmitter and Receiver), wavelength multiplexers (WDM MUXs), a mode multiplexer, a mode demultiplexer, and wavelength demultiplexers (WDM DEMUXs). Each optical transceiver consists of a transmit port and a receive port, independently generating and detecting individual wavelengths. A polarization controller (PC) is utilized to adjust the coupling of modes in fiber transmission. The measured maximum mode crosstalk of a typical back-to-back mode multiplexer-demultiplexer system with three modes (LP01, LP11a and LP21) across the O + E + S + C + L + U bands is below −8.87 dB (see Supplementary Fig. S7). Experimental validation of mode multiplexing and wavelength-mode hybrid multiplexing communication transmission in the O, E, S, C, L, and U bands was conducted under two typical optical transceiver mechanisms.

Fig. 4: The characterization of wavelength division multiplexing (WDM)-mode division multiplexing (MDM) and broadband MDM communication transmission.
Fig. 4: The characterization of wavelength division multiplexing (WDM)-mode division multiplexing (MDM) and broadband MDM communication transmission.
Full size image

a Schematic of ultra-wideband mode multiplexing communication system. WDM MUXs, wavelength multiplexers. PC, polarization controller. WDM DEMUXs, wavelength demultiplexers. Measured bit error rate (BER) curves b in O + E bands and c in U band. d Measured BER scatter in S + C + L bands. FEC, forward error correction.

In communication systems operating within the O and E bands, a cost-effective communication mechanism based on direct detection necessitates low mode coupling and stable transmission characteristics. A coarse WDM (CWDM)-MDM transmission system over a 5 km six-mode fiber using direct detection technology was demonstrated. In this scenario, six-wavelength channels (1271, 1291, 1311, 1331, 1351, and 1371 nm) and three-mode channels (LP01, LP11b, and LP21) were utilized for signal transmission. Signals modulated with on-off keying (OOK) were transmitted at a data rate of 10 Gbps per channel. Due to the strong coupling of modes between the same mode groups, one of the LP11a and LP11b mode channels was selected, posing a challenge for detection without a multiple-input multiple-output algorithm. The measured bit error rate (BER) values of all 18 channels in the CWDM-MDM transmission scenario are depicted in Fig. 4b. All BER values are below the forward error correction (FEC) threshold of 3.8E-3, indicating successful data signal transmission. These results validate that the designed mode multiplexer exhibits excellent compatibility with standard CWDM systems.

Based on an intensity modulation direct detection (IMDD) system, a three-mode multiplexed transmission of OOK signals with a channel rate of 10 Gbps over a 5 km four-mode fiber was conducted in the U band. In the absence of optical amplifiers, a 3 × 3 multiple-input multiple-output (MIMO) architecture with digital signal processing (DSP) was implemented at the receiver. The measured BER values remained below the FEC threshold of 2.4E-2 across the 1630 − 1670 nm wavelength range, as demonstrated in Fig. 4c.

Using coherent detection technology, the C band DWDM-MDM system, S band and L band broadband MDM transmission systems were constructed over a span of 20 km of four-mode fiber. The S band employed dual-polarization quadrature phase shift keying (DP-QPSK) modulation, whereas DP 16-quadrature amplitude modulation (DP-16QAM) was utilized in the C and L bands, both operating at a symbol rate of 32 Gbaud. All three systems employed 8×8 MIMO-DSP configuration at the receiver. At the same optical signal-to-noise ratio (OSNR), measured BER values for all mode channels were similar with low wavelength dependence, demonstrating the system’s ability to transmit high-quality broadband signals within a given OSNR range (see Supplementary Fig. S8). Figure 4d illustrates the BER scatters of 80-wavelength 4-mode multiplexing transmission in the C band (1530.33−1561.43 nm) and 4-mode multiplexing transmission in the S band (1460−1510 nm) and L band (1570−1608 nm). It can be seen that the BER values of all measured channels were below the FEC threshold of 2.4E-2, affirming that the proposed ultra-wideband mode multiplexer effectively supports mode multiplexing transmission in the S + C + L bands. Deeper MIMO equalizer enhancements (e.g., increased taps or advanced nonlinear compensation) demonstrate effective BER reduction (see Supplementary Fig. S9), enabling our system to achieve longer-distance transmission potential while using the current multiplexer. Since the device was designed to excite only the LP21a mode, it inherently implemented selective demultiplexing of LP21a at the receiver for coherent detection. When the multiplexer is specifically designed to excite both LP21a and LP21b modes simultaneously, the system enables joint detection of degenerate LP21 modes at the receiver without additional PC for suppressing coupling (see Supplementary Fig. S10).

Discussion

In conclusion, we propose an ultra-wideband mode multiplexing transmission system covering the O + E + S + C + L + U band. Employing an end-to-end inverse design approach and utilizing a multi-spectrum light diffraction mode multiplexer based on MPLC, we achieved 4-LP mode multiplexing with a bandwidth surpassing 420 nm while maintaining over 90% mode purity across six phase planes. Optical experimental characterization within the entire O + E + S + C + L + U bands confirmed that the fabricated multiplexer is compatible with commercially available six-mode and four-mode FMF standards, exhibiting extremely low wavelength-dependent fiber-to-fiber losses of less than 4.9 dB.

In our experiments, communication transmissions were carried out in four distinct trials, encompassing the O + E + U and S + C + L bands. Nonetheless, these trials aptly represent prevalent fiber communication systems commonly utilized today. The O + E band, characterized by shorter wavelengths, finds primary application in CWDM systems characterized by wide wavelength spacings and direct detection. Conversely, DWDM systems featuring narrower wavelength spacings are deployed in the S + C + L bands, leveraging coherent detection. For the U band, due to limitations of the associated components, it is primarily used in direct detection scenarios. Theoretical and experimental findings indicate that applicability of our system extends to other bands as well.

Compared to coherent detection, direct detection systems require decreased mode coupling in both the mode multiplexer and the transmission fiber to minimize mode crosstalk. Our communication transmission experiments have underscored the impact of FMF attributes on the selection of detection system. This delineates why four-mode fiber was not employed in the CWDM-MDM direct detection system discussed herein. Rather, the selected six-mode fiber provides increased versatility with improved mode coupling performance.

Ultra-wideband SDM transmission systems, amalgamating ultra-wideband WDM and ultra-wideband MDM with higher signal modulation formats, hold promise for substantial enhancements in channel capacity. The end-to-end inverse design strategy for multi-spectrum light diffraction mode multiplexer based on MPLC presents a foundational framework applicable across various scenarios. Expanding the bandwidth of the mode multiplexer to cover the entire communication band in our work results in a slight increase in the number of phase planes, but this does not significantly affect computational complexity. For future advancements, additional mode channels can be integrated within a fixed bandwidth. Theoretically, the impact of the number of modes and wavelengths associated with bandwidth on channel capacity is multiplicative, constrained by light diffraction, and the limit of modes and bandwidth merits further investigation. For a fixed bandwidth, compromising mode quality can accommodate more modes. Increasing the number of diffractive layers enhances mode quality by introducing additional structures for phase modulation. While this increase also raises reflection losses, the improved mode quality is beneficial for reducing fiber coupling loss and ensuring stable signal transmission. Channel limits for number of modes and bandwidth can be determined based on specific requirements for mode quality and the number of diffractive layers (see Supplementary Fig. S11). It should be noted that the design strategy yields the locally optimal modes- and bandwidth-dependent limitation, with potential for different capacity limits under varied conditions. Thus, a judicious trade-off between the number of modes, bandwidth, and diffraction layers must be carefully considered depending on the specific application requirements.

Furthermore, as an extension approach, the difficulties of ultra-wideband WDM transmission system are challenging to introduce new optical devices, such as modulators, transceivers and amplifiers operated in these bands. Some wideband devices have been investigated, such as the Raman amplifier covering the E, S, C, and L bands52 and the bismuth-doped fiber amplifier for O, E, S bands5,6. Therefore, the ultra-wideband SDM transmission scenarios deserve continued exploration and long-term research.

Methods

Optical propagation model of the ultra-wideband diffractive network

To model the diffractive network, its diffractive layers are treated as thin planar elements. For the i-th diffractive feature of the k-th layer at the spatial coordinates (xi, yi, zk), the wavelength-dependent complex transmission coefficient \(t^{k}_{\lambda^{c}} < /Emphasis > < /Subscript > \)(xi, yi, zk) at the c-th mode channel is represented as a function of the wide-spectrum modulation phase function \(\psi^{k}_{\lambda^{c}} < /Emphasis > < /Subscript > \)(λ) at the spatial coordinates (xi, yi, zk). The relationship can be mathematically expressed as:

$${t}_{{\lambda }^{c}}^{k}({x}_{i},{y}_{i},{z}_{k})={a}^{k}({x}_{i},{y}_{i},{z}_{k},\lambda )\exp (j{\psi }_{{\lambda }^{c}}^{k}({x}_{i},{y}_{i},{z}_{k}))$$
(8)

here, a and ψ represent the amplitude and phase coefficients, respectively. The diffractive layers are connected by free-space propagation, which is modeled through the band-limited angular approach. Consequently, the transfer function for free-space propagation over a distance d between two successive layers is given by:

$${H}^{k}({f}_{x},{f}_{y},d,\lambda )=\left\{\begin{array}{cc}\begin{array}{c}\exp \left(\frac{j2\pi d}{\lambda }\sqrt{1-{\lambda }^{2}{{f}_{x}}^{2}-{\lambda }^{2}{{f}_{y}}^{2}}\right),\\ 0,\end{array} & \begin{array}{c}{{f}_{x}}^{2}+{{f}_{y}}^{2} < \frac{1}{{\lambda }^{2}}\\ {{f}_{x}}^{2}+{{f}_{y}}^{2}\ge \frac{1}{{\lambda }^{2}}\end{array}\end{array}\right\}$$
(9)

where fx and fy represent the spatial frequencies along the x and y directions, respectively.

Assuming \(u^{k}_{\lambda^{c}} < /Emphasis > < /Subscript > \)(xi, yi, zk) represents the input complex light field of wavelength λ at the c-th mode channel on the k-th optical layer. The input complex light field of the k-th optical layer is obtained through the convolution operation between the scattered field generated by the (k-1)-th optical layer and the diffraction kernel. For the k-th layer, the forward optical field \(u^{k}_{\lambda^{c}} < /Emphasis > < /Subscript > \) (xi, yi, zk) can be expressed as:

$$\begin{array}{l}{U}_{{\lambda }^{c}}^{k-1}({f}_{x},{f}_{y},d)={\mathcal F} \left\{{u}_{{\lambda }^{c}}^{k-1}({x}_{i},{y}_{i},{z}_{k-1})\cdot \exp \left[j\cdot {\mbox{Arg}}({t}_{{\lambda }^{c}}^{k-1}({x}_{i},{y}_{i},{z}_{k-1}))\cdot \frac{{\lambda }_{r}}{\lambda }\right]\right\},\\ {u}_{{\lambda }^{c}}^{k}({x}_{i},{y}_{i},{z}_{k})={ {\mathcal F} }^{-1}\left\{{U}_{i}^{k-1}({f}_{x},{f}_{y},d)\cdot {H}_{{\lambda }^{c}}^{k-1}({f}_{x},{f}_{y},d)\right\}\end{array}$$
(10)

where ℱ{·} and ℱ⁻¹{·} denote the Fourier transform and inverse Fourier transform operations, respectively. λr represents the reference wavelength.

It is assumed that \(g^{k}_{\lambda^{c}} < /Emphasis > < /Subscript > \) (xi, yi, zk) represents the ouput complex light field of a wavelength of λ at the c-th mode channel on the k-th optical layer. The ouput complex light field of the k-th optical layer is obtained through the convolution between the scattered field generated by the (k + 1)-th optical layer and the diffraction kernel. The backward optical field \(g^{k}_{\lambda^{c}} < /Emphasis > < /Subscript > \) (xi, yi, zk) of the k-th layer can be written as follows:

$${G}_{{\lambda }^{c}}^{k+1}({f}_{x},{f}_{y},d) ={\mathcal F} \left\{{u}_{{\lambda }^{c}}^{k+1}({x}_{i},{y}_{i},{z}_{k+1})\cdot \exp \left[-j\cdot {{\mbox{Arg}}}({t}_{{\lambda }^{c}}^{k+1}({x}_{i},{y}_{i},{z}_{k+1}))\cdot \frac{{\lambda }_{r}}{\lambda }\right]\right\},\\ {g}_{{\lambda }^{c}}^{k}({x}_{i},{y}_{i},{z}_{k}) ={ {\mathcal F} }^{-1}\left\{{G}_{{\lambda }^{c}}^{k+1}({f}_{x},{f}_{y},d)\cdot {\left[{H}_{{\lambda }^{c}}^{k+1}({f}_{x},{f}_{y},d)\right]}^{\ast }\right\}.$$
(11)

The correlation function is described as follows:

$${\rho }_{{\lambda }^{c}}=\frac{{\sum }_{x}{\sum }_{y}{\big|{O}_{{\lambda }^{c}}\cdot {T}_{{\lambda }^{c}}^{\ast }\big|}^{2}}{\left({\sum }_{x}{\sum }_{y}\big|{O}_{{\lambda }^{c}}\cdot {O}_{{\lambda }^{c}}^{\ast }\big|\right)\times \left({\sum }_{x}{\sum }_{y}\big|{T}_{{\lambda }^{c}}\cdot {T}_{{\lambda }^{c}}^{\ast }\big|\right)}$$
(12)

The function is defined as mode purity, which is used to evaluate modulation efficiency. Here, \(O_{\lambda^{c}} < /Emphasis > < /Subscript > \) and \({T}_{{\lambda }^{c}}\) denote the generated and target modes light fields of wavelength λ at the c-th mode channel on the output plane, respectively.

Experimental setup for the communication transmission demonstration

In the implementation of the CWDM-MDM transmission, 18 optical transceivers are all driven by an optical switch (HUAWEI CE6865-48S8CQ-EI). 10 Gbit/s fundamental mode light signals using OOK at wavelengths of 1271, 1291, 1311, 1331, 1351, and 1371 nm were generated independently from the transmit port of different optical transceivers and coupled into six-wavelength CWDM broadband signals after passing through a commercial CWDM multiplexing module. The three channels of fundamental mode broadband signals were then modulated by the wideband mode multiplexer to generate LP01, LP11b and LP21 three spatial modes channels of broadband signals and transmitted over a 5 km six-mode fiber. At the receiving side, the broadband LP modes signals were split into three channels of broadband fundamental mode signal lights by a wideband mode demultiplexer. After passing through three CWDM demultiplexing modules, three channels of broadband signals channels were separated into 18-channel single wavelength signals and detected by the receive port of the corresponding transceiver.

In the broadband MDM transmission experiment in the U-band, an IMDD-based experimental setup was used. At the transmitter, the external cavity laser (ECL) was utilized to generate five optical carriers in the range of 1630−1670 nm with a 10 nm wavelength spacing. The OOK signals modulated by the DSP were loaded onto the arbitrary waveform generator (AWG) to generate the electrical waveform, which was amplified by the electrical amplifier (EA) to drive a Mach-Zehnder modulator (MZM) biased at the quadrature point to implement intensity modulation. Then, the signals were divided into three channels by a 1 × 4 optical coupler (OC) to be incident on the input ports of a mode multiplexer. The three channels of fundamental mode signals were further modulated by the wideband mode multiplexer to generate LP01, LP11, and LP21 spatial modes of broadband signals, which were transmitted over a 5 km four-mode fiber. At the receiver, followed by the fiber was a PC to adjust the coupling of modes in fiber transmission. Then, the LP modes signals were split into three channels by the wideband mode demultiplexer. Photoelectric signal conversion detection was performed by a photodiode (PD) and the waveform was displayed on the mixed signal oscilloscope (MSO) (see Supplementary Fig. S12). Given the use of a single polarization state per spatial mode and a total of three spatial channels, a 3×3 MIMO equalization structure was implemented.

In the implementation of C band DWDM-MDM transmission, ECLs were utilized to generate 80 optical carriers with a 50 GHz frequency spacing at the transmitter. These carriers spanned the wavelength range of 1530.33−1561.43 nm, following the international telecommunication union (ITU) standard. The 80 channels were divided into even and odd channels and underwent modulation in two IQ-modulators after passing through polarization-maintaining array waveguide gratings (PM-AWGs). The modulators were driven by a 4-channel arbitrary waveform generator (AWG) operating at a 64 GSa/s sampling rate. The modulation format of the generated optical signals was DP-16QAM with a symbol rate of 32 Gbaud. Post electro-optical modulation, the two sets of 40-channel optical carriers were combined into 80-channel DWDM signals via a 1 × 2 optical coupler. Subsequently, the DWDM signals traversed an analog polarization multiplexer (PM) containing a 1 m length of polarization-maintaining (PM) fiber serving as a delay line for decorrelation between the two polarization signals. They were then amplified in an erbium-doped fiber amplifier (EDFA) to equalize the power spectrum and divided into four channels by a 1 × 4 optical coupler to be incident on the input ports of a mode multiplexer. The four channels of fundamental mode signals were further modulated by the wideband mode multiplexer to generate LP01, LP11a, LP11b, and LP21 four spatial modes of broadband signals, which were transmitted over a 20 km four-mode fiber. SMF delay lines, with lengths of 10, 6, and 3 m, were introduced to three of the input optical signal channels of the mode multiplexer to facilitate decorrelation between signals of different modes. Meanwhile, a variable optical attenuator (VOA) was placed after each branch of the 1×4 optical coupler to equalize the optical power of the four modes. At the receiver, the broadband LP modes signals were split into four channels by the wideband mode demultiplexer. A time-division multiplexing (TDM) receiver was employed, receiving a single channel of four-mode broadband signals output from the mode demultiplexer for each time slot. The signals channel with single wavelength and single mode were separated after passing through a wavelength selector switch (WSS). Finally, the signals were received and demodulated by an integrated coherent receiver (ICR), and the waveform was displayed on an oscilloscope (OSC). Given the total eight spatial channels of four modes and two polarization states, an 8×8 MIMO-DSP configuration was employed (see Supplementary Fig. S13).

In the implementation of broadband MDM transmission in the S band, ECL generated 26 optical carriers with varying wavelengths intended for transmission into the IQ modulator. These carriers spanned the wavelength range of 1460−1510 nm with a wavelength spacing of 2 nm. Unlike the facilities used in C band transmission, a thulium-doped fiber amplifier (TDFA) is adopted to amplify the signals, which can be better adapted to the S band transmission. The sample rate of AWG is set at 64 GSa/s. The modulation format of the generated optical signals was DP-QPSK with a symbol rate of 32 Gbaud (see Supplementary Fig. S14).

Simultaneously, in the implementation of broadband MDM transmission in the L band, the ECL generated 20 optical carriers with varying wavelengths intended for transmission into the IQ modulator. These carriers spanned the wavelength range of 1570 − 1608 nm with a wavelength spacing of 2 nm. Parameters, such as the AWG sample rate, modulation format, and symbol rate remained consistent with those of the C band system.