Robust high-capacity free-space optical communication using OAM-based structured light and intelligent adaptive signal processing

Abstract

Free-space optical (FSO) communication enables high-speed, secure, and bandwidth-efficient data transmission for terrestrial and inter-satellite networks, outperforming traditional radio frequency (RF) systems in interference immunity and directional security. Atmospheric turbulence (AT), which causes beam distortion, intensity fading, and intermodal interference, remains a significant limitation for long-distance links. Existing approaches, such as Gaussian beam transmission, static equalization, and basic convolutional models, fail to provide real-time, adaptive resilience to these challenges. To overcome these limitations, this study proposes a novel hybrid FSO framework combining resilient structured light beams (Bessel, Airy, and orbital angular momentum (OAM) modes), adaptive optics (AO), and intelligent signal processing. A Dynamic Neural Fuzzy Inference System (DNFIS) provides robust equalization, and a Deep Convolutional Neural Network with Time-domain Correlation Sequence Generation (DCNN-TCSGm) predicts and compensates for turbulence effects in real time. Furthermore, the framework models compact optical metasurface-based OAM multiplexing combined with Wavelength Division Multiplexing (WDM) in the mid-infrared range to enhance spectral and spatial throughput. Simulation results demonstrate a 55% reduction in Bit Error Rate (BER), a 22% improvement in signal voltage stability, and up to 10 dB power gain over conventional Mode Division Multiplexing (MDM)-FSO and Decision Feedback Equalizer (DFE) systems, highlighting the proposed framework’s robustness and scalability under challenging atmospheric conditions.

Introduction

The rapid growth of multimedia applications has intensified the demand for high-speed wireless networks¹. Free-space optical (FSO) communication has emerged as a promising solution for high-speed terrestrial links, inter-satellite communications, and last-mile access, offering large unlicensed bandwidth, high data rates, and enhanced security compared to traditional radio frequency (RF) systems^2,3,4. Unlike optical fiber, FSO transmits data via modulated laser beams through the atmosphere or vacuum⁵, allowing for rapid and cost-effective deployment in areas where fiber installation is impractical or expensive⁶. However, atmospheric turbulence (AT) poses significant challenges to FSO performance. Fluctuations in the refractive index due to temperature and pressure variations cause beam wander, intensity scintillation, and phase distortions, which degrade signal quality^7,8,9. These impairments increase bit error rates (BER) and reduce link availability, particularly under adverse weather conditions such as fog, rain, and haze^10,11. Overcoming these challenges is critical for advancing FSO technology across academic, industrial, and commercial domains¹². Conventional FSO systems typically employ Gaussian beams, which are straightforward to generate but highly susceptible to atmospheric distortions, resulting in signal degradation^13,14. To address these limitations, structured light beams, especially those carrying Orbital Angular Momentum (OAM), have garnered increasing interest. OAM beams exhibit a helical phase front that enhances resilience against turbulence^15,16,17,18. This work focuses on OAM, Bessel, and Airy beams, which possess properties such as non-diffraction, self-healing, and self-acceleration, helping maintain signal integrity over longer distances and through obstructions. Further performance enhancements can be achieved through adaptive optics (AO) for real-time wavefront correction and intelligent optimization techniques such as Particle Swarm Optimization (PSO). Advanced signal processing methods, including Decision Feedback Equalizers (DFE) and Deep Convolutional Neural Networks (DCNN), are also essential for reliable signal recovery in turbulent conditions. Moreover, integrating Wavelength Division Multiplexing (WDM) with OAM and Mode Division Multiplexing (MDM) can significantly increase data throughput, making these technologies vital for future terrestrial and inter-satellite FSO communication systems.

Literature survey

This section reviews structured light and OAM-based multiplexing and demultiplexing techniques for enhancing FSO communication. Authors in¹⁹ examine OAM waveform generation methods, comparing them with communication requirements and performance metrics. They highlight OAM’s potential for energy-efficient transmission and low error rates, including indirect fiber-to-atmosphere transmission suited for short-range wireless links. The study emphasizes optical communication’s role in meeting broadband demands and its application in secure quantum key distribution (QKD). In²⁰, a simple and green detection method for OAM states is proposed using optical differentiation with weak measurements. Experiments on Laguerre-Gaussian beams $(l=5, l=8)$ show reduced sensitivity to disturbances, enabling direct detection with a simplified setup. While promising for FSO applications, real-time detection of multiple vortex modes remains a challenge.

Authors in²¹ introduce an optimized convolutional neural network (CNN) with transfer learning for recognizing OAM modes in distorted beams. Results show higher accuracy with reduced training time compared to conventional CNNs. The study also reports that turbulence and longer propagation distances gradually reduce recognition accuracy. In²², a novel modulation method is presented that exploits the phase differences between superposed OAM modes. Small phase shifts alter interference patterns, creating additional parameters for encoding information. This approach exponentially increases encoding capacity, and a neural network-based decoder enables accurate recovery of transmitted data from light intensity patterns.

A vortex modulation technique enhances minor variations among closely spaced OAM states, enabling CNNs to distinguish features more accurately and improve topological charge identification²³. Simulations demonstrate high recognition accuracy under strong AT and long transmission distances, with future work exploring alternative modulation schemes and neural network architectures for high-resolution fractional OAM modes. Building on this, co-scale reception with convergent beams²⁴ ensures OAM modes transmit with equal divergence, while a ring-shaped Airy compensation focuses energy onto a compact receiving region. Experiments demonstrate tunable OAM modes with improved received power and minimal crosstalk, while simulations indicate that lower-order OAM modes fluctuate less under weak time-varying AT (TV-AT), whereas higher-order modes perform better under stronger TV-AT. This highlights the need for accurate TV propagation models and phase-correction strategies. To mitigate turbulence-induced degradation in real-time systems, predictive diversity combining (PDC) with optimal dynamic channel processing (ODCP) maintains BER²⁵.

Table 1 Literature summary.

Complementary diversity gains techniques optimize capacity while reducing turbulence-induced fading by assigning channel weights and applying digital synchronization, with optical fiber delay lines enabling effective signal combination²⁶. These methods are validated in Multiple-Input Multiple-Output (MIMO) DWDM-FSO links using diversity coding, where an 8-channel, 2.5 Gbps per channel, 1500 m system performs efficiently under turbulent conditions²⁷. Multi-OAM-mode transmission from a main transmitter to distributed receivers employs planar arrays and precoding to deliver multiple channels simultaneously, though scalability limits the number of achievable OAM modes. High-speed communication is further enhanced by combining optical code division multiple access (OCDMA) with OAM modes²⁸. Two LG beams transmit three 10 Gbps channels each, with spatial diversity and beamforming reducing BER under turbulence, although dense fog and dust remain limiting factors²⁹. Real-time spatiotemporal acoustic communication using a single sensor and the rotational Doppler effect demonstrates efficient multiplexing of multiple OAM channels³⁰, illustrating the versatility of OAM-based systems. Terrestrial OAM-FSO links achieve up to 40 Gb/s using four OAM beams. Comparisons of Alternate Mark Inversion (AMI), Return to Zero (RZ), and Non-Return to Zero (NRZ) encodings indicate NRZ is optimal over 64–800 m³¹. Forward-backward dynamic mode decomposition (FBDMD) mitigates white noise interference, accurately recovering OAM topological charges and reducing crosstalk³², although real-time, high-data-rate applications requires significant computational resources. For UAV-to-ground communications, a 4-level quadrature amplitude modulation (4-QAM)-OFDM-FSO architecture³³ and a comprehensive pointing error model³⁴ demonstrate trade-offs between mode number and modulation order. Simulations show that moderate laser pulse powers preserve phase singularities under cubic nonlinearity, mitigating divergence, while higher powers accelerate transverse spreading³⁵. Partially coherent Airy beam analysis under jet exhaust turbulence shows that beam quality improves with smaller structure constants and outer scales³⁶.

The authors in³⁷ proposed coherent laser arrays with discrete vortices (CLA-DV) to reduce crosstalk in OAM transmission under turbulence, improving stability, though signal quality still degraded at longer distances. In³⁸, the authors introduced DFE with minimum mean square error (MMSE) optimization across Hermite–Gaussian channels, enhancing BER performance, but effectiveness decreased under strong turbulence. The authors in³⁹ developed the deep-learning model cGULnet to extract phase information from multiplexed Laguerre–Gaussian modes, lowering BER, while high computational cost limited real-time use. In⁴⁰, the authors demonstrated an 80 Gbps inter-satellite MDM-FSO link over 35,000 km, with potential gains using wavelength division multiplexing (WDM), though system complexity remained a challenge. The authors in⁴¹ presented single-layer dielectric metasurfaces for compact multi-dimensional demultiplexing of wavelength, spin angular momentum (SAM), and OAM across 132 channels, offering scalable integration. However, the integration issues and performance degradation under real atmospheric conditions still limit practical implementation. Recent advancements continue to explore hybrid techniques for turbulence mitigation. For instance, Elsayed⁴² integrated OAM with spatial modulation and L-ary PPM in a DWDM-MIMO FSO system to enhance throughput. Similarly, Elsayed⁴³ employed multi-hop MIMO with spatial modulation and M-ary PPM to improve spectral efficiency and combat turbulence-induced BER, highlighting the trend towards complex, co-optimized physical-layer designs. Table 1 lists the summary of the literature.

Motivation and contributions

FSO communication faces significant challenges from air turbulence, which severely degrades signal quality and image transmission, especially over long distances. Increasing air turbulence and transmission distance result in higher error rates and reduced data quality. Current equalization methods are insufficient to combat diverse AT in long-distance MDM-FSO links. Existing deep learning approaches for turbulence mitigation also lack optimal speed and accuracy. OAM beam transmission using CLA-DV³⁷ reduces crosstalk under severe turbulence, but image quality still degrades with increasing turbulence and distance. In MDM-FSO systems, the MMSE across Hermite-Gaussian channels³⁸ optimized by DFE is insufficient for diverse long-haul conditions highlighting the need for more advanced schemes. Deep learning models like cGULnet³⁹ have shown promise in accurately extracting phase information from spatially multiplexed LG modes, but improvements are still required in speed and performance. High-data-rate inter-satellite FSO links using MDM have achieved high speeds over long distances⁴⁰, though system complexity and integration remain bottlenecks. Additionally, single-layer dielectric metasurfaces allow compact multi-dimensional demultiplexing of wavelength, SAM, and OAM across 132 channels⁴¹, yet the practical integration of nanophotonic components remains a significant hurdle.

To address the aforementioned issues, this paper proposes a high-capacity and resilient optical communication system that integrates self-healing structured light beams with advanced adaptive equalization and deep learning techniques to enhance signal integrity over long distances. This paper aims to significantly improve the robustness, data rate, and adaptability of FSO links for both terrestrial and inter-satellite applications. The key objectives include mitigating nonlinear atmospheric distortions, enabling real-time turbulence prediction and compensation, and developing highly integrated multiplexing solutions. The main contributions of this paper are given below:

• A novel, intelligent FSO framework that synergistically integrates adaptive optics (AO) with particle swarm optimization (PSO) for dynamic wavefront correction, and structured OAM beams for inherent turbulence resilience.

• A closed-loop intelligent signal processing core that uniquely co-integrates a Deep Convolutional Neural Network (DCNN-TCSGm) for predictive turbulence channel estimation with a Dynamic Neural-Fuzzy Inference System (DNFIS) for adaptive, nonlinear equalization. This core enables real-time, proactive compensation.

• The unified integration of this intelligent core with compact, dielectric metasurface-based OAM multiplexers/demultiplexers in the mid-infrared spectrum, proposing a pathway to highly integrated transceivers.

• A hybrid WDM-MDM (wavelength and mode division multiplexing) scheme for the mid-infrared spectrum, significantly enhancing the spectral-spatial capacity for inter-satellite links.

The integration of these components provides a novel, intelligent, and turbulence-resilient optical transmission system that advances the state-of-the-art in channel robustness, data capacity, and system adaptability.

Paper organization

The remainder of this paper is structured as follows. Section 2 describes the system model. Section 3 details the proposed methodology. Section 4 presents and discusses the simulation results and comparative analysis. Finally, Section 5 provides the conclusion.

System model

The base of our OAM-based FSO communication framework relies on a formalized mathematical systems model that captures the physics of structured beam propagation through turbulence and signal degradation due to atmospheric impairments. This model also serves as a baseline for incorporating smart signal processing, equalization, and adaptive corrections discussed later in the methodology. We begin by modeling the structured light beam produced by the transmitter, which carries information encoded onto separate OAM modes. An optical field carrying an OAM mode with topological charge $({{E}_{l}})$ can be mathematically described by Eq. (1)

$$\begin{aligned} {{E}_{l}}(r,\phi ,z)=A(r,z)\cdot {{e}^{il\phi }} \end{aligned}$$

(1)

In this Equation, r is the radial distance from the beam’s center, A(r, z) denotes the radial amplitude distribution of the beam, typically defined by its type (e.g., Gaussian, Bessel, Airy). The azimuthal angle $\phi$ and propagation distance z determine the phase along the vortex beam. The azimuthally varying exponential term ${{e}^{il\phi }}$ represents the helically structured wavefront, producing the central “doughnut” intensity profile characteristic of OAM beams. In a multiplexed system, multiple OAM modes can be combined to carry parallel data streams. The resulting optical field at the transmitter, ${{E}_{tx}}$, is expressed as in Eq. (2).

$$\begin{aligned} {{E}_{tx}}(r,\phi ,z)=\sum \limits _{l=-L}^{L}{{{s}_{l}}}(t)\cdot {{E}_{l}}(r,\phi ,z) \end{aligned}$$

(2)

Where ${{s}_{l}}(t)$ is the symbol transmitted on the mode l, and $2L+1$ represents the total number of OAM modes used. This expression captures the concept of MDM, exploiting the orthogonality of OAM modes to transmit multiple independent data streams over a single optical channel. During propagation, these structured beams experience phase distortions and intensity fluctuations due to spatial and temporal variations in the atmospheric refractive index. The accumulated phase perturbation $\Phi (r,\phi ,z)$ over the propagation distance is modeled by the integral given in Eq. (3).

$$\begin{aligned} \Phi (r,\phi ,z)=\frac{2\pi }{\lambda }\int _{0}^{z}{\delta }n(r,\phi ,{z}')d{z}' \end{aligned}$$

(3)

In this expression, $\delta n(r,\phi ,{z}')$ represents the stochastic fluctuations in the refractive index due to turbulence, and $\lambda$ is the wavelength of the optical beam. This integral accounts for the total accumulated phase distortion along the propagation distance, leading to wavefront incoherence, which manifests as beam wandering, scintillation, and ultimately, mode crosstalk. To quantify turbulence severity, the Rytov variance $\sigma _{R}^{2}$ is introduced, defined in Eq. (4).

$$\begin{aligned} \sigma _{R}^{2}=1.23C_{n}^{2}{{k}^{7/6}}{{z}^{11/6}} \end{aligned}$$

(4)

Here, $C_{n}^{2}$ denotes the atmospheric structure constant characterizing turbulence strength, $k={2\pi }/{\lambda }\;$ is the optical wave number, and z the link distance. The Rytov variance directly reflects intensity scintillation, with values above one indicating strong turbulence, and signal loss could be expected. Atmospheric turbulence strength is categorized using the Rytov variance $\sigma _{R}^{2}$ (Eq. 4): weak turbulence $(\sigma _{R}^{2}<0.3)$, moderate turbulence $(0.3\le \sigma _{R}^{2}\le 5)$, and strong turbulence $(\sigma _{R}^{2}>5)$. The Fried parameter, ${{r}_{0}}={{(0.423{{k}^{2}}C_{n}^{2}z)}^{-3/5}}$, quantifies the transverse coherence length of atmospheric phase distortions. For terrestrial FSO links in the mid-infrared regime $(\lambda \sim 3\text {--}5\ \mu \text {m)}$ over distances of 1–5 km, typical ${{r}_{0}}$ values range from approximately 2 cm under strong turbulence to 10–20 cm under weak turbulence. The distortion severity scales as ${{\left( {D}/{{{r}_{0}}}\; \right) }^{{5}/{3}\;}}$, guiding AO requirements. Intensity fluctuations are characterized by the scintillation index, which for a plane wave is expressed as in Eq. (5).

$$\begin{aligned} \sigma _{I}^{2}=\exp \left( \frac{0.49\sigma _{R}^{2}}{{{(1+1.11\sigma _{R}^{12/5})}^{7/6}}}+\frac{0.51\sigma _{R}^{2}}{{{(1+0.69\sigma _{R}^{12/5})}^{5/6}}} \right) -1 \end{aligned}$$

(5)

The turbulence spectrum includes the inner scale ${{l}_{0}}$ $(\sim 1\ \text {m} \text { to } 100\ \text {m})$ and outer scale ${{L}_{0}}$ $(\sim 1\ \text {m} \text { to } 100\ \text {m})$ effects via the modified Von Kármán spectrum. Weather-induced attenuation follows established models: fog/haze (Kim model), rain $(\alpha =a{{R}^{b}})$, and snowfall, which are incorporated into the total path loss ${{\alpha }_{\text {total}}}$ in Eq. (23).

At the receiver, the structured beam is further degraded by mode-dependent fading, and the received signal r(t), as shown in Eq. (6).

$$\begin{aligned} r(t)=\sum \limits _{l=-L}^{L}{(}{{h}_{l}}(t){{s}_{l}}(t)+n(t)) \end{aligned}$$

(6)

Where ${{h}_{l}}(t)={{\alpha }_{l}}(t){{e}^{i{{\theta }_{l}}(t)}}$ is the complex fading coefficient for the mode l, incorporating both amplitude attenuation and phase shift, and n(t) is Additive White Gaussian Noise (AWGN). This model represents the TV effects of atmospheric conditions and serves as the input for equalization and error correction. Turbulence-induced aberrations diminish OAM mode orthogonality, leading to inter-modal interference; thus, the system response is best expressed in channel matrix form, as defined by Eq. (7)

$$\begin{aligned} {{H}_{m,n}}=\langle E_{m}^{\text {recv}}\mid E_{n}^{\text {trans}}\rangle \end{aligned}$$

(7)

Here $H\in {{\mathbb {C}}^{M\times M}}$ is the MDM channel matrix, with each element ${{H}_{m,n}}$ representing the crosstalk between the transmitted mode n and received mode m. The coupling coefficient is defined by the inner product projection.

In practical FSO systems, the received signal is impaired by the combined effects of path loss, AT-induced fading, and pointing errors. The composite channel coefficient for a single mode is modeled as the product:

$$\begin{aligned} h={{h}_{l}}\cdot {{h}_{a}}\cdot {{h}_{p}}, \end{aligned}$$

(8)

Where ${{h}_{l}}$ is the deterministic path loss, ${{h}_{a}}$ is the atmospheric turbulence fading coefficient, and ${{h}_{p}}$ is the pointing error loss factor.

Pointing Error Model: Misalignment due to platform vibration and tracking jitter is modeled by a Rayleigh-distributed radial displacement r:

$$\begin{aligned} {{f}_{r}}(r)=\frac{r}{\sigma _{s}^{2}}\exp \left( -\frac{{{r}^{2}}}{2\sigma _{s}^{2}} \right) ,\quad r\ge 0, \end{aligned}$$

(9)

Where $\sigma _{s}^{2}$ is the variance of the underlying Gaussian jitter. For a circular aperture of radius (a) and a Gaussian beam with a waist ${{\omega }_{z}}$ at the receiver, the fraction of power collected (the pointing loss factor) is approximated by:

$$\begin{aligned} {{h}_{p}}\approx {{A}_{0}}\exp \left( -\frac{2{{r}^{2}}}{\omega _{z}^{2}} \right) ,\quad \text {with }{{A}_{0}}={{\left[ \text {erf}\left( \frac{\sqrt{\pi }a}{\sqrt{2}{{\omega }_{z}}} \right) \right] }^{2}}. \end{aligned}$$

(10)

Atmospheric Turbulence Model: For moderate-to-strong turbulence conditions, the intensity scintillation I is well-characterized by the Gamma-Gamma distribution:

$$\begin{aligned} {{f}_{I}}(I)=\frac{2{{(\alpha \beta )}^{(\alpha +\beta )/2}}}{\Gamma (\alpha )\Gamma (\beta )}{{I}^{(\alpha +\beta )/2-1}}{{K}_{\alpha -\beta }}\left( 2\sqrt{\alpha \beta I} \right) ,\quad I>0, \end{aligned}$$

(11)

Where $\alpha$ and $\beta$ are the effective numbers of large-scale and small-scale eddies, respectively, and ${{K}_{v}}$ is the modified Bessel function of the second kind.

To rigorously quantify turbulence-induced impairments, we define the inter-modal crosstalk coefficient ${{X}_{m,n}}$ as the normalized power coupled from the transmitted mode n to received mode m:

$$\begin{aligned} {{X}_{m,n}}(z,C_{n}^{2})=\frac{|{{H}_{m,n}}{{|}^{2}}}{\sum \limits _{k}{|}{{H}_{m,k}}{{|}^{2}}},\quad m\ne n. \end{aligned}$$

(12)

This coefficient increases with propagation distance z and the refractive index structure constant $C_{n}^{2}$. The OAM mode purity ${{P}_{l}}$ for a mode with topological charge l is the fraction of power remaining in the intended mode after propagation through turbulence:

$$\begin{aligned} {{P}_{l}}(z,C_{n}^{2})=\frac{|{{H}_{l,l}}{{|}^{2}}}{\sum \limits _{k}{|}{{H}_{l,k}}{{|}^{2}}}. \end{aligned}$$

(13)

The degradation of ${{P}_{l}}$ is influenced by the turbulence spectrum, including the effects of inner ${{l}_{0}}$ and outer ${{L}_{0}}$ scales, which differentially impair the phase coherence of higher-order OAM modes.

To mitigate these distortions, an AO correction phase is applied, given as.

$$\begin{aligned} {{E}_{\text {corr}}}(r,\phi ,z)={{E}_{l}}(r,\phi ,z){{e}^{-i{{\Phi }_{\text {AO}}}(r,\phi )}} \end{aligned}$$

(14)

In Eq. (14), ${{\Phi }_{\text {AO}}}(r,\phi )$ is the phase correction applied by the AO system. The correction is iteratively optimized using PSO to minimize wavefront error and maximize signal quality at the receiver. System performance is ultimately evaluated through the signal-to-noise ratio (SNR) and BER. The instantaneous SNR for the OAM mode l is expressed as in Eq. (15).

$$\begin{aligned} \text {SN}{{\text {R}}_{l}}=\frac{|{{h}_{l}}(t){{|}^{2}}\mathbb {E}[|{{s}_{l}}(t){{|}^{2}}]}{{{\sigma }^{2}}} \end{aligned}$$

(15)

Here, the numerator denotes the signal power collected in that mode, while ${{\sigma }^{2}}$ represents the noise variance. A higher SNR corresponds to more reliable data reception and lower error probability. For Binary Phase Shift Keying (BPSK) and Quadrature Phase Shift Keying (QPSK) modulation, the average BER per mode is expressed using the Q-function, as shown in Eq. (16).

$$\begin{aligned} \text {BE}{{\text {R}}_{l}}=Q(\sqrt{2\text {SN}{{\text {R}}_{l}}}) \end{aligned}$$

(16)

These expressions describe the tail probability of a Gaussian distribution exceeding a given value, providing an analytic means to evaluate link reliability and compare network performance. Despite the TV-AT effects, it is important to characterize the temporal dynamics of fading. The autocorrelation function (ACF) of the fading coefficient ${{h}_{l}}(t)$ is modeled as in Eq. (17).

$$\begin{aligned} {{R}_{l}}(\tau )=\mathbb {E}[{{h}_{l}}(t)h_{l}^{*}(t+\tau )] \end{aligned}$$

(17)

Given in Eq. (17), which captures how the channel condition for a particular OAM mode evolves in delay $\tau$. For wind-driven turbulence, the ACF follows a Gaussian decay model as described by Eq. (18),

$$\begin{aligned} {{R}_{l}}(\tau )=\exp (-\frac{{{(\pi W\tau )}^{2}}}{4{{D}^{2}}}) \end{aligned}$$

(18)

where W is the transverse wind speed and D is the aperture diameter. These time-domain characteristics are crucial for training deep learning models, such as the proposed DCNN-TCSGm, for real-time turbulence prediction and compensation.

Modeling framework distinction

Our framework delineates two distinct optical link scenarios with fundamentally different physical impairments.

Uplink (Ground-to-Satellite) Channel: Employs multi-phase screen AT modeling with Rytov variance $\sigma _{R}^{2}$, Fried parameter ${{r}_{0}}$, and weather-dependent attenuation. Pointing errors arise from ground platform vibration and atmospheric tip-tilt.

Inter Satellite Link Channel: Assume negligible atmospheric turbulence $C_{n}^{2}\approx 0$. The primary impairment is pointing error due to spacecraft vibration and jitter, modeled via statistical pointing error distributions. Background radiation (solar, cosmic) is included in the noise budget.

For scenarios with significant jitter, the radial pointing error is modeled as a Rayleigh-distributed variable, and the normalized pointing jitter is defined as ${{{\sigma }_{p}}}/{{{w}_{z}}}\;$, where ${{\sigma }_{p}}$ is the standard deviation of the jitter and ${{w}\_{z}}$ is the receiver beam waist.

While the core adaptive signal processing algorithms (DCNN TCSGm, DNFIS) remain applicable to both, the physical layer impairment models differ substantially, as reflected in our simulation parameters.

Proposed methodology

The proposed methodology enhances FSO performance by strategically combining the inherent resilience of structured light beams with advanced mitigation strategies. Its integrated approach leverages AO, intelligent optimization, and sophisticated signal processing to overcome AT. The overall architecture of this research is illustrated in Fig. 1. The following subsection outlines the step-by-step actions taken as part of this work.

Fig. 1

Overall architecture of the proposed FSO framework. The optical domain comprises structured beam generation, multiplexing, atmospheric propagation, and wavefront sensing. The DSP domain at the receiver performs signal recovery and turbulence mitigation using deep learning and adaptive optimization modules, including DCNN, DNFIS, and PSO.

Data collection

A robust training dataset for turbulence mitigation was generated by simulating structured OAM beam propagation under varying AT using a multi-phase screen model. Each screen represented Kolmogorov-spectrum turbulence, parameterized by Rytov variance $\sigma _{R}^{2}$, refractive index structure constant $C_{n}^{2}$, wind speed, and propagation distance. Simulations spanned 500m to 5km, covering short to long-distance terrestrial and inter-satellite dynamics were captured via frozen flow and Taylor hypothesis models, yielding frame-by-frame distortion sequences. In total, 5000 OAM beam sequences across 10 turbulence profiles and 5 beam types (Gaussian, Bessel, Airy, Vortex, hybrid) with up to 50 frames per sequence were formatted as tensors for DCNN-TCSGm training. MATLAB R2023a with custom wave optics modules was used. This dataset enables the model to learn both spatial features and temporal fading patterns of turbulence-distorted beams.

Turbulence limits OAM image quality

The Proposed methodology addresses FSO communication challenges by first tackling turbulence limits on OAM image quality. To counter image degradation and unreliability under strong AT, we implement a multi-pronged solution. This applies to resilient, self-repairing structured beams, such as Airy and Bessel beams, as well as optimized Vortex OAM beams, which perform well under turbulent conditions. Combined with AO for real-time wavefront correction and PSO for intelligent parameter optimization, the system adapts to changing atmospheric conditions and preserves data over long distances. The proposed approach integrates three structured light beams: Bessel, Airy, and Vortex OAM. Implementing these beams requires modifying a conventional Gaussian-beam terminal. The key change is the addition of a wavefront-shaping element, such as a spatial light modulator (SLM) or a spiral phase plate, that imprints a specific spatial phase pattern (e.g., a helical profile for OAM) onto the laser output. This increases terminal complexity, cost, and alignment sensitivity. For future deployment, particularly in size, weight, and power-constrained (SWaP) applications like satellite communications, dielectric metasurfaces (discussed in section 3.5c) offer a promising path toward compact, integrated transceivers that can generate and manipulate structured light efficiently. these beams are selected for their unique ability to adapt to varying conditions and maintain data didelity across extended transmission ranges.

• (a) Besel-Gauss Beams (BGBs)

Bessel-Gauss beams are non-diffracting, meaning they can propagate over a finite range while maintaining their transverse intensity profile without divergence. A key trait is their self-reconstruction property, allowing them to reform after encountering obstacles, thus maintaining continuous signal propagation. The electric field distribution of a Bessel beam in cylindrical coordinates $(r,\phi ,z)$ is generally expressed as a solution of the Helmholtz equation, as shown in Eq. (19).

$$\begin{aligned} E(r,\phi ,z)={{A}_{0}}{{J}_{l}}({{k}_{t}}r)\cdot \exp (i{{k}_{z}}z)\cdot \exp (il\phi ) \end{aligned}$$

(19)

In this expression, ${{A}_{0}}$ is the constant amplitude, ${{J}_{l}}$ is the Bessel function of the first kind of order l (the topological charge), k_t is the transverse wave vector, ${{k}_{z}}$ is the longitudinal wave vector, and r and $\phi$ are the radial and azimuthal coordinates, respectively.

• (b) Finite-Energy Airy Beams (FABs)

Airy beams are unique for their self-accelerating behavior, allowing them to follow curvilinear paths, and for their strong resistance to diffraction. They also possess self-healing properties, enabling reconstruction after distortions during propagation through turbulence. Under the paraxial approximation, where light rays are nearly parallel to the optical axis, their propagation is governed by a specific form of the diffraction equation.

$$\begin{aligned} i\frac{\partial \psi }{\partial \varsigma }+\frac{1}{2}\frac{{{\partial }^{2}}\psi }{\partial {{\sigma }^{2}}}=0 \end{aligned}$$

(20)

In Eq. (20), i denotes the imaginary unit, while $\psi$ represents the complex amplitude of the light field, capturing both phase and magnitude. The variable $\varsigma$ is the normalized propagation distance along the beam axis, and $\sigma$ is the normalized transverse spatial coordinate, which renders the equation dimensionless for easier analysis. The exact non-diffracting solution for an ideal Airy beam, given by Eq. (21).

$$\begin{aligned} \psi (\sigma ,\varsigma )={Ai}\left( \sigma -\frac{{{\varsigma }^{2}}}{4} \right) \exp \left( i\left( \frac{\sigma \varsigma }{2}-\frac{{{\varsigma }^{3}}}{12} \right) \right) \end{aligned}$$

(21)

Here, Ai denotes the Airy function, defining the characteristic beam shape. The term ${{{\varsigma }^{2}}}/{4}\;$ describes the parabolic trajectory, illustrating its self-accelerating property. Exponential terms capture phase evolution during propagation.

• (c) Vortex Beams (VBs)

Vortex beams carry OAM via a helical phase front, producing a distinctive doughnut-shaped intensity profile due to a central phase singularity. Their OAM property is crucial for multiplexing data channels, increasing capacity, and enhancing the security of FSO systems. The complex electric field of a Vortex beam can be represented in terms of its azimuthal dependence as:

$$\begin{aligned} {{E}_{{field}}}(\phi )={{A}_{0}}\exp (il\phi ) \end{aligned}$$

(22)

In Eq. (22), ${{E}_{{field}}}$ is the complex electric field of the beam, and ${{A}_{0}}$ is its constant amplitude. The term $\exp (il\phi )$ represents the helical phase variation, where l is the topological charge (an integer) defining the phase winding and the OAM carried per photon. The symbol $\phi$ denotes the azimuthal angle in the plane perpendicular to propagation.

The implementation also incorporates a multi-faceted security strategy, including beam generation, shaping, and adaptive beam selection and switching. Spatial Light Modulators (SLMs) serve as reconfigurable optical elements, converting Gaussian input beams into desired structured profiles while precisely controlling amplitude, phase, and polarization. A dynamic switching mechanism adapts the transmitted beam type based on current atmospheric conditions and pre-agreed communication protocols, enhancing data protection and mitigating electromagnetic interference and signal attenuation in FSO links. The total attenuation, ${{\alpha }_{total}}$, experienced by a structured beam is the sum of contributions from individual atmospheric conditions and can be estimated to assess weather-induced signal loss, as detailed in Eq. (23).

$$\begin{aligned} {{\alpha }_{{total}}}={{\alpha }_{{mist}}}+{{\alpha }_{{flurry}}}+{{\alpha }_{{downpower}}}+{{\alpha }_{{dispersion}}} \end{aligned}$$

(23)

The total attenuation, ${{\alpha }_{{total}}}$, represents the overall signal intensity reduction over a given distance. Individual contributions include ${{\alpha }_{{mist}}}$ (fog or haze), ${{\alpha }_{{flurry}}}$ (snowfall), ${{\alpha }_{{downpower}}}$ (rainfall), and ${{\alpha }_{{dispersion}}}$ (scattering by atmospheric particles), which redirects light away from its intended path. Recovering the original Gaussian beam from the incoming structured light is typically a computational process at the receiver, compensating for structural changes and atmospheric wavefront distortions. The retrieval process can be expressed in Eq. (24).

$$\begin{aligned} {{G}_{{output}}}=\mathcal {F}\{{{E}_{{incident}}}\}\otimes \mathcal {F}\{{{T}_{{SLM}}}\} \end{aligned}$$

(24)

Here, $G_{{output}}$ is the complex electric field of the reconstructed Gaussian beam at the output, and $\mathcal {F}$ denotes the Fourier transform from the spatial to the frequency domain. $E_{{incident}}$ is the complex amplitude of the received structured light beam, and $T_{{SLM}}$ is the complex transmittance of the SLM, specifying how it modifies amplitude and phase. The symbol $\otimes$ represents a convolution operation, combining two functions to quantify their overlap. This approach enhances security and applies to various structured beams. According to experimental data, Bessel and Vortex beams maintain integrity over $>3$ km, while Airy beams sustain integrity over $\sim 2$ km under similar conditions.

• (d) PSO for System Parameter Tuning

The PSO algorithm is employed to intelligently optimize key parameters of the FSO system, enhancing its resilience to atmospheric turbulence. The proposed PSO algorithm dynamically tunes parameters such as the beam waist $({{w}_{0}})$, divergence angle, and AO correction factors in real-time. The objective is to maximize the SNR and minimize the BER under turbulent conditions. The PSO algorithm was implemented with the following parameters: swarm size $N=50$, inertia weight $w=0.8$, and acceleration coefficients ${{c}_{1}}={{c}_{2}}=1.5$. These values were selected based on preliminary grid search trials across representative turbulence scenarios to balance exploration and exploitation. Particle velocities were clamped to $\pm 20\%$ of the parameter search range to prevent divergence. The optimization typically converged within 100 iterations, with termination triggered when the global best fitness improvement remained below ${{10}^{-4}}$ for 10 consecutive iterations. The fitness function (F) for the PSO is formulated to maximize a composite performance score based on received signal power and stability. For a set of system parameters $\textbf{X}=[{{w}_{0}},{{\theta }_{{div}}},{{\alpha }_{{AO}}},\ldots ]$, the fitness is evaluated as shown in Eq. (25)

$$\begin{aligned} F(\textbf{X})={{w}_{1}}\cdot {SN}{{{R}}_{{avg}}}(\textbf{X})+{{w}_{2}}\cdot (1-{BE}{{{R}}_{{avg}}}(\textbf{X}))-{{w}_{3}}\cdot \sigma _{I}^{2}(\textbf{X}) \end{aligned}$$

(25)

Where ${SN}{{{R}}_{{avg}}}(\textbf{X})$ is the average SNR ratio, ${BE}{{{R}}_{{avg}}}(\textbf{X})$ is the average bit error rate and $\sigma _{I}^{2}(\textbf{X})$ is the variance of the signal intensity, which PSO seeks to minimize. The coefficients ${{w}_{1}},{{w}_{2}},{{w}_{3}}$ are weighting coefficients that prioritize the different performance metrics.

A swarm of particles, each representing a candidate solution $\textbf{X}$ , explores the parameter space. The position ${{\textbf{X}}_{i}}$ and velocity ${{\textbf{V}}_{i}}$ of each particle i are updated iteratively based on their personal best position ${{\textbf{P}}_{{best},i}}$ and the global best potion found by the swarm ${{\textbf{G}}_{{best}}}$, given by Eq. (26).

$$\begin{aligned} \textbf{V}_{i}^{(k+1)}=w\textbf{V}_{i}^{(k)}+{{c}_{1}}{{r}_{1}}({{\textbf{P}}_{{best},i}}-\textbf{X}_{i}^{(k)})+{{c}_{2}}{{r}_{2}}({{\textbf{G}}_{{best}}}-\textbf{X}_{i}^{(k)}) \end{aligned}$$

(26)

$$\begin{aligned} \textbf{X}_{i}^{(k+1)}=\textbf{X}_{i}^{(k)}+\textbf{V}_{i}^{(k+1)} \end{aligned}$$

(27)

Here, w is the inertia weight, ${{c}_{1}}$ and ${{c}_{2}}$ are the acceleration coefficient, and ${{r}_{1}}$, ${{r}_{2}}$ are random values. By converging towards the parameter set ${{\textbf{G}}_{{best}}}$ the maximizes $F(\textbf{X})$, the PSO algorithm ensures the FSO system is optimally configured to maintain link integrity, compensating for dynamic atmospheric effects more effectively than static parameter settings.

Dynamic neural-fuzzy equalization for MDM-FSO channels

To combat the nonlinear and time-varying distortions in long-haul MDM-FSO links, we implement a DNFIS as an intelligent equalizer. Unlike static equalizers, the DNFIS adapts in real-time to the dynamics of the atmospheric channels. Its function is to estimate and cancel the inter-modal crosstalk and nonlinear phase distortion introduced by turbulence, thereby recovering the original transmitted symbols $\textbf{s}(t)={{[{{s}_{-L}}(t),\ldots ,{{s}_{L}}(t)]}^{T}}$.

• (a) System inputs and outputs

The DNFIS is integrated at the receiver after OAM mode demultiplexing. For each received OAM mode $(\ell )$, the equalizer takes two primary inputs derived from the receiver signal:

• 1. Instantaneous normalized power $({{m}_{1}})$: The receiver power in mode $\ell$, normalized by the average power. This is a direct indicator of intensity scintillation, given by Eq. (28).

  $$\begin{aligned} {{m}_{1}}(t)=\frac{|{{r}_{\ell }}(t){{|}^{2}}}{{E}[|{{r}_{\ell }}(t){{|}^{2}}]} \end{aligned}$$

  (28)

• 2. Phase gradient $({{m}_{2}})$: The rate of change of the unwrapped phase of ${{r}_{\ell }}(t)$, which captures phase fluctuations and beam wander effects, given by Eq. (29).

  $$\begin{aligned} {{m}_{2}}(t)=\frac{d}{dt}\arg ({{r}_{\ell }}(t)) \end{aligned}$$

  (29)

  The output of the DNFIS for the mode $\ell$ is a complex-valued correction factor, ${{v}_{{fnn},\ell }}(t)$, which is applied to the received signal to produce the equalized symbol.

• (b) Fuzzy rule-based for FSO channel

The fuzzy logic core of the DNFIS uses rules that directly map atmospheric conditions to correction strategies. The input ${{m}_{1}}$ and ${{m}_{2}}$ are fuzzified into linguistic variables, such as Low, Medium, and High for power, and Stable, Fluctuating, and Erratic for phase gradient.

• (c) Neural network adaptation

The neural network component continuously optimizes the membership functions and rule weights based on the error between the equalized output and the known training symbols (or decisions from a preliminary equalizer). The cost functionJ(t) is the mean squared error between the equalized symbol and the target symbol, as shown in Eq. (30)

$$\begin{aligned} J(t)=\frac{1}{2}|{{\hat{s}}_{\ell }}(t)-{{s}_{\ell }}(t){{|}^{2}} \end{aligned}$$

(30)

The learning algorithm for adjusting the weights (W) of the output layer, considering a learning rate $\mu$ and a momentum factor $\omega$ for faster convergence, is given by Eq. (31).

$$\begin{aligned} \Delta W(t)=-\mu \frac{\partial J(t)}{\partial W}=\mu e(t){{L}_{3}} \end{aligned}$$

(31)

The updated weight can be expressed as in Eq. (32).

$$\begin{aligned} W(t)=W(t-1)+\Delta W(t)+\omega [W(t-1)-W(t-2)] \end{aligned}$$

(32)

To modify the core parameters of the association function for the hidden layer, the learning method, as given by Eq. (33), is used.

$$\begin{aligned} \Delta {{\eta }_{ij}}=-\mu \frac{\partial J(t)}{\partial {{\eta }_{ij}}}=\mu e(t)W{{L}_{3}}\frac{2({{m}_{i}}-{{\eta }_{ij}})}{V_{j}^{2}} \end{aligned}$$

(33)

The updated center parameter is given by Eq. (34).

$$\begin{aligned} {{\eta }_{ij}}(t)={{\eta }_{ij}}(t-1)+\Delta {{\eta }_{ij}}(t)+\omega [{{\eta }_{ij}}(t-1)-{{\eta }_{ij}}(t-2)] \end{aligned}$$

(34)

• (d) Integration with Overall System

The DNFIS operates in tandem with the DCNN-TCSGm. While the DCNN-TCSGm provides a predictive, frame-level correction, the DNFIS performs a sample-by-sample, adaptive equalization, making it highly effective against rapid scintillation and phase noise that characterize long-distance FSO links. This dual approach ensures robust signal recovery across both slow and fast fading conditions.

Enhanced deep learning turbulence mitigation

The proposed approach employs advanced deep learning for turbulence mitigation in FSO communication. To enhance the speed and efficiency of turbulence compensation, we utilize a DCNN-TCSGm framework. The DCNN, integrated with a generative component, learns temporal correlations of fading channels in AT, enabling accurate prediction and compensation of complex dynamic correlations, especially under slower fading conditions. This results in faster convergence, improved resilience, and higher system throughput. The architecture of the proposed DCNN-TCSGm method is shown in Fig. 2.

Fig. 2

DCNN-TCSGm.

Model architecture specifications

The DCNN TCSGm is engineered to process sequences of turbulence-distorted OAM beam images, extracting spatial features through convolutional layers. In contrast, its dedicated temporal convolution layers learn how turbulence evolves over consecutive frames. This enables predictive compensation, anticipating distortion before it fully develops, which is fundamental to its accuracy advantage over conventional reactive equalizers. The DNFIS provides complementary, sample-by-sample adaptive equalization. Table 2 illustrates the model architecture specifications for DCNN-TCSGm and DNFIS.

Table 2 Model Architecture Specifications for DCNN-TCSGm and DNFIS.

• (a) DCNN Architecture for Turbulence Compensation

To enhance the real-time turbulence mitigation in FSO Communication, we implement a DCNN that learns to correct phase and intensity distortions from TV-AT. The DCNN takes as input a sequence of distorted OAM beam images and produces compensated beam profiles, enabling robust signal recovery and reduced BER.

Let $X\in {{{R}}^{T\times H\times W}}$ be the input tensor representing T sequential OAM beam intensity frames of size $H\times W$, captured under turbulent conditions. Each convolutional layer extracts feature maps using the operation defined in Eq. (35)

$$\begin{aligned} {{F}^{(l)}}=\sigma ({{W}^{(l)}}{{F}^{(l-1)}}+{{b}^{(l)}}) \end{aligned}$$

(35)

where ${{F}^{(l)}}$ is the output of the layer l, ${{W}^{(l)}}$ is the convolution kernel, ${{b}^{(l)}}$ is the bias term, and $\sigma$ is the ReLU activation function. Temporal dependencies across frames are captured using 1D temporal convolution, as described by Eq. (36).

$$\begin{aligned} {{T}_{j}}=\sum \limits _{t=1}^{T}{{{K}_{j}}}(t)\cdot {{X}_{t}} \end{aligned}$$

(36)

Where ${{K}_{j}}(t)$ is the 1D kernel applied across the time axis, and ${{T}_{j}}$ is the temporal feature vector. The reconstructed output $\hat{X}\in {{{R}}^{H\times W}}$ representing the turbulence-compensated beam, is obtained using a transposed convolution (deconvolution) operation, as defined in Eq. (37).

$$\begin{aligned} \hat{X}={Deconv}({{F}^{(N)}}) \end{aligned}$$

(37)

Where ${{F}^{(N)}}$ is the final feature map from the last convolutional block. The training objective minimizes the pixel-wise MSE between the predicted output and the clean target image ${{X}_{{clean}}}$, given by Eq. (38).

$$\begin{aligned} {{L}_{{MSE}}}=\frac{1}{HW}\sum \limits _{i=1}^{H}{\sum \limits _{j=1}^{W}{{{\left( {{{\hat{X}}}_{i,j}}-{{X}_{{clean},i,j}} \right) }^{2}}}} \end{aligned}$$

(38)

To preserve fine structural details, we include a spatial gradient loss, formulated in Eq. (39).

$$\begin{aligned} {{L}_{{grad}}}=\left\| \nabla \hat{X}-\nabla {{X}_{{clean}}} \right\| _{2}^{2} \end{aligned}$$

(39)

The total loss function is then defined by Eq. (40)

$$\begin{aligned} {{L}_{{total}}}={{\lambda }_{1}}{{L}_{{MSE}}}+{{\lambda }_{2}}{{L}_{{grad}}} \end{aligned}$$

(40)

where ${{\lambda }_{1}}$ and ${{\lambda }_{2}}$ are hyperparameters determined empirically during training, optimization is performed using the Adam optimizer with a learning rate $\mu$. The training convergence is monitored using the Root Mean Square Error (RMSE), calculated as shown in Eq. (41).

$$\begin{aligned} {RMSE}=\sqrt{\frac{1}{N}\sum \limits _{k=1}^{N}{{{\left( {{{\hat{Y}}}_{k}}-{{Y}_{k}} \right) }^{2}}}} \end{aligned}$$

(41)

In this equation ${{\hat{Y}}_{k}}$ and ${{Y}_{k}}$ denote predicted and actual beam intensities for the training sample. Additionally, the Pearson correlation coefficient (CC) is used to assess the structural similarity, as expressed in Eq. (42)

$$\begin{aligned} CC=\frac{\sum {(}\hat{X}-\bar{X})({{X}_{{clean}}}-{{{\bar{X}}}_{{clean}}})}{\sqrt{\sum {(}\hat{X}-\bar{X}{{)}^{2}}\ \sum {(}{{X}_{{clean}}}-{{{\bar{X}}}_{{clean}}}{{)}^{2}}}} \end{aligned}$$

(42)

• (b) Time-domain correlation sequence generation model for turbulence fading channel

To generate the time-domain correlation sequence model for a turbulence fading channel, we consider the large-scale ${{C}_{\ln A}}(\rho )$ and small-scale ${{C}_{\ln B}}(\rho )$ logarithmic covariance functions. The ACF model for optical turbulence can be expressed by Eq. (43)

$$\begin{aligned} {{C}_{{intensity}}}(\rho )=\exp [{{C}_{\ln A}}(\rho )+{{C}_{\ln B}}(\rho )]-1 \end{aligned}$$

(43)

where the ${{C}_{{intensity}}}(\rho )$ represents the intensity of ACF at spatial separation $\rho$. The comprehensive expressions for ${{C}_{\ln A}}(\rho )$ and ${{C}_{\ln B}}(\rho )$ are provided in Eq. (44) and Eq. (46) through Eq. (48) and Eq. (49).

$$\begin{aligned} {{C}_{\ln A}}(\rho )=1.06\sigma _{{variance}}^{2}\int _{0}^{1}{\int _{0}^{\infty }{{{\zeta }^{-11/6}}}}\exp [-\frac{\zeta }{{{\zeta }_{A}}}{{Z}_{0}}(\frac{\rho \sqrt{\omega \zeta }}{D})(1-\cos \zeta \phi )]d\zeta d\phi \end{aligned}$$

(44)

$$\begin{aligned} {{C}_{\ln A}}(\rho )\simeq 0.16\sigma _{{variance}}^{2}\zeta _{A}^{7/6}{{F}_{1}}(\frac{7}{6};1;-\frac{\omega {{\rho }^{2}}{{\zeta }_{A}}}{4D}) \end{aligned}$$

(45)

$$\begin{aligned} {{C}_{\ln B}}(\rho )=1.06\sigma _{{variance}}^{2}\int _{0}^{1}{\int _{0}^{\infty }{{{\zeta }^{-11/6}}}}\exp [-\frac{\zeta }{{{\zeta }_{B}}}{{Z}_{0}}(\frac{\rho \sqrt{\omega \zeta }}{D})(1-\cos \zeta \phi )]d\zeta d\phi \end{aligned}$$

(46)

$$\begin{aligned} {{C}_{\ln B}}(\rho )\simeq 1.27\sigma _{{variance}}^{2}{{\left( \frac{\omega {{\rho }^{2}}}{D{{\zeta }_{B}}} \right) }^{5/12}}{{K}_{5/6}}(\sqrt{\frac{\omega {{\rho }^{2}}{{\zeta }_{B}}}{D}}) \end{aligned}$$

(47)

Where $\sigma _{{variance}}^{2}$ is the Rytov variance. D represents the atmospheric structure constant. $\omega ={2\pi }/{{{\lambda }'}}\;$ is the wave number, with ${\lambda }'$ being the wavelength. ${{F}_{1}}(.)$ signifies a confluent hypergeometric function. ${{K}_{5/6}}$ is denoted the modified Bessel function of the second kind. The parameters ${{\zeta }_{A}}$ and ${{\zeta }_{B}}$ are given by:

$$\begin{aligned} {{\zeta }_{A}}=\frac{2.61}{1+1.11\sigma _{{variance}}^{12/5}} \end{aligned}$$

(48)

$$\begin{aligned} {{\zeta }_{B}}=3\left( 1+0.69\sigma _{{variance}}^{12/5} \right) \end{aligned}$$

(49)

By substituting Eq. (44)-(49) into Eq. (43) and applying the transformation given in Eq. (50),

$$\begin{aligned} \rho ={{W}_{\bot }}{\tau }' \end{aligned}$$

(50)

where, ${{W}_{\bot }}$ is the transverse wind speed and ${\tau }'$ is the time index, we can derive the time-dependent ACF as shown in Eq. (51).

$$\begin{aligned} \begin{aligned} C_{{intensity}}({\tau }')&= \exp \left[ \frac{0.49\sigma _{{variance}}^{2}}{(1 + 1.11\sigma _{{variance}}^{12/5})^{7/6}} F_{1}\left( \frac{7}{6}; 1; -\frac{\omega W_{\bot }^{2} ({\tau }')^{2} \zeta _{A}}{4D} \right) \right. \\&\quad \left. + \frac{0.50\sigma _{{variance}}^{2}}{(1 + 0.69\sigma _{{variance}}^{12/5})^{5/6}} \left( \frac{\omega W_{\bot }^{2} ({\tau }')^{2} \zeta _{B}}{4D} \right) ^{5/12} K_{5/6}\!\left( \sqrt{\frac{\omega W_{\bot }^{2} ({\tau }')^{2} \zeta _{B}}{4D}} \right) \right] - 1 \end{aligned} \end{aligned}$$

(51)

The normalized ACF ${{\Psi }_{{intensity}}}({\tau }')$ is defined by Eq. (52).

$$\begin{aligned} {{\Psi }_{{intensity}}}({\tau }')=\frac{{{C}_{{intensity}}}({\tau }')}{{{C}_{{intensity}}}(0)} \end{aligned}$$

(52)

The normalized ACF represents a stationary stochastic process and its power spectrum (S) form can be derived as shown in Eq. (53).

$$\begin{aligned} {{S}_{{intensity}}}({\Omega }')=4\int _{0}^{\infty }{{{\Psi }_{{intensity}}}}({\tau }')\cos ({\Omega }'{\tau }')d{\tau }' \end{aligned}$$

(53)

Here, ${\Omega }'=2\pi {v}'$, where ${v}'=1/{\lambda }'$ denotes the light frequency. To further discretize the data, a normalized digital frequency $\phi =2\pi /{{f}_{\text {sample}}}$ is obtained by sampling at ${{f}_{{sample}}}$. When added with Gaussian noise AWGN, at a certain power. $\sigma _{{noise}}^{2}$ passes through a rational filter transfer function $H(\phi )$ as given by Eq. (54). The temporal signal can then be generated using the relations in Eq. (55) and (56).

$$\begin{aligned} {{S}_{{intensity}}}(\phi )=\sum \limits _{n=1}^{\infty }{{{\Psi }_{{intensity}}}}(n){{e}^{-j\phi n}}=\left| \frac{B(\phi )}{A(\phi )} \right| \sigma _{\text {noise}}^{2}=H(\phi )\sigma _{{noise}}^{2} \end{aligned}$$

(54)

$$\begin{aligned} A(\phi )=1+{{a}_{1}}{{e}^{-j\phi }}+\cdots +{{a}_{n}}{{e}^{-jn\phi }} \end{aligned}$$

(55)

$$\begin{aligned} B(\phi )=1+{{b}_{1}}{{e}^{-j\phi }}+\cdots +{{b}_{m}}{{e}^{-jm\phi }} \end{aligned}$$

(56)

From the covariance structure of the autoregressive moving average (ARMA) process, it can be deduced from Eq. (57).

$$\begin{aligned} \left\{ \begin{array}{l} {\Psi _{{filter}}}(0) + \sum \limits _{n=1}^{N} a_{{filter}} \Psi _{{filter}}(-n) = \sigma _{{noise}}^{2}, \\ {\Psi _{{filter}}}(m) + \sum \limits _{n=1}^{N} a_{n} \Psi _{{filter}}(m-n) = 0 \end{array} \right. \end{aligned}$$

(57)

Let the relations in Eqs. (58) and (59) hold.

$$\begin{aligned} {{\alpha }_{{{n}'}}}={{\Psi }_{{intensity}}}(n+1)+\Psi _{{intensity},1\times n}^{*}{{\phi }_{n\times 1}} \end{aligned}$$

(58)

$$\begin{aligned} \left[ \begin{matrix} \Psi _{\text {intensity}}^{(n+1)\times (n+1)} & \Psi _{\text {intensity}}(-n-1) \\ \Psi _{\text {intensity}}^{n\times 1} & \Psi _{\text {intensity}}(0) \end{matrix} \right] \left[ \begin{matrix} 1 \\ \phi _{n\times 1} \end{matrix} \right] = \left[ \begin{matrix} \sigma _{\text {noise}}^{2} \\ 0 \end{matrix} \right] \end{aligned}$$

(59)

Using a recursive solution method, we can obtain the results shown in Eqs. (60), (61), and (62).

$$\begin{aligned} \phi _{(n+1)\times 1} = \left[ \begin{matrix} \phi _{n} \\ 0 \end{matrix} \right] + \xi _{n+1} \left[ \begin{matrix} \phi _{n} \\ 0 \end{matrix} \right] \end{aligned}$$

(60)

$$\begin{aligned} \sigma _{\textrm{noise}}^{2}(n+1)=\sigma _{\textrm{noise}}^{2}(n)\bigl (1-|\xi (n+1)|^{2}\bigr ) \end{aligned}$$

(61)

$$\begin{aligned} \sigma _{{noise}}^{2}(n)\phi (n)=Z(n) \end{aligned}$$

(62)

Where, $\xi (n+1)=-{{{\alpha }_{{{n}'}}}}/{\sigma _{{noise}}^{2}(n)}\;$ allows us to determine the coefficients, ${{\phi }_{n}}$ of the turbulence fading channels transfer function, and calculate the time-domain correlation sequence Z using the above given equations. However, this calculation is based on AWGN and does not account for the probability density function (PDF), meaning the generated sequence may not conform to turbulence disturbance theory. To address this, assuming a free-space receiver without aperture smoothing, the Gamma-Gamma (G-G) function is used as the PDF for the turbulence fading channel. Its parameters ${{\beta }_{GG}}$ and ${{\gamma }_{GG}}$ can be related to the turbulence structure constant $C_{n}^{2}$ as shown in Eq. (63)-Eq. (66).

$$\begin{aligned} {{P}_{{intensity}}}(J)=\frac{2{{({{\beta }_{GG}}{{\gamma }_{GG}})}^{({{\beta }_{GG}}+{{\gamma }_{GG}})/2}}}{\Gamma ({{\beta }_{GG}})\Gamma ({{\gamma }_{GG}})}J{{\left( \frac{J}{\langle J\rangle } \right) }^{({{\beta }_{GG}}+{{\gamma }_{GG}})/2}} \end{aligned}$$

(63)

$$\begin{aligned} {{K}_{{{\beta }_{GG}}-{{\gamma }_{GG}}}}(2\sqrt{\frac{{{\beta }_{GG}}{{\gamma }_{GG}}J}{\langle J\rangle }}),\quad J>0 \end{aligned}$$

(64)

$$\begin{aligned} {{\beta }_{GG}}=\frac{1}{\sigma _{A}^{2}}=\frac{1}{\exp (\sigma _{\ln A}^{2})-1} \end{aligned}$$

(65)

$$\begin{aligned} {{\gamma }_{GG}}=\frac{1}{\sigma _{B}^{2}}=\frac{1}{\exp (\sigma _{\ln B}^{2})-1} \end{aligned}$$

(66)

Where the expressions for the logarithmic variances $\sigma _{\ln A}^{2}$ and $\sigma _{\ln B}^{2}$, which are required for these calculations, are provided in Eqs. (67) and (68).

$$\begin{aligned} \sigma _{\ln A}^{2}=\frac{0.49\sigma _{{variance}}^{2}}{{{\left( 1+1.11\sigma _{{variance}}^{12/5} \right) }^{7/6}}} \end{aligned}$$

(67)

$$\begin{aligned} \sigma _{\ln B}^{2}=\frac{0.51\sigma _{{variance}}^{2}}{{{\left( 1+0.69\sigma _{{variance}}^{12/5} \right) }^{9/6}}} \end{aligned}$$

(68)

Here, $\langle {{J}_{n}}\rangle$ represents the ${{n}^{th}}$ moment of the light intensity J, and $\Gamma (\cdot )$ denotes the Gamma function. When the normalized random light intensity sequence ${{\left\{ {{{{\hat{J}}}_{n}}}/{\langle \hat{J}\rangle }\; \right\} }_{1\times N}}$ From the simulation, the G-G PDF follows, which the vector relation can express in Eq. (69).

$$\begin{aligned} \widehat{\textbf{J}}=\left[ \begin{matrix} {{{\hat{J}}}_{1}}/\langle \hat{J}\rangle \\ {{{\hat{J}}}_{2}}/\langle \hat{J}\rangle \\ \vdots \\ {{{\hat{J}}}_{n}}/\langle \hat{J}\rangle \\ \end{matrix} \right] \in {PD}{{\text {F}}_{{Gamma--Gamma}}} \end{aligned}$$

(69)

Where $i=1,2,\ldots ,n$. By rearranging $\hat{J}$ according to the rank of Z, the CC of the sequence ${{\rho }_{J,Z}}$ can be calculated using the formula given in Eq (70).

$$\begin{aligned} {C}{{{C}}_{{{\rho }_{J,Z}}}}=\frac{\langle (J(i)-\langle J\rangle )(Z(i)-\langle Z\rangle )\rangle }{\sqrt{\langle {{(J(i)-\langle J\rangle )}^{2}}{{(Z(i)-\langle Z\rangle )}^{2}}\rangle }}=1 \end{aligned}$$

(70)

This perfect mapping ensures that the temporal correlation information from Z is transferred to J, thereby satisfying the dual criteria defined in Eq. (71).

$$\begin{aligned} \hat{J}\in {PD}{{{F}}_{{Gamma--Gamma}}},\quad {{\Psi }_{{intensity}}}(n)=\langle (J(i)-\langle J\rangle )(Z(i-n)-\langle Z\rangle )\rangle \end{aligned}$$

(71)

Eq. (51) verifies that the ACF generated by $\hat{J},{{\Psi }_{{intensity}}}(n)$ matches the numerical simulation of the theoretical solution Eq. (52), while preserving the PDF information. Filter coefficients are obtained using the Yule–Walker algorithm, and a temporal Gaussian correlation sequence is produced by integrating AWGN. Finally, through the rank mapping of the time-domain Gaussian correlation, the random light intensity sequence following the G–G PDF is reordered to accurately represent turbulence disturbance information across frequency bands.

Multiplexing and demultiplexing techniques

To maximize the throughput of inter-satellite FSO links, we propose a hybrid approach that combines WDM with OAM-based MDM. This system is modeled to operate natively in the mid-infrared (mid-IR) spectral region, leveraging its advantages for FSO communications. The implemented model relies on compact, single-layer dielectric metasurface structures, specifically designed for the mid-IR, which are modeled to function as integrated transceivers for efficient OAM multiplexing and demultiplexing, generating and receiving orthogonal coaxial OAM beams as depicted in Fig. 3.

Fig. 3

Multiplexing and Demultiplexing.

• (a) WDM in Mid-IR

WDM enhances channel capacity by simultaneously transmitting multiple data streams on different optical wavelengths. For the mid-IR regime, this is efficiently achieved using native optical components which provide direct emission within the 3-5$\mu m$ atmospheric transmission window. This approach avoids the inefficiencies of wavelength conversion and is well-suited for high-capacity, multi-gigabit-per-second data links.

• (b) OAM-based MDM in Mid-IR

OAM-based MDM boosts capacity by using orthogonal spatial modes to transmit multiple channels with minimal crosstalk. In the mid-IR, OAM beams of different orders remain orthogonal. Spiral phase plates (SPPs) and compact near-IR OAM generation methods can be adapted for mid-IR use. Combining WDM with OAM-based MDM enables higher capacities through simultaneous transmission of multiple high-capacity channels across wavelengths and orthogonal OAM modes.

• (c) Optical Metasurface-based OAM Multiplexing/ Demultiplexing in Mid-IR

To achieve a highly integrated and compact form factor for our FSO transceiver, we model single-layer dielectric metasurfaces designed to operate in the mid-infrared spectrum. These sub-wavelength structures provide unprecedented control over the phase, amplitude, and polarization of light, enabling them to efficiently generate and separate multiple coaxial OAM beams. For multiplexing, the metasurface is modeled to impart a spatially varying phase profile that converts an array of incident Gaussian beams (each potentially at a different WDM wavelength) into a set of coaxial OAM beams with distinct topological charges (l). The phase function for generating an OAM beam with charge l is given by $\phi (x,y)=l\cdot \arctan \left( {y}/{x}\; \right)$, which defines the required spatial phase distribution.

For demultiplexing at the receiver, the complementary metasurface model performs the inverse operation. It applies a phase correction that transforms each incoming OAM mode $\ell$ into a focused Gaussian spot at a distinct lateral position on a multi-pixel detector array. This spatial separation allows for the simultaneous detection of all multiplexed channels with minimal crosstalk. The metasurface’s efficiency in our model is characterized by its ability to direct optical power into the desired diffraction order. The performance is quantified by the Signal-to-Crosstalk Ratio (SXR) between different OAM modes, which we optimize to be >15 dB across the operational mid-IR band (e.g., 3-5 $\mu m$). Unlike bulky conventional optics like spiral phase plates, the modeled metasurfaces represent an ultra-thin (on the order of the wavelength) and lightweight alternative, suggesting a potential path for future compact, next-generation FSO terminals that are suitable for size, weight, and power (SWaP)-constrained applications like satellite communications. To contextualize the trade-offs between conventional SLM-based approaches and the proposed metasurface implementation, Table 3 compares their key performance parameters. This comparison highlights the advantages of metasurfaces for compact, SWaP-constrained applications despite their static phase profiles. Both approaches present practical implementation considerations that would require careful engineering in physical prototypes, but are suitable for the system-level modeling presented in this work.

Table 3 Comparison of OAM Manipulation Technologies for FSO Terminals.

Simulation results and comparative analysis

This section presents a simulation-based performance evaluation of the proposed framework. Due to the complexity and cost of real-world multi-component FSO deployment, simulations provide a controlled and repeatable methodology essential for analyzing the complex interactions within such an integrated system. The performance was evaluated under various atmospheric turbulence conditions using established and widely accepted models for turbulence and beam propagation. The results demonstrate significant improvements in signal quality, data rate, and link reliability compared to conventional systems. All simulations were conducted in MATLAB R2023a. A discussion on the necessary path toward experimental validation is provided in the Conclusion.

Experimental visualizations and results

This section presents extensive visualizations of the simulations, detailing the results of the simulated experiments and highlighting the performance of the proposed OAM-based FSO communication system in the following subsections.

Fig. 4

Implemented OAM-based Free-Space Optical (FSO) communication model.

• (a) OAM-FSO Communication Model Implementation

Figure 4 shows the spatial intensity and phase profile of OAM Mode 1 at a propagation distance of $Z=4.00m$. The intensity map in the x–y plane exhibits the characteristic doughnut-shaped pattern of vortex beams, while the phase front confirms the helical structure associated with OAM. This baseline visualization verifies beam fidelity and symmetry in a controlled FSO channel, serving as a reference for evaluating the effects of turbulence and compensation mechanisms during transmission.

• (b) Error Vector Magnitude (EVM) Analysis

The EVM graph illustrates modulation accuracy by comparing in-phase and quadrature (I–Q) deviations under four polarization and interference scenarios: No Crosstalk (EVM = 0.0007), Single Polarization (EVM = 0.0022), Dual Polarization–X (EVM = 0.0028), and Dual Polarization–Y (EVM = 0.0032). The tight clustering of symbols near the ideal constellation points indicates superior signal fidelity, while the low EVM values confirm the system’s robustness against polarization-induced distortions and inter-mode crosstalk, ensuring reliable high-quality demodulation in FSO links.

Figure 5 presents the EVM graph as a function of SNR, turbulence strength, and propagation distance, with curves for different mitigation strategies and OAM modes. Results show consistently low EVM under varying AT conditions, confirming the proposed system’s robustness and reliable demodulation in FSO links.

Fig. 5

Error Vector Magnitude (EVM) analysis graph.

Fig. 6

(a) Turbulence limits OAM image quality, (b) Bessel beam, (c) Airy beam, and (d) Vortex beam .

• (c) Integration of Structured Light, AO, and PSO for Turbulence Mitigation

The integration of resilient structured light beams (Bessel, Airy, and optimized OAM vortex beams) with AO and PSO effectively mitigates AT. Structured beams inherently resist image degradation, while AO provides real-time wavefront correction, and PSO optimizes system parameters intelligently. Experimental results demonstrate that this combined approach preserves signal integrity and significantly reduces AT-induced distortions over long-distance FSO links. Figure 6a–d illustrates the integrated system architecture, high- lighting the joint use of structured beams, AO, and PSO for enhanced turbulence compensation.

• (d) BER Performance as a Function of OAM Modes

The system’s BER performance was analyzed with respect to the number of OAM modes employed for MDM under varying AT conditions. BER was measured before and after applying the DNFIS equalization scheme, showing significant improvements in error correction and signal recovery. While conventional FSO systems experience notable BER degradation with increasing turbulence and mode count, the proposed system, which integrates structured light and advanced equalization, maintains a consistently low BER across multiple OAM modes. Figure 7 shows the BER performance, where the final BER (after DNFIS) is plotted against initial BER, with separate curves representing different OAM modes.

Fig. 7

BER based on OAM modes graph.

• (e) Real-time Turbulence Compensation using DCNN-TCSGm

The DCNN-TCSGm was implemented for real-time compensation of turbulence-induced effects. The training curve plots RMSE and loss across iterations. Initially, high RMSE values indicate poor prediction of turbulence degradation; however, as training progresses, the model learns temporal patterns in the channel, enabling accurate prediction and compensation. The sharp decline in RMSE and loss demonstrates fast convergence, strong generalization, and more effective distortion reduction compared to traditional equalization. Figure 8 shows the training performance of the proposed DCNN-TCSGm framework in terms of RMSE and loss over 500 iterations. The RMSE drops rapidly during the first 50 iterations and stabilizes below 0.05 after about 100 iterations, while the training loss decreases from 0.18 to nearly 0.01. The smooth convergence without significant fluctuations indicates stable learning and the absence of overfitting, confirming that the network effectively captures the underlying signal features. Table 4 summarizes the performance before and after applying DCNN-TCSGm. The BER decreases from 0.0032 to 0.00144, representing a 55% improvement. The MSE is reduced from 0.015 to 0.0117, yielding a 22% gain in signal stability. In addition, the SNR improves from 18.5 dB to 28.5 dB, providing a 10 dB power gain. These results clearly demonstrate the effectiveness of DCNN-TCSGm in mitigating channel impairments and enhancing the reliability of OAM-based optical transmission.

Fig. 8

Mitigate the data using DCNN-TCSGm methods image.

Table 4 DCNN-TCSGm performance summary.

Fig. 9

Multiplexed and Demultiplexed output image.

• (f) Performance of Combined WDM-MDM with Optical Metasurface-based OAM Multiplexing/Demultiplexing

The simulation results demonstrate that integrating WDM with OAM-based MDM in the mid-infrared spectrum significantly enhances data transmission capacity for inter-satellite FSO links. Modeled compact dielectric metasurfaces enable efficient OAM multiplexing and demultiplexing, supporting multiple channels across different wavelengths and OAM modes. Figure 9 shows the enhanced throughput, illustrating multiplexed wavelengths carrying multiple OAM modes and the simulated metasurface demultiplexing incoming beams into distinct spatial positions based on their topological charge.

OAM mode purity under turbulence

To evaluate the robustness of OAM modes under turbulent conditions, we analyze the mode purity as defined in Eq. (13). Figure 10a plots the mode purity against the topological charge $(\ell )$ for three turbulence strengths, demonstrating the proposed framework’s compensation capabilities. It represents Mode purity versus topological charge $\ell$ for the proposed framework under weak $\left( C_{n}^{2}\approx 1\times 10^{-14}\,\text {m}^{-2/3}\right)$, moderate $\left( C_{n}^{2}\approx 3\times 10^{-14}\,\text {m}^{-2/3}\right)$, and strong $\left( C_{n}^{2}=5\times 10^{-14}\,\text {m}^{-2/3}\right)$ turbulence conditions. Error bars represent Monte Carlo simulation uncertainty $(\pm 1.5-2.5\%)$. Dashed lines indicate performance thresholds: $\text {P}>0.90$ (excellent), $\text {P}>0.75$ (good), $\text {P}>0.50$ (marginal). Under strong turbulence, the framework maintains exceptional purity for low-order modes $(\ell =0$: P$=0.90$, $\ell =\pm 1$: P$=0.85)$ while exhibiting graceful parabolic degradation for higher-order modes, with $\ell =\pm 5$ retaining P$=0.42$. This demonstrates the system’s ability to support up to five reliably usable channels $(|\ell |\le 2$, P$>0.75)$ even under severe turbulence conditions.

Figure 10b OAM mode purity degradation with propagation distance for different topological charges $|\ell | = 1$–5 under moderate atmospheric turbulence $(C_n^2 = 3\times 10^{-14}\,\text {m}^{-2/3})$. Mode purity shows approximately linear degradation with distance, with higher-order modes exhibiting faster degradation rates. Lower-order modes $(|\ell | \le 2)$ maintain $P_\ell > 0.70$ at 10 km, supporting long-distance transmission, while $|\ell | = 5$ degrades to $P_\ell = 0.38$ at the same distance. These findings highlight a critical trade-off in free-space optical communication that employs OAM multiplexing. While higher-order modes offer greater spectral efficiency, their susceptibility to both turbulence and propagation loss makes lower-order modes substantially more robust for long-distance links. These findings highlight a critical trade-off in free-space optical communication that employs OAM multiplexing. While higher-order modes offer greater spectral efficiency, their susceptibility to both turbulence and propagation loss makes lower-order modes substantially more robust for long-distance links. This analysis also establishes the framework’s sensitivity to propagation distance, quantifying the graceful degradation of mode purity essential for long-haul link planning.

Impact of pointing errors on system performance

We evaluate BER degradation under the combined effects of turbulence and pointing errors. Figure 11 shows BER versus the normalized pointing jitter $\sigma _p/\omega _z$ under moderate turbulence. Without compensation, BER exhibits approximately exponential growth with pointing error, reaching 0.095 at $\sigma _p/\omega _z = 0.5$. With our proposed hybrid mitigation, where tip/tilt correction is handled by the adaptive optics subsystem and residual distortions are compensated by the DCNN-TCSGm equalizer, the BER is significantly suppressed to 0.018 at the same pointing error level, representing an $81\%$ improvement. This demonstrates the system’s resilience to misalignment, a critical factor for long-distance terrestrial links.

Fig. 10

OAM mode purity characterization under atmospheric turbulence.

Fig. 11

BER vs. normalized pointing error for different compensation schemes.

Comparative analysis

To evaluate the improvements of the proposed methodology, we compare its performance with existing approaches, namely DFE³⁸ and conventional MDM-FSO⁴⁴ systems. This comparison highlights enhancements in signal quality, BER, and data throughput under varying atmospheric conditions. The Time $\text {(s)}$ versus Error $(\%)$ metric is used to assess how error percentage varies with time, reflecting the impact of dynamic AT on FSO links. A reduced and more stable error profile demonstrates the effectiveness of the proposed mitigation techniques.

$$\begin{aligned} \text {Error}(\%)\text {=}\frac{\text {Number of Erroneous Bits }(\text {t})}{\text {Total Number of Transmitted Bits }(\text {t})}\times \text {100}\% \end{aligned}$$

(72)

In Eq. (72), Error $(\%)$ denotes the ratio of the Number of Erroneous Bits $(\text {t})$, i.e., bits incorrectly received within a time interval around $\text {t}$, to the Total Number of Transmitted Bits $(\text {t})$ during the same interval. The error percentage over time reflects the system’s temporal stability and robustness under dynamically varying atmospheric conditions. As shown in Fig. 12, the proposed method starts with an error rate of 1.8 $\%$ at 0.1s, gradually increasing to 4.5 $\%$ at 1.0s. In contrast, MDM-FSO shows a sharp increase from 6.1 $\%$ to 9.5$\%$, while DFE rises from 5.3$\%$ to 8.0$\%$. This consistently lower error trend highlights the superior turbulence adaptation achieved through the integration of structured light beams, AO, and DCNN-TCSGm, which collectively enhance signal clarity and resilience against rapid environmental fluctuations.

Time $\text {(s)}$ vs. Signal $\text {(V)}$ graph examines the variation of received optical signal voltage under turbulence-induced refractive index fluctuations, leading to intensity scintillation. Since scintillation is time-dependent, tracking the signal voltage over time reflects the effectiveness of AO and mitigation techniques in maintaining stable power for reliable decoding. A relatively flat signal trace indicates effective suppression of turbulence effects.

Fig. 12

Numerical Outcomes of Time (s) vs. Error (%).

Fig. 13

Numerical Outcomes of Time (s) vs. Signal (V).

$$\begin{aligned} \text {Signal}\ (\text {V})={{V}_{\text {received}}}(t) \end{aligned}$$

(73)

In Eq. (73), $\text {Signal}\ (\text {V})$ is the instantaneous voltage of the received optical signal at time (t) and ${{V}_{\text {received}}}$ is the measured voltage of the signal at the receiver.

Figure 13 shows the numerical results of time vs. signal. The proposed system demonstrates steady voltage growth from 0.93V to 1.02V within 1s, indicating strong resilience to atmospheric fading. By contrast, MDM-FSO starts at 0.70V and peaks at 0.80V, while DFE marginally increases from 0.75V to 0.84V. This improvement arises from the adaptive correction mechanism via AO and PSO, which stabilizes beam power and mitigates fluctuations, ensuring robust demodulation and decoding for high-speed links.

Wavelength $\text {(m)}$ vs. Power $\text {(dBm)}$ graph evaluates WDM performance in OAM-based MDM by mapping power distribution across multiple wavelength channels, often in the mid-IR spectral region. This ensures sufficient signal strength for reliable multi-channel, high-capacity transmission.

$$\begin{aligned} {{P}_{\text {dBm}}}=10{{\log }_{10}}\left( \frac{P(\lambda )}{1\text {mW}} \right) \end{aligned}$$

(74)

In Eq. (74), ${{P}_{\text {dBm}}}$ represents the optical power of the signal at a specific wavelength $\lambda$, and $P(\lambda )$ is the measured optical power of the signal at the wavelength. Figure 14 presents the results. The proposed model shows significant power enhancement, with peak values reaching -18 dBm at $1.55\text \!\!\mu \!\!\text { m}$ and rising to 0 dBm at $1.61\text \!\!\mu \!\!\text { m}$, outperforming MDM-FSO and DFE by up to 10 dB. This gain stems from efficient metasurface-based OAM demultiplexing and optimized spectrum usage, which enhances channel separation, improves SNR, and enables energy-efficient communication.

Fig. 14

Wavelength (m) vs. Power (dBm).

OAM modes with different topological charges are orthogonal, enabling MDM to transmit multiple data streams simultaneously. The analysis examines the impact of adding OAM modes on BER, particularly under AT, where crosstalk may occur.

$$\begin{aligned} BER({{N}_{\text {OAM}}})=f\left( {{N}_{\text {OAM}}},\text {Turbulence Strength} \right) \end{aligned}$$

(75)

In Eq. (75), $BER({{N}_{\text {OAM}}})$ denotes the BER as a function of the number of OAM modes used, ${{N}_{\text {OAM}}}$ is the number of distinct OAM modes being multiplexed, and $\text {Turbulence Strength}$ is a parameter that represents the severity of AT.

Figure 15 shows that the proposed model maintains a low and stable BER ( 0.00025 to 0.00050 for up to 10 OAM modes), whereas MDM-FSO and DFE exhibit higher and fluctuating BER, especially at higher mode counts. This demonstrates that DNFIS-based equalization effectively mitigates inter-modal interference and preserves OAM orthogonality. Combined with AO, it supports scalable multi-mode transmission, crucial for high-bandwidth applications such as inter-satellite links. Thus, Fig. 15 provides a crucial sensitivity analysis on OAM mode count, demonstrating the framework’s capacity-robustness trade-off and its consistent advantage over baselines across multiplexed channels.

Fig. 15

Number of OAM modes vs. BER.

Fig. 16

Normalized signal quality versus Rytov variance $(\sigma _{R}^{2}$) under atmospheric turbulence.

AT, caused by refractive index fluctuations from pressure and temperature variations, induces beam wander, scintillation, and phase distortions that impair signal quality. In this study, turbulence is varied across weak, moderate, and strong regimes, with performance assessed using SNR and received power. The objective is to demonstrate that the proposed approaches, including structured light beams and AO, can sustain or even improve signal quality under increasing turbulence conditions.

$$\begin{aligned} {{Q}_{\text {Signal}}}=Q\left( \text {Turbulence}\ \text {Strength} \right) \end{aligned}$$

(76)

In Eq. (76), ${{Q}_{\text {Signal}}}$, is a quantitative measure of the received signal’s integrity.

Figure 16 shows the proposed framework consistently outperforms baseline methods across all turbulence regimes, exhibiting improved robustness against turbulence-induced signal degradation compared to MDM-FSO and DFE. Although the proposed framework significantly mitigates turbulence-induced degradation, residual performance loss is still observed under strong turbulence conditions, reflecting the fundamental limits imposed by atmospheric channel impairments. Figure 16 provides the sensitivity analysis for turbulence strength. Collectively, Figures 10b, 11, 15, and 16 evaluate system performance against propagation distance, pointing error, OAM mode count, and turbulence strength, confirming that the reported performance gains are robust across the operational parameter space.

Computational complexity and real-time performance

The practical deployment of intelligent compensation requires an assessment of computational overhead. The complexity of the core algorithms was evaluated. The DCNN-TCSGm, with the architecture detailed in Table , requires approximately 2.5 Giga-FLOPS (GFLOPs) per inference. Based on standard benchmarks for embedded AI platforms (e.g., NVIDIA Jetson AGX), this complexity translates to a projected processing latency of $<10\text {ms}$ per frame. The DNFIS equalizer exhibits a complexity that is quadratic in the number of OAM modes, $\text {O(}{{\text {M}}^{\text {2}}}\text {)}$, enabling sub-millisecond symbol-rate adaptation.

The total projected system latency encompassing wavefront sensing, AO correction, PSO optimization, DCNN inference, and DNFIS equalization is therefore estimated to be under 20 ms. This is critically compared to the characteristic coherence time of atmospheric turbulence, which is typically on the order of $1-10\text {ms}$ for terrestrial links. Therefore, the proposed processing pipeline is estimated to operate within the required real-time window, enabling effective compensation before the turbulent channel state decorrelates.

Conclusion

This study proposes an advanced FSO communication framework integrating structured light beams, intelligent optimization, AO, and deep learning-based turbulence mitigation to overcome key limitations of conventional FSO systems. By leveraging the self-healing and non-diffracting properties of Bessel, Airy, and optimized vortex beams, the system inherently enhances robustness against AT. Adaptive wavefront correction, combined with PSO and real-time compensation using a DNFIS, further improves signal fidelity in long-distance MDM scenarios. To address temporal channel variations, a DCNN-TCSGm is implemented to predict and compensate turbulence effects in real-time. Simulation results demonstrate up to 55% reduction in BER, 22% improvement in received signal voltage, and up to 10 dB power gain over baseline MDM-FSO and DFE systems. Additionally, the integration of WDM with metasurface-based OAM multiplexing significantly enhances data throughput, particularly for inter-satellite communication. The proposed framework demonstrates significant robustness, scalability, and resilience under challenging atmospheric conditions, highlighting its potential for next-generation optical communication networks. Future work may explore advanced ML for real-time turbulence mitigation and hybrid multiplexing strategies to further enhance throughput, scalability, and reliability.

Limitations and future validation

This study establishes a comprehensive simulation-based proof-of-concept. A key limitation is the absence of experimental validation under real atmospheric conditions. Future work will therefore focus on implementing a laboratory-scale prototype. This experimental phase will utilize a programmable turbulence phase screen, SLMs for dynamic beam generation, and high-bandwidth coherent detectors to empirically validate the predicted performance gains. Furthermore, future experimental and deployment efforts will also investigate practical integration challenges, such as the stringent alignment requirements for high-order OAM modes and the integration of adaptive optics within robust pointing, acquisition, and tracking PAT subsystems. It will also serve to identify and address practical integration challenges related to hardware latency, alignment precision, and component non-idealities not captured in simulation.