Introduction

Recently, the coordination between light fields carrying orbital angular momentum (OAM) and inter-band quantum well transitions has gained significant attention1. With the increasing demand for optical interconnects to enhance data transmission, various properties of light such as time, amplitude, phase, wavelength, and polarization have been utilized to increase transmission capacity. However, challenges including signal degradation, interference, and bandwidth limitations continue to impact effective communication2,3,4. OAM, a new aspect of spatial multiplexing, is widely utilized in fiber optics due to its virtually unlimited orthogonal modes. Three fiber types for transmitting OAM modes have been proposed: step-index fibers (SIFs), graded-index fibers (GIFs), and photonic crystal fibers (PCFs)5. Optical vortices with OAM are light beams with phase singularities, enabling new phenomena in light-object interactions6. OAM beams offer extra physical dimensions that improve bandwidth, data rate, security, and spectral efficiency, while reducing noise and bit error rate (BER)7,8. OAM beams have found extensive applications in the fields of quantum optics, Nano-optics, biomedicine, optical communication, and microscopic imaging. Optical vortex beams that contain longitudinal OAM are well recognized as one of the most prevalent forms of optical vortices. These beams have an OAM orientation that is either parallel or antiparallel to the direction in which the light is propagating9. OAM transmission technique can be regarded as an innovative method of mode-division multiplexing, which brings a higher degree of adaptability to improve the performance of existing communication systems, expanding from optical to radio frequency (RF) domains10,11,12.

The recently created Spotted Hyena Optimizer (SHO) is used for fine-tuning the hyperparameters of machine-learning models that provide functions such as load balancing, and quarter sensing. This approach is susceptible to variations in traffic and costs, as well as several other network indicators, including interference13,14. Focused Vector Beams, with distinct frequency and amplitude dispersion, are utilized in fields like electron manipulation, quantum communications, and optical communication. Fractional OAM (FOAM) modes in multiplexed communication have shown potential to improve communication capacity, however, there are still difficulties in precisely defining distorted modes15,16. Atmospheric turbulence (AT) results in the phase distortion structure of focused vector beams (FVBs), leading to mode dispersion and intensity profile distortion17. The unlimited dimension of the topological charge and the inherent orthogonality between optical modes or quantum states have significant potential for enhancing communication systems18. These properties aim to increase photon information efficiency in quantum communication systems or address capacity limitations in classical ones. Highly dimensional quantum states are also utilized, which are more resistant to noise and thus beneficial for transmission in challenging environments such as photon scarcity or complete saturation zones19,20. However, optical transmission methods face limitations, including low OAM mode capacity, high blocking rates, inefficient error management, inadequate bandwidth, slow optical switches, and insufficient models for addressing AT, which cause reduced performance.

To address the aforementioned issues, this paper proposes a secure and optimized optical communication system integrating quantum well-based modulators with OAM modes and advanced modulation techniques for enhanced performance. This paper aims to enhance the efficiency, reliability, and security of optical transmission systems. The key objectives include maximizing OAM mode support, improving transmission efficiency, minimizing outages through real-time routing adjustments, and ensuring resilience against environmental disturbances. The proposed approach also emphasizes optimizing quantum well structures and achieving nanosecond-level responses for real-time network monitoring. The main contributions of this paper are given below:

  • This work integrates Helical Phase Spatial Light Modulator (HPSLM) with Orthogonal Frequency Division Multiplexing and Quadrature Amplitude Modulation (OFDM-QAM) and precise vector phase modulation to reduce BER and improve the purity and capacity of OAM modes.

  • A Hybrid traffic prediction and optimization algorithm named Traffic Prediction assisted with a Spotted Hyena Optimizer (TPAR-SHO) is proposed for load balancing and congestion mitigation in optical networks. TPAR-SHO enables the network to perform real-time rerouting and minimize the blocking probability.

  • To enhance the transmission privacy with physical layer encryption, a secure Quantum Well Structure with Injection Locking Synchronization (QWS-ILCS) is proposed. QWS-ILCS provides high coherence, wide spectral gain, and resistance to unauthorized access.

  • This work also adopts the Optical Frequency Division Multiplexing with Proportional Integral Derivative Controller (OFDM-PID) technique is proposed to mitigate atmospheric turbulence effects by dynamically adjusting transmission parameters.

  • A nanosecond-level optical monitoring framework is proposed by using Fast Fourier Transform (FFT) with a Fiber Optical Performance Monitoring (FOPM) tool. This integration enables real-time signal integrity assessment and system self-correction, which is absent in most traditional systems.

The integration of these five modules provides a novel, secure, and turbulence-resilient optical transmission system that advances the state-of-the-art across performance, security, and adaptability.

The remaining section of this article is organized as follows: “State of the art”, lists previous studies and limitations in optical transmission systems. “Proposed secure and optimized optical transmission system” states the problem statement and the proposed approach. “Simulation results and comparative analysis” discusses the simulation results and analysis and the “Conclusion” illustrates the conclusion.

Table 1 Summary of related work and their limitations.

State of the art

The authors presented a Semiconductor Optical Amplifier (SOA) with wide bandwidth, low division, and deep stressed wells as its foundation21. The use of new quartz wells for TM modes reduces transmission susceptibility and expands bandwidth. By combining a sloped broadband construction with a hollow optical thin film, they decrease internal reflectivity and transmission losses. Although the SOA increases bandwidth by three decibels, it doesn’t fully cover the L-band, requiring further improvements to shift the quantum wells to higher wavelengths. In22, a secure chaotic OFDM encryption strategy for OAM systems is proposed, focusing on high-order modulation and security. Using Zhan’s hyper-chaotic model, the system prevents eavesdropping and maintains a low BER below a certain forward error correction (FEC) threshold. The proposed technique offers high security, but managing BER under difficult conditions remains a challenge for system reliability.

The Diffractive Deep Neural Network (D2NN) for OAM signal multiplexing and demultiplexing is proposed to automatically modify phase and amplitude values and control the wavefront of light beams across multiple diffractive panels23. The D2NN model can be rebuilt to address problems like layer disarray, optical abnormalities, and wavefront discrepancies by feeding experimental outputs back into the model for further optimization. The authors explain an OAM visible light communication (OAMVLC) network with a 45 Gbps capacity, utilizing 3 Laguerre-Gaussian (LG) modes at 5 Gbps24. The system uses red/green/blue (RGB) Tunable Visible Light Laser Diodes (TVL-LDs) and Non-Return-to-Zero (NRZ) line coding. OAM aids minimize mode crosstalk and inter-channel interference, but the approach is limited by high data rate challenges, requiring further optimization.

In25, the authors suggest a deep-learning classification system for LG beam detection in turbulent and scattering surroundings with varying topological charges (l). An experimental setup demonstrates that the approach can accurately detect LG beam modes even in difficult circumstances. However, the interaction mechanism may affect the BER, requiring additional research for various contexts. A detailed efficiency analysis of the OAM Multiple-Input Multiple-Output (OAM-MIMO) multiplexing system is presented, showing its potential for parallel computation due to the orthogonality of OAM modes26. The system combines OAM multiplexing with MIMO technology for wireless communication. However, scalability issues and compatibility with current infrastructure and wireless standards need further development and optimization.

The authors improve Dense Wavelength Division Multiplexing (DWDM) transmission by optimizing optical amplifier characteristics, as discussed in27. Simulations show that using multiple Erbium-Doped Fiber Amplifiers (EDFA) over longer distances performs better than single amplifier systems, offering superior transmission qualities. However, scalability and infrastructure compatibility remain challenges. Unidirectional OAM waves transmit multiple channels to distributed receiving units, with pre-coding ensuring the interference-free transmission of OAM modes and allowing simultaneous delivery to different receivers, as demonstrated in28. Despite these advantages, challenges like mode alteration and antenna limitations remain. An amplitude optimization method corrects OAM beam distortion using a Single-intensity-measurement Phase Retrieval Algorithm (SPRA), which simplifies the system but requires prior knowledge of beam expansion and has high implementation costs29.

The authors proposed using OAM in mode division multiplexing (MDM) to increase optical communication capacity, focusing on Ring-Core Photonic Crystal Fiber (RC-PCF), as discussed in30. This innovative structure faces integration challenges with existing communication infrastructure due to compatibility concerns. In31, the authors extended OAM signal modulation for antenna transmission to multiconductor transmission lines. The OAM modal network enables multiple signal transmissions without mode conversion, although it doesn’t eliminate mode changes entirely. A high-speed system using meta-surface-based OAM multiplexing and THz signals was proposed, achieving up to 48 Gbps data rates32. However, intermodal crosstalk and limited transmission distance pose challenges. In33, the authors presented a meta-surface design for controlling non-diffractive OAM photons at 0.1 THz, with potential for high versatility but challenges in scalability and manufacturing. A GRU-based chaotic encryption method was proposed for small-scale datasets34, enhancing security but requiring further research to assess its robustness. Lastly, the OAM amplitude spectrum was used to measure the azimuth of a target, though environmental changes and resolution limits affect accuracy35.

A novel approach to enhance multiuser optical wireless network capacity, focusing on a communication system that supports several users is proposed36. It uses OAM and optical geometric transformation (OGT) to calculate the channel impulse response for OAM-carrying Laguerre-Gaussian beams using the angular spectrum method. The photonic receiver adjusts light shape via spatial dispersion and simultaneous processing. However, the fiber faces limitations with decreasing beam concentration and a restricted number of OAM modes. The terahertz (THz) frequency band PCF is a promising candidate for high-reliability optical fiber communication and THz OAM transmissions. In37, the authors explored the impact of load-aware boundary decisions on dynamic service delivery in elastic optical networks (EONs). They presented a mixed-integer linear programming (MILP) framework for routing, modulation, spectrum, and power assignment (RMSPA) while considering nonlinear interferences and noise. This method aims to reduce spectrum consumption and transmission impairments by selecting the optimal channel. However, the approach experiences significant outage levels, whereas ODA, despite high blocking rates, has no outages. A hybrid chaos-based 3D constellation scrambling mechanism is proposed to improve COOFDM system security and signaling efficiency38. Utilizing a 3D hexahedron signal constellation and a hybrid chaotic system increases encryption dimensions and error performance. However, the proposed method has a high bit error rate (BER) of over 40% when using the wrong decryption key, posing security vulnerabilities during data transmission.

The authors proposed a platform for simulating atmospheric turbulence with high precision, operating at a rate of one megahertz (MHz) and accuracy above 0.1 dB39. Despite its flexibility using the TCP/IP protocol, the system is limited by its performance in dynamic network conditions. This author investigates the fast-switching method of an optically controlled RF-OAM beam, simplifying systems compared to purely electrical methods40. While the optical switch offers quick response times, it has a maximum speed of 10 ms but can only record at a rate of 20 ms, limiting its ability to detect phase conversion.

The authors proposed41, an optical chaos communication scheme based on semiconductor lasers subject to intensity modulation optical injection for the secure transmission of quadrature amplitude modulation (QAM) messages. Two distinct chaotic sources with different modulation parameters are utilized to drive two receivers across separate channels. Simulation results showed successful recovery of a 20 GBaud 16-QAM signal over 120 km of standard single-mode fiber (SMF), while the 64-QAM signal was limited to 20 km. However, the study remains limited to maintaining synchronization under high turbulence and fiber impairments over longer distances. In42, dual-channel secure free-space optical communication is performed using two orbital angular momentum (OAM) modes multiplexed with all-optical chaotic modulation. The author achieved high-quality chaos synchronization and a transmission capacity using the wideband optical chaos generated in semiconductor lasers subject to intensity modulation optical injection. Results show that the communication security through physical-layer encryption is improved. However, the approach is limited by scalability challenges, particularly in maintaining stable OAM mode alignment and synchronization in dynamic atmospheric conditions. A summary of the literature survey is provided in Table 1.

Proposed secure and optimized optical transmission system

To address these aforementioned limitations and increase system performance, the proposed approach aims to enhance optical transmission systems’ efficiency, security, and reliability. It starts with an input light source as shown in Fig. 1, which passes through a HPSLM for generating the OAM signal. The data is then encoded using OFDM-QAM, which optimizes the transmission process. The system uses the TPAR-SHO algorithm to manage traffic effectively, analyzing and optimizing the routing. Atmospheric disturbances are mitigated using the OFDM-PID technique. For secure transmission, the QWS-ILCS method is applied. Finally, real-time network monitoring is performed through FFT combined with FOPM, ensuring continuous performance analysis. This comprehensive system addresses key challenges in optical communication, including traffic management, atmospheric interference, and data security. In an optical communication environment, optimizing the optical transmission approach aims to improve transmission efficiency. The following subsection consists of step-by-step actions that are taken as a part of this work.

OAM mode generation and data encoding

Initially, an HPSLM method is proposed to generate OAM beams by incorporating a spiral phase pattern into the incoming light. This method modifies the phase of light waves, enabling multiple OAM modes for high capacity and secure data delivery. OAM serves as a reliable multiplexing method due to its various modes, generated through a linear combination of odd and even modes with a phase shift of \(\pm {\varepsilon }/{2}\;\) for the same vector mode AB or BA variant which can be written as:

$$\begin{aligned} \begin{aligned}&OAM_{\pm k,l}^{\frac{M}{N}}=AB_{k+1,l}^{\text {even}}\pm iAB_{k+1,l}^{\text {odd}} \\&OAM_{\pm k,l}^{\frac{N}{M}}=BA_{k-1,l}^{\text {even}}\pm iBA_{k-1,l}^{\text {odd}} \end{aligned} \end{aligned}$$
(1)

Equation (1) represents the correlation between left-hand (M) and right-hand (N) circular polarization, corresponding to the spin angular momentum. The lowercase letters k and l denote the topological charge number (which represents the number of twists) and the axial index (which represents the number of zeros radially in the amplitude profile) of the OAM mode. The topological charge \(\ell\) of each OAM mode is defined as \(\ell =\pm (k+l)\), where k and l represent the spatial mode indices. For simulation purposes, we considered \(\ell\) values ranging from \(\pm 1\) to \(\pm 50\), resulting in 100 multiplexed OAM modes. Using a Spatial Light Modulator (SLM), complex light patterns are created by precisely altering phases and amplitudes. The incident light field \({{A}_{0}}\) is polarized at \(45{}^\circ\) relative to the horizontally oriented SLM in the x-direction. The SLM’s active zone is divided into two parts for dual modulation with phase volume displays (PVDs) referred to as \({{j}_{1}}\) and \({{j}_{2}}\). During initial modulation, the x-axis light component intensity is modulated as: \(\frac{1}{2}{{B}_{0}}\left\{ \exp (k{{\beta }_{1}}(x,y))+1 \right\}\), While the unmodulated y-axis component exhibits a Gaussian beam which can be expressed as: \(\frac{1}{\sqrt{2}}{{B}_{0}}\), After modulation, the reflected light passes through a polarizer at \(45{}^\circ\) to the x-axis, producing a light field distribution which is given in equation (2).

$$\begin{aligned} F(x,y)={{B}_{0}}\frac{\exp (k{{\beta }_{1}}(x,y))}{\sqrt{2}}\times \frac{1}{\sqrt{2}}+\frac{1}{\sqrt{2}}{{B}_{0}}\times \frac{1}{\sqrt{2}}=\frac{1}{2}{{B}_{0}}\left\{ \exp (k{{\beta }_{1}}(x,y))+1 \right\} \end{aligned}$$
(2)
Fig. 1
figure 1

Overall architecture of this research. Created using Microsoft Visio 2013.

The current standard operating procedure involves rotating the polarization state of the one-time modulated light by \(45{}^\circ\) relative to the working axis (x-direction) of the Spatial Light Modulators (SLMs). This rotation is accomplished using a half-wave plate, which leads to a second modulation of the wavefront. The spatial arrangement of the electromagnetic (EM) field of light, after this second manipulation by the SLM, can be mathematically represented in Eq. (3).

$$\begin{aligned} F(x,y)=\frac{1}{2}{{B}_{0}}\left\{ \exp (k{{\beta }_{1}}(x,y))+1 \right\} \exp \left( k{{\beta }_{1}}(x,y) \right) ={{B}_{0}}\cos \left( \frac{{{\beta }_{1}}}{2} \right) \exp \left\{ k\left( \frac{{{\beta }_{1}}+2{{\beta }_{2}}}{2} \right) \right\} \end{aligned}$$
(3)

The vector beam in Eq. (2) undergoes modulation in intensity and range along the x-component. The cosine term in Eq. (3) controls intensity distribution, while the compound number governs the phase range \({{{\beta }_{1}}+2{{\beta }_{2}}}/{2}\;\). By adjusting the phase profiles on the SLM, the amplitude and phase of the light beam are simultaneously calculated. This method effectively controls scalar fields with variable amplitude and phase, enabling manipulation of THz OAM beams with 59 modes over a frequency range of 0.80 THz (0.30 THz–0.90 THz).

The data encoding is achieved by integrating an OFDM-QAM into the OAM signal generation process along with an adaptive signal processing method. This allows the simultaneous transmission of several data streams by dividing the available bandwidth into orthogonal subcarriers. The parameter is dynamically adjusted based on the input signal and environmental conditions. This adaptive processing approach enhances the efficiency of the utilized resources and improves the quality of the generated OAM beams. Furthermore, to ensure the integrity and authenticity of the generated OAM signals, our approach maintains high OAM mode purities. The Inverse Fast Fourier Transform (IFFT), parallel-to-serial (P/S), and cyclic prefix (CP) are used. While performing the OFDM encoding an arbitrary waveform generator, namely the Tektronix AWG 70001, is employed to convert the OFDM data sequence into a digital signal. The SLM is kept about one meter away from the two identically specified photodiodes (PD, EOT ET-2030A). Both users had beam diameters of around 2 mm. The PDs are focused with the lenses in front of them before the reception. The two signals can be recorded at the same time when the photodetectors (PDs) are connected to a real-time oscilloscope (RTO), such as the Teledyne LeCroy 816ZI-B model. Following the OFDM decoding protocol, the decoding procedure includes one-tap equalization, serial-to-parallel (S/P), fast Fourier transform (FFT), elimination of CP, and signal to resample.

Figure 2 illustrates the Simplified OFDM system block diagram. The digital OFDM symbol waveform is generated by the time-domain output data produced through the IFFT operation, which converts frequency-domain input data into time-domain signals. In a cognitive OFDM system, the input bits are grouped and distributed to create data symbols. The modulation points are represented by complex numbers corresponding to the signal values. The signals used are typically QAM or BPSK symbols, which are common in single subcarrier systems. These complex signals are processed as if they were within the bandwidth range. The data is then mapped onto the time domain network using the IFFT module. The IFFT analyzes N input symbols at once, where N represents the total number of subcarriers in the system. Each of these N symbols have a time duration of T seconds. It’s important to note that the IFFT produces N orthogonal sinusoids, each with different frequencies, with the lowest frequency corresponding to direct current (DC). The input signals include values that represent the modulation points, defining the speed and amplitude of each sinusoid for every subcarrier. The sum of these N sinusoids form the IFFT output. Therefore, the IFFT block effectively divides data into N parallel subcarriers, and the collection of N output samples from the IFFT constitute the OFDM signal. The time-domain signal produced by the IFFT is transmitted over a wireless connection after further processing.

Fig. 2
figure 2

Simplified OFDM system block diagram.

At the receiver, the input signal is processed and converted into the frequency domain using an FFT unit, allowing for the recovery of the original information bits. This step involves scaling two separate normalized power 4-QAM data sequences with different powers, and then adding them together. Since each of the two users undergoes two decoding operations, four SNR values need to be determined: SNR1st user1, SNR2nd user1, SNR1st user2, and SNR2nd user2. As data-2 has higher encoding power, the first step in the decoding process is to consider the received data-2 for both users. Maximum Likelihood (ML) detection is applied, treating data-1 as noise. Mathematically this process can be represented as:

$$\begin{aligned} & \sqrt{{{Q}_{\text {large}}}}{{\hat{y}}_{2,\text {user }1}}(p)={{\arg }_{m\in \{1,2,3,4\}}}\max f\left( {{a}_{\text {user }1}}(p)|\sqrt{{{Q}_{\text {large}}}}{{y}_{2,m}} \right) \end{aligned}$$
(4)
$$\begin{aligned} & \sqrt{{{Q}_{\text {large}}}}{{\hat{y}}_{2,\text {user }2}}(p)={{\arg }_{m\in \{1,2,3,4\}}}\max f\left( {{a}_{\text {user }2}}(p)|\sqrt{{{Q}_{\text {large}}}}{{y}_{2,m}} \right) \end{aligned}$$
(5)

Where the \(p-\text {th}\) received signal for users 1 and 2, after processing are denoted by \({{a}_{\text {user }1}}(p)\) and \({{a}_{\text {user }2}}(p)\) respectively. The set \(\sqrt{{{Q}_{\text {large}}}}{{y}_{2,m}}\) where \(m\in \{1,2,3,4\}\) denotes the 4-QAM scaled normalized signals using \({{Q}_{\text {large}}}\). The p-th approximated data for users 1 and 2 are represented by \({{\hat{y}}_{2,\text {user }1}}(p)\), and \({{\hat{y}}_{2,\text {user }2}}(p)\), and these follow the Gaussian probability density function (PDF), represented by f(x). The initial decoding’s SNR distribution is estimated as:

$$\begin{aligned} & \text {SNR}_{\text {user }1}^{\text {1st}}=\frac{1}{N}\sum \limits _{z}{\frac{{{\left| \sqrt{{{Q}_{\text {large}}}}{{y}_{2,m}} \right| }^{2}}}{{{\left| {{a}_{\text {user }1}}(p)-\sqrt{{{Q}_{\text {large}}}}{{y}_{2,m}} \right| }^{2}}}} \end{aligned}$$
(6)
$$\begin{aligned} & \text {SNR}_{\text {user 2}}^{\text {1st}}=\frac{1}{N}\sum \limits _{z}{\frac{{{\left| \sqrt{{{Q}_{\text {large}}}}{{y}_{2,m}} \right| }^{2}}}{{{\left| {{a}_{\text {user 2}}}(p)-\sqrt{{{Q}_{\text {large}}}}{{y}_{2,m}} \right| }^{2}}}} \end{aligned}$$
(7)

Where in Eq. (7), N is the data length, and \(\sqrt{{{Q}_{\text {large}}}}{{y}_{2,m}}\) scales the data-2 utilizing the \(i-th\) representation. Next, the successive interference cancellation (SIC) is run using \(\sqrt{{{Q}_{\text {large}}}}\)to estimate the data-2 which is then used to recover data-1. This implies that the approximated data-2 sequence is subtracted from the signals that are received for users 1 and 2. This can be expressed as:

$$\begin{aligned} & \sqrt{{{Q}_{\text {small}}}}{{\hat{y}}_{1,\text {user}1}}(p)=\underset{m\in \{1,2,3,4\}}{\mathop {\arg \max }}\,f\left( {{a}_{\text {user}1}}(p)\sqrt{{{Q}_{\text {small}}}}{{{\hat{y}}}_{2,\text {user}1}}(p),\sqrt{{{Q}_{\text {small}}}}{{y}_{1,m}} \right) \end{aligned}$$
(8)
$$\begin{aligned} & \sqrt{{{Q}_{\text {small}}}}{{\hat{y}}_{1,\text {user}2}}(p)=\underset{m\in \{1,2,3,4\}}{\mathop {\arg \max }}\,f\left( {{a}_{\text {user}2}}(p)\sqrt{{{Q}_{\text {small}}}}{{{\hat{y}}}_{2,\text {user}2}}(p),\sqrt{{{Q}_{\text {small}}}}{{y}_{1,m}} \right) \end{aligned}$$
(9)

In Eqs. (8) and (9), where \(\sqrt{{{Q}_{\text {small}}}}{{y}_{1,m}}\) for \(m\in \{1,2,3,4\}\) represents the set of standardized 4 QAM scales with data for users 1 and 2 corresponding to \(\sqrt{{{Q}_{\text {small}}}}{{\hat{y}}_{1,\text {user}1}}(p)\) and \({{\hat{y}}_{1,\text {user}1}}(p)\). The second decoding SNR distributions \(\text {SNR}_{\text {user}1}^{\text {2nd}}\) and \(\text {SNR}_{\text {user 2}}^{\text {2nd}}\), are calculated using the Eqs. (10) and (11).

$$\begin{aligned} & \text {SNR}_{\text {user}1}^{\text {2nd}}=\frac{1}{N}\sum \limits _{z}{\frac{{{\left| \sqrt{{{Q}_{\text {small}}}}{{y}_{1,m}} \right| }^{2}}}{{{\left| {{a}_{\text {user}1}}(p)-\sqrt{{{Q}_{\text {small}}}}{{{\hat{y}}}_{2,\text {user}1}}(p)-\sqrt{{{Q}_{\text {small}}}}{{y}_{1,m}} \right| }^{2}}}} \end{aligned}$$
(10)
$$\begin{aligned} & \text {SNR}_{\text {user2}}^{\text {2nd}}=\frac{1}{N}\sum \limits _{z}{\frac{{{\left| \sqrt{{{Q}_{\text {small}}}}{{y}_{1,m}} \right| }^{2}}}{{{\left| {{a}_{\text {user2}}}(p)-\sqrt{{{Q}_{\text {small}}}}{{{\hat{y}}}_{2,\text {user2}}}(p)-\sqrt{{{Q}_{\text {small}}}}{{y}_{1,m}} \right| }^{2}}}} \end{aligned}$$
(11)

Where N is the data length and (\(\sqrt{{{Q}_{\text {small}}}}\)) represents the scale data-1 symbol. During the data transfer, the power ratio (PR), which is the ratio of the larger signal (\(\sqrt{{{Q}_{\text {large}}}}\)) to the smaller signal (\(\sqrt{{{Q}_{\text {small}}}}\)), remains constant, as indicated in the channel estimate. During the encoding process, the data sequences for data-1 and data-2 are generated according to the \(\text {SN}{{\text {R}}^{\text {2nd}}}\)and \(\text {SN}{{\text {R}}^{\text {1st}}}\) distributions of \(\text {user }1\) and \(\text {user 2}\), respectively. These sequences are then encoded using the (\(\sqrt{{{Q}_{\text {small}}}}\)), and (\(\sqrt{{{Q}_{\text {large}}}}\)). During decoding, the sequences are obtained using Eqs. (12) and (13).

$$\begin{aligned} & \sqrt{{{Q}_{\text {large}}}}{{\hat{y}}_{2,\text {user}1}}(p)=\underset{m}{\mathop {\arg \max }}\,f\left( {{a}_{\text {user}1}}(p),\left| \sqrt{{{Q}_{\text {large}}}}{{y}_{2,m}} \right| \right) \end{aligned}$$
(12)
$$\begin{aligned} & \sqrt{{{Q}_{\text {small}}}}{{\hat{y}}_{\text {1,user1}}}(p)=\underset{m\in \{1,2,3,4\}}{\mathop {\arg \max }}\,f\left( {{a}_{\text {user1}}}(p)-\sqrt{{{Q}_{\text {small}}}}{{{\hat{y}}}_{\text {2,user1}}}(p),\left| \sqrt{{{Q}_{\text {small}}}}{{y}_{1,m}} \right| \right) \end{aligned}$$
(13)
$$\begin{aligned} & \sqrt{{{Q}_{\text {large}}}}{{\hat{y}}_{2,\text {user}2}}(p)=\underset{m\in \{1,2,3,4\}}{\mathop {\arg \max }}\,f\left( {{a}_{\text {user}2}}(p),\left| \sqrt{{{Q}_{\text {large}}}}{{y}_{2,m}} \right| \right) \end{aligned}$$
(14)

Where \({{y}_{1,m}}\) and \({{y}_{2,m}}\) are sets of values whose cardinalities depend on the distributions of \(\text {SNR}_{\text {user}1}^{\text {2nd}}\) and \(\text {SNR}_{\text {user2}}^{\text {1st}}\), respectively.

Traffic pattern analysis using TPAR and SHO

The TPAR-SHO algorithm analyzes optical transmission traffic, dynamically adjusting routing based on current and projected conditions. This adaptive method mitigates congestion and enhances network performance.

  1. (a)

    Traffic prediction assisted routing

The TPAR algorithm aims to balance the loads, preventing congestion and ensuring resource allocation for handling traffic demands. Routing through the shortest paths in EONs and using optimal modulation formats, maximizes resource utilization and minimizes the total amount of trips.

In Algorithm 1 three variables are initialized. The first variable, crowded links, is a set that includes the overloaded network links. The second variable, not redirected origin-destination (OD), consists of all potential network OD pairs that are responsible for rerouting. The third variable is a logical variable that determines when the algorithm should terminate. Rerouting can be performed because of two reasons. The first reason is to identify solutions that closely match the ideal solution, which was determined in the first phase using j-shortest paths; and the second reason is to avoid unnecessary iterations during the algorithm’s execution. In the next step, TPAR calculates the expected load on each network link using the traffic map provided by the traffic prediction and uncertainty load allocation (TPULA) method and the routing table. The traffic map, generated by the IP layer, represents the highest data transfer rates estimated between network OD pairs over a given timeframe. However, this map does not include the GB limit between traffic requests and the type of transmission being used, both of which are essential for accurately estimating the load on each network connection. To refine this estimate, the highest anticipated load value in the traffic map is divided by a specific amount of traffic demands, denoted as \(num\_re{{q}_{pq}}\), which is calculated as:

$$\begin{aligned} num\_re{{q}_{pq}}=\frac{pred\_loa{{d}_{pq}}}{mean\_re{{q}_{pq}}} \end{aligned}$$
(15)

Let p represent the origin node and q represent the destination node \(pred\_load\)is the predicted upper limit of the OD pair’s traffic load, and \(mean\_req\) seen for that particular OD pair within the given time frames is the typical bitrate for each request. The value is used to calculate the expected load, expressed in frequency slots (FSs), on each network connection.

$$\begin{aligned} {{L}_{{{e}_{c}}}}=\sum \limits _{cepat{{h}_{o{{d}_{pq}}}}}{\left[ \frac{mean\_re{{q}_{pq}}}{FS\cdot {{M}_{pq}}}+GB \right] }\cdot num\_re{{q}_{pq}} \end{aligned}$$
(16)

The \(pat{{h}_{o{{d}_{pq}}}}\) refers to the primary paths in the routing table that include the link a. FS represents the data of a particular time slot for frequencies, GB represents the bandwidth of the security band, and M represents the modulation type utilized. In this study, the value of M is assumed to range from 1 to 5, representing the modulation formats BPSK, quadrature phase shift keying (QPSK), 8-QAM, 16-QAM, and 32-QAM, respectively. Next, the predicted load is compared to a predetermined threshold number to determine whether a network connection is overcrowded. To stop crowded connections from being used for routing in the next rounds, the technique removes them from the actual network topology. Algorithm 2 displays the link congestion calculation procedure in its entirety. After calculating predicted network link congestion, the algorithm stops if no links are congested. Otherwise, it initiates decongestion by rerouting pathways with overloaded connections. Congested links are first sorted by anticipated demand in descending order. Then OD pairs are identified that are not yet rerouted. These pairs are added to a candidate OD set, sorted by distance. To meet the continuity and contiguity requirements of EONs, priority is given to rerouting distant OD pairs along less crowded channels, ensuring sufficient resources for their traffic demands. Once an alternative path is found, it is added to the routing table for the OD pair and set as the primary path.

Alternative routes, identified using the K-paths approach, are evaluated for rerouting. If a feasible path is found, the OD pair is removed from the candidate set, and the routing database is updated if the busiest link’s load decreases. The process ends when rerouting options are exhausted or congestion is resolved. The Primary TPAR Algorithm begins by analyzing traffic predictions and network topology to identify potential paths (k-paths) between origin-destination (OD) pairs. The algorithm then monitors link traffic levels and identifies congested connections. For each congested link, it evaluates candidate OD pairs and searches for alternative viable routes. If a better path is found, it updates the routing table and reduces congestion; otherwise, it continues searching. The process repeats until congestion is minimized or no further improvements are possible.

Algorithm 1
figure a

Primary TPAR algorithm.

Algorithm 2
figure b

Methodology for calculating link congestion.

  1. (b)

    Spotted hyena optimizer (SHO)

The SHO algorithm, inspired by hyenas’ natural hunting behavior, is a novel meta-based heuristic optimization method used for network reconfiguration. Unlike other methods, SHO’s unique model structure enables spectrum defragmentation (DF) by consolidating fragmented frequency slots (FSs) into a continuous set. Spectrum fragmentation occurs when discontinuous FSs cannot accommodate additional links, leading to inefficient utilization. SHO-DF mimics hyenas’ precise targeting and surrounding of prey, using mathematical formulas to represent this behavior and achieve optimal solutions.

$$\begin{aligned} \overrightarrow{{{D}_{f}}}=\left| \vec {N}.{{L}_{s}}(y)-\overrightarrow{{{L}_{y}}} \right| \end{aligned}$$
(17)

The distance (D)between the connectivity and the free FSs in the links is evaluated using the coefficient vector (M) and location vectors (Lp) for the target position and hyenas’ positions. Spotted hyenas rely on group collaboration and precise prey location, with the leading agent guiding the search. Following agents cluster around the leader, forming a network to retain and update optimal solutions using equations like Eqs. (18) and (19).

$$\begin{aligned} & \overrightarrow{{{D}_{f}}}=\left| \vec {N}\cdot {{L}_{p}}-\overrightarrow{{{L}_{q}}} \right| \end{aligned}$$
(18)
$$\begin{aligned} & Gh={{L}_{q}}+{{L}_{q+1}}+\ldots +{{L}_{q+M}} \end{aligned}$$
(19)

The initial best hyena position is represented by Lp, while other positions are \({{L}_{q}}\), and M denotes the spotted hyena population. The optimal solution in the search area is h, where N optimal solutions are placed. The SHO method adjusts exploration agents positions to move closer to the target, emulating hyenas’ hunting patterns by observing clusters and selecting prey. A random number N (less than or greater than 1) diversifies the search. To tackle fragmentation, the algorithm considers external fragmentation, denoted as \({{F}_{Et}}\), in Eq. (20), reducing blockage likelihood effectively.

$$\begin{aligned} {{F}_{Et}}=1-\frac{\sum {T{{F}_{{{s}_{s}}}}}}{\sum {F{{F}_{{{s}_{s}}}}}} \end{aligned}$$
(20)

The total number of frequency slots in the largest frequency block is represented by \(T{{F}_{{{s}_{s}}}}\) while \(F{{F}_{{{s}_{s}}}}\) denotes the number of available slots. A frequency block is a cluster of adjacent unused spectrum slots. The efficiency factor \({{F}_{Et}}\) equals 1 when bandwidth is distributed evenly across adjacent unused slots, and approaches zero when gaps consolidate into a single location. The holding time, which tracks the interval between a connection’s entry and exit, is crucial for reconfiguring the network and converting \({{F}_{{{s}_{s}}}}\) for active links. This duration is determined using Eq. (21).

$$\begin{aligned} ht=\left( \sum \limits _{i=0}^{F-1}{\frac{\left| {{t}_{j}}+1-{{t}_{j}} \right| }{\sigma }} \right) \times \left( \frac{Cp{{c}_{h}}(1)-Cpc(1)}{Cpc(1)}+1 \right) \end{aligned}$$
(21)

Frequency Slots in a connection are denoted as \({{F}_{{{s}_{s}}}}\), with the holding time of a link represented by ht. The duration of the ith \({{F}_{s}}\) is marked by \({{t}_{j}}\), and the holding time difference between two adjacent \({{F}_{{{s}_{s}}}}\) in a connection can be written as \(\left| {{t}_{j}}+1-{{t}_{j}} \right|\). The average connection duration is symbolized by \(\sigma\), while Cpc indicates the dimension of preserved contiguous slots, with \(Cp{{c}_{h}}(1)\)representing pre-assignment connections. The period when all current network connections are closed shows the holding data period. Algorithm 3 represents the pseudocode of the SHO algorithm which begins by initializing network parameters and activating light paths. It iteratively adjusts the positions of “spotted hyenas” (representing candidate solutions) using mathematical equations to explore the solution space. During each iteration, it evaluates occupied frequency slots, identifies suboptimal connections, and applies a Hop-tuning technique to reassign resources. The loop continues until the maximum number of iterations is reached or an optimal solution is found.

Algorithm 3
figure c

SHO.

The termination condition is met when fragmentation reduction reaches a satisfactory level or a predefined number of iterations is completed. Inspired by the cooperative hunting behavior of spotted hyenas, the SHO algorithm adapts to traffic-varying fragmentation in EONs.

QWS structure-based secure data transmission

After analyzing traffic patterns, the proposed QWS-ILCS approach enhances the security of the optical transmission system. By detecting unauthorized access and maintaining BER within acceptable limits, the method significantly improves the system’s security and reliability.

  1. (a)

    Quantum well structure (QWS)

Quantum wells are semiconductor structures that confine carriers (electrons and holes) in one dimension, enabling discrete energy states and efficient photon emission. Injection locking synchronizes the frequency and phase of a slave laser with a master laser to ensure coherent light output. The QWS-ILCS combines these technologies to enable high-speed optical communication with large data capacity and minimal noise. Strained quantum wells with optimized potential barrier configurations are incorporated into the active region of SOA to achieve low polarization and wide wavelength ranges. This approach enhances the light cavity sub-band power while reducing frequency mismatch with the heavy-hole sub-band. The spectrum boundaries for heavy and light hole channels at the Brillouin zone center are defined mathematically for \(k=0\).

$$\begin{aligned} & {{E}_{HH}}(j=0)={{b}_{v}}\left( {{\alpha }_{x}}+{{\alpha }_{y}}+{{\alpha }_{z}} \right) -\frac{a}{2}\left( {{\alpha }_{x}}+{{\alpha }_{y}}-2{{\alpha }_{z}} \right) \end{aligned}$$
(22)
$$\begin{aligned} & {{E}_{HH}}(j=0)={{b}_{v}}\left( {{\alpha }_{x}}+{{\alpha }_{y}}+{{\alpha }_{z}} \right) +\frac{a}{2}\left( {{\alpha }_{x}}+{{\alpha }_{y}}-2{{\alpha }_{z}} \right) \end{aligned}$$
(23)

In Eq. (22) the variable b represents the deformation potential of tangential strain while \({{b}_{v}}\) is the deformation potential of the conduction band. The energy at the boundary of the electron conduction band in a band structure can be written as:

$$\begin{aligned} {{E}_{c}}(j=0)={{E}_{q}}+{{b}_{c}}\left( {{\alpha }_{x}}+{{\alpha }_{y}}+{{\alpha }_{z}} \right) \end{aligned}$$
(24)

In Eq. (23) the variables \(b_v\) represent the deformation potentials of the conduction band while in Eq. (24) The variables \({{\alpha }_{x}},{{\alpha }_{y}}\), and \({{\alpha }_{z}}\) represent the amounts of lattice misfit. In addition, the net transition energy from the conduction band to the heavy hole and light hole bands can be represented as:

$$\begin{aligned} {{E}_{c-HH}}= & {{E}_{q}}+\left( {{b}_{c}}-{{b}_{v}} \right) \left( {{\alpha }_{x}}+{{\alpha }_{y}}+{{\alpha }_{z}} \right) -\frac{a}{2}\left( {{\alpha }_{x}}+{{\alpha }_{y}}-2{{\alpha }_{z}} \right) , \end{aligned}$$
(25)
$$\begin{aligned} {{E}_{c-HH}}= & {{E}_{q}}+\left( {{b}_{c}}-{{b}_{v}} \right) \left( {{\alpha }_{x}}+{{\alpha }_{y}}+{{\alpha }_{z}} \right) +\frac{a}{2}\left( {{\alpha }_{x}}+{{\alpha }_{y}}-2{{\alpha }_{z}} \right) \end{aligned}$$
(26)

In Eq. (25), \({{E}_{q}}\) is the band gap when there is no tension. The preceding section explains the impact of strain on quantum well energy bands. To meet the frequency demands of \(C+L\) band communications, further enhancements to the quantum well structure are necessary. These enhancements should either shift the quantum well emission spectrum toward longer wavelengths or expand the gain spectrum to encompass the entire \(C+L\) band.

  1. (b)

    Injection locking synchronization (ILCS)

Injection locking is the process of synchronizing the frequency of a slave laser with a master laser. In this technique, the slave laser’s light beam is frequency- and phase-locked to the master laser’s signal. Figure 3 shows the setup of an intensity modulation (IM) optical injection dual-channel chaotic communication system in which a single-mode distributed feedback (DFB) semiconductor laser acts as the master laser (ML). After passing through Mach-Zehnder modulators (MZMs) and variable optical attenuators (VOAs), its output is directed into two single-mode DFB lasers, TL1 and TL2. These signals are modulated by MZM3 and MZM4 using 16-QAM (Message 1) and 64-QAM (Message 2) respectively, and amplified by an EDFA before transmission over a fiber link. The receiver includes two photodetectors (PD1 and PD2). Message 1 is decoded by comparing the outputs of PD1 and PD2, while Message 2 is decoded by comparing the output of PD3 and PD4.

Fig. 3
figure 3

Dual-channel chaotic communication system scheme with IM optical injection.

The Quantum Well Structure in a QWS-ILCS system generates photons with distinct energy levels, while injection locking ensures the stability and coherence of these photons. The key advantages include fast transmission, where high-speed photon emission is achieved through rapid electron-hole recombination in quantum wells and higher data speeds facilitated by injection locking, which synchronizes the emitted photons and reduces jitter. High-capacity transmission is enabled by the discrete energy levels in quantum wells, allowing for wavelength-division multiplexing (WDM) to emit photons at different wavelengths, increasing bandwidth. Injection locking also minimizes crosstalk by maintaining the coherence of multiple wavelengths. Additionally, the system offers low-profanity functionality, as quantum wells emit photons with small line widths, reducing noise, while injection locking further stabilizes the output, reducing phase noise and frequency variations. By combining the discrete energy levels of quantum wells for efficient photon emission with injection-locking stability for coherent light output, QWS-ILCS becomes a promising solution for advanced optical networks, which enhance optical data transfer with increased speed, bandwidth, and reduced noise.

The flowchart in Fig. 4 illustrates the integrated operation of the QWS and ILCS process. It begins with the QWS process, where semiconductor materials confine carriers to create discrete energy states, optimized via strain to enable efficient photon emission. Simultaneously, the ILCS process starts with a master laser, modulates and injects the signal into slave lasers to achieve coherent, phase-locked output. These two systems merge in the QWS-ILCS integration, combining QWS’s high-speed photon generation with ILCS’s stability to enable WDM, noise reduction, and enhanced optical network performance. The final output feeds into a dual-channel chaotic communication system.

Fig. 4
figure 4

Flowchart of QWS-ILCS.

AT effect mitigation

The OFDM-PID method is proposed to protect the optical transmission system from the AT effect by adaptively adjusting transmission parameters based on atmospheric conditions, ensuring signal quality. Atmospheric turbulence causes random fluctuations in the transmission medium’s refractive index, leading to beam wandering, scintillation, and phase distortions in FSO systems. OFDM-PID controllers are widely used to adjust system parameters dynamically in response to such disturbances43,44. In optical communication, an OFDM-PID controller can regulate variables like beam alignment, power levels, and timing offsets to compensate for turbulence-induced errors. The proportional term reacts to the immediate error, the integral term corrects the cumulative offset and the derivative term anticipates future trends, enabling fast and stable corrections. This adaptive behaviour helps maintain signal integrity and synchronization despite rapid channel variations. Previous studies confirmed PID’s effectiveness in stabilizing optical links under dynamic turbulence conditions by reducing signal variance and improving BER performance. OFDM-PID enables efficient data transmission under dynamic network conditions, allowing for faster transfer of large data volumes. The OFDM frame structure is used which includes training symbols, using Long Training Symbols (LTSs) to estimate Sampling Clock Synchronization (SCO) and extract crucial information. The LTS contains fixed values applied to each subcarrier, and for N subcarriers (indices 0 to \((N-1)\)), the LTS can be represented as:

$$\begin{aligned} L=[k(0),\ k(1),\ \ldots ,\ k(N-1)] \end{aligned}$$
(27)

In Eq. (27), the variable L denotes the value of LTS appended to the ith subcarrier. Similarly, the received Long-Term Support (LTS) can be written as:

$$\begin{aligned} \hat{L}=[\hat{k}(0),\ \hat{k}(1),\ \ldots ,\ \hat{k}(N-1)] \end{aligned}$$
(28)

The association technique is used to determine the received LTS phase offsets using Eq. (29), while the phase offset is represented by the matrix \(\rho\) which are given in equation (30).

$$\begin{aligned} \mu (j)= & \frac{k(j)\cdot k(j)}{\Vert k(j)\Vert \Vert k(j)\Vert } \end{aligned}$$
(29)
$$\begin{aligned} \rho= & [\mu (0),\ \mu (1),\ \ldots ,\ \mu (N-1)] \end{aligned}$$
(30)

Based on the theoretical derivation presented in Eq. (30), it can be concluded that is directly proportional to the subcarrier index. Mathematically this relation can be represented as:

$$\begin{aligned} \mu (n)=s(n-1) \end{aligned}$$
(31)

The variable s represents the slope. According to Equation 6, if there is no phase rotation, the value of sshould be zero. Thus, the goal slope as \(s=0\) can be established. Since we can measure the error slope from each received OFDM frame, it can be denoted as:

$$\begin{aligned} \hbar =s-\hat{s} \end{aligned}$$
(32)
  1. (a)

    Proportional-integral-derivative controller

In FSO systems, the PID controller dynamically modifies system parameters to counteract air turbulence effects. The master clock is considered to have a fixed frequency due to the high signal rate of the Optical OFDM (OOFDM) system, allowing us to ignore clock variations at the broadcaster. According to PID theory, the error in adjusting the object y(t)to a target value \(\bar{y}\) can be represented as \(\hbar (t)\) which can be used to obtain the output z(t).

$$\begin{aligned} & \hbar (t)=y(t)-\bar{y} \end{aligned}$$
(33)
$$\begin{aligned} & z(t)={{K}_{p}}\cdot \hbar (t)+{{K}_{i}}\int _{0}^{t}{\hbar }(v)dv+{{K}_{d}}\frac{d\hbar (t)}{dt} \end{aligned}$$
(34)

In Eq. (34), \({{K}_{p}},{{K}_{i}}\) and \({{K}_{d}}\) are specified and adjusted for the particular system. Since our method operates as a discontinuous system, the PID controller must be converted to its discrete form. Therefore, y(t) is replaced with y(n), and Eq. (34) is modified as:

$$\begin{aligned} h(n)=y(n)-\bar{y} \end{aligned}$$
(35)

Furthermore, the discrete representation of Eq. (35) can be expressed as:

$$\begin{aligned} y(n)={{K}_{p}}\cdot \hbar (n)+{{K}_{i}}\sum \limits _{u=0}^{n}{\hbar }(u)+{{K}_{d}}[\hbar (n)-\hbar (n-1)] \end{aligned}$$
(36)

In our systems, y(n) is updated once per frame, as per Eq. (36). The OOFDM system uses intensity modulation and direct detection (IMDD), with a received optical power of -11 dBm. The arbitrary waveform generator (AWG) operates at 2 GHz with 1 ppm precision, sending an OFDM message repeatedly. At the receiver, the ADC samples the received symbol, synchronized with the Voltage-Controlled Oscillator (VCO), set to 2 GHz. The OOFDM technology achieves a data throughput of 2.5 Gbps with a single wavelength. Matlab and Simulink are used for frame coordination, slot coordination (SCO), channel component estimation, and demodulation, including the FFT.

  1. (b)

    Measuring and documenting AT Channels

To accurately replicate the AT channels, it is necessary to do measurements and collect data. Fig. 5 depicts the schematic design of the equipment used to measure and record data from the AT channel. The system comprises a laser, a transmitter, a receiver, a high dynamic range photodetector, a recorder, and an atmospheric channel.

Fig. 5
figure 5

Diagrammatic representation of the system for measuring and documenting AT. Created using Microsoft PowerPoint 2016.

The diagram compares the system’s framework to that of an FSO communication system. The chaotic atmospheric conditions monitoring system works as follows: The AT channel receives a light signal from the transmitters, which is then converted into an electrical signal by a high dynamic range photodetector. After processing, the signal is saved locally. For consistency with the FSO system, the optical features of both the transmitter and receiver must match the FSO system being studied.

Optimizing optical switching for real-time network monitoring

We enhance optical switching to the nanosecond range for real-time monitoring. The FFT-FOPM tool is proposed for effective signal processing and network monitoring which are discussed below.

  1. (a)

    Fast Fourier transform (FFT)

FFT is an effective approach for calculating the Discrete Fourier Transform (DFT) and its inverse, widely utilized in optical transmission for spectrum analysis, performance monitoring, and signal processing. It helps identify issues like distortion, noise, and signal degradation. FFT is also utilized in modulation/demodulation and optical performance monitoring (OPM), tracking key performance indicators (KPIs) such as chromatic dispersion (CD), polarization mode dispersion (PMD), and optical signal-to-noise ratio (OSNR). FFT algorithm steps for optical transmission:

  1. (1).

    Sampling: A rate of sampling that meets the Nyquist criteria is employed to the continuous optical signal. The sampled signal y(n)is the optical signal represented in discrete time.

  2. (2).

    Windows: Before applying FFT, the sampled signal is often multiplied by a window function (such as a Hanning, Blackman, or Hamming window) to lower spectral leakage.

$$\begin{aligned} {{y}_{z}}(n)=y(n)\cdot z(n) \end{aligned}$$
(37)

where, z(n) indicates the window function.

  1. (3).

    FFT computation: To calculate the frequency-domain representation, the FFT algorithm is used for the windowed signal.

    $$\begin{aligned} X(j)=\sum \limits _{n=0}^{N-1}{{{y}_{z}}}(n)\cdot {{e}^{-{{k}_{2}}\sigma jn/N}} \end{aligned}$$
    (38)
  2. (4).

    Phase calculation and magnitude: The FFT complex output helps to compute the magnitude and phase of the frequency.

    $$\begin{aligned} \left| X(j) \right| =\sqrt{{{\left( \operatorname {Re}(X(j)) \right) }^{2}}+{{\left( \operatorname {Im}(X(j)) \right) }^{2}}}\ \end{aligned}$$
    (39)
  3. (5).

    Extraction of performance metrics: The FFT output is used to extract important performance parameters for analysis and monitoring purposes. One way to determine the OSNR is to compare the signal-to-noise ratio in the frequency domain.

  4. (6).

    IFFT: Converting a signal from the frequency domain to the time domain using the IFFT is a common practice in certain applications.

    $$\begin{aligned} y(n)=\frac{1}{N}\sum \limits _{j=0}^{N-1}{X}(j)\cdot {{e}^{-{{k}_{2}}\sigma jn/N}}\ \end{aligned}$$
    (40)
  5. (b)

    FOPM tool.

To handle growing traffic in data centers, networks relying on electrical switching face bandwidth bottlenecks and inefficiency. Optical switching is a promising solution due to its large bandwidth, low power, and cost efficiency. However, nanosecond switch control, optical buffers for packet congestion, and clock/data recovery pose challenges. Fiber performance monitoring (FPMT) detects fiber flaws by generating an experimental signal sent with traffic flow. A laser diode (LD) transmits FPMT signals through a separate frequency band to avoid interference with C-band signals. A hybrid filter (HF) separates send and receive path signals. After filtering, the photodetector (PD) converts the reflected signal into a digital stream for processing. Signal back monitoring is based on energy levels, receiving time, bit errors, and signal shape. The Optical Supervisory Channel (OSC) manages fiber networks, operating at 2 Mbps over a 32-slot frame (1490-1510 nm). The Small Form Factor Pluggable (SFP) module handles signal conversion, while transmit/receive management ports handle signals. The system accounts for a 1 dB insertion loss from fiber interface boards (FIUs). The FPMT detection circuit integrates with the OSC board to transmit test signals over the same fiber in the optical DWDM network, as shown in Fig. 6.

Fig. 6
figure 6

FOPM tool.

Simulation results and comparative analysis

This section describes the experimental investigation that is done to assess the proposed. This part is divided into three subsections: simulation setup, comparative analysis, and research summary. To simulate the proposed research method, Matlab-R2023a is utilized. This tool is efficient and provides all specifications for the proposed technique. Table 2 indicates the Simulation parameters.

Table 2 Simulation parameters.

The proposed approach is compared with existing methods OGT36 and DWDM27, to evaluate its effectiveness through performance metrics. The performance is evaluated in terms of OAM modes with respect to the BER, the network throughput obtained over time, data integrity rate concerning the transmission distance, signal quality in different atmospheric turbulence conditions, and blocking probability over time. The relationship between the BER and OAM is using an empirical theoretical model which represents the BER as a function of the number of OAM modes N. This relationship is often exponential or logarithmic due to increasing complexity and disruption which is written in Eq. (41).

$$\begin{aligned} BER(N)=\gamma {{e}^{\delta N}} \end{aligned}$$
(41)

where, \(\gamma\) and \(\delta\) are fixed values determined by system properties like the quantum well, transmission medium, and detection mechanism. The x-axis represents the number of OAM modes, and the y-axis represents the corresponding BER. As the number of OAM modes increases, the BER exhibits an exponential rise, highlighting the challenge of maintaining a low BER with more modes.

Fig. 7
figure 7

No of OAM mode vs. BER.

Figure 7 compares the number of OAM modes vs. BER for the proposed system and existing methods, with the BER values now represented on a logarithmic scale. The proposed system achieves a log-scale BER of \(3.13 \times 10^{-4}\) at 100 OAM modes, while DWDM and OGT achieve values of \(3.71 \times 10^{-4}\) and \(3.52 \times 10^{-4}\), respectively, indicating the effectiveness of the proposed method in reducing BER. On average, the proposed method shows a 17.63% improvement compared to DWDM and a 13.07% improvement compared to OGT. This improved BER performance is due to the integration of an HPSLM with OFDM-QAM, which enhances the precision and efficiency of OAM mode generation. This combination reduces phase distortion and improves mode purity, resulting in a lower log-scale BER. The network throughput denoted as T(t), represents the amount of data sent across the network at a given time (t). The variable T(t) can be written as:

$$\begin{aligned} T(t)={{T}_{0}}\cdot {{e}^{-\gamma t}} \end{aligned}$$
(42)

where, the initial throughput \({{T}_{0}}\) is \(t=0\), and the attenuation coefficient \(\gamma\) is determined by the quantum well characteristics and OAM light. The performance of the proposed system is evaluated in terms of network throughput over time as shown in Fig. 8. The x-axis represents the time (t) in seconds, while the Y-axis shows network throughput in Mbps. The graph illustrates the fluctuation in throughput over time, represented by T(t), which indicates the amount of data transmitted at a given time (t). Figure 8 demonstrates that the proposed method surpasses both DWDM and OGT in network throughput, achieving 0.96 Mbps, compared to 0.79 Mbps for DWDM and 0.90 Mbps for OGT. This shows that the proposed approach is more effective in terms of network throughput. The improved throughput achieved by the TPAR-SHO algorithm, compared to DWDM and OGT, is due to its adaptive and predictive routing strategies. By dynamically adjusting routes based on real-time and projected traffic conditions, the algorithm effectively balances loads and mitigates congestion. It uses optimal modulation formats and efficiently allocates frequency slots, maximizing resource utilization. Additionally, prioritizing rerouting of distant OD pairs along less crowded channels ensures sufficient resources and reduces bottlenecks. These features make the proposed method more effective in enhancing network throughput compared to traditional static approaches.

Fig. 8
figure 8

Time (s) vs. network throughput (Mbps).

The data integrity rate can be measured by considering the initial integrity rate and the attenuation coefficient, which can be written as:

$$\begin{aligned} I(d)={{I}_{0}}{{e}^{-\gamma d}} \end{aligned}$$
(43)

Equation (43) shows that the initial data integrity rate at \((d=0)\) is denoted by I, and the attenuation coefficient \(\gamma\) measures the decline in data integrity with distance. Figure 9 shows the achieved data integrity as transmission distance increases, starting from 100%. The x-axis represents transmission distance in kilometers, and the y-axis shows the data integrity rate in percentage.

Fig. 9
figure 9

Transmission distance (km) vs. data integrity rate (%).

It can be seen in Fig. 9 that after about 45 km, DWDM reaches a data integrity rate of approximately 43% after 45 km, which started from 79%. OGT starts with a data integrity rate of about 85%, gradually fluctuating with transmission distance before stabilizing at around 55% when the distance is 45 km. The proposed approach outperformed the existing schemes starting at 96% and eventually stabilizing at 75% with a transmission distance of 45 km. The fluctuation in the data integrity rate of the proposed approach is less compared to the existing OGT and DWDM approaches.

The log-normal distribution models the relationship between atmospheric turbulence and signal quality in optical communication, which is given in Eq. (44).

$$\begin{aligned} Q(p)={{Q}_{0}}-\gamma \log (p+1) \end{aligned}$$
(44)

Signal quality Q(p), in (dB) depends on the turbulence level \({{Q}_{0}}\) is the standard quality without turbulence, and \(\gamma\) is a constant quantifying system responsiveness to turbulence. The variable p represents the turbulence parameter.

Fig. 10
figure 10

Atmospheric turbulence (m) vs. Signal quality (dB).

Figure 10 illustrates the relationship between atmospheric turbulence (m) and signal quality (dB). The proposed system achieves a signal quality of 0.3 dB, which shows the effectiveness of the proposed approach as compared to DWDM and OGT, which achieve 0.09 dB and 0.18 dB, respectively, due to its adaptive adjustment of transmission parameters using PID control. Although this value may appear modest in absolute terms, it reflects meaningful improvement in high-OAM and turbulence-affected environments, where signal degradation is common and even small gains can significantly enhance overall system robustness. This dynamic response to atmospheric turbulence and accurate synchronization via Long Training Symbols (LTSs) ensures lower blocking probability and more stable signal quality compared to DWDM and OGT. To graph the correlation between time (in seconds) and blocking probability (percentage) in optical transmission with OAM light in quantum wells the blocking probability can be formulated as given in Eq. (45).

$$\begin{aligned} P(t)={{P}_{0}}{{e}^{-\gamma t}} \end{aligned}$$
(45)

Where P(t) is the blocking probability at the time (t), \({{P}_{0}}\) is the initial blocking probability, and \(\gamma\) is a constant that controls the rate of decrease.

Fig. 11
figure 11

Time(s) vs. blocking probability (%).

It can be seen in Fig. 11 that the blocking probability decreases exponentially over time as transmission improves. Fig. 11 illustrates the comparison of blocking probability at different times using DWDM, OGT, and the proposed method. The proposed method achieves the lowest blocking probability at 0.03%, outperforming DWDM and OGT, which have blocking probabilities of 0.19% and 0.07%, respectively. The reduction in blocking probability shows the effectiveness of the proposed approach. This improvement is primarily due to the integration of the TPAR-SHO, which enables dynamic traffic forecasting and efficient routing decisions. By anticipating network load and optimizing resource allocation in real time, the proposed approach minimizes congestion and ensures smoother data transmission, thereby significantly lowering the blocking probability compared to conventional methods.

Table 3 Comparison of the proposed system with benchmark methods.

To further clarify the performance improvements of the proposed system, a detailed comparison with benchmark techniques27,36 across key metrics is presented in Table 3. This includes BER, network throughput, signal quality, data integrity, blocking probability, transmission security, and scalability.

Computational complexity and scalability analysis

The computational complexity and scalability of the proposed approaches, such as TPAR-SHO, QWS-ILCS, OFDM-PID, and FFT-FOPM, are analyzed. The TPAR-SHO algorithm combines TPAR with the SHO, computing multiple paths and traffic distribution for OD pairs, resulting in a time complexity of \(O\left( {{\left| V \right| }^{2}}\log \left| V \right| +T.M.\left| E \right| \right)\), where V is the number of nodes, E is the number of links, and M are the hyenas agents. The QWS-ILCS algorithm operates at the physical layer. Its core complexity arises from spectral estimation, injection synchronization, and energy level analysis within quantum wells. These computations are done per frame and primarily involve matrix operations; thus, the time complexity is \(O(N\log N)\). The time complexity of OFDM-PID and FFT-FOPM is \(O(N\log N)\). Thus, the overall system complexity can be written as:

$$\begin{aligned} O\left( N\log N{{\left| V \right| }^{2}}\log \left| V \right| +T.M.\left| E \right| \right) \end{aligned}$$
(46)

The proposed system is designed with modular, parallel components (e.g., OFDM-PID, FFT-FOPM) that can be scaled horizontally. Therefore, the scalability is linear in N subscriber and \(\left| E \right|\) links, while Quadratic in \(\left| V \right|\) nodes.

Conclusion

In conclusion, optical transmission in quantum wells is optimized to enhance communication network performance through the integration of light-based OAM beams. The HPSLM is utilized to generate OAM beams, manipulating the light wave phase to improve data encoding, while OFDM-QAM facilitates the simultaneous transmission of multiple data streams. The TPAR-SHO and QWS-ILCS algorithms ensure real-time traffic management, secure data flow, and minimal bit error rates. The OFDM-PID algorithm accounts for atmospheric disturbances, guaranteeing high-quality data transmission even under adverse conditions. Additionally, the FFT-FOPM tool significantly enhances optical switching speeds to the nanosecond range. The collaboration of these technologies enables comprehensive network monitoring and efficient signal processing. The proposed methodology demonstrates notable performance metrics, including a BER of 17.63%, network throughput of 0.96 Mbps, data integrity rate of 75%, the signal quality of 0.3 dB, and a blocking probability of 0.03%. This work also presents opportunities for future research, particularly in integrating machine learning techniques to optimize system performance and dynamically enhance network efficiency. Furthermore, the development of hybrid algorithms for optical traffic management and signal integrity could contribute to the scalability and overall efficiency of the system.