A robust direction of arrival estimation method based on the chaotic MUSIC algorithm

Abstract

This article proposes a robust direction of arrival (DOA) estimation method in which the traditional MUSIC algorithm is enhanced with the Tukey Biweight cost function. The proposed scheme is specifically designed to perform well in chaotic signal situations. The chaotic nature of real world signal environments is modeled using a chebyshev-based chaotic number generator, which effectively captures signal variability and non-linearity. To assess the performance of the proposed methods, extensive Monte Carlo simulations have been performed on four configurations: standard multiple signal classification (MUSIC), MUSIC with Tukey Biweight, Chaotic MUSIC, and Chaotic MUSIC with Tukey Biweight. In this work, performance was measured using root mean square error (RMSE) and probability of resolution (PR) metrics. The simulation results demonstrate that the incorporation of chaotic signal modeling combined with the robust Tukey Biweight function significantly enhances DOA estimation accuracy, especially in challenging scenarios with low signal-to-noise ratio (SNR) and high signal correlation. Among the tested techniques, Chaotic MUSIC with Tukey Biweight consistently outperformed others, showing improved resolution capability and robustness. In practical wireless communication and radar systems, the proposed approach presents a strong candidate for reliable DOA estimation.

Introduction

Signal processing is a concept that focuses on analyzing, modifying, and synthesizing signals such as sound, seismic data, and altimeter measurements. The fundamental challenges in signal processing include: a) determining how best the signal is sensed to ensure accurate and efficient data acquisition, b) maximizing system performance while minimizing cost to achieve optimal resource utilization, and c) extracting the original information from the signal with high fidelity for accurate interpretation and application.

There are many applications of signal processing like wireless sensor networks (WSN), acoustics, sonar, video processing etc. WSN is a self-configured system comprises of sensor nodes designed for monitoring remotely for a specific purpose. Wireless source localization has many applications such as target tracking, wireless security, signal routing, emergency response and interference alignment¹. Source localization involves in estimating the position of the source, from the signal received at the sensor nodes, where each sensor in the array receives the signal data to estimate the position (source location) of the source². Direction of arrival (DOA) is one of the key issues in array signal processing for analysing source location. This DOA can be used for determining either a single source or multiple sources by using a sensor array. A sensor array (or array of sensors) is group of sensors, arranged in certain geometric patterns, used in many fields of science and engineering, particularly where the goal is to study propagating fields. Numerous techniques have been developed in the array signal processing for estimating the DOA of the signals by using different algorithms for different array models such as linear array, rectangular array, non-uniform and other arbitrary arrays.

Over the past few decades DOA has been extensively studied and numerous algorithms have been presented. Algorithms for DOA estimation are mainly divided into three different categories namely classical, sub-space and maximum likelihood (ML) techniques². Among the most popular methods, multiple signal classification (MUSIC) algorithm part of the subspace method are widely used which involves the estimation of co-variance matrix and its eigen decomposition³. MUSIC is a well analysed and mostly used algorithm and also computationally efficient as compared to maximum-likelihood (ML) method. It is essentially used for calculating the DOA of signal impinge on the array. The MUSIc algorithm perform well for narrow-band signal with normal distributed noise, whereas the perfoamnce degraded for coherent signals and non-Gaussian noise environment^4,5. In fact, real-time signals contain unpredictable noise or variations in parameters, making them inherently non-deterministic in nature.

In literature, many variants of MUSIC algorthms are proposed to make robust against different noise environment [6]. The DOA is estimated from the signal covariance matrix and it is sensitive to the non-Gaussian noise and other disturbances present in the channel. Deep learning methods are used now to estimate the DOA in multipath and impulse noise wireless environment⁶. Recently, researchers were used Tukey Biweight cost function to minimize the error. The same cost function can be used to estimate the robust covariance matrix to make the MUSIC algorithm robust.

Chaotic signal models are employed to represent complex real-world phenomena. A chaotic signal arises from a nonlinear autonomous deterministic system whose dynamics are highly sensitive to initial conditions. By introducing structured randomness, such models bridge the gap between purely random noise and fully predictable signals⁷. In array signal processing, chaotic signals have been applied to tasks such as weak signal detection and beamforming, exploiting their sensitivity to initial conditions and inherent nonlinearity⁸. The use of chaotic modeling enhances the robustness of covariance-based DOA estimation algorithms against noise, signal correlation, and low-SNR conditions⁹.

This paper proposes a novel DOA estimation approach by modeling received signals as chaotic in nature. A modified MUSIC algorithm is utilized, incorporating chaotic signal snapshots from a uniform linear array (ULA) to improve robustness against noise and nonlinearities. Further to enhance the accuracy, a Tukey Biweight cost function is integrated, minimizing the error at lower signal-to-noise ratio (SNR). The method demonstrates superior accuracy and consistency compared to coventional MUSIC, showing lower variability and greater robustness even when standard methods perform better on average metrics. Extensive simulations validate the proposed chaotic MUSIC algorithm with cost function optimization, especially under low SNR and limited snapshot conditions. The main contribution of the paper is summarized below.

• Proposes a novel chaotic signal modeling based DOA estimation framework to better capture real-world non-deterministic and nonlinear signal characteristics.

• Introduces a robust modified MUSIC algorithm integrating chaotic snapshots and a Tukey biweight cost function to reduce RMSE and improve SNR under challenging conditions.

• Demonstrates superior localization accuracy, higher probability of resolution, and enhanced robustness compared to conventional and DL-based methods, especially at low SNR and with limited snapshots.

Literature review

In literature many efforts are made by various authors for DOA estimation. The spatial spectrum plays a pivotal role in array signal processing, as it characterizes the angular power distribution of incident signals across the spatial domain. Classical and subspace-based methods such as Capon’s spectrum, MUSIC, and ESPRIT exploit this property to achieve high-resolution DOA estimation². To make this feasible, an eigen decomposition method is used to estimate the DOA of narrow-band signal³. Several extensions of MUSIC have been proposed to improve robustness under noise and coherent sources.

For example, multi-stage wiener filter (MSWF)-based formulations eliminate the need for covariance matrix estimation, while weighted-MUSIC approaches investigate robustness using asymptotic covariance analysis¹⁰. G-MUSIC further extends statistical analysis to large-array asymptotics, demonstrating consistent estimation and separation of closely spaced sources¹¹. To address practical noise environments, iterative subspace-based estimators jointly refine signal and noise subspaces⁴, and real-valued root-MUSIC (RV-root-MUSIC) reduces computational complexity through structural optimizations of covariance matrices¹². For multi-source scenarios, partial-relaxation approaches achieve superior performance under low SNR and limited snapshots¹³, while robust MUSIC variants incorporating Jordan canonical matrices enhance coherent source estimation with fewer antennas and reduced computational load¹⁴. More recently, deep learning–based frameworks have been integrated into DOA estimation, with CNN-based models achieving strong low-SNR performance¹⁵.

Table 1 summarizes the robustness and effectiveness of various MUSIC-like algorithms for DOA estimation of narrow-band complex non-circular sources. Building on this foundation, Nanda et al.⁹ introduced a chaotic variant of the algorithm to reduce computational complexity. A Chaotic Salp Swarm Algorithm (CSSA) was further developed to achieve faster convergence, enhance solution diversity, and mitigate the risk of local optima¹⁶. Similarly, a multi-objective distributed Sailfish Optimizer has been proposed to improve the diversification of search agents and thereby enhances DOA estimation accuracy¹⁷.

Table 1 Previous Studies.

N is number of sensors, K is number snapshots, L is the deep learning model dependent factor and G is the number of grids.

Earlier methods assume signals as noise with normal distributions, recent research explores the use of chaos in communication systems. Chaotic signals, though deterministic yet random-like, enable robust modulation and synchronization, effectively handling multipath and underwater acoustic challenges. These approaches enhance security, and reduce computational complexity, offering a promising alternative to conventional techniques. We propose a method wherein the received signal is assumed to be chaotic in nature. We use a snapshot of this signal from the ULA to validate and determine the location of the source. Modified MUSIC algorithm is used to increase the robustness of the system while determining the DOA. We further include a cost function to minimize the RMSE and the SNR.

Data model for DOA estimation

This section briefly describes the data and noise model used for the localization of narrow-band sources.

Considering a ULA with N elements placed with a distance of half wavelength each. Assuming the base-band signal S(t) is received at different sensors are correlated and the phase of the signal at the first element is zero (0). The time delay of arrival i.e, the time difference between the first element and the z element is computed with the help of trigonometric and wave propagation.

The time delay $\Delta \left( t_{z} \right)$ at the z^th array element defined in Eq. (1) which depends on the inter-element spacing D, angle of arrival $\theta$, and speed of light c. Here, $z=0, 1,..., N-1$.

$$\begin{aligned} \Delta \left( t_{z} \right) = \frac{z\;D\;sin\theta }{C} \end{aligned}$$

(1)

The transmitted narrow-band modulated signal S(t) in terms of its low-pass equivalent $S_{l}(t)$ and carrier frequency $f_c$ is expressed as.

$$\begin{aligned} S(t)= Re{\left\{ S_{l}(t)e^{j2\pi f_{c}t}\right\} } \end{aligned}$$

(2)

The received signal $x_Z(t)$ at the z^th sensor element is expressed in Eq. (3).

$$\begin{aligned} x_{Z}(t) = Re\left\{ S_{l}\left( t-\Delta t_{z} \right) e^{j\omega _{c}\left( t-\Delta t_{z} \right) }\right\} \end{aligned}$$

(3)

Finally, the down-converted and sampled base-band signal at the zth element in discrete time, modeled as the product of the signal s(n) and the array response $a_{z}(\theta )$. The signal received at the zth element is,

$$\begin{aligned} x_{z}\left( n \right) = s(n)a_{z}(\theta ) \end{aligned}$$

(4)

Fig. 1

A array model receiving a narrow-band signal from a distant source at an angle of arrival.

The Fig. 1 displays the arrangements of the sensors from which the data received is modeled as Eq. (5):

The signal model for a narrow band array system is illustrated by a matrix which is formed by the above equation for considering N elements $z=0,1,2,3....N-1$ is as follows in Eq. (5):

$$\begin{aligned} \begin{bmatrix} x_{0}(n) \\ x_{1}(n) \\ x_{2}(n) \\ \cdots \\ x_{N-1}(n) \\ \end{bmatrix} = \begin{bmatrix} a_{0}(\theta _{0}) & a_{0}(\theta _{1}) & a_{0}(\theta _{2})\;\;\cdots a_{0}(\theta _{m-1}) \\ a_{1}(\theta _{0}) & a_{1}(\theta _{1}) & a_{1}(\theta _{2})\;\;\cdots a_{1}(\theta _{m-1}) \\ a_{2}(\theta _{0}) & a_{2}(\theta _{1}) & a_{2}(\theta _{2})\;\;\cdots a_{2}(\theta _{m-1}) \\ \cdots & \cdots \cdots & \cdots \\ a_{N-1}(\theta _{0}) & a_{N-1}(\theta _{1}) & a_{N-1}(\theta _{2})\;\;\cdots a_{N-1}(\theta _{m-1}) \\ \end{bmatrix} \begin{bmatrix} s_{0}(n)\\ s_{1}(n)\\ s_{2}(n)\\ \cdots \\ s_{m-1}(n)\\ \end{bmatrix} + \begin{bmatrix} v_{o}(n)\\ v_{1}(n)\\ v_{2}(n)\\ \cdots \\ v_{N-1}(n)\\ \end{bmatrix} \end{aligned}$$

(5)

where $v_{z}(n)$ is the noise present at each element, added to the signal. The above equation can be written in compact matrix notation using $N \times 1$ vector $x_{n}$, $N \times M$ matrix $\textbf{A}$ along with the signal and noise vectors $\textbf{s}_{n}$ and $\textbf{v}_{n}$ respectively in Eq. ((6)):

$$\begin{aligned} \textbf{X}_{n}=[\textbf{a}(\theta _{0})\;\textbf{a}(\theta _{1})\;.\;.\;.\;\textbf{a}(\theta _{m-1})]\;\textbf{s}_{n}\;+\;\textbf{v}_{n}\; =\;\textbf{A} \textbf{s}_{n}\;+\;\textbf{v}_{n} \end{aligned}$$

(6)

The columns of matrix $\textbf{A}$, denoted as $\textbf{a}(\theta _{i})$, represent the steering vectors corresponding to signals $s_{i}(t)$. Assuming each of the m signals arrives at a distinct angle, these vectors constitute a linearly independent set. The vector $v_{n}$ signifies the uncorrelated noise present at individual antenna elements. Since the steering vectors depend on the arrival angles of the signals, these angles can be determined if the steering vectors are known or if a basis for the subspace spanned by these vectors is available²⁰.

Eigen structure of the spatial covariance matrix

The antenna array’s spatial covariance matrix can be computed as follows. Assuming the vectors $\textbf{s}_{n}$ and $\textbf{v}_{n}$ are uncorrelated where $\textbf{v}_{n}$ is a vector of Gaussian white noise with zero mean and correlation matrix $\sigma ^{2}I$

As,

$$\begin{aligned} R_{ss}=E\left[ \textbf{s}_{n}\textbf{s}_{n}^{H}\right] \end{aligned}$$

the matrix can be written as,

$$\begin{aligned} R_{xx}&=E\left[ \textbf{X}_{n}\textbf{X}_{n}^{H}\right] \end{aligned}$$

(7)

$$\begin{aligned}&= \textbf{A}E\left[ \textbf{s}_{n}\textbf{s}_{n}^{H} \right] A^{H}+E\left[ \textbf{v}_{n}\textbf{v}_{n}^{H} \right] \nonumber \\&=\textbf{A}R_{ss}\textbf{A}^{H}+\sigma ^{2}I_{N\times N}. \end{aligned}$$

(8)

Since the matrix $R_{xx}$ is Hermitian (Complex Conjugate Transpose), it has real eigen values and can be unitarily decomposed.

For $N>M$ any vector $\textbf{q}_{n}$ which is orthogonal to the columns of $\textbf{A}$, is also eigenvector of $R_{xx}$ it can be represented by the following Eq.

$$\begin{aligned} R_{xx}\textbf{q}_{n} = \left( \textbf{A}R_{ss}\textbf{A}^{H} + \sigma ^{2}I \right) \textbf{q}_{n} = 0+\sigma ^{2}I \textbf{q}_{n} = \sigma ^{2}\textbf{q}_{n} \end{aligned}$$

(9)

Further the $\textbf{q}_s$ is an eigenvector of $\textbf{A}R_{ss}\textbf{A}$, then we get :

$$\begin{aligned} R_{xx}\textbf{q}_{s} = \left( \textbf{A}R_{ss}A^{H}+\sigma ^{2}I \right) \textbf{q}_{s} = \sigma _{s}^{2}\textbf{q}_{s} + \sigma ^{2}I \textbf{q}_{s} = \left( \sigma _{s}^{2}+\sigma ^{2} \right) \textbf{q}_{s} \end{aligned}$$

(10)

notably the eigen decomposition of $R_{xx}$ can then be expressed as follows:

$$\begin{aligned} R_{xx}= QDQ^{H} = \left[ Q_{s}Q_{n} \right] \begin{bmatrix} D_{s} & 0 \\ 0& \sigma ^{2}I \\ \end{bmatrix}\left[ Q_{s}Q_{n} \right] ^{H}. \end{aligned}$$

(11)

The matrix Q is divided into two parts: an $N \times M$ matrix $Q_s$, whose columns are the M eigenvectors that correspond to the signal subspace, and an $N \times (N - M)$ matrix $Q_n$, whose columns correspond to the noise eigenvectors. The diagonal matrix D consists of the eigenvalues of $R_{xx}$ and is partitioned into an $M \times M$ diagonal matrix $D_s$, containing the signal eigenvalues, and an $(N - M) \times (N - M)$ scaled identity matrix $\sigma ^2 I_{N \times N}$, whose diagonal elements are the noise eigenvalues.

Due to the availability of stable SVD algorithms, many methods prefer decomposing the data matrix over diagonalizing the spatial covariance matrix to derive the signal subspace basis. In practical scenarios, the $( Z - M )$ smallest eigenvalues won’t exactly equal $( \sigma ^2 )$; they will be small relative to the signal eigenvalues because $R_{xx}$ is estimated from data rather than known perfectly [19].

Robust MUSIC algorithm using Tuky’s biweight cost function

A common approach to estimate the spatial covariance matrix is using the sample spatial covariance matrix, obtained by averaging rank-one data matrices of the form $( \textbf{x}_p \textbf{x}_p^H)$, as follows in Eq. (12):

$$\begin{aligned} R_{xx}= \frac{1}{K}\sum _{p=0}^{K-1}\textbf{x}_{p}\textbf{x}_{p}^{H} \end{aligned}$$

(12)

where K is the number of snapshots available from the sensors.

The covariance matrix $\textbf{R}_{xx}$ is employed in the MUSIC algorithm to estimate the directions of arrival (DOAs) of signals impinging on the antenna array. However, conventional method used in Eq. (12) to compute $\textbf{R}_{xx}$ is sensitive to impulsive noise and channel disturbances, which can result in inaccurate DOA estimates. To address this issue, a robust approach is adopted to estimate $\textbf{R}_{xx}$ using the Tukey biweight cost function, as presented in the following section.

Tuky’s Biweight robust cost function

In the MUSIC-based DOA estimation framework, the Tukey biweight function is employed to robustly suppress the influence of outliers and impulsive noise during covariance matrix estimation²¹. The resulting cost function quantifies the deviation between the true value and mean of the distribution, enabling more accurate separation of the signal and noise subspaces. By minimizing this cost function, the robustness of the covariance matrix is improved, leading to enhanced DOA estimation accuracy in low-SNR and disturbed channel conditions. The Tuky’s Biweight Robust Cost Function is given as²²

$$\begin{aligned} f_{s}(e)={\left\{ \begin{array}{ll} \frac{b^2}{6}\left[ 1-\left[ 1-(\frac{e}{b})^2\right] ^3\right] & \parallel e\parallel \le b\\ \frac{b^2}{6}& \parallel e\parallel > b\end{array}\right. } \end{aligned}$$

(13)

Where e is the error used as the argument to the function and b is a positive constant used as a threshold parameter. In literature, a common choice of this parameter is $b = 4.685$.

Robust MUSIC algorithm

Multiple SIgnal Classification (MUSIC) is one of the most widely used subspace method for DOA estimation which uses the eigen structure of the covariance matrix. The algorithm uses the signal covariance matrix to estimate the DOA which is usually sensitive to the outliers. In order to make the algorithm is robust, we have used Tuky’s Biweight score as a weight to the each sample while calculating the signal covariance matrix in Eq. (12).

The score function is is defined as the derivative of Tuky’s Biweight Robust Cost Function given in Eq. (13)²³

$$\begin{aligned} f^{'}_{s}(e)={\left\{ \begin{array}{ll}e\left[ 1-(\frac{e}{b})^2\right] ^2 & \parallel e\parallel \le b\\ 0 & \parallel e\parallel > b\end{array}\right. } \end{aligned}$$

(14)

where e is the deviation of the data from the actual distribution and is calculated as

$$\begin{aligned} e=\frac{x-\mu }{\sigma ^2} \end{aligned}$$

(15)

$\mu$ is the mean and $\sigma ^2$ is the variance of the random data we assumed. In this paper, a zero mean and unity variance signal is considered. The associated weight w for the corresponding input signal x which have the deviation of e is given by

$$\begin{aligned} w={\left\{ \begin{array}{ll}\left[ 1-(\frac{e}{c})^2\right] ^2 & \parallel e\parallel \le c\\ 0 & \parallel e\parallel > c\end{array}\right. } \end{aligned}$$

(16)

In MUSIC algorithm performance directly depends on the covariance matrix. The Tukey’s Biweight function provides a robust covariance matrix estimate, providing the signal subspace necessary for MUSIC which improves the DOA accuracy. Robust covariance with Tukey’s Biweight function is given by

$$\begin{aligned} R _{robust}=\frac{\sum _{k=1}^{N}w\left\| x_k\right\| x_kx_k^H}{\sum _{k=1}^{N}w\left\| x_k\right\| } \end{aligned}$$

(17)

Now the robust covariance matrix $R _{robust}$ is used to estimate the DOA. We find the steering vectors which are orthogonal to the noise subspaces. $a(\theta )$ is the steering vector to one of the incoming signal as in Eq. (18)³,

$$\begin{aligned} a(\theta )^{H}Q_{n} = 0 \end{aligned}$$

(18)

practically $a(\theta )$ is not precisely orthogonal to noise subspaces due to the errors in estimating $Q_(n)$, therefore in Eq. (19) we get,

$$\begin{aligned} P_{MUSIC}(\theta ) = \frac{1}{a^{H}\left( \theta \right) Q_{n}Q_{n}^{H}a\left( \theta \right) } \end{aligned}$$

(19)

when $\theta$ becomes equal to any one of the signals then the above function gives a very large value.

This is called as MUSIC spectrum.

Methodology

The process begins with generation of random signal and chaotic signal, initialization of key parameters, including the number of sensors, Monte Carlo trials, and the signal-to-noise ratio (SNR) range. It also specifies values such as the actual signal arrival angles, carrier frequency, wavelength, and sensor spacing. A chaos factor is introduced to simulate random variations in noise, and a Tukey biweight constant is set to improve estimation accuracy. A steering vector function is defined to model the response of the sensor array for angles 130 AND 138. Additionally, a separate function is implemented to compute a cost function based on projection errors in the covariance matrix which utilizes Tukey’s biweight method. This technique reduces the impact of outliers, thereby improving the stability of the MUSIC algorithm. The flow chart of Algorrithm is provided in Fig. 2.

Signal generation

A signal received at the sensor array is assumed to be a random signal. Few signals due to its chaotic properties are called chaotic signals. Hence these signals can be initialised as follows:

Chaotic signal generation

The Chaotic Random data generated by using chebyshev model in range [0,1] with logistic map is

$$\operatorname {Chaotic}(i+1)=a \times \operatorname {Chaotic}(i) \times (1-\operatorname {Chaotic}(i))$$

where a is the bifurcation parameter taken as 4 and Chaotic (i) is the chaotic number value in the ith iteration (its initial value Chaotic(0) is taken as 0.4).

The chebyshev code used for the generation of chaotic data is as follows in Eqs. (20), (21), (22):

$$\begin{aligned} & x(1)=0.1 \end{aligned}$$

(20)

$$\begin{aligned} & x(n)= cos\left[ ncos^{-1}\left( x(n-1) \right) \right] \end{aligned}$$

(21)

$$\begin{aligned} & x\left( 1+ n \right) = cos\left[ ncos^{-1}\left( x(n) \right) \right] \end{aligned}$$

(22)

where $n = 1,2,3,....,N$

Fig. 2

Flow chart for proposed MUSIC with Tukey biweight method.

For each SNR level, 100 Monte Carlo simulations are performed. Within each simulation, synthetic signals are generated, normalized, and processed through a sensor array. The response matrix of the array is computed based on the true arrival angles. The received signal undergoes noise addition, incorporating both Gaussian and chaotic noise components, and the resulting covariance matrix is subjected to eigen decomposition to extract the signal subspace, which is fundamental to the MUSIC algorithm. The MUSIC spectrum is then computed by scanning across possible angles, leveraging the orthogonality between noise and signal subspaces. The Tukey biweight function is applied to further refine the spectrum by mitigating the influence of anomalies. Peaks in the refined spectrum are detected to estimate the DOAs. If at least two peaks are identified, they are recorded as estimated angles for the trial. The probability of accurate resolution is determined by the proportion of trials where both DOAs are successfully distinguished. After all trials are completed, the RMSE is calculated by comparing the estimated and actual angles. The resolution probability is determined based on the number of cases where both angles were correctly identified. Finally, graphical representations are generated, showing the variation of RMSE and resolution probability with respect to SNR. The proposed method is depicted as the Algorithm 1 given below.

Algorithm 1

MUSIC with TukyBiweight,

Simulation results

In this work, we considered two number of sources in the position of $130^{\circ }$ and $138^{\circ }$. Number of sensor nodes used is $N=16$. Number of Montecarlo simulation considered is 50 and 100 those sufficiently larger than $2\times N$.

The simulations presented in this section demonstrates the performance of robust DOA estimation in chaotic scenario and random data model. The robustness of the MUSIC algorithm is analyzed. The robustness of MUSIC is tested by root mean square error (RMSE) and probability of resolution (PR). The performance of all the algorithms is compared in terms of: (a) RMSE and (b) PR. The RMSE is calculated by using 100 Monte-Carlo simulation where M is the number of sources used in simulation²⁰.

$$\begin{aligned} \operatorname {RMSE}=\sqrt{\frac{1}{M N_{\text{ run } }} \sum _{i=1}^{N_{\text{ ran }}} \sum _{l=1}^M\left( \hat{\theta }_l(i)-\theta _l\right) ^2} \end{aligned}$$

(23)

$N_{r u n}$ is the number of Monte-Carlo simulations, $\hat{\theta }_I(i)$ is the estimate of the l th DOA in the i th run and $\theta _l$ is the true DOA of the l th source. The simulated RMSE is compared with theoretical CRLB.

PR is the ability of DOA estimation algorithms to resolve closely spaced sources. Two sources are said to be resolved in a given run if $\left| \hat{\theta }_1-\theta _1\right|$ and $\left| \hat{\theta }_2-\theta _2\right|$ is less than $\left| \theta _1-\theta _2\right| / 2$.

RMSE versus SNR

RMSE is a crucial performance metric for evaluating the accuracy of source localization methods in sensor networks. This analysis compares the RMSE statistics of four different methods: RMSE MUSIC, RMSE MUSIC with cost, RMSE Chaotic MUSIC, and RMSE Chaotic MUSIC with cost. The evaluated statistical parameters include the mean, standard deviation, standard error, variance, and median. This section presents the simulation analysis which illustrates the effectiveness of chaotic based DOA estimation of correlated signals. The simulation of MUSIC algorithm is not considered here as it fails for fully correlated signals.

Fig. 3

RMSE performance comparison.

Table 2, shows the over all comparative statistical analysis of all the methods, The simulation result along with the statistical analysis displays that The RMSE Chaotic MUSIC methods with Tuky’s Biweight improves the precision of the mean RMSE estimate and reduces variability across methods. Figure 3 depicts the RMSE performance comparison using 50 and 100 snapshots, respectively.

Table 2 Statistical analysis of RMSE.

Analysis of the RMSE vs. SNR graphs for 50 and 100 snapshots shows that the accuracy of DOA estimation depends strongly on the quantity of snapshots. When the number of snapshots was increased from 50 to 100, all the methods displays clear improvement in the RMSE values, especially in the low to moderate SNR range (-10 dB to 10 dB). This improvement was more prominent in chaotic based techniques, which suggests that having more snapshots benefits from temporal information, and chaos based estimators become more robust. Also, the RMSE curves of all algorithms converge closely to the theoretical CRLB when the number of snapshots increases.

Table 3 shows the statistical values for different algorithms at 5 dB SNR and 100 number of Montecarlo iteration.

Table 3 Statistical values of algorithms at 5dB SNR.

Probability of resolution

The probability of resolution (PR) is a critical performance metric in sensor networks, particularly in applications such as target detection and localization. In this analysis, we compare The four different methods for calculating the resolution here is PR MUSIC, PR MUSIC with cost, PR Chaotic MUSIC, and PR Chaotic MUSIC with cost. The statistical parameters evaluated include the mean, standard deviation, standard error, variance, and median. The table 2, summarizes the statistical properties of the PR for each method: Table 3, summarizes the statistical properties of the PR for each method:

Table 4 Statistical analysis of probability of resolution.

Fig. 4

Probability of resolution.

The PR MUSIC and PR MUSIC with cost methods show similar performance in terms of mean PR, with a slight improvement in variability and accuracy when the cost function is added. The PR Chaotic MUSIC methods, both with and without cost, have significantly lower mean PR and median values. However, they demonstrate reduced variability and higher accuracy, particularly with the addition of the cost function. The addition of cost functions generally improves the precision of the mean PR estimates and reduces variability across all the methods. In Fig. 4, the probability of resolution results are presented, while Fig. 5 shows the RMSE performance under partially correlated sources.

RMSE versus correlation coefficient

For partially correlated multipath signals varying from 0.1 to 1 where CRLB is used as theoretical reference. statistics states that the proposed method improves their accuracy and robustness. The performance of the Chaotic MUSIC standalone version was poor, but when a Tuky’s Biweight was added, it performs close to CRLB–which is an ideal condition. This means that the use of robust statistics is very useful in DOA estimation to efficiently handle challenges such as highly correlated signals and practical antenna errors. Table 4, shows that statistical comparison of RMSE and correlation coefficient(Table 5). The comparative performance of source correlation is shown in the Fig. 5.

Fig. 5

RMSE performance under partially correlated sources.

Table 5 Statistical comparison of RMSE and correlation coefficient.

Conclusion

In sensor networks, the choice of RMSE calculation method impacts the average localization error, variability, and accuracy. While standard MUSIC methods yield lower mean RMSE values, chaotic methods provide more consistent and accurate estimates with lower variability. The incorporation of cost functions enhances performance metrics across both standard and chaotic MUSIC methods. The probability of resolution calculation method also impacts the average probability of resolution, variability, and accuracy. While standard MUSIC methods yield higher mean PR, chaotic methods provide more consistent and accurate estimates with lower variability. The incorporation of cost functions enhances performance metrics across both standard and chaotic MUSIC methods. Further research could explore the trade-offs between lower mean RMSE and reduced variability, as well as the specific applications where each method may be most effective.