Introduction

Since the inception of artificial intelligence (AI) studies in the 1950s, AI techniques have undergone significant development over the past several decades. The 1990s and 2000s saw a rapid advancement in AI techniques, driven by the increasing computational power made possible by semiconductor technology. The milestone achievement of AlphaGo by Google DeepMind, which competed in the game of Go against human players, marked a pivotal moment where AI began to integrate more closely into everyday life. In recent years, AI techniques have become ubiquitous in various fields, revolutionizing data processing in areas such as medical diagnoses1,2, big data analysis3,4, and the Internet of Things (IoT)5,6. Academic communities have swiftly embraced AI to tackle complex problems and automate tedious tasks7,8,9. One such example is the application of AI in extended X-ray absorption fine structure (EXAFS) analysis, a technique that provides detailed information on the local structural properties of materials. EXAFS analysis is crucial for studying a wide range of materials, from amorphous substances to crystals10,11,12,13, and is versatile enough to be used in different environmental conditions such as solutions, external fields, and high-pressure environments. However, the quantitative analysis of EXAFS data can be challenging, especially for those new to the field. To address this challenge, research groups have turned to machine learning techniques to efficiently analyze EXAFS data from in-situ and operando experiments14,15,16. Timoshenko and colleagues have successfully demonstrated the use of supervised machine learning, specifically artificial neural networks (NNs), to quantitatively analyze X-ray absorption near edge structure (XANES) and EXAFS data15. By training the NNs with a large dataset generated from theoretical calculations that incorporate variations in local structures such as interatomic distances and buckling angles, they have shown the potential of machine learning in enhancing the analysis of complex EXAFS data sets. Generating a substantial number of EXAFS datasets, either experimentally or theoretically, in advance for NNs, coupled with the considerable experimental or computational time required, presents a significant challenge to the practical application of machine learning in EXAFS data analysis. Additionally, the need for accurate and reliable data sets to train the NNs effectively adds another layer of complexity to the challenge. On the other hand, generating theoretical EXAFS datasets computationally also requires significant computational time, especially for complex systems. Terry and colleagues have highlighted instances where machine learning techniques have proven ineffective in analyzing EXAFS data14. Furthermore, the NNs used for EXAFS data analysis may need to be retrained when applied to different systems, even if it was adequately trained for a specific system initially.

In this study, the use of deep reinforcement learning (RL) techniques is first employed to quantitatively and automatically analyze series data from EXAFS. Unlike traditional NN approaches, deep RL methods do not require a large quantity of experimental or calculated datasets with similar parameter values in advance for training neural networks. Various RL algorithms, such as deep Q-network, deep deterministic policy gradient, proximal policy optimization, and asynchronous advantage actor critic (A3C), have been developed for this purpose17,18,19,20. The A3C algorithm is a deep RL algorithm that integrates the benefits of both value-based and policy-based methods. In this approach, state values are estimated, and the agent directly optimizes the policy. In A3C, the NNs consists of a critic network for estimating the value function and an actor network for determining the actions of agents. The weights of both the critic and actor networks are updated during training using the following formula:

$$\:{W}_{i+1}\approx\:\:{W}_{i}+\alpha\:\left\{{\nabla\:}_{w}\left[log{P}_{w}\left({a}_{i}|{s}_{i}\right)\right]\left[\left(R{(s}_{i+1}\right)+\gamma\:V\left({s}_{i+1}\right)-V\left({s}_{i}\right)\right]\right\}$$
(1)

Here, \(\:W\) is the weight of NNs, \(\:{\upalpha\:}\) is the learning rate, \(\:{P}_{w}\left(a|s\right)\) is the policy for the selected action of \(\:a\) at the state of \(\:s\), R(s) is the reward at the s state, \(\:\gamma\:\) is the discount factor, and \(\:V\left(s\right)\) is the value function which is obtained by a critic network. In this study, the learning rate α and discount factor γ were established at 0.0001 and 0.5, respectively, following testing to achieve the optimal convergence of EXAFS data fits. The policy determines the agent’s strategy by associating each state with a probability distribution of potential actions. The value function of the environment calculates a reward based on a given state (experimental and theoretical EXAFS data) and helps the agent select the new value that maximizes the reward in the subsequent trial. In the A3C algorithm, agents interact asynchronously with the environment, exploring possible actions independently and simultaneously to find optimal actions. This approach aims to improve the rationality and quality of EXAFS data analysis by leveraging deep RL techniques. Additional details on the A3C algorithm can be found in the existing literature20.

In the EXAFS data fits, NNs with three hidden layers of 256, 256, and 64 nodes, respectively, are currently being used to generate and evaluate policies. The Rectified Linear Unit (ReLU) activation function is applied between the hidden layers. The actor network of NNs selects a policy using the softmax function, which converts action preferences into quantitative probabilities. As a result, actions become stochastic and exploratory rather than purely random trials. This approach can save computing time in EXAFS data fits, especially when dealing with large amounts of series data sets, compared to a simple random Monte Carlo method. It also helps avoid getting stuck at a local minimum of the R-factor, that often occurs in EXAFS data fits with a gradient descent method. The critic network generates the value function for the current state. Theoretical EXAFS data are computed using the FEFF codes21,22 with an initially provided structural model and cluster size. In the process of fitting EXAFS data using deep RL, the atomic paths that are predicted to contribute to the measured EXAFS are initially selected with estimated values of fitting parameters, similar to a traditional EXAFS fitting procedure. The theoretically-calculated EXAFS is then obtained by summing the EXAFS of the selected paths at the environment of the A3C using the Larch code23. Subsequently, the reward value, which is the reciprocal of the R-factor, is calculated based on the comparison between the measured and theoretically-calculated EXAFS data. Upon receiving the calculated EXAFS and the reward value, the agent determines a new value for a fitting parameter, and a new action is stochastically chosen based on the policy. This iterative process continues until the termination criteria, which include reaching the maximum reward value and achieving repeatability in the fitting results, are satisfied. The Larch code is utilized for summing the theoretical EXAFS of selected paths and performing the Fourier transform of the EXAFS at the environment. Throughout the fitting process of the EXAFS data, the NNs of the agent and the fitting parameter value are continuously updated with the calculation results from the environment using Eq. (1). In the deep RL approach, since the fitting parameter value for the analysis of EXAFS data is autonomously generated by the agent, there is no need for constraints to reduce correlations between the fitting parameters. At the environment, paths that make insignificant contributions to the EXAFS signal are excluded based on specific exclusion criteria. The criteria include considerations such as the distance between an atomic pair exceeding the fitting range, the backscattering amplitude ratio of photoelectrons (F/Ffirst neighboring atom) < 0.15, the Debye-Waller factor > 0.1, the coordination number < 0.1, or within the fitting uncertainty.

In the deep RL method for EXAFS data analysis, the agent and the environment in A3C play distinct roles and closely cooperate, as illustrated in Fig. 1 (a). The NNs generate a new value for a fitting parameter based on previous fitting results under given fitting conditions and sends this new value to the environment. This process allows the NNs to be trained, and the optimal value of the fitting parameter is continuously updated until the best fit is achieved. On the other hand, the reciprocal value of the R-factor of the EXAFS data fit is defined as the reward in deep RL. Figure 1 (b) displays the calculated reward using the R-factor of EXAFS24 for Fourier-transformed EXAFS with real and imaginary parts, as described by Eq. (2):

$$\:R-factor=\frac{\sum\:_{i=1}^{i=N}{\left|{Re\chi\:}_{data}\left({r}_{i}\right)-\:{Re\chi\:}_{theory}\left({r}_{i}\right)\right|}^{2}}{\sum\:_{i=1}^{i=N}{Re\chi\:}_{theory}^{2}\left({r}_{i}\right)}+\:\frac{\sum\:_{i=1}^{i=N}{\left|{Im\chi\:}_{data}\left({r}_{i}\right)-\:{Im\chi\:}_{theory}\left({r}_{i}\right)\right|}^{2}}{\sum\:_{i=1}^{i=N}{Im\chi\:}_{theory}^{2}\left({r}_{i}\right)},$$
(2)

where \(\:{\chi\:}_{data}\left({r}_{i}\right)\) and \(\:{\chi\:}_{theory}\left({r}_{i}\right)\) represent the measured and theoretically-calculated EXAFS in the r-space, respectively. The theoretically-calculated EXAFS is obtained using the Larch codes23 with a specified structural model. The inset in Fig. 1 (b) shows the reward values as a function of sequences for the EXAFS data fit of a Pt foil, where the reward values begin to increase. The reward value changes as the fitting parameter receives a new value from the NNs at each sequence. Figure 1 (c) illustrates the changes in reward during the fitting process of the EXAFS data for a Pt foil. The fitting process automatically concludes based on the termination criteria outlined earlier. Deep RL techniques are particularly beneficial for analyzing a series of EXAFS data sets with similar theoretical EXAFSs. The deep RL method has been applied to EXAFS data series of PtOx and Zn-O complexes. The deep RL method successfully determined the local structural properties of the systems, including interatomic distances (d), Debye-Waller factors (σ2, including thermal vibration and static disorder), and coordination numbers (N).

Results and discussion

In the EXAFS data analysis process, the EXAFS data is extracted from measured XAFS as a function of photoelectron wave number, k. The atomic absorption background is properly determined using AUTOBK, which is a part of the Demeter package21. The measured EXAFS data in the k-space is then Fourier-transformed to the r-space and analyzed, compared to theoretically-calculated EXAFS23 using the deep RL method as previously described. In this analysis, both single- and multiple-scattering paths that make a significant contribution to the EXAFS within the fitting range are considered. The X-ray absorption edge energy shift, ΔE0, is considered as an additional parameter. The parameters of d, N, σ2, and ΔE0 are varied singly and repeatably in sequence during the analysis of the EXAFS data. For the deep RL method, an optional constraint on the width (e.g., 0.4 Å) of a distance parameter is applied.

In the EXAFS data analysis of a Pt foil, theoretical paths are generated using FEFF within the Larch code, utilizing an initially provided structural model of Fm-3 m phased Pt and a cluster radius of 6.0 Å. The first four single- and four multiple-scattering paths within the distance range of r = 2.0–5.8 Å are selected based on the exclusion criteria discussed earlier. The parameters Ns and ds for the selected atomic pairs are initially set to the same values as those in the structural model. Initially, ΔE0 is set to 0, and the values of σ2s are set to 0.005 Å2 and 0.012 Å2 for single and multiple paths, respectively. In the EXAFS data analysis using the deep RL method, the fitting process starts from the nearest neighboring atoms of a probe atom. Each scattering path is defined by four parameters: ΔE0, N, d, and σ2, and these parameters are sequentially optimized through the deep RL fitting process as described earlier. Once the parameter values for the first nearest neighboring atoms are optimized, the fitting process for the second neighboring atoms begins with the same four parameters. It is crucial to emphasize that a single ΔE0 is fitted across all paths included in the fit because ΔE0 is independent of the paths for unpolarized EXAFS. The fitting process advances through each selected path sequentially until it satisfies the termination criteria for all paths. The NNs are continuously trained throughout this process, as depicted in Fig. 1 (a). The NNs continuously generate new values of a parameter with the given fitting conditions during the fitting process. This approach helps avoid convergence of the data fit into a local optimal value. After the best fit is determined using the deep RL method, the uncertainty of a fitting parameter is estimated using a conventional method in EXAFS data fits22. This comprehensive approach allows for accurate analysis and interpretation of the EXAFS data obtained from the experimental measurements. Figure 2 shows the representative fits of EXAFS data of a Pt foil at different reward values. The fits show good agreement when the reward value is greater than 50 (R-factor ≤ 0.02), satisfying the crystallographic criterion of phase matching ≤ 15%. The data analysis was automatically concluded when a reward value of approximately 295 (corresponding to an R-factor of approximately 0.0034) was achieved after approximately 34,000 sequences were executed, as depicted in Fig. 1 (c). The EXAFS data and the best fit in k-space are visualized in the supplementary materials. The analysis results are organized in Table 1. The N and d values of the atomic pairs determined using the deep RL method are quite similar to those calculated for Pt metal based on XRD measurements. Single-scattering paths of photoelectrons predominantly contribute to EXAFS peaks at short distances, while peaks at further distances are affected by single- and multiple-scattering paths. The N and d values of Pt-Pt-Pt multiple scattering paths with different distances also correspond well to the calculations. The analysis results of EXAFS data of Pt foil using the deep RL method agree well with previous reports where conventional EXAFS analysis methods were used25,26.

Fig. 1
Fig. 1
Full size image

(a) Schematic illustrating the EXAFS data fitting procedure for a parameter using the deep RL method. (b) Calculated reward in the deep RL method as a function of the R-factor, as described in the main text. (c) Evolution of rewards during the fitting process. The inset in (b) displays reward changes indicating the point where the reward starts to increase.

Fig. 2
Fig. 2
Full size image

The EXAFS data of a Pt foil at the Pt L3 edge was used for successive fits at the reward values of (a) 3, (b) 10, (c) 30, (d) 50, (e) 75, (f) 295 using the deep RL method. The data in the range of 2.5–11.0 Å-1 was utilized for the Fourier transform, and the data in the range of r = 2.0–5.8 Å was fitted.

Table 1 Fit results of EXAFS data for a Pt foil. The EXAFS data and fits are illustrated in Fig. 2. \(\:{\mathbf{S}}_{0}^{2}\) = 0.86 was derived from fitting the EXAFS data of the first three single paths of the Pt foil. ΔE0 = -0.3 eV was determined from the best fit.
Full size table

Using the NNs of the deep RL method, which was trained with the analysis of EXAFS data of the Pt foil, the in-situ EXAFS data sets of PtOx are analyzed. Figure 3 (a) displays the XANES spectra of PtOx on SiO2 supports measured at the Pt L3 edge during heating. Strong white lines at lower temperatures indicate oxidation of Pt atoms. The EXAFS data in the region of k = 2.5–10.5 Å−1 is Fourier-transformed to r-space, as shown in Fig. 3 (b) and (c). The in-situ EXAFS data sets of PtOx nanoparticles in the k-space are provided in the supplementary materials. In the Fourier-transformed EXAFS data, a dominant peak at ~ 1.7 Å is observed in the temperature range of room temperature (RT) to 100 °C, which decreases significantly at 150 °C. Additionally, peaks within the range of 2–3 Å appear at 150 °C and become more pronounced with further heating. These peaks remain consistent when cooled back to room temperature (RTc). It is important to note that the peak positions are approximately 0.3 Å shorter than the true atomic distances because the overall phase shift of back-scattered photoelectrons has not been accounted for yet. In Fig. 3 (c), the peaks observed around 1.7 Å and 2–3 Å may correspond to Pt-O and Pt-Pt pairs, respectively. This is because PtOx consists of only O and Pt atoms, and the distance between Pt-Pt pairs should be greater than that of Pt-O pairs. A structural model of Pt-O and Pt-Pt pairs is selected to analyze the EXAFS data sets of PtOx. Theoretical EXAFS calculations with the structural model of Pt-O and Pt-Pt pairs are carried out using the Larch code23. In systems with significant structural disorder, the contribution of multiply-scattered photoelectrons to EXAFS can be negligible due to their lack of coherence. The NNs of the deep RL method start fitting the EXAFS data of PtOx at RT within the distance range of r = 1.3–3.5 Å. This is done using a structural model of Pt foil and a Pt-O pair with the initial values of distance d = 2.0 Å and coordination number N = 1. Since the distances of Pt-Pt-Pt pairs are beyond the fitting range, only the single-scattering paths of Pt-O and Pt-Pt pairs are initially employed with an σ2 value of 0.005 Å2. In the temperature range from RT to 100 °C, two distances of Pt-Pt pairs, denoted as Pt-Pt(1) and Pt-Pt(2), significantly enhance the fitting quality compared to a single distance. When fitting the EXAFS data in the temperature range from 200 °C to RTc, the Pt-O path is automatically excluded due to the N value of Pt-O pairs < 0.1 or within fitting uncertainty. As a result, the three single-scattering paths of Pt-O, Pt-Pt(1), and Pt-Pt(2) are utilized to fit the series of EXAFS data for the PtOx nanoparticles. Once the final structural model of the three paths is established, the NNs of the deep RL sequentially fit the entire sets of EXAFS data, applying the exclusion criteria until the termination criteria are met. The fitting procedure for the EXAFS data of PtOx is similar to that of the Pt foil, as described earlier. In the EXAFS data analysis of PtOx at RT, the NNs of the deep RL start with the four parameters of ΔE0, N, d, and σ2 for the shortest path of Pt-O. After achieving an optimal fit for a path, the NNs proceed to fit the subsequent paths with the same four parameters, as described above. If the difference in distance (Δd) of an atomic pair between the structural model and the fit result generated by the NNs exceeds 0.2 Å, the FEFF in the Larch code automatically recalculates the theoretical EXAFS of the path with the new distance obtained from the optimal fit. The specifics of data fitting for the series of EXAFS data are outlined in the Supplementary materials.

Fig. 3
Fig. 3
Full size image

(a) XANES from PtOx at the Pt L3 edge during heating. (b) and (c) the EXAFS data in the k-space and the Fourier-transformed EXAFS data in the r-space. The solid lines in (c) represent the best fits obtained using the deep RL method. The data in the k range of 2.5–10.5 Å-1 was used for the Fourier transform with a Hanning window and a window sill width of 0.5 Å-1. The data in the region of r = 1.3–3.5 Å was fitted.

Figure 3 (c) demonstrates the successful simultaneous fitting of the series of EXAFS data sets using the deep RL method. The best fits and the EXAFS data sets in k-space are displayed in the supplementary materials. The analysis results are summarized in Table 2. The local structural properties of PtOx on SiO2 supports are in good agreement with previous reports on PtOx supported on transition metal oxides25,27. The quantitative analysis of the EXAFS data sets indicates the presence of two different bond lengths of Pt-Pt pairs below 100 °C, with the Pt-O peak dominating over the Pt-Pt peaks. At 150 °C, the N value of oxygen decreases significantly and is mostly absent above 200 °C. The mean size of Pt nanoparticles at RTc was found to be around 20 nm based on transmission electron microscopy (TEM) measurements28. Prior research has indicated that Pt nanoparticles on metal-oxide supports often exhibit shapes resembling a half oval or a pancake. Additionally, these Pt nanoparticles were found to be incompletely crystallized. The coordination number of approximately 11 aligns with these earlier observations. This size, combined with N = 11.0 ± 0.5 for Pt atoms at RTc as determined by EXAFS, indicates that the Pt atoms on SiO2 supports have a tendency to aggregate into clusters at elevated temperatures. In the scenario where oxygen atoms exclusively bond to the surface of the 20 nm Pt nanoparticles, the N of Pt atoms would remain constant across temperatures ranging from RT to 350 °C, and the oxygen coordination would be minimal even at room temperature. This is because EXAFS measurements capture the average structural characteristics of the entire sample. A high coordination number value for oxygen at lower temperatures suggests that oxygen atoms are positioned between Pt atoms within the PtOx nanoparticles. The segregation of oxygen between Pt atoms leads to the creation of Pt-O-Pt pairs at lower temperatures, accompanied by a significant degree of disorder in the buckling angle of Pt-O-Pt pairs. Upon heating the Pt nanoparticles in an environment containing hydrogen (H2), the Pt atoms undergo a gradual reduction and agglomeration process, resulting in the formation of Pt clusters with a size of 20 nm and a bond length of approximately 2.77 Å. The details regarding the formation and dispersion of Pt clusters on supports have been extensively discussed in previous literature sources25,27,28,29.

Table 2 Fit results of EXAFS data for PtOx/SiO2 at the Pt L3 edge with \(\:{S}_{0}^{2}\) = 0.86. The EXAFS data and the best fits are depicted in Fig. 3. RTc refers to room temperature when heated up to 350 °C and subsequently cooled down.
Full size table

In the subsequent step, more complex systems are investigated by examining EXAFS data sets using deep RL methods. A series of in-situ EXAFS data sets were collected during the crystallization of ZnO in an aqueous solution at the Zn K edge. Figure 4 displays the XANES and EXAFS spectra of Zn-O complexes under different solution conditions. In the XANES spectra (Fig. 4a), a consistent absorption edge of the Zn-O complexes suggests minimal changes in the oxidation state of Zn atoms during ZnO crystallization in the solution. The analysis focuses on the EXAFS data in the region of k = 2.5–10.5 Å−1, as depicted in Fig. 4 (b) and (c). The theoretical EXAFS of the single- and multiple-scattering paths involving wurtzite ZnO and a cluster radius of 6.0 Å are calculated using FEFF in the Larch code. The EXAFS data analysis of Zn-O complexes follows a similar methodology to that of PtOx nanoparticles, as discussed previously. The analysis begins with the EXAFS data of ZnO powder and progresses from the Sol1 to Nano EXAFS data sets utilizing the trained NNs with the fits of the ZnO powder EXAFS data. For the analysis of the ZnO powder EXAFS data, the NNs automatically select the scattering paths of Zn-O(1), Zn-O(2), Zn-Zn(1), Zn-Zn(2), and Zn-O(3) within the fitting range of r = 1.3–3.4 Å based on exclusion criteria. The Ns and ds of the scattering paths are initially derived from the wurtzite phased ZnO structure. The ΔE0 and the σ2 of the atomic pairs are initially set to 0 and 0.005 Å2, respectively. The NNs of the deep RL method fit the theoretically calculated EXAFS to the measured data by iteratively adjusting the four parameters of ΔE0, N, d, and σ2 for each scattering path. The exclusion criteria are automatically applied to new trials after achieving an optimal fit. The fitting range for the Zn-O complexes starts at r = 1.3–3.9 Å and is adjusted based on the exclusion criteria. Ultimately, the fitting ranges are set as r = 1.3–2.3 Å for Sol1, r = 1.3–3.9 Å for Sol2 and Sol3, and r = 1.3–3.4 Å for Sol4, Nano, and ZnO powder, respectively. Once the fitting ranges and paths for each EXAFS data set are determined, the NNs iterate the fitting process for the series of Zn-O complexes EXAFS data until the termination criteria are met. In the fitting of a series of EXAFS data, the NNs select the optimal fit results from the preceding EXAFS as the initial parameter values for the subsequent EXAFS data fitting, as elaborated in the supplementary materials. Figure 4 (c) illustrates the series of in-situ EXAFS data and the best fits achieved simultaneously through the deep RL method. The best fits and the EXAFS data sets of Zn-O complexes in k-space are provided in the supplementary materials. The fitting results are summarized in Table 3. During the Sol1 and Sol2 stages, both N and σ2 values determined using the deep RL method are slightly larger than those obtained through conventional methods10. However, the distances of the Zn-O pairs in the solutions remain nearly identical regardless of the fitting methods used. In EXAFS data fitting, there are correlations between parameters, particularly N and σ2. To mitigate this correlation, a k-weight fit is commonly employed in conventional EXAFS data fitting methods30. It is important to note that in the deep RL method, the fitting process progresses iteratively with one parameter at a time until the termination condition of the fit is satisfied. The slight discrepancies in the fit results of the Zn-O complex may be attributed to differences in the fitting processes between conventional methods and the deep RL approach.

Fig. 4
Fig. 4
Full size image

(a) XANES data from the Zn-O complex at the Zn K edge during the crystallization of ZnO in an aqueous solution. (b) and (c) the EXAFS data in the k-space and the Fourier-transformed EXAFS data in the r-space. The solid lines in (c) represent the best fits. The data in the k range of 2.5–10.5 Å-1 was used for the Fourier transform with a Hanning window and a windowsill width of 0.5 Å-1.

Table 3 Fit results of EXAFS data for the Zn-O complex at the Zn K edge. The EXAFS data and the best fits are illustrated in Fig. 4. \(\:{S}_{0}^{2}\) = 0.95 was derived from the EXAFS data fit of a ZnO powder and was subsequently utilized as a fixed parameter in the subsequent data fits of the Zn-O complex.
Full size table

The analysis outcomes of the EXAFS data series for PtOx and Zn-O complexes utilizing the deep RL method indicate that AI techniques have the potential to replace manual labor in EXAFS data analysis, although the process is not entirely automated in this particular study. The research demonstrates that once the neural networks (NNs) of the deep RL methods are trained to fit the EXAFS data of a known system, they can automatically fit a series of EXAFS data sets from similar systems. This approach proves to be valuable for analyzing numerous data sets, particularly those generated from in-situ or operando experiments, to comprehend instantaneous changes in phenomena. The ability of AI techniques to streamline and expedite the analysis of complex data sets holds promise for enhancing the efficiency and accuracy of material characterization and understanding dynamic processes in various scientific fields. Several research groups have made progress in developing AI techniques for analyzing EXAFS data sets14,15,16,17, and our further development of AI techniques using deep RL has shown promise. Indeed, while AI techniques show promise in automating the analysis of EXAFS data sets, there are significant challenges that must be overcome for their full utilization. Firstly, AI methods need to be able to accurately select structural parameters such as space groups, lattice constants, and atomic positions for theoretical EXAFS calculations. This is crucial for ensuring that the calculated EXAFS data aligns well with the experimental data. Secondly, the selection of scattering paths, both single and multiple, for fitting complex systems is a critical aspect that requires careful consideration, especially in cases where there is a high degree of structural disorder. Thirdly, AI techniques must verify the physical significance of fitting results and decide whether to accept the best fits or retry fits with new structural models and fitting conditions. Additionally, the NNs of an AI technique should be well-trained with diverse EXAFS data sets to ensure robust performance. Furthermore, AI techniques need to evaluate and determine the reliability of measured EXAFS data, the appropriate ranges in k- and r-spaces for analysis, the fitting parameters, \(\:{S}_{0}^{2}\), and other factors to ensure the trustworthy application of AI in EXAFS data analysis. While deep RL techniques show promise, especially in cases where there is insufficient data in advance to train the NNs of an AI technique, further development is still necessary. The use of AI techniques for big data is rapidly expanding across various fields, including sales and consumer analysis31, medical data2, and public institution data32. In academic research, AI techniques are increasingly being utilized in experimental designs33, scientific data analyses14,15,16, and various other research areas. The integration of AI techniques in EXAFS data analysis holds great potential for advancing our understanding of complex systems and materials.

Conclusions

The use of RL methods represents a powerful approach for the quantitative analysis of EXAFS data sets. Deep RL techniques offer the advantage of not requiring a large number of similar data sets in advance to train the NNs. This flexibility allows for additional theories and fitting conditions to be easily applied and modified within the techniques. One key strength of deep RL methods is the ability for the agent and the environment of the A3C algorithm to work both separately and collaboratively. This enables deep RL techniques to potentially determine data quality, select appropriate structural models, and interpret the physical significance of fitting results autonomously as they are further developed. The progress of deep RL techniques in the analysis of EXAFS data holds the promise of revolutionizing the field and making substantial contributions to academic research. By leveraging the capabilities of deep RL methods, there is the potential to redefine the boundaries of data analysis and introduce a cutting-edge approach that can significantly enhance our comprehension of phenomena, especially those related to instantaneous changes.

Methods

Preparation of specimens

For EXAFS data analysis using a deep RL method, the first series of EXAFS data sets were obtained during the reduction process of PtOx using in-situ XAFS measurements. PtOx nanostructures were synthesized by impregnating tetraamineplatinum(II) nitrate (Pt(NH3)4(NO3)2) with a purity of 99% on SiO2 supports using an incipient wetness method28,29. The second series of EXAFS data was obtained during ZnO crystallization. The process involved starting with a zinc nitrate hexahydrate (Zn(NO3)2․6H2O) solution at room temperature (RT) (Sol1). Hexamethylenetetramine (HMT, C6H12N4) was then added to Sol1 at RT (Sol2). Sol2 was heated up to 60 °C in a water bath (Sol3) and continuously kept under the same condition for 6 h (Sol4). When Sol4 was heated up to 80 °C, a precipitate was formed at the bottom, which was then dried in air (Nano). Additional information regarding the crystallization process of ZnO in an aqueous solution can be found in the literature reference10.

In-situ XAFS measurements

In-situ XAFS measurements with a transmission mode were performed from PtOx/SiO2 at the Pt L3 edge during heating from RT to 350 °C under an H2 environment. In-situ XAFS measurements with a fluorescence mode were performed from the solutions during ZnO crystallization and the dried precipitate at the Zn K edge. To ensure accurate data collection, the total absorption lengths of the PtOx powder and the Zn-O solutions were kept to be less than unitary at the Pt L3 and the Zn K edges, respectively, to exclude unexpected errors in the XAFS data. Further details of the in-situ XAFS measurements of PtOx nanostructures and Zn-O complex solutions can be found in the literature references provided10,25,27.