Machine learning methods for predicting residual strength in corroded oil and gas steel pipes

Abstract

This review examines machine learning approaches for predicting pipeline residual strength, which is crucial for assessing operational lifespan and safety in industrial applications. We analyze various machine learning models, data preprocessing methods, and evaluation metrics used in existing research. The study highlights how data characteristics and model selection influence prediction accuracy, providing practitioners with guidelines for model implementation, while discussing current challenges and future research directions.

Introduction

Pipelines are pivotal in the transportation system, serving as the primary means of conveying oil and gas^1,2,3. As the operational time of pipelines increases, they inevitably suffer from corrosion^4,5,6,7,8. Corrosion can significantly diminish the structural integrity of pipelines, ultimately resulting in the possibility of pipeline leakage or fracture^9,10,11. The failure of oil and gas steel pipelines can result in catastrophic economic and environmental consequences^12,13,14,15. Figure 1 shows oil and gas pipeline incidents reported by the Pipeline and Hazardous Materials Safety Administration (PHMSA) of the U.S. Department of Transportation¹⁶. The residual strength of a pipeline refers to its maximum capacity to withstand loads^17,18. Considering the severe consequences of failure, accurately predicting the residual strength of corroded pipelines holds significant importance.

Fig. 1: Pipeline incident statistics from multiple perspectives¹⁶.

The number of accidents, fatalities, injuries, and total cost of pipelines reported by PHMSA from 2004 to 2023.

Predicting residual strength in pipelines typically involves three primary methods: empirical formula, finite element analysis, and machine learning. Introduced in the late 1960s, the NG-18 formula was instrumental in calculating pipeline residual strength and served as a foundation for the development of later empirical formulas¹⁹. The ASME introduced the ASME B31G evaluation criteria, which were established through a combination of theoretical research and extensive burst test^20,21,22. Formulated in the 1980s, the formula relies on a limited amount of corrosion data, which reduces its versatility. Due to the conservatism of ASME B31G, which often leads to unnecessary pipeline replacements, Kiefner proposed the ‘0.85dl’ method in the PRCI report²³. DNV and BG Technology jointly developed DNV RPF-101 by integrating data from full-scale burst tests and 3D nonlinear finite element simulations. This model accounts for the impact of external loads on the pipeline’s load-bearing capacity, and extensive empirical research has validated the DNV formula as one of the most effective methods for predicting pipeline residual strength²⁴. However, this method is inapplicable to pipelines with X80 or higher grades. Liu et al. extended this formula by introducing a modified defect depth to account for the impact of defect quantity on the results²⁴. The Battelle Laboratory in the United States developed the renowned PCORRC method, based on the theory of plastic collapse failure, for calculating residual strength of medium to high-toughness corroded pipelines²⁵. Due to the simplicity of these formulas, they have been widely applied in engineering practice²⁶. These formulas show considerable evaluation errors and a high degree of conservatism. Although greater conservatism enhances pipeline operational safety, it may reduce operational efficiency.

Due to the significant errors in empirical formulas, more researchers in recent years have adopted finite element analysis to evaluate the residual strength of corroded pipelines. In the early years, limited computational power led researchers to use 2D finite element models to calculate pipeline residual strength^27,28. The use of 2D finite element analysis models cannot account for defect shapes and geometries, which substantially reduces prediction accuracy. With advancements in computational power, more researchers have shifted to using 3D finite element modeling to study pipeline residual strength. Zhang et al. incorporated a safety factor into the pipeline assessment process and employed 3D finite element analysis to model the pipeline under triple failure pressure²⁹. Phan et al. utilized the finite element method to calculate and analyze the burst pressure of corroded pipelines under different conditions³⁰. Ma et al. developed a novel finite element-based predictive formula to estimate the failure pressure of high-strength pipelines, demonstrating strong agreement with experimental data³¹. However, this approach does not consider defect width and is not applicable for evaluating low-strength pipelines. Huang et al. proposed a finite element-based model to calculate the failure pressure of pipelines with dent corrosion defects. The results indicate that dents reduce the failure pressure³². Qin et al. developed a model that considers mechanical-electrochemical (ME) interactions to predict the variation of failure pressure over time and investigated the effects of parameters such as internal pressure, axial tensile stress, and initial defect length³³. While FEA has provided satisfactory results, it is essential to consider the specific characteristics of real-world pipelines and various failure criteria. Achieving high accuracy requires a large number of meshes, which significantly increases computational load and time. Additionally, changes in defect parameters require model reconstruction, making this method inconvenient and inefficient.

With advancements in artificial intelligence technology, numerous researchers have integrated machine learning methods into the prediction of pipeline residual strength. The general framework for predicting residual strength of pipelines using machine learning is shown in Fig. 2. Unlike empirical formulas, machine learning methods can learn the complex relationship between corrosion influencing factors and residual strength from abundant observational data. The research paradigm of residual strength assessment has shifted from the “reasoning” and “knowledge” stages to the current “learning” stage. Ma et al. developed an ensemble learning model using existing empirical formulas to optimize LightGBM through TPE, aimed at studying the impact of varying pipeline and corrosion conditions on pipeline burst pressure³⁴. Using ensemble learning to optimize empirical formulas effectively addresses the issue of overly conservative empirical formulas. The presence of empirical formulas can also help mitigate the poor interpretability of machine learning models. Ma et al. incorporated empirical formulas into the neural network training process, leveraging the prior knowledge from these formulas to guide the training, thereby proposing a theory-guided machine learning model³⁵. Lu et al. proposed a novel theoretical formula, specifically designed for multipoint corroded pipelines, by modifying the existing pipeline corrosion criterion. In addition, a residual strength prediction model for multipoint corroded pipelines was developed using RBFNN and data from existing literature machine learning models⁶. Lu et al. combined PCA and SVM techniques, optimized SVM parameters through multi-objective optimization methods, and introduced a novel pipeline residual strength prediction framework, achieving higher accuracy and stability¹⁸. Wang and Lu integrated the prediction outputs of multiple individual models through the Stacking model, establishing a residual strength prediction model with higher accuracy and generalization performance¹⁷.

Fig. 2: General framework for predicting residual strength of pipelines using machine learning.

Flowchart for predicting the residual strength of corroded pipelines using machine learning, including: Data Collection, Preprocessing, Model Training, Testing, and Burst Pressure Prediction.

Machine learning has become a research hotspot in the field of residual strength prediction for oil and gas pipelines. Nevertheless, the field currently lacks a comprehensive systematic review. Therefore, it is necessary to conduct a comprehensive review of the current development and application scope of machine learning technologies in this field. To address this issue, this paper provides a comprehensive review of the application of machine learning techniques in predicting the residual strength of oil and gas pipelines. The review aims to highlight the most widely used techniques, emphasizing the essential machine learning approaches for residual strength prediction. Covering topics from raw data acquisition and preprocessing techniques to commonly used prediction models and their performance evaluation. This review emphasizes the limitations of current machine learning research in predicting the residual strength of corroded oil and gas pipelines, while providing comprehensive and innovative perspectives on future advancements in the field.

The remainder of this paper is organized as follows. The “Research methodology” section introduces the research methods used in this study. The “Data collection and preprocessing” section reviews methods for collecting and processing data for machine learning. The “Machine learning models” section summarizes commonly used machine learning models, including single and hybrid models. The “Evaluation metrics” section provides an overview of the indicators that have been used to evaluate model performance in existing studies. The “Discussion” section discusses the impact of data size and division ratio, and machine learning model type on model performance, and summarizes the frequency of use of each evaluation metric. The “Challenges and perspectives” section identifies the challenges in current research and outlines directions for future studies. The meanings of the abbreviations in the text are found in Table 1.

Table 1 List of abbreviations

Research methodology

The methodology of previous research reviews provides valuable guidance for this research^14,36,37. The methodology used in this research is the systematic literature review (SLR)¹⁴ and consists of three essential stages: databases and keywords, inclusion and exclusion criteria, and analysis of publications.

Databases and keywords

To conduct a detailed search for articles related to the application of machine learning in residual strength prediction of oil and gas pipelines, three databases were used in this research: Scopus, Web of Science, and ScienceDirect. By summarizing the keywords in the related articles, the search was performed using the following keywords: “pipeline” AND (“residual strength” OR “burst pressure” OR “failure pressure” OR “remaining strength”) AND (“machine learning” OR “artificial intelligence” OR “neural network”). The scope of the search was conference papers and journals published between 2000 and 2024.

Inclusion and exclusion criteria

After the first search, the retrieved articles were organized based on PRISMA criteria³⁸. The inclusion criteria include that the articles are peer-reviewed and in English, the articles are based on machine learning models, and the articles predict the residual strength of oil and gas pipelines. Articles that do not meet the above criteria will be excluded.

Analysis of publications

Through a detailed analysis of the retrieved articles, 36 papers closely aligned with the research content were manually selected. Figure 3 shows the yearly distribution of the research articles. As can be seen from the figure, more and more researchers have started to work in this field in the last few years. It also shows that the field is evolving and that it makes sense to engage in relevant research.

Fig. 3: Number of articles published per year.

The number of articles published annually on the application of machine learning in predicting the residual strength of pipelines.

Data collection and preprocessing

Data parameters

Data plays a crucial role in machine learning, serving as the foundation for developing machine intelligence. Similar to the human learning and decision-making process, machine learning methods require large amounts of data to discover patterns, extract features, and generate accurate predictions. The quality, diversity, and comprehensiveness of data greatly influence the model’s performance and its ability to generalize. In machine learning, approximately 80% of the modeling time is spent on data-related tasks³⁹. Therefore, data collection, cleansing, and processing are crucial steps for ensuring the success of machine learning. Table 2 summarizes the data parameters used in previous studies to predict pipeline residual strength using machine learning methods. These parameters are categorized into three main groups: Pipe size parameters, Material parameters, and Defect parameters. When investigating the residual strength of a pipeline under specific conditions, specific external condition parameters are required.

Table 2 Summary of data parameters

Data collection methods

Full-scale burst test

The corroded pipeline full-scale burst test is the most important experimental method for evaluating the structural safety of pipelines. This test considers key parameters such as the material, pipe size, and corrosion of the pipeline to ensure the reliability and applicability of the test results. By utilizing advanced equipment such as high-speed cameras and pressure sensors to record and analyze critical data from the bursting process, researchers can gain insight into the pipeline’s load-bearing capacity, damage patterns, and causes of failure, as shown in Fig. 4. This testing methodology provides valuable reference and decision support for pipeline design and maintenance, while also offering substantial data and a foundation for pipeline safety management and risk assessment.

Fig. 4: The schematic diagram of full-scale burst test.

The schematic diagram of the setup and connections for a full-scale burst test on corroded pipelines.

Full-scale burst test data plays a crucial role in predicting pipeline residual strength. On one hand, it directly supports the training and validation of models. On the other hand, it serves to verify the accuracy of models developed using finite element methods and other methods. To investigate the behavior of pipelines with corrosion defects, Benjamin et al. conducted burst tests on nine tubular samples, each featuring a single simulated external corrosion defect⁴⁰. The experimentally measured burst pressures were compared and analyzed against the calculated results from empirical formulas, revealing that traditional empirical formulas tend to be overly conservative in estimating burst pressure. Benjamin et al. conducted full-scale burst tests on seven pipelines with corrosion defects and compared the results with six empirical formulas⁴¹. However, the availability of such data in open literature remains limited. On the one hand, due to the complexity and risks of testing, it is necessary to strictly comply with relevant safety standards and regulations under the guidance and supervision of professionals. On the other hand, conducting full-scale burst tests on corroded pipeline samples requires substantial time and financial resources. Machine learning requires a large amount of data, and insufficient data prevents the model from capturing nonlinear relationships in the data, which significantly impacts its performance.

Finite element model

Directly obtaining pipeline burst pressure data through burst tests is both inconvenient and costly. In such case, the finite element method serves as a crucial approach for obtaining pipeline burst pressure data. The finite element model simulates the mechanical behavior of pipelines under varying pressures by defining the geometric model, material properties, and operational boundary conditions. The accuracy of the model can be verified by comparing the finite element model with the real pipeline burst test data, facilitating the simulation of rupture pressure under different working conditions. Vijaya Kumar et al. generated data using a finite element model and developed an ANN model to predict the residual strength of corroded pipelines. The results demonstrated significant accuracy⁴². Zhang et al. obtained a burst pressure dataset through finite element analysis and utilized it to train neural networks for predicting pipeline failure pressure²⁹.

While finite element models effectively generate burst pressure datasets, they face notable limitations in practical applications. Assumptions in the modeling process, material parameter determination, selection of ontological models, meshing strategies, and model complexity can influence the results. Moreover, finite element models require high-performance computing resources, and obtaining large amounts of data incurs significant time and financial costs. Therefore, these factors need to be carefully considered when using a finite element model to obtain pipeline burst pressure data and combined with other reliable information and methods for comprehensive evaluation and analysis.

Data generation model

Pipeline burst tests are time-consuming and costly, while finite element models have limited applicability to various pipeline configurations. Literature research indicates that many researchers employ data generation models to create previously unavailable datasets^43,44. In recent years, GAN models have emerged as a new approach for generating synthetic data^{45,46,47,48,49,50}. Figure 5 shows the basic framework of GAN. GAN is a deep neural network model that learns the data distribution from user-provided training data and generates synthetic data based on this learned distribution. Jain et al. utilized a GAN architecture to generate synthetic data representing six types of pipeline surface defects⁵¹. Xu et al. employed GAN to synthesize fault data, which was then used to evaluate and classify pipeline conditions, aiming to prevent leaks⁵². In the area of corroded pipeline burst pressure research, He and Zhou utilized 258 real burst pressure data collected from the literature to train a Table GAN model for generating a dataset of pipeline burst test pressures⁵³. The generated database was used to train RF and DT models for predicting pipeline residual strength.

Fig. 5: The basic framework of GAN¹¹⁹.

The basic framework of GAN involves a generator that creates synthetic data and a discriminator that distinguishes between real and fake data, with both components being trained through adversarial competition to improve data generation.

Compared to full-scale pipeline burst tests and finite element methods, GAN significantly reduces experimental and computational costs, enabling the rapid generation of large-scale datasets without additional expenses. However, as a machine learning model, the effectiveness of GANs also depends on the availability of extensive and reliable residual strength datasets for corroded pipelines.

Data processing

Employing appropriate data preprocessing methods is crucial for ensuring reliable and accurate research outcomes. Many researchers have applied standard methods or developed specific methods for data processing as it directly affects the prediction results and their accuracy and reliability. The main work of data processing is the processing of features, which consists of two main steps: feature engineering and normalization.

Feature engineering

An excess or deficiency of model features has the potential to diminish prediction accuracy, and the inclusion of redundant or repetitive features can likewise impact computational speed. Feature engineering enables identifying the most important features for the model, reducing feature count, lowering computational complexity, and improving both performance and interpretability¹⁴. The common feature engineering methods include feature selection and feature transform. Feature selection is primarily used to identify and rank the most relevant features in a model. The Pearson correlation coefficient is the most commonly used feature selection method. And it can be calculated using the Eq. (1)⁵⁴:

$${p}_{c}=\frac{\mathop{\sum }\nolimits_{i=1}^{N}\left({x}_{i}-\bar{{x}_{i}}\right)\left({y}_{i}-\bar{{y}_{i}}\right)}{\sqrt{\mathop{\sum }\nolimits_{i=1}^{N}{\left({x}_{i}-\bar{{x}_{i}}\right)}^{2}\mathop{\sum }\nolimits_{i=1}^{N}{\left({y}_{i}-\bar{{y}_{i}}\right)}^{2}}}$$

(1)

where ${x}_{i}$ is the feature value; ${y}_{i}$ is the label value. The larger the absolute value of ${p}_{c}$, the higher the correlation between the two factors⁵⁵. Ma et al. used Pearson correlation coefficient to study the correlation of 14 features in the feature space and ranked the correlation of the features from small to large³⁴. Ma et al. applied Pearson correlation analysis to select the three most relevant features out of eight for machine learning modeling to predict the residual strength of corroded oil and gas pipelines³⁵.

Unlike feature selection, feature transform maps the original features to a lower-dimensional space to achieve feature dimensionality reduction. Dimensionality reduction minimizes data dimensions while retaining its essential information, with PCA being a widely adopted method for this purpose^56,57. PCA maps high-dimensional data to a lower-dimensional subspace while maximizing the variance of the data. The principal components’ ability to explain the variance in the original data is represented by Eq. (2)⁵⁸:

$${g}_{i}=\frac{{\lambda }_{i}}{{\lambda }_{1}+{\lambda }_{2}+\ldots +{\lambda }_{m}}$$

(2)

where ${g}_{i}$ denotes the explained variance of principal component $i$; $m$ denotes the number of principal components; ${\lambda }_{i}$ denotes the feature value of the principal component.

Lu et al. utilized PCA to reduce eight features from the raw pipeline burst pressure data and ranked the contribution of each parameter to burst pressure in descending order. The results show that the top seven features account for 99.235% of the contribution to burst pressure. These parameters were selected as input variables to reduce the model’s complexity¹⁸. Phan and Duong combined PCA with ANFIS to predict pipeline burst pressure, and the ANFIS-PCA model outperformed existing models, achieving a determination coefficient of 0.9919 and reducing the RMSE to 0.9883 MPa⁵⁹.

In recent years, several manifold learning algorithms have been developed for data dimensionality reduction, including ISOMAP⁶⁰, LLE, and KPCA⁶¹. Li et al. utilized LLE to reduce the dimensionality of submarine pipeline corrosion data and then employed the preprocessed data to train a BRANN model for predicting the residual strength of underwater pipelines with dual corrosion defects⁶¹. The LLE-processed data and KCPA-processed data were separately fed into the BRANN for comparison. The BRANN model trained with raw data alone exhibited the lowest predictive performance, highlighting the importance of data dimensionality reduction.

Normalization

Due to the different data dimensions of each parameter, normalization helps ensure the consistency of scales between different features and variables, speeding up the training process, and reducing the model’s sensitivity to specific features. The calculation formula is shown in Eq. (3)⁶².

$${z}_{n}=\frac{{z}_{i}-{z}_{\min }}{{z}_{\max }-{z}_{\min }}$$

(3)

where ${z}_{n}$ is the normalized data, range from 0 to 1; ${z}_{i}$ is the original data; ${z}_{\max }$ is the maximum original data; ${z}_{\min }$ is the minimum original data.

Data division

Since the accuracy of predictions cannot be predetermined, validation is usually performed using established datasets. The raw data needs to be divided into a training set, a validation set, and a test set before model training. The training set is used to train the model, the validation set is used to tune the model’s hyperparameters and for performance evaluation, and the test set is used for the final evaluation of the model’s performance. In this paper, the data sizes and test set ratios of the 30 literature collected are summarized in Table 3.

Table 3 Data size and division in the existing literature

Table 3 indicates that during the modeling process for predicting the residual strength of pipelines, the available data is significantly limited. Dividing the dataset into three subsets enables an accurate evaluation of the model’s generalization ability and mitigates overfitting. In scenarios with limited data, the dataset is often divided into a training set and a test set, and K-fold cross-validation is used to optimize the model on the training set. This approach provides a more stable evaluation of the model’s performance under limited data conditions. Figure 6 shows the dataset splitting strategy using 5-fold cross-validation.

Fig. 6: Schematic diagram of 5-fold cross-validation process for model training and validation.

The 5-fold cross-validation process splits the dataset into five subsets, using each once for validation and averaging the results to ensure robust model evaluation.

Machine learning models

In the context of pipeline residual strength prediction, a machine learning model refers to an algorithm that transforms pipeline data with fixed characteristics into predictions of burst pressure. Common machine learning models can be classified into “Single Models” and “Hybrid Models”. The former involves a straightforward prediction process using a single algorithm, while the latter combines multiple simple models to enable more complex predictions. Hybrid models can be further divided into “Ensemble Models” and “Improved Models”. “Ensemble model” combines multiple machine learning techniques into a unified predictive model, whereas “Improved model” integrates a single machine learning model with an optimization method.

Single models

Single model refers to an approach that uses only one machine learning model for prediction. Based on a review and analysis of the literature, the commonly used single models for pipeline residual strength prediction are outlined below.

Statistical regression model

Statistical regression models are employed to model the relationship between one or more independent variables and a dependent variable. The linear regression model is the simplest statistical regression model, as shown in Fig. 7a. The linear model predicts the residual strength by establishing a linear relationship, as shown in the Eq. (4) and Eq. (5):

$$y={w}_{0}{x}_{0}+{w}_{1}{x}_{1}+\ldots {w}_{n}{x}_{n}+b$$

(4)

$$J(W,b)=\frac{1}{n}\mathop{\sum }\limits_{i=1}^{m}({y}_{i}-{(W\cdot {X}_{i}+b))}^{2}$$

(5)

where ${w}_{n}$ is the weight; ${x}_{n}$ is the characteristic value; $b$ is bias. $y$ is the predicted value; ${y}_{i}$ is the real value; $J(W,b)$ is the value of the loss function.

Fig. 7: Schematic representation of four basic machine learning models.

a The schematic diagram of the LR model¹²⁰. b The schematic diagram of the KNN model¹²¹. c The schematic diagram of the SVM model⁹⁷ and d the schematic diagram of the ANN model²⁹.

Linear models assume linear relationships, which means they may fail to capture complex nonlinear relationships accurately. For high-dimensional data like pipeline data, linear models need more degrees of freedom to fit the data, which may lead to overfitting. Regularization penalties are typically applied to prevent overfitting. Cai used three regression methods with different regularization strategies, including Ridge⁶³, Lasso⁶⁴ and ElasticNet⁶⁵ for model training⁶⁶. As shown in Fig. 8, the ElasticNet strategy shows high accuracy on the training set, validation set, and test set. A comparison of ANN, SVR, and LR models revealed that the ANN model achieved the highest training accuracy but exhibited significant overfitting. The training accuracy of the LR model decreased slightly, but it maintained relatively high validation accuracy. Among the three models, the linear model demonstrated the best predictive performance. However, this conclusion is based on a limited dataset and may not always be generalizable. Additional experiments are needed to confirm its reliability. Another statistical regression model applied in the field of pipeline residual strength prediction is GPR⁶⁷.

Fig. 8: Accuracy of LR models with different regularization strategies⁶⁶.

a The R² of the model and b the MSE of the model.

K nearest neighbor

KNN performs regression prediction by calculating the distance between a given data point and its nearest K neighbors in the training set, enabling it to handle data intuitively and straightforwardly⁶⁸, as shown in Fig. 7b.

A review of the existing literature reveals that Wang et al.⁶⁹ and Abyani et al.⁶⁷ used the KNN method in their studies. However, it was primarily used as a benchmark for comparison against the main models discussed in the respective works. Because KNN is sensitive to the selection of K value and high computational complexity, it is not very effective in practical application. The DT is a frequently employed classification-based technique. The literature review has revealed a scarcity of studies utilizing a single DT for modeling in the domain of pipeline residual strength prediction. It is common to use multiple decision trees for ensemble learning, which will be detailed in the following sections.

Support vector machine

SVM performs classification and regression tasks by determining a hyperplane that best fits the data while minimizing prediction errors^70,71, as shown in Fig. 7c. SVR was introduced as an extension of SVM to deal with regression problems by fitting a function that closely approximates the actual data points. Since residual strength prediction is a regression task, SVR is widely used in the prediction of pipeline residual strength.

There are three important parameters in SVR, including the regularization coefficient (C), the kernel width coefficient (γ), and the type of kernel, all of which are used for grid search. Cai et al. found that the SVR model with an RBF kernel outperformed other models when compared with SVR models using different kernel types⁶⁶, as shown in Fig. 9.

Fig. 9: SVR model accuracy with different kernel functions⁶⁶.

a The R² of the model and b the MSE of the model.

To address the issues of excessive support vectors and challenging hyperparameter selection in traditional SVM, Tipping proposed the sparse kernel method known as RVM⁷². Lu et al. was the first to apply RVM to pipeline residual strength prediction, optimizing the correlation vector machine using the multi-objective salp swarm algorithm, which enhanced both the accuracy and stability of the prediction¹³.

Artificial neural networks

ANN, a widely used technique in pipeline residual strength prediction, consists of three layers: input, hidden, and output, as illustrated in Fig. 7d. The calculation formula is shown in Eq. (6):

$${H}_{l}=\sigma ({w}_{l}X+{b}_{l})$$

(6)

where ${w}_{l}$ represents the weight, ${b}_{l}$ represents the bias, $\sigma$ is the activation function, and ${H}_{l}$ is the output of the neural network.

ANN is a generic term encompassing various specific types of networks tailored for different situations, each with varying complexity and accuracy levels. Table 4 lists the ANN models commonly used in pipeline residual strength prediction.

Table 4 Common types of ANN

Simple ANN structures face significant limitations in addressing highly nonlinear problems. With advancements in technology, deep learning models are increasingly being applied to pipeline residual strength prediction^73,74. Deep learning models contain multiple hidden layers that enable deeper feature extraction from data, significantly enhancing the model’s representational capacity and performance. Chen et al. utilized MLP and PSO-FFNN models to predict the residual strength of the pipeline⁷⁵. The MSE of the MLP model was only 0.15, demonstrating strong predictive performance on the dataset. Gholami et al. utilized MLP and SVR with spline and Gaussian kernel to build ANN models and predict pipeline burst pressure. The results indicate that MLP-ANN is moderately accurate for predicting actual results but is less accurate compared to the Gaussian kernel⁷⁶. Miao and Zhao proposed a novel method for predicting the residual strength of defective pipelines using DELM and enhanced the model’s performance through HTLBO optimization. The model achieved a relative error of less than 6%, indicating superior predictive accuracy compared to other models⁷⁷. Su et al. developed a deep neural network with seven hidden layers, each comprising 250 nodes. The results demonstrate that this deep learning model delivers high prediction accuracy²⁶.

Hybrid models

Ensemble models

To improve predictive models, strategies such as ensemble modeling can be used. Instead of a single machine learning algorithm, ensemble learning enhances the predictive performance by integrating multiple individual models. By reviewing literature, the ensemble methods applied in the field of pipeline residual strength prediction are RF, ETR, GBDT, AdaBoost, XGBoost, LightGBM, and Stacking.

RF is a robust machine learning method capable of performing regression analysis by integrating a collection of decision trees to improve the accuracy of predictive results^78,79. The core concept of RF involves training multiple distinct decision tree models independently and averaging their outputs to predict the test example⁸⁰.

ETR is a regression model based on the concept of random forests, proposed by Geurts et al. in 2006⁸¹. Its core idea is to build decision trees by introducing additional randomness on top of random forests, aiming to enhance the model’s generalization ability and reduce training time. The primary difference from traditional random forests is that ETR randomly selects split thresholds at node splitting instead of performing optimal split searches on candidate features.

GBDT is a gradient boosting algorithm based on decision trees. Its core idea is to gradually optimize model performance through ensemble learning^82,83,84. GBDT combines the iterative optimization ability of weak learners with the theoretical foundation of gradient descent. By using the negative gradient of the loss function under the current model as pseudo-labels, it builds a series of independent decision trees, enabling the model to continuously approximate the true target function.

AdaBoost is an ensemble learning method based on the weighted combination of weak learners, proposed by Freund and Schapire in 1995^82,85,86,87. Its core idea is to combine multiple weak learners into a strong learner by progressively adjusting sample weights and learner weights, thereby improving the model’s generalization performance. Unlike traditional simple weighted averaging, the key to AdaBoost lies in iteratively focusing on samples that are difficult to classify, enabling the model to progressively optimize its performance on these samples.

XGBoost is an efficient gradient boosting framework proposed by Chen and Guestrin in 2016, focusing on balancing computational performance and model predictive capability. As an extension of GBDT, XGBoost incorporates techniques such as regularization, parallel computing, and distributed training, offering significant advantages in large-scale data analysis and high-dimensional feature learning^88,89,90.

LightGBM is an efficient gradient boosting framework proposed by the Microsoft team, focusing on improving training efficiency on large-scale datasets and high-dimensional sparse data^82,91,92,93. Compared to traditional GBDT, LightGBM introduces unique optimization strategies, such as histogram-based decision tree construction and efficient data sampling methods, significantly reducing computational and memory overhead while maintaining model prediction accuracy.

Stacking is an ensemble learning method that combines the predictions of multiple base learners to construct a final strong learner^93,94,95. Unlike traditional ensemble methods, stacking does not simply rely on voting or weighted averaging of base learners, but rather merges the predictions of base learners by training a new learner, often referred to as a meta-learner. The core idea of this method is to improve model prediction performance by utilizing the strengths of different models through multi-level learning.

Ma et al. utilized LightGBM to build an oil and gas pipeline burst pressure prediction model, and the prediction results showed that this model can significantly improve the prediction accuracy³⁴, which was found to be the most accurate by comparing this method with other ensemble learning methods such as RF and XGBoost. Xiao et al. modeled various machine learning models, including RF, AdaBoost, and GBDT, and among all the models, the ANN model exhibited the best predictive performance⁸⁵. Wang and Lu established a Stacking model for predicting the residual strength of corroded oil and gas pipelines. Compared to a single model, the Stacking model exhibits higher prediction accuracy and generalizability¹⁷.

Ensemble methods are divided into three categories: Bagging, Boosting, and Stacking. Categorically, RF and ETR belong to the Bagging algorithm in ensemble learning, GBDT, AdaBoost, XGBoost, and LightGBM belong to the Boosting algorithm in ensemble learning, as shown in Fig. 10. Bagging divides the dataset into multiple datasets by sampling with putbacks to train multiple models separately. For regression problems such as residual strength prediction, Bagging gives a prediction by averaging the results of multiple tests. Boosting employs the same weak learners as Bagging but trains them sequentially in an adaptive manner, with each new learner relying on the previous model and integrating them based on a specific deterministic strategy.

Fig. 10: Schematic illustration of three major ensemble learning methods.

a The schematic diagram of the Bagging model⁶⁷. b The schematic diagram of the Boosting model¹²² and c the schematic diagram of the Stacking model¹⁷.

Improved models

Improved models enhance performance through advanced hyperparameter optimization techniques, significantly improving prediction accuracy. The use of improved models in the literature is shown in Table 5.

Table 5 Improved modeling in pipeline residual strength prediction

Evaluation metrics

Supervised learning models for pipeline residual strength prediction inherently face a trade-off between bias and variance⁹⁶. Variance measures the degree to which a model’s predictions vary when applied to different datasets. A higher variance suggests the model is more prone to overfitting, as it struggles to generalize across different data distributions. In contrast, high bias often leads to underfitting, where the model fails to capture the underlying patterns, resulting in poor performance on both training and new datasets.

Adjusting model complexity is key to addressing this trade-off. Increasing complexity can reduce bias but may lead to higher variance, while simplifying the model decreases variance at the cost of greater bias. Therefore, achieving good generalization performance requires finding an optimal balance between bias and variance.

The performance of supervised learning models can be evaluated using the following metrics:

$${R}^{2}=1-\frac{\mathop{\sum }\nolimits_{i=1}^{n}{\left({y}_{i}-{p}_{i}\right)}^{2}}{\mathop{\sum }\nolimits_{i=1}^{n}{\left({y}_{i}-\bar{y}\right)}^{2}}$$

(7)

$${\rm{MSE}}=(1/n)* \sum {({y}_{i}-{p}_{i})}^{2}$$

(8)

$${\rm{RMSE}}=\sqrt{\frac{1}{{\rm{n}}}\mathop{\sum }\limits_{i=1}^{n}{({p}_{i}-{y}_{i})}^{2}}$$

(9)

$${\rm{MAE}}=\frac{1}{n}\mathop{\sum }\limits_{i=1}^{n}|{y}_{i}-{p}_{i}|$$

(10)

$$\text{MAPE}=\frac{1}{n}\mathop{\sum }\limits_{i=1}^{n}\left|\frac{{p}_{i}-{y}_{i}}{{y}_{i}}\right|\times 100 \%$$

(11)

where $i$ represents each sample; $n$ represents the total number of samples; ${y}_{i}$ is the true value, ${p}_{i}$ is the predicted value, $\bar{y}$ is the mean value of the samples.

Discussion

The previous section covered data collection and processing, commonly used models for pipeline residual strength prediction, and model evaluation metrics. In practice, the performance of a machine learning model is influenced by factors such as dataset size, preprocessing methods, and model type. This section will explore these aspects in detail.

Data size and division ratio

The size of the dataset is a critical factor that influences the accuracy of the model. The largest dataset of pipeline burst pressure in the reviewed literature contains 1843 data points. These data are far from sufficient for machine learning models. Generally, the performance of a model increases with the size of the data. However, the accuracy of the model does not continuously improve with an increase in data size, and there is a limit to how much data size can enhance the model’s accuracy. In most of the literature, the burst pressure data used for predicting pipeline residual strength is very limited. Without the utilization of FEM and other data generation methods, relying solely on data from full-scale burst tests may result in predictions that fall short of the required accuracy.

In machine learning, the original data is typically divided into three parts: training set, validation set, and test set. A review of the literature reveals that only 12 out of 30 papers employed this three-part division, while the remaining 18 papers divided the data into only training and test sets, as illustrated in Fig. 11. Due to the limited dataset, the original data is often divided into training and test sets only. K-fold cross-validation is employed during training for hyperparameter tuning, which helps mitigate the accuracy limitations caused by the insufficient data volume. As shown in Fig. 11, in pipeline residual strength prediction, the training set typically constitutes more than 70% of the dataset to ensure a high degree of model accuracy.

Fig. 11: Ratio of dataset divisions.

The distribution of dataset division ratios in literature.

Data preprocessing analyze

Preprocessing raw data has been demonstrated to enhance data quality, thereby increasing the model’s accuracy. Table 6 presents three sets of comparative data extracted from literature^35,59,61, grouped as ①②③, ④⑤, and ⑥⑦. Comparison of the data in the table indicates that preprocessing techniques such as PCA, KPCA, and LLE enhance model accuracy, particularly when the initial accuracy of the model is relatively low. When the model’s accuracy is relatively low, certain data preprocessing techniques can significantly enhance predictive accuracy. However, when the model’s accuracy is already high, traditional preprocessing techniques show limited direct improvement in performance. Nonetheless, data processed by these techniques can serve as a foundation for further enhancing model performance.

Table 6 Comparison of data preprocessing performance

Comparative analysis of models

This literature review categorizes models into two main types: single models and hybrid models. Statistical analysis of the literature has been conducted, and the test performance of single, ensemble, and improved models is summarized in Tables 7–9. Due to variations in external factors such as dataset size and quality across different studies, direct comparison and analysis of all statistics are not feasible. Therefore, this review adopts a “partial-then-whole” analysis strategy: first, comparing the performance of different models within the same study, followed by observing their performance under various application conditions. Phan compared the performance of SVR, ANN, and RF in predicting residual strength, and the experimental results showed that the SVR model achieved the best predictive performance⁹⁷. However, in Cai’s study, the performance of the ANN model surpassed that of the SVR model⁶⁶.

Table 7 Single models performance for pipeline residual strength prediction

Table 8 Ensemble models performance for pipeline residual strength prediction

Table 9 Improved models performance for pipeline residual strength prediction

While ensemble models often perform well in practical applications, their performance is not always superior to that of single models. In the study by Abyani et al., the performance of the GPR and MLP models outperforms that of models like KNN and RF⁶⁷. In the research by Wang and Lu, the performance of the SVR model outperforms that of ensemble models such as XGBoost and LightGBM¹⁷. In Xiao et al.’s study, the ANN model demonstrated the best performance among all models⁸⁵.

Optimization algorithms can enhance model performance, but their applicability varies under different data conditions. Lu et al. used three optimization algorithms to optimize the SVM. The results showed that only NSGA-II-SVM reduced the RMSE and MAPE values, achieving model optimization, while the PSO and WOA optimization methods increased the RMSE and MAPE values, negatively affecting the model¹³.

According to the theorem of “No free lunch”, no one machine learning model is suitable for all situations. In practical applications, it is essential to compare different algorithms and select the most appropriate one based on the scale and quality of the available data.

Statistical analysis of evaluation metrics

Model evaluation metrics assess model performance, and five commonly used metrics were identified through a statistical analysis of the literature. Figure 12 shows that R² is the most commonly used evaluation metric for the pipeline residual strength prediction model. In practical evaluations, model assessment typically involves the analysis of multiple metrics rather than relying on a single one. Among these evaluation metrics, MSE, RMSPE, and MAE vary with data dimensions and do not have fixed benchmarks. In contrast, R² and MAPE represent relative errors and can be used to make comparisons in different scenarios. R² ranges from 0 and 1, with values closer to 1 indicating better model performance. The smaller the value of MAPE, the better the performance of the model. However, a higher R² value does not necessarily imply higher predictive accuracy for the model. R² is sensitive to outliers, and the presence of outliers may lead to an overestimation of the model’s fitting ability. In practical applications, potential outliers in the data can be identified using boxplots and scatter plots. For datasets that may contain outliers, caution should be exercised when using R² as the sole performance evaluation metric. Supplementing R² with additional metrics is recommended for a more comprehensive assessment of the model’s performance.

Fig. 12: Frequency of use of model evaluation metrics in literature.

The statistical distribution of commonly used evaluation metrics in literature.

Chen categorized the predictive ability of the model into four classes based on the magnitude of the MAPE⁹⁸. The MAPE of less than 10% indicates that the model is very accurate. The MAPE of between 11% and 20% indicates that the model performs well. The MAPE of between 21% and 50% indicates that the model performs reasonably well. The MAPE of greater than 50% indicates that the model performance is unacceptable. The maximum value of MAPE counted in this study is 31.520%, and the MAPE in the rest of the studies is lower than 20%. It shows that the machine learning model shows good performance in the field of pipeline residual strength prediction.

In addition to the five evaluation metrics mentioned above, other evaluation metrics have also been used to assess the accuracy of the corrosion oil and gas pipeline residual strength prediction models. Lu et al. used three new metrics—root mean squared percentage error (RMSPE), Theil U statistic 1 (U1), and Theil U statistic 2 (U2)—to evaluate the prediction accuracy of the model¹³. Since RMSPE and Theil U statistics are mainly used in specific fields, they are relatively complex and less intuitive. Specifically, RMSPE, which involves calculating the square root of percentage errors, may amplify errors when handling predictions close to zero, making it difficult to provide an intuitive performance interpretation. Although Theil U statistics can be used to measure the relative performance of model predictions compared to a baseline model, their involvement with baseline predictions and complex formulas means they are less commonly applied in general regression analysis, and may make interpretation of results more difficult. In contrast, commonly used metrics such as R², MSE, RMSE, MAE, and MAPE not only have simple calculations but also have strong applicability in various regression tasks, making them widely familiar to researchers and engineers, and thus more frequently used for evaluating regression models.

Challenges and perspectives

Machine learning models have been widely used in residual strength prediction of corroded pipelines due to their high efficiency and accuracy. However, machine learning models for residual strength prediction of corroded pipelines face some challenges in practical applications, which may affect the accuracy, stability, and usefulness of the prediction models. Given the work done so far and its limitations, the following challenges in the field of pipeline residual strength prediction need to be addressed.

Interpretability of the model

The widespread use of machine learning methods enables the construction of complex and efficient predictive models. However, these models often lack interpretability, making it challenging to understand how predictions are made. In the field of pipeline management and maintenance, the challenge of model interpretability can have an impact on the trust and adoption of decision makers. Pipeline residual strength prediction models typically involve a large number of features and data, such as pipeline parameters, corrosion parameters, and material parameters. However, complex models may not be able to directly explain which features have influenced the prediction results and how they are related. By using model-agnostic interpretability tools such as SHAP and LIME, models can achieve greater reliability and transparency in practical applications. The SHAP method, based on the Shapley value theory in game theory, explains model outputs by allocating feature importance^99,100,101. It not only quantifies each feature’s contribution to a single prediction but also provides insights into the model’s decision-making logic by plotting global feature importance or interaction effect graphs. LIME explains the prediction behavior of complex models for specific inputs by fitting an interpretable linear model based on locally perturbed data¹⁰².

Testing standards

In current research, the evaluation of machine learning model performance lacks unified standards, making it difficult to directly compare experimental results across studies. Researchers often build models based on independently collected datasets, which vary significantly in terms of scale, distribution, features, and preprocessing methods. This lack of standardized benchmarks not only limits fair comparisons of model performance but also hinders the promotion and application of the technology to some extent. To address this issue, authoritative organizations could lead the establishment of a model performance testing platform in the future. This platform could offer standardized datasets, evaluation metrics, and testing procedures to regulate the performance evaluation of machine learning models. Through this platform, researchers could validate model performance in a consistent environment, thereby facilitating the comparison and exchange of research outcomes and advancing the scientific development and practical application of machine learning technology in the residual strength prediction of corroded oil and gas pipelines.

Quality of data

From the review, it can be observed that most current studies focus on the development of machine learning algorithms, with relatively little attention given to model data research. Data quality directly affects the accuracy, stability and credibility of the prediction model. Problems in data quality may lead to distortion of the model’s prediction results, affecting the accurate judgment of the pipeline’s health status. To improve data quality and enhance the performance of predictive models, several measures can be implemented: Firstly, to address the class imbalance in datasets, the SMOTE can be employed to generate minority class samples, while random undersampling can be used to reduce majority class samples, achieving a more balanced class distribution and enhancing the model’s ability to recognize minority classes. Secondly, handling missing values is a critical step in improving data quality, and methods like multiple imputation can be applied to fill in missing values, preserving as much original information as possible.

Topicality

As urban infrastructures continue to develop and operate, real-time prediction of the residual strength of pipelines has become critical. In practical applications, achieving real-time prediction encounters various technical and operational challenges. Real-time prediction depends on instantaneous data acquisition and transmission. However, factors like network latency and data processing delays can compromise the real-time performance of sensor data acquisition and transmission. Delays can result in a time gap between prediction outcomes and actual conditions, compromising real-time accuracy. Additionally, complex predictive models often demand significant computational resources and processing time, conflicting with real-time requirements. In the future, edge computing technology and model compression techniques can be integrated to reduce model computational complexity and enhance data transmission speed, thereby meeting the real-time requirements of the model.