Introduction

Motivation and incitement

Fossil fuels such as coal and oil currently satisfy about 80% of the world’s energy needs. However, their use significantly contributes to greenhouse gas emissions, accelerating global warming and causing environmental harm, which has led to the ongoing climate crisis. Over the past 30 years, renewable energy sources (RESs), such as solar, wind, and hydropower, have emerged as viable alternatives to mitigate the harmful environmental impacts of fossil fuels.

Due to the economic efficacy, utilities are expressing interest in generating power from RES in the wake of the Covid-19 outbreak. Renewable energy installed capacity is estimated to reach 3600GW by 2030, an increase of roughly 1900GW over 20201. However, the intermittent nature of RES raises a lot of exciting challenges, including instantaneous grid management, energy market clearance, ancillary service limits, stability issues of the power system, and reliability2,3,4,5. Uncertainties remain, particularly regarding the predicted output of renewable energy sources (RES) and its deviation from actual performance, known as imbalance6. To tackle these challenges, enhanced prediction methods are needed. Addressing these issues is crucial for ensuring stability, dispatchability, and resolving market concerns, especially when bidding in the day-ahead electricity market7. The Energy Storage System (ESS), with its reliable energy storage and flexible charging and discharging capabilities, plays a vital role in modern power systems. ESS is a helpful technique to make RES dispatchable, and it’s often used with precise prediction approaches to forecast demand and generation profiles8,9. In addition to reducing investment costs and energy losses from long-distance power generation, Energy Storage Systems (ESS) enhance the efficiency of power system operations and meet increasing reserve requirements. By providing ancillary services, ESS are crucial for maintaining grid stability, lowering energy transmission costs, improving operational efficiency, and managing growing reserve needs. Furthermore, ESS enhance power quality by minimizing fluctuations and ensuring a reliable energy supply.

Distributed energy resources (DERs), such as smaller-scale generators and storage facilities located near energy usage sites, are made possible through the integration of renewable energy sources (RESs) and energy storage systems (ESSs). However, the growing demand for DERs increases load variability, leading to grid instability. To ensure safe operation, distribution operators must manage and monitor the grid in real-time. Demand-side management, generator adjustments, and energy storage technologies contribute to enhancing system flexibility. Effective coordination is crucial to maximize the benefits of these resources for the grid10,11,12,13,14,15.

Literature review

The VPP idea can be an excellent tool for finding a realistic alternative to the challenging difficulties faved by DERs, and regarded as the most successful approach to DER management. A VPP is a decentralized medium-scale power source comprising solar photovoltaic (PV), wind energy production, combined heat and power (CHP) units, energy storage devices, and demand-responsive loads. VPPs are controlled from one location through communication links and play a significant role in incorporating RES into the deregulated electricity markets. The goal is to combine several distributed generation sources and RESs to mitigate their volatility. The VPP combines small distributed generation units to produce a single virtual generating unit equivalent to a traditional one and may react and function independently16,17. On one side, VPP alleviates the external smart grid’s stability and dispatchability issues since it may be controlled separately while seeming like a unified system. VPP, on the other side, increases flexibility from all associated units and allows marketeers to improve renewable energy forecasts and trading programs.

While virtual power plants (VPPs) have significantly enhanced the electricity grid, their development still faces numerous challenges. A key issue is the uncertainty caused by the intermittent nature of renewable energy sources (RESs), necessitating more accurate forecasting techniques. Additionally, VPPs encounter regulatory hurdles such as ambiguous market rules, strict grid compliance requirements, complex ownership structures, privacy concerns, inconsistent interconnection standards, outdated tariff systems, slow regulatory updates, and lengthy licensing processes. These challenges make it difficult for VPPs to seamlessly integrate into and operate effectively within the energy market, highlighting the need for improved regulatory frameworks and technological advancements. A forecasting technique must be developed to ensure the validity of the abovementioned combination method to permit the competent usage of VPPs18. Neural networks are a good fit for several prediction issues19,20,21,22, with outstanding performance over various time horizons utilizing feedforward, recurrent, machine learning, and deep learning architectures. In23, solar power forecasting uses multiple deep neural networks. Six hours ahead of solar power generation, multi-site forecasting is illustrated in24, using a non-parametric machine learning method. Physical hybrid artificial neural network (ANN) approaches25 and hybrid models26 are the approaches that can be found in the literature for forecasting the power output of RESs, such as solar and wind plants. Some of the further vital machine learning and deep learning techniques used to predict RESs with accurate outcomes are illustrated in Refs.27,28,29,30,31.

Reference27 introduces a short-term wind power forecasting model enhanced by Support Vector Machines (SVMs), which accounts for seasonal, diurnal fluctuations, and historical patterns to address wind ramp dynamics. In28, a probabilistic wind power forecasting method is proposed, using gradient-boosted machines (GBM) for independent quantile and zone modeling. This method incorporates correlated wind farm data through a two-layer approach, offering a simple yet effective solution for day-ahead forecasting. To enhance probabilistic forecasting accuracy29, presents a Naive Bayesian model combined with particle swarm optimization (PSO) and rough set (RS) theory, achieving improved reliability at different confidence levels with narrower bandwidths and superior coverage of prediction intervals. Reference30 improves wind power forecasting accuracy and memory efficiency by incorporating an extended correntropy criterion into the stacked extreme learning machine (SELM) model within a deep learning framework. Lastly31, proposes a probabilistic wind power forecasting model based on relevance vector machines (RVMs) that integrates probabilistic risk (PR) and wind power mismatch balancing cost (WPMBC), demonstrating enhanced accuracy and cost-effectiveness compared to existing machine learning methods.

It’s also helpful in generating arbitrage by releasing storage at suitable times with better forecasting, which boosts the wind producer’s income32[,36. References34[–36 describes many energy storage metrics such as life cycle, power, energy transfer, self-discharge rate, storage length, etc. In the last two decades, numerous new ESSs have emerged37. describes an effective day-ahead electricity market bidding technique integrating RES and storage devices. The energy storage system might store low-cost peak energy and then release it as prices increase. It can assist reduce market instability to fluctuating on-peak pricing and regulate high-cost energy unbalanced charges38. VPPs can enhance the reliability and efficiency of ancillary services by utilizing LSTM networks for forecasting. Accurate, data-driven predictions enable optimized reserve allocation and better management of renewable energy variability, contributing to improved grid stability.Table 1 presented the brief analysis of various literature survey.

Table 1 Literature survey brief analysis.
Full size table

Contribution and paper organization

From the literature survey it is observed that there has been no discussion of an inclusive approach for maximizing power producers’ profits by combining the VPP generation units forecasting model with energy storage to the best of the authors’ knowledge. Traditional LSTM approaches can struggle with noisy and variable data from heterogeneous generating units, leading to limited robustness and slow convergence. The AOLSTM model offers a more reliable and adaptable solution to these issues, addressing the shortcomings of current forecasting methods for VPP generating units. By effectively handling challenges such as nonlinear dynamics, noisy data, and diverse inputs, AOLSTM significantly improves forecasting accuracy, scalability, and efficiency. This results in more effective management of renewable energy sources, ultimately enhancing the operational and financial performance of VPPs. In this paper, we first model VPP generating units, including wind power, solar PV, and CHP, and forecast them using the RNN-based AOLSTM model. Additionally, we integrate a Monte Carlo optimization model to further increase the power producer’s overall income. This study contributes to boosting the revenue of power producers through improved forecasting and optimization strategies.

  1. 1.

    VPP generating units integrative model: A comprehensive framework is developed to integrate solar PV, wind power, and CHP systems into a VPP, optimizing their combined efficiency.

  2. 2.

    AOLSTM for enhanced prediction: The advanced AOLSTM model is utilized to improve the accuracy of forecasting VPP generating units.

  3. 3.

    Revenue maximization with Monte Carlo optimization: A novel application of Monte Carlo optimization is used to identify the optimal revenue frontier under locational marginal pricing (LMP) scenarios for the IEEE-14 bus system.

  4. 4.

    Robust performance validation: The proposed method is validated through various error metrics, including Mean Absolute Error (MAE), Mean Squared Error (MSE), and Root Mean Square Error (RMSE), along with the Wilcoxon signed-rank test to ensure prediction accuracy and economic feasibility.

The following is the structure of the rest of this article: The second section covers mathematical models. The proposed approach’s technique is outlined in section three. Section four describes the results and discussions, and Sect. 5 conclusion.

Mathematical models of the proposed approach

AOLSTM model

Hchreiter and Schmidhuber invented LSTM networks in 1997, and they have shown to be quite beneficial in machine learning. LSTM networks can understand long-term dependencies and are classified as RNNs with memory. LSTMs can recall information for extended periods of time. The problem of vanishing gradients with ordinary RNNs was handled by employing LSTM networks in conjunction with a gating mechanism39. With each gate, the LSTM configuration raises more complex yet efficient. Figure 1 depicts the architecture of the LSTM model with weights. The novelty in this work is AOLSTM developed, and Adam optimizer is a stochastic gradient descent substitute for machine learning model training. Adam combines the best elements of the AdaGrad and RMSProp methods to create a sparse gradient optimization approach for noisy problems. This strategy is quite efficient and consumes little memory when dealing with significant issues with many data or features.

Fig. 1
Fig. 1
Full size image

LSTM architecture with weights.

On every instant t, \(\:{\prime\:}{i}_{t}{\prime\:}\) stands for ‘input’, \(\:{\prime\:}{f}_{t}{\prime\:}\) stands for ‘forget’, \(\:{\prime\:}{o}_{t}{\prime\:}\) stands for output gates, and \(\:{\prime\:}{\stackrel{\sim}{c}}_{t}{\prime\:}\:\)stands for candidate values, which are classified as follows:

$$\:{i}_{t}=\sigma\:({W}_{i-x}{x}_{t}+{W}_{i-h}{h}_{\left(t-1\right)}+{b}_{i}$$
(1)
$$\:{f}_{t}=\sigma\:({W}_{f-x}{x}_{t}+{W}_{f-h}{h}_{\left(t-1\right)}+{b}_{f}$$
(2)
$$\:{o}_{t}=\sigma\:({W}_{o-x}{x}_{t}+{W}_{o-h}{h}_{\left(t-1\right)}+{b}_{o}$$
(3)
$$\:{\stackrel{\sim}{c}}_{t}=tantan\:h\:({W}_{\stackrel{\sim}{c}-x}{x}_{t}+{W}_{\stackrel{\sim}{c}-h}{h}_{\left(t-1\right)}+{b}_{\stackrel{\sim}{c}}$$
(4)

where \(\:W\:\:\:\:\:\:\:\:\) matrix representing the weights, \(\:b\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\) bias, \(\:{x}_{t}\:\:\:\:\:\:\:\:\:\:\:\:\:\:\)present time step input to the AOLSTM, \(\:{h}_{(t-1)}\:\:\:\:\:-\:\)past time step hidden state to the AOLSTM, \(\:\left(.\right)\:\:\:\:\:\:\:\:\:\:\:-\) activation function, \(\:\sigma\:\left(.\right),\:tantan\:h\:\left(.\right)\:\--\) functions of Sigmoid & Tangent hyperbolic.

The mathematical representations of sigmoid and hyperbolic tangent functions are shown below, with incoming values ranging from 0 to 1; A value of 1 signifies that the current time step input holds greater relevance and warrants integration into the cell state. Conversely, a value of 0 indicates that the current time input lacks significance and is disregarded in the determination of the cell state.

$$\:\sigma\:\left(z\right)=\frac{1}{1-{e}^{-z}}$$
(5)
$$\:tantan\:h\left(z\right)=\:\frac{{e}^{z}-{e}^{-z}}{{e}^{z}+{e}^{-z}}$$
(6)

\(\:{f}_{t}\) resolves memory removal from cell state based on actual input; \(\:{o}_{t}\) determines the AOLSTM output, and \(\:{\stackrel{\sim}{c}}_{t}\) determines the current candidate values. When the candidate value and the outputs of the three gates are resolved, the AOLSTM changed cell state memory is approximated as:

$$\:{c}_{t}={f}_{t}^{*}{c}_{(t-1)}+{i}_{t}^{*}{\stackrel{\sim}{c}}_{t}$$
(7)

Here \(\:{c}_{(t-1)}-\) past time step cell state.

The AOLSTM model employs the subsequent equations to evaluate both the predicted output and the hidden state:

$$\:{h}_{t}={o}_{t}^{*}tantan\:h\left({c}_{t}\right)\:$$
(8)
$$\:\widehat{{y}_{t}}={W}_{(h-y)}{h}_{t}$$
(9)

Where \(\:{\:W}_{(h-y)}-\)weight matrix.

Indeed, this constitutes the fundamental AOLSTM model utilized for forecasting VPP generating units. AOLSTM enhances LSTM network training by making it more efficient, stable, and adaptable. This approach addresses the limitations of traditional LSTMs and outperforms other machine learning models, particularly in complex forecasting tasks involving nonlinear patterns, noisy data, and temporal dependencies. The use of Adam optimization accelerates convergence, improves accuracy, and boosts robustness, making it ideal for real-world forecasting applications.

Using LSTMs for VPP forecasting presents a trade-off between computational complexity and cost-effectiveness. While LSTMs provide accurate predictions, their resource-intensive nature may not be ideal for low-budget or small-scale VPPs. However, their utility can be enhanced through innovations like pre-trained models, transfer learning, and edge computing, which help lower training costs and improve deployment efficiency. Research and implementation discussions often include empirical comparisons of costs and performance across various forecasting methods, such as ARIMA, conventional neural networks, and ensemble approaches, to assess their suitability for different scenarios.

Energy arbitrage model

Energy storage has attracted a lot of attention to mitigate the risks associated with increased usage of renewable energy sources. In this article, the best practicable income for the VPP is evaluated in the day-ahead electricity market using wind power, solar irradiation, CHP, and energy storage devices. VPP (14th bus) and energy storage (13th bus) are all considered in the IEEE-14 bus. LMPs for the storage system are determined for the next 24 h by performing optimal power flow (OPF). The model is tested using historical data from the PJM Interconnection regional transmission organization (RTO)40.

In this study, the storage owner will know exactly how much upper limit of income will earn for one day ahead by applying the perfect forecast of historical wind power, solar irradiance, and CHP. Monte Carlo optimization is used to identify the best possible solution for the upper revenue limit. Optimization is accomplished by considering storage system characteristics such as efficiency, self-discharge, power limits, and capacity. For developing the model, the following assumptions are considered:

  • The storage device has no impact on the total market pricing.

  • Network capacity limits do not influence the storage device.

  • The values of the device constraints are assumed to be constant.

  • Compared to the spot pricing period, the storage device time for changing the charging and discharge rate in the power limitations is minimal.

Proposed methodology

This chapter discusses the entire approach of VPP forecasting and Monte Carlo model optimization.

VPP forecasting

VPP integrates several distributed energy sources to increase power output and power exchange in the electricity market. Power generation, storage of the energy units, and flexible loads are the main components of a VPP41. Furthermore, prediction (demand, production, and pricing) is critical to VPP’s improved performance.

Fig. 2
Fig. 2
Full size image

Schematic structure of the VPP.

The schematic diagram of a VPP structure is presented in Fig. 2. The proposed AOLSTM-RNN model anticipates three distinct types of generating unit forecasting: wind power, solar PV, and CHP. Six years of historical data on wind power generation, solar irradiance, and net hourly electrical energy output are collected from the wind farm42, solar PV arrays43, and CHP44 in the Washington region. In processing, 70% of the data is utilized for training, 20% for testing, and 10% for validation.

Figures 3 and 4, and Fig. 5 describe impact factor heat maps with various parameters used to forecast wind power, solar irradiance, and net hourly electrical energy output of CHP.

Fig. 3
Fig. 3
Full size image

Heat map analysis of wind power and impact factors.

Fig. 4
Fig. 4
Full size image

Heat map analysis of solar irradiance and impact factors.

Fig. 5
Fig. 5
Full size image

Heat map analysis of CHP and impact factors.

Monte Carlo optimization model

Monte Carlo optimization is utilized to determine the best possible solution for the energy storage arbitrage upper border revenue45, as stated in Chap. 2. The Monte Carlo optimization model’s flowchart is shown in Fig. 6.

Fig. 6
Fig. 6
Full size image

Flow chart of the Monte Carlo optimization.

The storage system’s positive revenue is based on the price difference between purchasing and selling electrical energy and the system’s round-trip efficiency (\(\:{\eta\:}_{r-t})\). Within the periods \(\:{t}_{1\:}\) and \(\:{t}_{2\:}\), the storage system round trip efficiency is given by

$$\:{\eta\:}_{(r-t)}\left(\varDelta\:t\right)={\eta\:}_{in}\times\:{\eta\:}_{out}\times\:{e}^{\frac{({t}_{1\:}-{t}_{2\:})}{\tau\:}}$$
(10)

Here, \(\:\tau\:\:\:\:\:\:\:-\) Storage system time constant, \(\:{\eta\:}_{in}\:\:-\) Input charging efficiency of the storage system, \(\:\:\:\:\:\:\:\:\:\:\:\:{\eta\:}_{out}-\)Storage system output discharging efficiency, and

$$\:\varDelta\:t\:\:\:={t}_{1\:}-{t}_{2\:}$$

The exponential component in Eq. 10 represents the time-dependent loss rate of the stored energy. The \(\:{t}_{2\:}\) period output energy Δ\(\:{E}_{2}\) from the storage system is derived by assuming a stable penalty for transporting energy into and out of the storage system.

$$\:\varDelta\:{E}_{2}=\varDelta\:{E}_{1}\times\:{\eta\:}_{(r-t)}$$
(11)

\(\:\varDelta\:{E}_{1}\) denotes the quantity of input energy during the time interval \(\:{t}_{1\:}\). The time-dependent self-discharge storage mechanism is provided by

$$\:\frac{d}{dt}\left(SOC\right)=-\frac{SOC\left(t\right)}{\tau\:}$$
(12)

The difference in the energy contained in the storage system Δ\(\:{E}_{3}\) is provided by for the intermediate period \(\:{t}_{3}\).

Δ\(\:{E}_{3}={\eta\:}_{in}\times\:\varDelta\:{E}_{1}\times\:{e}^{\frac{({t}_{1\:}-{t}_{3\:})}{\tau\:}}\) (13)

Briefly, a fraction of \(\:\frac{1}{{\eta\:}_{(r-t)}\varDelta\:t}\) incremental electricity price at\(\:\:{t}_{2}\:\) in contrast to pricing at \(\:{t}_{1\:}\) leads to purchasing energy at \(\:{t}_{1\:}\) and selling at \(\:{t}_{2}\), resulting in additional revenue for the storage system operator.

The revenue generated by the model for each time t, is provided by

$$\:R\left(t\right)=OTG\left(t\right)\times\:Price\left(t\right)$$
(14)

The output ‘to the’ or ‘taken from’ the grid is OTG(t).

Revenue is calculated using the optimizing method in each cycle. This process is continued until the maximum profit is achieved. The model code implementations were built and performed in PYTHON software-GOOGLE COLAB version using Intel Core TM i7-4770 CPU 3.40 GHz and 8 GB RAM as hardware specifications.

Results & discussions

VPP forecasting

This section illustrates the improved performance analysis of the AOLSTM-based forecasting model for the realistic VPP generating units case study by comparing it to the other three models, namely the persistence, Gradient Boost, and Random Forest methods. In processing, 70% of the data is utilized for training, 20% for testing, and 10% for validation. The AOLSTM representation is executed using forwarding propagation after processing the data. For the work presented in this article, tuning of the hyper-parameters was made with grid search. In designing the AOLSTM hyper-parameters, for 60 epochs, the total number of hidden layers used is 4, with batch size 256. The softMax activation function is used for this AOLSTM. Figure 7 shows a schematic representation of the dataset partition utilized for all the proposed forecasting models.

Fig. 7
Fig. 7
Full size image

Dataset division of the proposed method.

Wind power, solar irradiation, and CHP forecasting are done 24 h ahead of time using an RNN-designed AOLSTM model, as seen in Figs. 8 and 9, and Fig. 10. The blue line corresponds to the actual generation, whereas the green line corresponds to the expected generation. Based on error measurements such as MAE, MSE, and RMSE, these findings are compared with persistence, Gradient Boost, and Random Forest approaches. Tables 2 and 3, and Table 4 shows the error metrics values for the AOLSTM model and the other approaches of wind power, solar irradiation, and CHP forecasting. The suggested AOLSTM model has very little error and makes more accurate predictions. For 60 epochs, the variation in training and testing loss is measured using the Adam optimizer and the SoftMax activation function.

Fig. 8
Fig. 8
Full size image

One-day ahead wind power forecasting.

Table 2 Error metrics comparison of wind power forecast.
Full size table
Fig. 9
Fig. 9
Full size image

One-day ahead solar irradiance forecasting.

Table 3 Solar irradiance forecast error comparison table.
Full size table
Fig. 10
Fig. 10
Full size image

CHP forecasting for 24 h ahead horizon.

Table 4 CHP forecast error comparison table.
Full size table

Energy storage arbitrage

The battery storage system is used for arbitrage in this article, and the parameters of the battery storage system are listed in Table 5, including capacity, charging and discharging power, and charging and discharging efficiencies. Monte Carlo optimization optimizes the model, and revenue upper bounds are estimated. The model is run for the appropriate number of iterations, and the following are the results:

Table 5 Specifications of the battery storage system.
Full size table
Fig. 11
Fig. 11
Full size image

Real VPP optimum schedule of storage and charging/discharging.

Figure 11 and 12 depicts the differences in the optimal storage and charging/discharging schedules for real and predicted VPP. The energy storage system changing/discharging is represented based on market prices and VPP conditions. When there is a change in the market price, the variations in the energy storage, energy input, and energy transfer are shown in the graph for both real and predicted VPP systems. If the market price is low, the battery gets charged, and energy will be stored; in case of a higher market price, the battery discharges, and the energy will be delivered. In this way, the arbitrage of energy storage is achieved. The predicted energy storage arbitrage revenue of VPP is estimated at around $ 7.749 per hour, and it’s about $8.197 per hour in real-time values.

Fig. 12
Fig. 12
Full size image

Predicted VPP optimum schedule of storage and charging/discharging.

Profit maximization

The profit maximization using the proposed approach is explained in Table 6. The total forecasted power from wind turbines, solar cells, and combined heat and power (CHP) are calculated 24 h ahead of time. The predicted income utilizing the AOLSTM model for the VPP model is $21.175 per hour and for the real-time VPP model is about $21.202 per hour, respectively. The predicted energy storage arbitrage revenue of VPP is estimated using Monte Carlo optimization, with a value of around $ 7.749 per hour, and it’s about $8.197 per hour in real-time values. The total upper boundary revenue is predicted to be $28.924 per hour using this bi-level recommended strategy, which is much closer to the entire real revenue of $ 29.399 per hour.

Table 6 Real-time and predicted models revenue comparison table.
Full size table

Figure 13 depicts the variance in absolute actual and predicted power profits. The proposed technique produces a very accurate forecast, with an overall RMSE of 0.016. The Wilcoxon’s signed-rank test is conducted for real-time and predicted models. The Proposed method is giving the p-value of 0.025 which is less than 0.05 always considered as statistically significant.

Fig. 13
Fig. 13
Full size image

Real and predicted profit ($/H) variation.

Conclusion

The current situation has a greater demand for sustainable, green, and low-cost power. The market is expected to be driven by this demand in the future. As a result, improving the system’s prediction accuracy will aid in more efficient project planning and operations. Furthermore, future predictions with the lowest error will be advantageous in real-time and day-ahead energy trading. This paper illustrates an effective strategy for predicting maximum revenue with three generating units of VPP and energy storage in the day-ahead power market. The proposed method effectively forecasts the power of the VPP-generating units using an RNN-based AOLSTM. The proposed approach calculates the maximum revenue from storage arbitrage using Monte Carlo optimization techniques and market pricing.

This two-fold original contribution helps evaluate the non-linear uncertainties due to the RES and estimate the maximum revenue power producers will receive using energy storage. The RNN-AOLSTM model is compared with the Persistence, Gradient Boost, and Random Forest approaches, and error metrics such as MAE, MSE, and RMSE values are observed with the lowest value.

The maximum VPP and storage arbitrage revenue is calculated using real-time and forecasted values. When the results are compared, the RMSE is found to be 0.016, indicating that the proposed model is adequate. As a result, the suggested model may help predict power accurately and maximize revenue in the electricity market.