Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Designing the future electricity spot market with high renewables via reliable simulations

Nature Reviews Electrical Engineering volume 2, pages 320–337 (2025)Cite this article

Subjects

Abstract

The global transition to renewable energy is crucial for mitigating climate change, but the increasing penetration of renewable sources introduces challenges such as uncertainty and intermittency. The electricity market plays a vital role in encouraging renewable generation while ensuring operational security and grid stability. This Review examines the optimization of market design for power systems with high renewable penetration. We explore recent innovations in renewable-dominated electricity market designs, summarizing key research questions and strategies. Special focus is given to multi-agent reinforcement learning (MARL) for market simulations, its performance and real-world applicability. We also review performance evaluation metrics and present a case study from the Horizon 2020 TradeRES project, exploring European electricity market design under 100% renewable penetration. Finally, we discuss unresolved issues and future research directions.

Key points

  • The global transition to renewable energy presents challenges such as uncertainty and intermittency, requiring innovative electricity market designs.

  • Market designs for high renewable penetration need optimization, with special focus on operational security, grid stability and incentivizing renewable generation.

  • Multi-agent reinforcement learning (MARL) has emerged as a promising approach for simulating renewable-dominated electricity markets.

  • Real-world performance evaluation metrics are crucial for assessing the effectiveness of market simulations, ensuring their scalability and applicability.

  • The Horizon 2020 TradeRES project offers valuable insights into the feasibility of achieving 100% renewable penetration in European electricity markets.

  • Despite advancements, key issues such as uncertainty management and scalability of simulation models remain unsolved, necessitating further research.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to the full article PDF.

USD 39.95

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Framework of the market design methodology.
Fig. 2: Multi-agent reinforcement learning algorithm implementation.

Similar content being viewed by others

References

  1. MacDonald, A. E. et al. Future cost-competitive electricity systems and their impact on US CO2 emissions. Nat. Clim. Change 6, 526–531 (2016).

    Article  Google Scholar 

  2. Sovacool, B. K., Schmid, P., Stirling, A., Walter, G. & MacKerron, G. Differences in carbon emissions reduction between countries pursuing renewable electricity versus nuclear power. Nat. Energy 5, 928–935 (2020).

    Article  Google Scholar 

  3. Middlemiss, L. Taking control of energy as a solar prosumer. Nat. Energy 8, 13–14 (2023).

    Article  Google Scholar 

  4. Reese, M. O. et al. Increasing markets and decreasing package weight for high-specific-power photovoltaics. Nat. Energy 3, 1002–1012 (2018).

    Article  Google Scholar 

  5. Bunn, D. W. & Gianfreda, A. Integration and shock transmissions across European electricity forward markets. Energy Econ. 32, 278–291 (2010).

    Article  Google Scholar 

  6. Li, B., Basu, S., Watson, S. J. & Russchenberg, H. W. J. Quantifying the predictability of a “dunkelflaute” event by utilizing a mesoscale model. J. Phys. Conf. Ser. 1618, 62042 (2020).

    Article  Google Scholar 

  7. Ghiassi-Farrokhfal, Y., Ketter, W. & Collins, J. Making green power purchase agreements more predictable and reliable for companies. Decis. Support. Syst. 144, 113514 (2021).

    Article  Google Scholar 

  8. Guo, F. et al. Implications of intercontinental renewable electricity trade for energy systems and emissions. Nat. Energy 7, 1144–1156 (2022).

    Article  Google Scholar 

  9. Mays, J., Morton, D. P. & O’Neill, R. P. Asymmetric risk and fuel neutrality in electricity capacity markets. Nat. Energy 4, 948–956 (2019).

    Article  Google Scholar 

  10. Parag, Y. & Sovacool, B. K. Electricity market design for the prosumer era. Nat. Energy 1, 16032 (2016).

    Article  Google Scholar 

  11. Widuto, A. Improving the design of the EU electricity market. https://www.europarl.europa.eu/RegData/etudes/BRIE/2023/745694/EPRS_BRI(2023)745694_EN.pdf (European Parliament, 2024).

  12. Department for Energy Security and Net Zero. Review of Electricity Market Arrangements (REMA): second consultation. GOV.UK https://www.gov.uk/government/consultations/review-of-electricity-market-arrangements-rema-second-consultation (2024).

  13. Wolak, F. A. An empirical analysis of the impact of hedge contracts on bidding behavior in a competitive electricity market. Int. Economic J. 14, 1–39 (2000).

    Article  Google Scholar 

  14. Borenstein, S. & Bushnell, J. An empirical analysis of the potential for market power in California’s electricity industry. J. Ind. Econ. 47, 285–323 (1999).

    Article  Google Scholar 

  15. Dai, T. & Qiao, W. Trading wind power in a competitive electricity market using stochastic programing and game theory. IEEE Trans. Sustain. Energy 4, 805–815 (2013).

    Article  Google Scholar 

  16. Lise, W. et al. A game theoretic model of the northwestern European electricity market — market power and the environment. Energy Policy 34, 2123–2136 (2006).

    Article  Google Scholar 

  17. Xia, X. & Elaiw, A. M. Optimal dynamic economic dispatch of generation: a review. Electr. Power Syst. Res. 80, 975–986 (2010).

    Article  Google Scholar 

  18. PRIMES. E3 Modelling https://e3modelling.com/modelling-tools/primes/ (accessed 26 February 2023).

  19. Zhou, Z., Chan, W. K. & Chow, J. H. Agent-based simulation of electricity markets: a survey of tools. Artif. Intell. Rev. 28, 305–342 (2007).

    Article  Google Scholar 

  20. Thimmapuram, P. R. & Kim, Jinho Consumers’ price elasticity of demand modeling with economic effects on electricity markets using an agent-based model. IEEE Trans. Smart Grid 4, 390–397 (2013).

    Article  Google Scholar 

  21. Ruhnau, O. et al. Why electricity market models yield different results: carbon pricing in a model-comparison experiment. Renew. Sustain. Energy Rev. 153, 111701 (2022).

    Article  Google Scholar 

  22. Le Coq, C., Orzen, H. & Schwenen, S. Pricing and capacity provision in electricity markets: an experimental study. J. Regulatory Econ. 51, 123–158 (2017).

    Article  Google Scholar 

  23. Ringler, P., Keles, D. & Fichtner, W. Agent-based modelling and simulation of smart electricity grids and markets — a literature review. Renew. Sustain. Energy Rev. 57, 205–215 (2016).

    Article  Google Scholar 

  24. Zhou, Z., Zhao, F. & Wang, J. Agent-based electricity market simulation with demand response from commercial buildings. IEEE Trans. Smart Grid 2, 580–588 (2011).

    Article  Google Scholar 

  25. Franco, C. J., Castaneda, M. & Dyner, I. Simulating the new British electricity-market reform. Eur. J. Operational Res. 245, 273–285 (2015).

    Article  Google Scholar 

  26. Damien, P. et al. Impacts of day-ahead versus real-time market prices on wholesale electricity demand in Texas. Energy Econ. 81, 259–272 (2019).

    Article  Google Scholar 

  27. Alahyari, A., Ehsan, M. & Mousavizadeh, M. A hybrid storage-wind virtual power plant (VPP) participation in the electricity markets: a self-scheduling optimization considering price, renewable generation, and electric vehicles uncertainties. J. Energy Storage 25, 100812 (2019).

    Article  Google Scholar 

  28. Fernandes, C., Frías, P. & Reneses, J. Participation of intermittent renewable generators in balancing mechanisms: a closer look into the Spanish market design. Renew. Energy 89, 305–316 (2016).

    Article  Google Scholar 

  29. Eicke, A. & Schittekatte, T. Fighting the wrong battle? A critical assessment of arguments against nodal electricity prices in the European debate. Energy Policy 170, 113220 (2022).

    Article  Google Scholar 

  30. Chakravarthi, K. et al. Real time congestion management using generation re-dispatch: modeling and controller design. IEEE Trans. Power Syst. 38, 1–14 (2023).

    Article  Google Scholar 

  31. Zhu, Z. et al. Analysis of evolutionary dynamics for bidding strategy driven by multi-agent reinforcement learning. IEEE Trans. Power Syst. 36, 5975–5978 (2021).

    Article  Google Scholar 

  32. Zhu, Z. et al. Optimal bi-level bidding and dispatching strategy between active distribution network and virtual alliances using distributed robust multi-agent deep reinforcement learning. IEEE Trans. Smart Grid 13, 2833–2843 (2022).

    Article  Google Scholar 

  33. Haring, T. W., Kirschen, D. S. & Andersson, G. Incentive compatible imbalance settlement. IEEE Trans. Power Syst. 30, 3338–3346 (2015).

    Article  Google Scholar 

  34. Martin, S., Smeers, Y. & Aguado, J. A. A stochastic two settlement equilibrium model for electricity markets with wind generation. IEEE Trans. Power Syst. 30, 233–245 (2015).

    Article  Google Scholar 

  35. Ghorani, R., Fotuhi-Firuzabad, M. & Moeini-Aghtaie, M. Main challenges of implementing penalty mechanisms in transactive electricity markets. IEEE Trans. Power Syst. 34, 3954–3956 (2019).

    Article  Google Scholar 

  36. Milano, F. & Ortega, A. A method for evaluating frequency regulation in an electrical grid — part I: theory. IEEE Trans. Power Syst. 36, 183–193 (2021).

    Article  Google Scholar 

  37. Rebours, Y. G., Kirschen, D. S., Trotignon, M. & Rossignol, S. A survey of frequency and voltage control ancillary services — part I: technical features. IEEE Trans. Power Syst. 22, 350–357 (2007).

    Article  Google Scholar 

  38. Canizes, B., Silveira, V. & Vale, Z. Demand response and dispatchable generation as ancillary services to support the low voltage distribution network operation. Energy Rep. 8, 7–15 (2022).

    Article  Google Scholar 

  39. Riesz, J. & Milligan, M. in Advances in Energy Systems 479–489 (Wiley, 2019).

  40. Lisi, F., Grossi, L. & Quaglia, F. Evaluation of cost-at-risk related to the procurement of resources in the ancillary services market. the case of the italian electricity market. Energy Econ. 121, 106625 (2023).

    Article  Google Scholar 

  41. Anaya, K. L. & Pollitt, M. G. A social cost benefit analysis for the procurement of reactive power: the case of power potential. Appl. Energy 312, 118512 (2022).

    Article  Google Scholar 

  42. Rancilio, G., Rossi, A., Falabretti, D., Galliani, A. & Merlo, M. Ancillary services markets in Europe: evolution and regulatory trade-offs. Renew. Sustain. Energy Rev. 154, 111850 (2022).

    Article  Google Scholar 

  43. Petropoulos, G. & Willems, B. Long-term transmission rights and dynamic efficiency. Energy Econ. 88, 104714 (2020).

    Article  Google Scholar 

  44. Kell, N. P., Santibanez-Borda, E., Morstyn, T., Lazakis, I. & Pillai, A. C. Methodology to prepare for UK’s offshore wind contract for difference auctions. Appl. Energy 336, 120844 (2023).

    Article  Google Scholar 

  45. Koolen, D., Bunn, D. & Ketter, W. Renewable energy technologies and electricity forward market risks. Energy J. 42, 43–67 (2021).

    Article  Google Scholar 

  46. Kim, J. H., Bolinger, M., Mills, A. D. & Wiser, R. Rethinking the role of financial transmission rights in wind-rich electricity markets in the central U.S. Energy J. 44, 1–20 (2023).

    Article  Google Scholar 

  47. Xiao, D., do Prado, J. C. & Qiao, W. Optimal joint demand and virtual bidding for a strategic retailer in the short-term electricity market. Electr. Power Syst. Res. 190, 106855 (2021).

    Article  Google Scholar 

  48. Xiao, D. & Chen, H. Stochastic up to congestion bidding strategy in the nodal electricity markets considering risk management. IEEE Access. 8, 202428–202438 (2020).

    Article  Google Scholar 

  49. Bunn, D. W. & Oliveira, F. S. Modeling the impact of market interventions on the strategic evolution of electricity markets. Oper. Res. 56, 1116–1130 (2008).

    Article  MathSciNet  Google Scholar 

  50. Ambrosius, M. et al. Uncertain bidding zone configurations: the role of expectations for transmission and generation capacity expansion. Eur. J. Oper. Res. 285, 343–359 (2020).

    Article  MathSciNet  Google Scholar 

  51. Guo, N. & Lo Prete, C. Cross-product manipulation with intertemporal constraints: an equilibrium model. Energy Policy 134, 110851 (2019).

    Article  Google Scholar 

  52. Hampton, H. & Foley, A. A review of current analytical methods, modelling tools and development frameworks applicable for future retail electricity market design. Energy 260, 124861 (2022).

    Article  Google Scholar 

  53. Boscan, L. R. European Union retail electricity markets in the green transition: the quest for adequate design. Wiley Interdiscip. Rev. Energy Environ. 9, e359 (2020).

    Google Scholar 

  54. Van Zoest, V., El Gohary, F., Ngai, E. C. H. & Bartusch, C. Demand charges and user flexibility — exploring differences in electricity consumer types and load patterns within the Swedish commercial sector. Appl. Energy 302, 117543 (2021).

    Article  Google Scholar 

  55. Enrich, J., Li, R., Mizrahi, A. & Reguant, M. Measuring the impact of time-of-use pricing on electricity consumption: evidence from Spain. J. Environ. Econ. Manag. 123, 102901 (2024).

    Article  Google Scholar 

  56. Samimi, A., Nikzad, M. & Mohammadi, M. Real-time electricity pricing of a comprehensive demand response model in smart grids. Int. Trans. Electr. Energy Syst. 27, e2256 (2017).

    Article  Google Scholar 

  57. Jenner, S., Groba, F. & Indvik, J. Assessing the strength and effectiveness of renewable electricity feed-in tariffs in European Union countries. Energy Policy 52, 385–401 (2013).

    Article  Google Scholar 

  58. Smith, K. M., Koski, C. & Siddiki, S. Regulating net metering in the United States: a landscape overview of states’ net metering policies and outcomes. Electricity J. 34, 106901 (2021).

    Article  Google Scholar 

  59. Brown, P. US renewable electricity: how does the production tax credit (PTC) impact wind markets? (Congressional Research Service, 2012); https://crsreports.congress.gov/product/details?prodcode=R42576.

  60. Priesmann, J., Spiegelburg, S., Madlener, R. & Praktiknjo, A. Does renewable electricity hurt the poor? Exploring levy programs to reduce income inequality and energy poverty across German households. Energy Res. Soc. Sci. 93, 102812 (2022).

    Article  Google Scholar 

  61. Liu, D., Jiang, Y., Peng, C., Jian, J. & Zheng, J. Can green certificates substitute for renewable electricity subsidies? A Chinese experience. Renew. Energy 222, 119861 (2024).

    Article  Google Scholar 

  62. Zhao, Z., Cai, W., Wang, Y., Xiong, J. & Liu, W. Day-ahead electricity pricing mechanism considering the conflict between distribution network congestion and renewable produce. Int. Trans. Electr. Energy Syst. https://doi.org/10.1002/2050-7038.13218 (2021).

  63. Mousavi, M. & Wu, M. A DSO framework for market participation of DER aggregators in unbalanced distribution networks. IEEE Trans. Power Syst. 37, 2247–2258 (2022).

    Article  Google Scholar 

  64. Klein, M., Ziade, A. & de Vries, L. Aligning prosumers with the electricity wholesale market — the impact of time-varying price signals and fixed network charges on solar self-consumption. Energy Policy 134, 110901 (2019).

    Article  Google Scholar 

  65. Li, X. et al. Sustainability or continuous damage: a behavior study of prosumers’ electricity consumption after installing household distributed energy resources. J. Clean. Prod. 264, 121471 (2020).

    Article  Google Scholar 

  66. Lu, X. Multi-Agent Reinforcement Learning in Games. Thesis, Carleton Univ. (2012).

  67. Ye, Y. et al. Multi-period and multi-spatial equilibrium analysis in imperfect electricity markets: a novel multi-agent deep reinforcement learning approach. IEEE Access. 7, 1 (2019).

    Article  Google Scholar 

  68. Du, Y. et al. Approximating Nash equilibrium in day-ahead electricity market bidding with multi-agent deep reinforcement learning. J. Mod. Power Syst. Clean Energy 9, 534–544 (2021).

    Article  Google Scholar 

  69. Ren, K., Liu, J., Liu, X. & Nie, Y. Reinforcement learning-based bi-level strategic bidding model of gas-fired unit in integrated electricity and natural gas markets preventing market manipulation. Appl. Energy 336, 120813 (2023).

    Article  Google Scholar 

  70. Harder, N., Qussous, R. & Weidlich, A. Fit for purpose: modeling wholesale electricity markets realistically with multi-agent deep reinforcement learning. Energy AI 14, 100295 (2023).

    Article  Google Scholar 

  71. Zhu, Z. et al. Analysis of strategic interactions among distributed virtual alliances in electricity and carbon emission auction markets using risk-averse multi-agent reinforcement learning. Renew. Sustain. Energy Rev. 183, 113466 (2023).

    Article  Google Scholar 

  72. Wang, J., Wu, J. & Kong, X. Multi-agent simulation for strategic bidding in electricity markets using reinforcement learning. CSEE J. Power Energy Syst. 9, 1051–1065 (2023).

    Google Scholar 

  73. Ye, Y. et al. Multi-agent deep reinforcement learning for coordinated energy trading and flexibility services provision in local electricity markets. IEEE Trans. Smart Grid 14, 1541–1554 (2023).

    Article  Google Scholar 

  74. Rokhforoz, P., Montazeri, M. & Fink, O. Multi-agent reinforcement learning with graph convolutional neural networks for optimal bidding strategies of generation units in electricity markets. Expert. Syst. Appl. 225, 120010 (2023).

    Article  Google Scholar 

  75. Kiran, P. & Vijaya Chandrakala, K. R. M. New interactive agent based reinforcement learning approach towards smart generator bidding in electricity market with micro grid integration. Appl. Soft Comput. 97, 106762 (2020).

    Article  Google Scholar 

  76. Harder, N., Weidlich, A. & Staudt, P. Finding individual strategies for storage units in electricity market models using deep reinforcement learning. Energy Inf. 6, 41–21 (2023).

    Article  Google Scholar 

  77. Roesch, M. et al. Smart grid for industry using multi-agent reinforcement learning. Appl. Sci. 10, 6900 (2020).

    Article  Google Scholar 

  78. Ochoa, T. et al. Multi-agent deep reinforcement learning for efficient multi-timescale bidding of a hybrid power plant in day-ahead and real-time markets. Appl. Energy 317, 119067 (2022).

    Article  Google Scholar 

  79. Yu, Y. et al. White-box transformers via sparse rate reduction: compression is all there is? Preprint at https://doi.org/10.48550/arXiv.2311.13110 (2023).

  80. Ferigo, A., Custode, L. L. & Iacca, G. Quality–diversity optimization of decision trees for interpretable reinforcement learning. Neural Comput. Appl. https://doi.org/10.1007/s00521-023-09124-5 (2023).

  81. Rashedi, N., Tajeddini, M. A. & Kebriaei, H. Markov game approach for multi-agent competitive bidding strategies in electricity market. IET Gener. Transm. Distrib. 10, 3756–3763 (2016).

    Article  Google Scholar 

  82. Xu, H. et al. Joint bidding and pricing for electricity retailers based on multi-task deep reinforcement learning. Int. J. Electr. Power Energy Syst. 138, 107897 (2022).

    Article  Google Scholar 

  83. Zhu, Z. et al. Nash equilibrium estimation and analysis in joint peer-to-peer electricity and carbon emission auction market with microgrid prosumers. IEEE Trans. Power Syst. 38, 5768–5780 (2023).

    Article  Google Scholar 

  84. Yan, L. et al. A hierarchical deep reinforcement learning-based community energy trading scheme for a neighborhood of smart households. IEEE Trans. Smart Grid 13, 4747–4758 (2022).

    Article  Google Scholar 

  85. May, R. & Huang, P. A multi-agent reinforcement learning approach for investigating and optimising peer-to-peer prosumer energy markets. Appl. Energy 334, 120705 (2023).

    Article  Google Scholar 

  86. Reichenberg, L., Siddiqui, A. S. & Wogrin, S. Policy implications of downscaling the time dimension in power system planning models to represent variability in renewable output. Energy 159, 870–877 (2018).

    Article  Google Scholar 

  87. Cho, S., Li, C. & Grossmann, I. E. Recent advances and challenges in optimization models for expansion planning of power systems and reliability optimization. Comput. Chem. Eng. 165, 107924 (2022).

    Article  Google Scholar 

  88. Li, X. et al. A novel optimization framework for integrated local energy system multi-objective dispatch problem based on dynamic knowledge base. Int. J. Electr. Power Energy Syst. 128, 106736 (2021).

    Article  Google Scholar 

  89. Mezghani, I., Stevens, N., Papavasiliou, A. & Chatzigiannis, D. I. Hierarchical coordination of transmission and distribution system operations in european balancing markets. IEEE Trans. Power Syst. 38, 1–13 (2023).

    Article  Google Scholar 

  90. Pavičević, M. et al. The potential of sector coupling in future European energy systems: soft linking between the Dispa-SET and JRC-EU-TIMES models. Appl. Energy 267, 115100 (2020).

    Article  Google Scholar 

  91. Zheng, W., Zhu, J. & Luo, Q. Distributed dispatch of integrated electricity-heat systems with variable mass flow. IEEE Trans. Smart Grid 14, 1907–1919 (2023).

    Article  Google Scholar 

  92. Helgeson, B. & Peter, J. The role of electricity in decarbonizing European road transport — development and assessment of an integrated multi-sectoral model. Appl. Energy 262, 114365 (2020).

    Article  Google Scholar 

  93. Yu, H., Nord, L. O., Yu, C., Zhou, J. & Si, F. An improved combined heat and power economic dispatch model for natural gas combined cycle power plants. Appl. Therm. Eng. 181, 115939 (2020).

    Article  Google Scholar 

  94. Peng, M., Liu, L. & Jiang, C. A review on the economic dispatch and risk management of the large-scale plug-in electric vehicles (PHEVs)-penetrated power systems. Renew. Sustain. Energy Rev. 16, 1508–1515 (2012).

    Article  Google Scholar 

  95. Yin, X., Ye, C., Ding, Y. & Song, Y. Exploiting Internet data centers as energy prosumers in integrated electricity-heat system. IEEE Trans. Smart Grid 14, 167–182 (2023).

    Article  Google Scholar 

  96. Devine, M. T. & Siddiqui, S. Strategic investment decisions in an oligopoly with a competitive fringe: an equilibrium problem with equilibrium constraints approach. Eur. J. Operational Res. 306, 1473–1494 (2023).

    Article  MathSciNet  Google Scholar 

  97. Chen, H. et al. distribution market-clearing and pricing considering coordination of DSOs and ISO: an EPEC approach. IEEE Trans. Smart Grid 12, 3150–3162 (2021).

    Article  Google Scholar 

  98. Hong, Q., Meng, F., Liu, J. & Bo, R. A bilevel game-theoretic decision-making framework for strategic retailers in both local and wholesale electricity markets. Appl. Energy 330, 120311 (2023).

    Article  Google Scholar 

  99. Wang, J., Xin, H., Xie, N. & Wang, Y. Equilibrium models of coordinated electricity and natural gas markets with different coupling information exchanging channels. Energy 239, 121827 (2022).

    Article  Google Scholar 

  100. Kwon, J. et al. The impact of market design and clean energy incentives on strategic generation investments and resource adequacy in low-carbon electricity markets. Renew. Energy Focus. 47, 100495 (2023).

    Article  Google Scholar 

  101. Pan, Z., Song, Y. & Jing, Z. A visualization method for bidding games in the electricity spot market. Energy Rep. 8, 1305–1312 (2022).

    Article  Google Scholar 

  102. Tsimopoulos, E. G. & Georgiadis, M. C. Nash equilibria in electricity pool markets with large-scale wind power integration. Energy 228, 120642 (2021).

    Article  Google Scholar 

  103. Sun, X. et al. An equilibrium capacity expansion model for power systems considering GENCOs’ coupled decisions between carbon and electricity markets. Appl. Energy 359, 122386 (2024).

    Article  Google Scholar 

  104. Kim, J., Mieth, R. & Dvorkin, Y. Computing a strategic decarbonization pathway: a chance-constrained equilibrium problem. IEEE Trans. Power Syst. 36, 1910–1921 (2021).

    Article  Google Scholar 

  105. Chen, D., Tian, C., Chen, Z. & Zhang, D. Competition among supply chains: the choice of financing strategy. Operational Res. 22, 977–1000 (2022).

    Article  Google Scholar 

  106. Ravichandiran, S., Saito, S., Shanmugamani, R. & Yang, W. Python Reinforcement Learning (Packt, 2019).

  107. Sewak, M. Deep Reinforcement Learning: Frontiers of Artificial Intelligence (Springer, 2019).

  108. Wilbur, J. & Salzberg, S. L. Multi-Agent Reinforcement Learning in Markov Games. Thesis, Johns Hopkins Univ. (1997).

  109. Su, J. et al. Deep multi-agent reinforcement learning for multi-level preventive maintenance in manufacturing systems. Expert. Syst. Appl. 192, 116323 (2022).

    Article  Google Scholar 

  110. Wan, C.-H. & Hwang, M.-C. Value-based deep reinforcement learning for adaptive isolated intersection signal control. IET Intell. Transp. Syst. 12, 1005–1010 (2018).

    Article  Google Scholar 

  111. Son, D. B. et al. Value-based reinforcement learning approaches for task offloading in delay constrained vehicular edge computing. Eng. Appl. Artif. Intell. 113, 104898 (2022).

    Article  Google Scholar 

  112. Fontanesi, L. et al. A reinforcement learning diffusion decision model for value-based decisions. Psychon. Bull. Rev. 26, 1099–1121 (2019).

    Article  Google Scholar 

  113. Zolfpour-Arokhlo, M. et al. Modeling of route planning system based on Q value-based dynamic programming with multi-agent reinforcement learning algorithms. Eng. Appl. Artif. Intell. 29, 163–177 (2014).

    Article  Google Scholar 

  114. Wu, X. et al. A value-based deep reinforcement learning model with human expertise in optimal treatment of sepsis. NPJ Digit. Med. 6, 15 (2023).

    Article  Google Scholar 

  115. Pearce, A. L., Fuchs, B. A. & Keller, K. L. The role of reinforcement learning and value-based decision-making frameworks in understanding food choice and eating behaviors. Front. Nutr. 9, 1021868 (2022).

    Article  Google Scholar 

  116. Li, H. et al. Value-based multi-agent deep reinforcement learning for collaborative computation offloading in internet of things networks. Wireless Netw. 30, 6915–6928 (2023).

    Article  Google Scholar 

  117. Plaat, A. Deep Reinforcement Learning 69–100 (Springer, 2022).

  118. Hwang, H.-S., Lee, M. & Seok, J. Deep reinforcement learning with a critic-value-based branch tree for the inverse design of two-dimensional optical devices. Appl. Soft Comput. 127, 109386 (2022).

    Article  Google Scholar 

  119. Liang, Y. et al. Agent-based modeling in electricity market using Deep Deterministic Policy Gradient algorithm. IEEE Trans. Power Syst. 35, 4180–4192 (2020).

    Article  Google Scholar 

  120. Lapan, M. Deep Reinforcement Learning Hands-On: Apply Modern RL Methods, with Deep Q-Networks, Value Iteration, Policy Gradients, TRPO, AlphaGo Zero and More (Packt, 2018).

  121. Lazić, N. et al. Optimization issues in KL-constrained approximate policy iteration. Preprint at https://doi.org/10.48550/arXiv.2102.06234 (2021).

  122. Wu, C., Bi, W. & Liu, H. Proximal Policy Optimization algorithm for dynamic pricing with online reviews. Expert. Syst. Appl. 213, 119191 (2023).

    Article  Google Scholar 

  123. Wang, X.-N. et al. Optimal reward baseline for policy-gradient reinforcement learning. Jísuánjī Xuébào 28, 1021–1026 (2005).

    Google Scholar 

  124. Estanqueiro, A. et al. Performance assessment of current and new market designs and trading mechanisms for national and regional markets. TradeRES https://traderes.eu/wp-content/uploads/2022/12/D5.3_TradeRES_PerformanceAssessmentMarkets.pdf (2024).

  125. Malehmirchegini, L. & Farzaneh, H. Incentive-based demand response modeling in a day-ahead wholesale electricity market in Japan, considering the impact of customer satisfaction on social welfare and profitability. Sustain. Energy Grids Netw. 34, 101044 (2023).

    Article  Google Scholar 

  126. Nibedita, B. & Irfan, M. Analyzing the asymmetric impacts of renewables on wholesale electricity price: empirical evidence from the Indian electricity market. Renew. Energy 194, 538–551 (2022).

    Article  Google Scholar 

  127. Wolak, F. A. Measuring the competitiveness benefits of a transmission investment policy: the case of the Alberta electricity market. Energy Policy 85, 426–444 (2015).

    Article  Google Scholar 

  128. Lin, Y., Barooah, P. & Mathieu, J. L. Ancillary services through demand scheduling and control of commercial buildings. IEEE Trans. Power Syst. 32, 186–197 (2017).

    Article  Google Scholar 

  129. Bersalli, G., Menanteau, P. & El-Methni, J. Renewable energy policy effectiveness: a panel data analysis across Europe and Latin America. Renew. Sustain. Energy Rev. 133, 110351 (2020).

    Article  Google Scholar 

  130. About TradeRES. TradeRES https://traderes.eu/about/ (accessed 26 February 2023).

  131. TradeRES Consortium. TradeRES https://traderes.eu/consortium/ (accessed 26 February 2023).

  132. Estanqueiro, A. et al. Performance assessment of current and new market designs and trading mechanisms for national and regional markets. TradeRES 2–85 https://traderes.eu/wp-content/uploads/2022/12/D5.3_TradeRES_PerformanceAssessmentMarkets.pdf (2022).

  133. The Nordic aFRR capacity markets. ENTSO-E https://www.entsoe.eu/network_codes/eb/nordic-afrr-capacity-markets/ (accessed 26 February 2023).

  134. Helistö, N. et al. Backbone — an adaptable energy systems modelling framework. Energies 12, 3388 (2019).

    Article  Google Scholar 

  135. TradeRES project. MASCEM multi-agent simulator of competitive electricity markets. TradeRES https://traderes.eu/wp-content/uploads/2024/03/5-TradeRES-User-Guide-MASCEM_ISEP_vf.pdf (accessed 26 February 2023).

  136. Multi-agent trading of renewable energy sources. TradeRES https://traderes.eu/wp-content/uploads/2024/03/6-TradeRES-User-Guide-RESTrade_LNEG_vf.pdf (2021).

  137. Prashanth L. A. & Fu, M. C. Risk-Sensitive Reinforcement Learning via Policy Gradient Search (Now, 2022).

  138. Corrado, N. E. & Hanna, J. P. On-policy policy gradient reinforcement learning without on-policy sampling. Preprint at https://doi.org/10.48550/arXiv.2311.08290 (2023).

  139. Ishihara, S. & Igarashi, H. Policy gradient reinforcement learning for policy represented by fuzzy rules: application to simulations on speed control of an automobile. J. Jpn. Soc. Fuzzy Theory Intell. Inform. 32, 801–810 (2020).

    Google Scholar 

  140. Mousavi, S. S., Schukat, M. & Howley, E. Traffic light control using deep policy-gradient and value-function-based reinforcement learning. IET Intell. Transp. Syst. 11, 417–423 (2017).

    Article  Google Scholar 

  141. Yang, X. et al. Data-based optimal consensus control for multiagent systems with policy gradient reinforcement learning. IEEE Trans. Neural Netw. Learn. Syst. 33, 3872–3883 (2022).

    Article  MathSciNet  Google Scholar 

  142. Yang, X. et al. Data-based predictive control via multistep policy gradient reinforcement learning. IEEE Trans. Cybern. 53, 1–11 (2023).

    Article  Google Scholar 

  143. Shen, W. & Huan, X. Bayesian sequential optimal experimental design for nonlinear models using policy gradient reinforcement learning. Comput. Methods Appl. Mech. Eng. 416, 116304 (2023).

    Article  MathSciNet  Google Scholar 

  144. Liu, Q. et al. Data-driven optimal bipartite consensus control for second-order multiagent systems via policy gradient reinforcement learning. IEEE Trans. Cybern. 54, 3468–3478 (2023).

    Article  Google Scholar 

  145. Khoshkholgh, M. G. & Yanikomeroglu, H. Faded-experience Trust Region Policy Optimization for model-free power allocation in interference channel. IEEE Wirel. Commun. Lett. 10, 659–663 (2021).

    Article  Google Scholar 

  146. Engstrom, L. et al. Implementation matters in deep policy gradients: a case study on PPO and TRPO. Preprint at https://doi.org/10.48550/arXiv.2005.12729 (2020).

  147. Zhu, Z. et al. Cooperative dispatch of microgrids community using risk-sensitive reinforcement learning with monotonously improved performance. Preprint at https://doi.org/10.48550/arXiv.2310.10997 (2023).

  148. Khodadadian, S. et al. On linear and super-linear convergence of Natural Policy Gradient algorithm. Syst. Control. Lett. 164, 105214 (2022).

    Article  MathSciNet  Google Scholar 

  149. Liu, H., Wu, Y. & Sun, F. Extreme Trust Region Policy Optimization for active object recognition. IEEE Trans. Neural Netw. Learn. Syst. 29, 2253–2258 (2018).

    Article  MathSciNet  Google Scholar 

  150. Zhang, Y. & Ross, K. W. On-policy deep reinforcement learning for the average-reward criterion. Preprint at https://doi.org/10.48550/arXiv.2106.07329 (2021).

  151. Roostaie, S. & Ebadzadeh, M. M. EnTRPO: Trust Region Policy Optimization method with entropy regularization. Preprint at https://doi.org/10.48550/arXiv.2110.13373 (2021).

Download references

Author information

Authors and Affiliations

  1. Department of Electrical and Electronics Engineering, The Hong Kong Polytechnic University, Kowloon, Hong Kong SAR

    Ziqing Zhu, Siqi Bu, Ka Wing Chan & Chi Yung Chung

  2. Research Centre for Grid Modernisation, The Hong Kong Polytechnic University, Kowloon, Hong Kong SAR

    Ziqing Zhu, Siqi Bu, Ka Wing Chan & Chi Yung Chung

  3. Research Institute for Smart Energy, The Hong Kong Polytechnic University, Kowloon, Hong Kong SAR

    Siqi Bu

  4. Policy Research Centre for Innovation and Technology, The Hong Kong Polytechnic University, Kowloon, Hong Kong SAR

    Siqi Bu

  5. International Centre of Urban Energy Nexus, The Hong Kong Polytechnic University, Kowloon, Hong Kong SAR

    Siqi Bu

  6. Department of Electrical Engineering and Computer Science at The University of Tennessee, Knoxville, TN, USA

    Fangxing Li

  7. School of Electrical Engineering, Southeast University, Nanjing, China

    Yujian Ye

  8. Department of Electrical and Electronic Engineering, Imperial College London, London, UK

    Goran Strbac

Authors
  1. Ziqing Zhu
    View author publications

    Search author on:PubMed Google Scholar

  2. Siqi Bu
    View author publications

    Search author on:PubMed Google Scholar

  3. Ka Wing Chan
    View author publications

    Search author on:PubMed Google Scholar

  4. Fangxing Li
    View author publications

    Search author on:PubMed Google Scholar

  5. Yujian Ye
    View author publications

    Search author on:PubMed Google Scholar

  6. Chi Yung Chung
    View author publications

    Search author on:PubMed Google Scholar

  7. Goran Strbac
    View author publications

    Search author on:PubMed Google Scholar

Contributions

Z.Z. was responsible for conceptualization, formal analysis and writing the original draft. S.B. was responsible for conceptualization, supervision and validation, and reviewed and edited the manuscript. K.W.C., C.Y.C. and G.S. were responsible for supervision and validation. F.L. was responsible for validation, and reviewed and edited the manuscript. Y.Y. was responsible for supervision and validation, and reviewed and edited the manuscript.

Corresponding authors

Correspondence to Siqi Bu or Yujian Ye.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhu, Z., Bu, S., Chan, K.W. et al. Designing the future electricity spot market with high renewables via reliable simulations. Nat Rev Electr Eng 2, 320–337 (2025). https://doi.org/10.1038/s44287-025-00163-9

Download citation

  • Accepted:

  • Published:

  • Version of record:

  • Issue date:

  • DOI: https://doi.org/10.1038/s44287-025-00163-9

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing