Abstract
Quantifying the effect of non-pharmaceutical interventions (NPIs) is essential for formulating lessons from the COVID-19 pandemic. To enable a more reliable and rigorous evaluation of NPIs based on time series data, we reanalyse the official evaluation of NPIs in Germany. As the first part of a multi-step validation and verification project, we focus on properly analysing statistical uncertainties for time series data. Using a set of 9 competitive statistical methods for estimating the effects of NPIs and other determinants of disease spread on the effective reproduction number \(\mathcal {R}(t)\), we find significantly wider confidence intervals than the official evaluation. In addition to vaccination and seasonality, only few NPIs – such as restrictions in public spaces – can be confidently associated with variations in \(\mathcal {R}(t)\), but even then effect sizes have large uncertainties. Furthermore, due to multicollinearity in NPI activation patterns, it is difficult to distinguish potential effects of NPIs in public spaces from other interventions that came into force early, such as physical distancing. In future, NPIs should be more carefully designed and accompanied by plans for data collections to allow for a timely evaluation of benefits and harms as a basis for an effective and proportionate response.
Data availability
Our Python code is freely available on Zenodo (https://doi.org/10.5281/zenodo.14761068). The code utilises processed data from the original StopptCOVID project [23], which in turn uses NPI data from infas (https://datenkatalog.infas360.de/dataset/cdp), which are freely available after registration. Our code repository contains instructions to download the required third-party data and code to generate the requisite input data for our analysis.
References
UK Covid-19 Inquiry. Module 1 report: The resilience and preparedness of the United Kingdom (2024). https://covid19.public-inquiry.uk/reports/module-1-report-the-resilience-and-preparedness-of-the-united-kingdom/.
Murphy, C. et al. Effectiveness of social distancing measures and lockdowns for reducing transmission of COVID-19 in non-healthcare, community-based settings. Philos. Trans. A Math. Phys. Eng. Sci. 381, 20230132 (2023).
Funk, M. et al. Wirksamkeit von Corona-Massnahmen in der Schweiz (2022). https://www.seco.admin.ch/seco/de/home/Publikationen_Dienstleistungen/Publikationen_und_Formulare/Strukturwandel_Wachstum/Wachstum/wirksamkeit-corona-massnahmen-schweiz.html.
Gesundheit, B. Evaluation der Rechtsgrundlagen und Maßnahmen der Pandemiepolitik. Bericht des Sachverständigenausschusses nach §5 Abs. 9 IfSG (2022). https://www.bundesgesundheitsministerium.de/fileadmin/Dateien/3_Downloads/S/Sachverstaendigenausschuss/BER_lfSG-BMG.pdf.
Iezadi, S. et al. Effectiveness of non-pharmaceutical public health interventions against COVID-19: A systematic review and meta-analysis. PLoS One 16, e0260371 (2021).
Flaxman, S. et al. Estimating the effects of non-pharmaceutical interventions on COVID-19 in europe. Nature 584, 257–261 (2020).
Sharma, M. et al. Understanding the effectiveness of government interventions against the resurgence of COVID-19 in europe. Nat. Commun. 12, 5820 (2021).
Bendavid, E., Oh, C., Bhattacharya, J. & Ioannidis, J. P. A. Assessing mandatory stay-at-home and business closure effects on the spread of COVID-19. Eur. J. Clin. Invest. 51, e13484 (2021).
Bendavid, E. & Patel, C. J. Epidemic outcomes following government responses to covid-19: Insights from nearly 100,000 models. Science Advances 10, eadn0671 (2024). https://www.science.org/doi/abs/10.1126/sciadv.adn0671.
Ramirez, D. W. E., Klinkhammer, M. D. & Rowland, L. C. COVID-19 transmission during transportation of 1st to 12th grade students: Experience of an independent school in virginia. J. Sch. Health 91, 678–682 (2021).
Helsingen, L. M. et al. Covid-19 transmission in fitness centers in norway - a randomized trial. BMC Public Health 21, 2103 (2021).
Hunter, P.R., Colón-González, F.J., Brainard, J. & Rushton, S. Impact of non-pharmaceutical interventions against COVID-19 in europe in 2020: a quasi-experimental non-equivalent group and time series design study. Euro Surveill. 26 (2021).
Wang, G. Stay at home to stay safe: Effectiveness of stay-at-home orders in containing the COVID-19 pandemic. Production and Operations Management 31, 2289–2305 (2022). https://ideas.repec.org/a/bla/popmgt/v31y2022i5p2289-2305.html.
Guyatt, G. et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J. Clin. Epidemiol. 64, 383–394 (2011).
Banholzer, N. et al. The methodologies to assess the effectiveness of non-pharmaceutical interventions during covid-19: a systematic review. European Journal of Epidemiology 37, 1003–1024. https://doi.org/10.1007/s10654-022-00908-y (2022).
Raftery, A. E., Madigan, D. & Hoeting, J. A. Bayesian Model Averaging for Linear Regression Models. Journal of the American Statistical Association 92, 179–191. https://doi.org/10.1080/01621459.1997.10473615 (1997).
Edeling, W. et al. The impact of uncertainty on predictions of the CovidSim epidemiological code. Nat. Comput. Sci. 1, 128–135 (2021).
Chin, V., Ioannidis, J. P. A., Tanner, M. A. & Cripps, S. Effect estimates of COVID-19 non-pharmaceutical interventions are non-robust and highly model-dependent. J. Clin. Epidemiol. 136, 96–132 (2021).
Bundesverfassungsgericht. decisions 1 BvR 971/21 and 1 BvR 1069/21. https://www.bundesverfassungsgericht.de/SharedDocs/Entscheidungen/DE/2021/11/rs20211119_1bvr097121.html (2021).
an der Heiden, M., Hicketier, A. & Bremer, V. Wirksamkeit und Wirkung von anti-epidemischen Maßnahmen auf die COVID-19-Pandemie in Deutschland (StopptCOVID-Studie) (2023). https://www.rki.de/DE/Content/InfAZ/N/Neuartiges_Coronavirus/Projekte_RKI/StopptCOVID_studie.html.
Meyer, G., Mühlhauser, I., Brinks, R. & Müller, B. Wirksame Kontrollmaßnahmen in der SARS-CoV-2-Pandemie? KVH Journal 10/2023 (2023).
Baumgarten, W., Beige, O., Haake, D., Merkl, J. & Wieland, T. Was die ‘StopptCOVID’-Studie des RKI sagt - und was nicht. Cicero Online (2023). https://www.cicero.de/innenpolitik/corona-pandemie-robertkochinstitut-studie.
Hicketier, A. & an der Heiden, M. StopptCOVID-Studie - Daten, Analyse und Ergebnisse (2024). https://zenodo.org/records/10888033.
Bodderas, E. Kanzleramt drängt Lauterbach zur Offenlegung von Corona-Studie. WELT, 8.2.2024 (2024). https://www.welt.de/politik/deutschland/plus250799234/Pandemie-Aufarbeitung-Kanzleramt-draengt-Lauterbach-zur-Offenlegung-von-Corona-Studie.html.
Stevens, G. A. et al. Guidelines for accurate and transparent health estimates reporting: The GATHER statement. Lancet 388, e19–e23 (2016).
Akaike, H. Information Theory and an Extension of the Maximum Likelihood Principle, 199–213 (Springer New York, New York, NY, 1998). https://doi.org/10.1007/978-1-4612-1694-0_15.
Roser, M. What is the covid-19 stringency index? Our World in Data (2021). https://ourworldindata.org/metrics-explained-covid19-stringency-index.
State Government of Mecklenburg-Vorpommern. Kitas und Kindertagespflege ab Montag flächendeckend geschlossen (2020). https://www.regierung-mv.de/Landesregierung/sm/Aktuell/?id=158498&processor=processor.sa.pressemitteilung.
Durbin, J. & Watson, G. S. Testing for serial correlation in least squares regression. i. Biometrika 37, 409–428. https://doi.org/10.1093/biomet/37.3-4.409 (1950).
Greene, W.H. Econometric Analysis (Pearson Education, 2003), fifth edn. http://pages.stern.nyu.edu/~wgreene/Text/econometricanalysis.htm.
Johnston, R., Jones, K. & Manley, D. Confounding and collinearity in regression analysis: a cautionary tale and an alternative procedure, illustrated by studies of British voting behaviour. Quality & Quantity 52, 1957–1976. https://doi.org/10.1007/s11135-017-0584-6 (2018).
Gostic, K. M. et al. Practical considerations for measuring the effective reproductive number, rt. PLOS Computational Biology 16, 1–21. https://doi.org/10.1371/journal.pcbi.1008409 (2020).
Neipel, J., Bauermann, J., Bo, S., Harmon, T. & Jülicher, F. Power-law population heterogeneity governs epidemic waves. PLoS ONE 15, e0239678 (2020). arXiv:2008.00471.
Gomes, M.G.M. et al. Individual variation in susceptibility or exposure to SARS-CoV-2 lowers the herd immunity threshold. Journal of Theoretical Biology 540, 111063 (2022). https://www.sciencedirect.com/science/article/pii/S0022519322000613.
den Boon, S. et al. Guidelines for multi-model comparisons of the impact of infectious disease interventions. BMC Medicine 17, 163 (2019).
Balshem, H. et al. GRADE guidelines: 3. rating the quality of evidence. J. Clin. Epidemiol. 64, 401–406 (2011).
Dekker, M.M., Coffeng, L.E., Pijpers, F.P., Panja, D. & de Vlas, S.J. Reducing societal impacts of sars-cov-2 interventions through subnational implementation. eLife 12, e80819 (2023). https://doi.org/10.7554/eLife.80819.
Driscoll, J.C. & Kraay, A.C. Consistent Covariance Matrix Estimation With Spatially Dependent Panel Data. The Review of Economics and Statistics 80, 549–560 (1998). https://ideas.repec.org/a/tpr/restat/v80y1998i4p549-560.html.
Hoechle, D. Robust standard errors for panel regressions with cross-sectional dependence. Stata Journal 7, 281–312 (2007). https://ideas.repec.org/a/tsj/stataj/v7y2007i3p281-312.html.
Ebisuzaki, W. A Method to Estimate the Statistical Significance of a Correlation When the Data Are Serially Correlated. Journal of Climate 10, 2147–2153 (1997).
Politis, D. N. & Romano, J. P. The stationary bootstrap. Journal of the American Statistical Association 89, 1303–1313. https://doi.org/10.1080/01621459.1994.10476870 (1994).
Wooldridge, J.M. Two-way fixed effects, the two-way mundlak regression, and difference-in-differences estimators (2021). https://ssrn.com/abstract=3906345.
Harvey, A. State space models, 269–275 (Palgrave Macmillan UK. London https://doi.org/10.1057/9780230280830_30 (2010).
Schwarz, G. Estimating the Dimension of a Model. The Annals of Statistics 6, 461–464 (1978). http://www.jstor.org/stable/2958889.
Breiman, L. Random forests. Machine Learning 45, 5–32 (2001).
Zou, H. & Hastie, T. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology 67, 301–320. https://doi.org/10.1111/j.1467-9868.2005.00503.x (2005).
Hansen, P. C. Truncated Singular Value Decomposition Solutions to Discrete Ill-Posed Problems with Ill-Determined Numerical Rank. SIAM Journal on Scientific and Statistical Computing 11, 503–518. https://doi.org/10.1137/0911028 (1990).
Seabold, S. & Perktold, J. statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference (2010).
Pedregosa, F. et al. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, 2825–2830 (2011).
Abbott, B. et al. Analysis of first LIGO science data for stochastic gravitational waves. Phys. Rev. D 69, 122004 (2004) arXiv:gr-qc/0312088.
Rizzuto, A.C. et al. Zodiacal Exoplanets in Time (ZEIT). VIII. A Two-planet System in Praesepe from K2 Campaign 16. Astron. J. 156, 195 (2018). arXiv:1808.07068.
Bundgaard, H. et al. Effectiveness of adding a mask recommendation to other public health measures to prevent SARS-CoV-2 infection in danish mask wearers: A randomized controlled trial. Ann. Intern. Med. 174, 335–343 (2021).
Abaluck, J. et al. Impact of community masking on COVID-19: A cluster-randomized trial in bangladesh. Science 375, eabi9069 (2022).
Jefferson, T. et al. Physical interventions to interrupt or reduce the spread of respiratory viruses. Cochrane Database of Systematic Reviews (2023). https://doi.org/10.1002/14651858.CD006207.pub6.
Polack, F. P. et al. Safety and efficacy of the BNT162b2 mRNA covid-19 vaccine. N. Engl. J. Med. 383, 2603–2615 (2020).
Baden, L. R. et al. Efficacy and safety of the mRNA-1273 SARS-CoV-2 vaccine. N. Engl. J. Med. 384, 403–416 (2021).
Chemaitelly, H. et al. mRNA-1273 COVID-19 vaccine effectiveness against the B.1.1.7 and B.1.351 variants and severe COVID-19 disease in Qatar. Nat. Med. 27, 1614–1621 (2021).
Shah, A.S. et al. Effect of Vaccination on Transmission of SARS-CoV-2. New England Journal of Medicine 385, 1718–1720 (2021). https://www.nejm.org/doi/full/10.1056/NEJMc2106757.
Spiliopoulos, L. On the effectiveness of COVID-19 restrictions and lockdowns: Pan metron ariston. BMC Public Health 22, 1842 (2022).
Perra, N., Balcan, D., Gonçalves, B. & Vespignani, A. Towards a characterization of behavior-disease models. PLoS One 6, e23084 (2011).
Agaba, G., Kyrychko, Y. & Blyuss, K. Mathematical model for the impact of awareness on the dynamics of infectious diseases. Mathematical Biosciences 286, 22–30 (2017). https://www.sciencedirect.com/science/article/pii/S0025556417300433.
Munro, A. P.S. & House, T. Cycles of susceptibility: Immunity debt explains altered infectious disease dynamics post-pandemic. Clinical Infectious Diseases ciae493 (2024). https://doi.org/10.1093/cid/ciae493.
Feng, Z., Towers, S. & Yang, Y. Modeling the effects of vaccination and treatment on pandemic influenza. AAPS J. 13, 427–437 (2011).
Worby, C.J., Wallinga, J., Lipsitch, M. & Goldstein, E. Population effect of influenza vaccination under co-circulation of non-vaccine variants and the case for a bivalent a/h3n2 vaccine component. Epidemics 19, 74–82 (2017). https://www.sciencedirect.com/science/article/pii/S1755436517300208.
Backer, J., van Boven, M., van der Hoek, W. & Wallinga, J. Vaccinating children against influenza increases variability in epidemic size. Epidemics 26, 95–103. https://doi.org/10.1016/j.epidem.2018.10.003 (2019).
de Boer, P. T., Backer, J. A., van Hoek, A. J. & Wallinga, J. Vaccinating children against influenza: overall cost-effective with potential for undesirable outcomes. BMC Med. 18, 11 (2020).
Dürr, H.-P. & Eichner, M. Corona-Pandemie: Zukunfts-Überlegungen aus der Sicht epidemiologischer Modellierung. Monitor Versorgungsforschung 02/22, 57 (2022).
Castioni, P., Gómez, S., Granell, C. & Arenas, A. Rebound in epidemic control: how misaligned vaccination timing amplifies infection peaks. npj Complexity 1, 20 (2024).
Anderson, R. M. & May, R. M. Vaccination against rubella and measles: quantitative investigations of different policies. J. Hyg. (Lond.) 90, 259–325 (1983).
Cohen, T. & Lipsitch, M. Too little of a good thing: a paradox of moderate infection control. Epidemiology 19, 588–589 (2008).
Heffernan, J. M. & Keeling, M. J. Implications of vaccination and waning immunity. Proc. Biol. Sci. 276, 2071–2080 (2009).
Chiolero, A., Tancredi, S. & Ioannidis, J. P. A. Slow data public health. Eur. J. Epidemiol. 38, 1219–1225 (2023).
Acknowledgements
We thank M. an der Heiden and V. Bremer for answering technical questions on the original StopptCOVID project. We acknowledge helpful discussions with W. Baumgarten, O. Beige, G. Meyer, I. Mühlhauser, D. Schuricht, and T. Wieland. We are grateful to the German Network for Evidence-Based Medicine, the German Society for Epidemiology and the German Reproducibility Network for help in distributing the project call.
Author information
Authors and Affiliations
Contributions
B.M.: Conceptualisation, Project Management, Software, Formal analysis, Visualisation, Writing - Original draft, Review and Editing. I.P.: Conceptualisation, Procedures, Literature Review, Writing - Original draft, Review and Editing. ML: Conceptualisation, Code checks (models DK and Ebisuzaki), Writing - Review. R.B.: Conceptualisation, Writing - Review and Editing. S.C.: Conceptualisation, Writing - Review and Editing. M.G.M.M.: Conceptualisation, Writing - Review. D.H.: Conceptualisation, Writing - Review and Editing. J.P.A.I.: Conceptualisation, Writing - Review and Editing.
Corresponding author
Ethics declarations
Ethical approval
This study did not involve research on humans or animals, and only used publicly available, non-personal data sets.
Competing interests
B.M., I.P., M.L., and R.B. are signatories of a call for a non-partisan pandemic review in Germany (https://pandemieaufarbeitung.net). B.M. has been engaged in discussion and consultation of pandemic policy and science policy with members of several German parties (CDU, CSU, FDP, SPD, BSW, Greens), but is not receiving remuneration. S.C., M.G.M.G., D.H., and J.P.A.I. have no competing interests to declare.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
About this article
Cite this article
Müller, B., Padberg, I., Lorke, M. et al. Uncertainty and inconsistency of COVID-19 non-pharmaceutical intervention effects with multiple competitive statistical models. Sci Rep (2026). https://doi.org/10.1038/s41598-026-36265-z
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41598-026-36265-z