Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Data, measurement and empirical methods in the science of science

Nature Human Behaviour volume 7, pages 1046–1058 (2023)Cite this article

Subjects

Abstract

The advent of large-scale datasets that trace the workings of science has encouraged researchers from many different disciplinary backgrounds to turn scientific methods into science itself, cultivating a rapidly expanding ‘science of science’. This Review considers this growing, multidisciplinary literature through the lens of data, measurement and empirical methods. We discuss the purposes, strengths and limitations of major empirical approaches, seeking to increase understanding of the field’s diverse methodologies and expand researchers’ toolkits. Overall, new empirical developments provide enormous capacity to test traditional beliefs and conceptual frameworks about science, discover factors associated with scientific productivity, predict scientific outcomes and design policies that facilitate scientific progress.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to the full article PDF.

USD 39.95

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Science of science data and linkages.
Fig. 2: An example of prediction with machine learning.
Fig. 3: Quasi-experiment methods.

Similar content being viewed by others

References

  1. Bush, V. Science–the Endless Frontier: A Report to the President on a Program for Postwar Scientific Research (National Science Foundation, 1990).

  2. Mokyr, J. The Gifts of Athena (Princeton Univ. Press, 2011).

  3. Jones, B. F. in Rebuilding the Post-Pandemic Economy (eds Kearney, M. S. & Ganz, A.) 272–310 (Aspen Institute Press, 2021).

  4. Wang, D. & Barabási, A.-L. The Science of Science (Cambridge Univ. Press, 2021).

  5. Fortunato, S. et al. Science of science. Science 359, eaao0185 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  6. Azoulay, P. et al. Toward a more scientific science. Science 361, 1194–1197 (2018).

    Article  PubMed  Google Scholar 

  7. Clauset, A., Larremore, D. B. & Sinatra, R. Data-driven predictions in the science of science. Science 355, 477–480 (2017).

    Article  CAS  PubMed  Google Scholar 

  8. Zeng, A. et al. The science of science: from the perspective of complex systems. Phys. Rep. 714, 1–73 (2017).

    Article  Google Scholar 

  9. Lin, Z., Yin. Y., Liu, L. & Wang, D. SciSciNet: a large-scale open data lake for the science of science research. Sci. Data, https://doi.org/10.1038/s41597-023-02198-9 (2023).

  10. Ahmadpoor, M. & Jones, B. F. The dual frontier: patented inventions and prior scientific advance. Science 357, 583–587 (2017).

    Article  CAS  PubMed  Google Scholar 

  11. Azoulay, P., Graff Zivin, J. S., Li, D. & Sampat, B. N. Public R&D investments and private-sector patenting: evidence from NIH funding rules. Rev. Econ. Stud. 86, 117–152 (2019).

    Article  PubMed  Google Scholar 

  12. Yin, Y., Dong, Y., Wang, K., Wang, D. & Jones, B. F. Public use and public funding of science. Nat. Hum. Behav. 6, 1344–1350 (2022).

    Article  PubMed  Google Scholar 

  13. Merton, R. K. The Sociology of Science: Theoretical and Empirical Investigations (Univ. Chicago Press, 1973).

  14. Kuhn, T. The Structure of Scientific Revolutions (Princeton Univ. Press, 2021).

  15. Uzzi, B., Mukherjee, S., Stringer, M. & Jones, B. Atypical combinations and scientific impact. Science 342, 468–472 (2013).

    Article  CAS  PubMed  Google Scholar 

  16. Zuckerman, H. Scientific Elite: Nobel Laureates in the United States (Transaction Publishers, 1977).

  17. Wu, L., Wang, D. & Evans, J. A. Large teams develop and small teams disrupt science and technology. Nature 566, 378–382 (2019).

    Article  CAS  PubMed  Google Scholar 

  18. Wuchty, S., Jones, B. F. & Uzzi, B. The increasing dominance of teams in production of knowledge. Science 316, 1036–1039 (2007).

    Article  CAS  PubMed  Google Scholar 

  19. Foster, J. G., Rzhetsky, A. & Evans, J. A. Tradition and innovation in scientists’ research strategies. Am. Sociol. Rev. 80, 875–908 (2015).

    Article  Google Scholar 

  20. Wang, D., Song, C. & Barabási, A.-L. Quantifying long-term scientific impact. Science 342, 127–132 (2013).

    Article  CAS  PubMed  Google Scholar 

  21. Clauset, A., Arbesman, S. & Larremore, D. B. Systematic inequality and hierarchy in faculty hiring networks. Sci. Adv. 1, e1400005 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  22. Ma, A., Mondragón, R. J. & Latora, V. Anatomy of funded research in science. Proc. Natl Acad. Sci. USA 112, 14760–14765 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  23. Ma, Y. & Uzzi, B. Scientific prize network predicts who pushes the boundaries of science. Proc. Natl Acad. Sci. USA 115, 12608–12615 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  24. Azoulay, P., Graff Zivin, J. S. & Manso, G. Incentives and creativity: evidence from the academic life sciences. RAND J. Econ. 42, 527–554 (2011).

    Article  Google Scholar 

  25. Schor, S. & Karten, I. Statistical evaluation of medical journal manuscripts. JAMA 195, 1123–1128 (1966).

    Article  CAS  PubMed  Google Scholar 

  26. Platt, J. R. Strong inference: certain systematic methods of scientific thinking may produce much more rapid progress than others. Science 146, 347–353 (1964).

    Article  CAS  PubMed  Google Scholar 

  27. Ioannidis, J. P. Why most published research findings are false. PLoS Med. 2, e124 (2005).

    Article  PubMed  PubMed Central  Google Scholar 

  28. Simonton, D. K. Career landmarks in science: individual differences and interdisciplinary contrasts. Dev. Psychol. 27, 119 (1991).

    Article  Google Scholar 

  29. Way, S. F., Morgan, A. C., Clauset, A. & Larremore, D. B. The misleading narrative of the canonical faculty productivity trajectory. Proc. Natl Acad. Sci. USA 114, E9216–E9223 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  30. Sinatra, R., Wang, D., Deville, P., Song, C. & Barabási, A.-L. Quantifying the evolution of individual scientific impact. Science 354, aaf5239 (2016).

    Article  PubMed  Google Scholar 

  31. Liu, L. et al. Hot streaks in artistic, cultural, and scientific careers. Nature 559, 396–399 (2018).

    Article  CAS  PubMed  Google Scholar 

  32. Liu, L., Dehmamy, N., Chown, J., Giles, C. L. & Wang, D. Understanding the onset of hot streaks across artistic, cultural, and scientific careers. Nat. Commun. 12, 5392 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  33. Squazzoni, F. et al. Peer review and gender bias: a study on 145 scholarly journals. Sci. Adv. 7, eabd0299 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  34. Hofstra, B. et al. The diversity–innovation paradox in science. Proc. Natl Acad. Sci. USA 117, 9284–9291 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  35. Huang, J., Gates, A. J., Sinatra, R. & Barabási, A.-L. Historical comparison of gender inequality in scientific careers across countries and disciplines. Proc. Natl Acad. Sci. USA 117, 4609–4616 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  36. Gläser, J. & Laudel, G. Governing science: how science policy shapes research content. Eur. J. Sociol. 57, 117–168 (2016).

    Article  Google Scholar 

  37. Stephan, P. E. How Economics Shapes Science (Harvard Univ. Press, 2012).

  38. Garfield, E. & Sher, I. H. New factors in the evaluation of scientific literature through citation indexing. Am. Doc. 14, 195–201 (1963).

    Article  CAS  Google Scholar 

  39. de Solla Price, D. J. Networks of scientific papers. Science 149, 510–515 (1965).

    Article  CAS  Google Scholar 

  40. Etzkowitz, H., Kemelgor, C. & Uzzi, B. Athena Unbound: The Advancement of Women in Science and Technology (Cambridge Univ. Press, 2000).

  41. Simonton, D. K. Scientific Genius: A Psychology of Science (Cambridge Univ. Press, 1988).

  42. Khabsa, M. & Giles, C. L. The number of scholarly documents on the public web. PLoS ONE 9, e93949 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  43. Xia, F., Wang, W., Bekele, T. M. & Liu, H. Big scholarly data: a survey. IEEE Trans. Big Data 3, 18–35 (2017).

    Article  Google Scholar 

  44. Evans, J. A. & Foster, J. G. Metaknowledge. Science 331, 721–725 (2011).

    Article  CAS  PubMed  Google Scholar 

  45. Milojević, S. Quantifying the cognitive extent of science. J. Informetr. 9, 962–973 (2015).

    Article  Google Scholar 

  46. Rzhetsky, A., Foster, J. G., Foster, I. T. & Evans, J. A. Choosing experiments to accelerate collective discovery. Proc. Natl Acad. Sci. USA 112, 14569–14574 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  47. Poncela-Casasnovas, J., Gerlach, M., Aguirre, N. & Amaral, L. A. Large-scale analysis of micro-level citation patterns reveals nuanced selection criteria. Nat. Hum. Behav. 3, 568–575 (2019).

    Article  PubMed  Google Scholar 

  48. Hardwicke, T. E. et al. Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition. R. Soc. Open Sci. 5, 180448 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  49. Nagaraj, A., Shears, E. & de Vaan, M. Improving data access democratizes and diversifies science. Proc. Natl Acad. Sci. USA 117, 23490–23498 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  50. Bravo, G., Grimaldo, F., López-Iñesta, E., Mehmani, B. & Squazzoni, F. The effect of publishing peer review reports on referee behavior in five scholarly journals. Nat. Commun. 10, 322 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  51. Tran, D. et al. An open review of open review: a critical analysis of the machine learning conference review process. Preprint at https://doi.org/10.48550/arXiv.2010.05137 (2020).

  52. Tshitoyan, V. et al. Unsupervised word embeddings capture latent knowledge from materials science literature. Nature 571, 95–98 (2019).

    Article  CAS  PubMed  Google Scholar 

  53. Yang, Y., Wu, Y. & Uzzi, B. Estimating the deep replicability of scientific findings using human and artificial intelligence. Proc. Natl Acad. Sci. USA 117, 10762–10768 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  54. Mukherjee, S., Uzzi, B., Jones, B. & Stringer, M. A new method for identifying recombinations of existing knowledge associated with high‐impact innovation. J. Prod. Innov. Manage. 33, 224–236 (2016).

    Article  Google Scholar 

  55. Leahey, E., Beckman, C. M. & Stanko, T. L. Prominent but less productive: the impact of interdisciplinarity on scientists’ research. Adm. Sci. Q. 62, 105–139 (2017).

    Article  Google Scholar 

  56. Sauermann, H. & Haeussler, C. Authorship and contribution disclosures. Sci. Adv. 3, e1700404 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  57. Oliveira, D. F. M., Ma, Y., Woodruff, T. K. & Uzzi, B. Comparison of National Institutes of Health grant amounts to first-time male and female principal investigators. JAMA 321, 898–900 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  58. Yang, Y., Chawla, N. V. & Uzzi, B. A network’s gender composition and communication pattern predict women’s leadership success. Proc. Natl Acad. Sci. USA 116, 2033–2038 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  59. Way, S. F., Larremore, D. B. & Clauset, A. Gender, productivity, and prestige in computer science faculty hiring networks. In Proc. 25th International Conference on World Wide Web 1169–1179. (ACM 2016)

  60. Malmgren, R. D., Ottino, J. M. & Amaral, L. A. N. The role of mentorship in protege performance. Nature 465, 622–626 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  61. Ma, Y., Mukherjee, S. & Uzzi, B. Mentorship and protégé success in STEM fields. Proc. Natl Acad. Sci. USA 117, 14077–14083 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  62. Börner, K. et al. Skill discrepancies between research, education, and jobs reveal the critical need to supply soft skills for the data economy. Proc. Natl Acad. Sci. USA 115, 12630–12637 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  63. Biasi, B. & Ma, S. The Education-Innovation Gap (National Bureau of Economic Research Working papers, 2020).

  64. Bornmann, L. Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. J. Informetr. 8, 895–903 (2014).

    Article  Google Scholar 

  65. Cleary, E. G., Beierlein, J. M., Khanuja, N. S., McNamee, L. M. & Ledley, F. D. Contribution of NIH funding to new drug approvals 2010–2016. Proc. Natl Acad. Sci. USA 115, 2329–2334 (2018).

    Article  CAS  Google Scholar 

  66. Spector, J. M., Harrison, R. S. & Fishman, M. C. Fundamental science behind today’s important medicines. Sci. Transl. Med. 10, eaaq1787 (2018).

    Article  PubMed  Google Scholar 

  67. Haunschild, R. & Bornmann, L. How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data. Scientometrics 110, 1209–1216 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  68. Yin, Y., Gao, J., Jones, B. F. & Wang, D. Coevolution of policy and science during the pandemic. Science 371, 128–130 (2021).

    Article  CAS  PubMed  Google Scholar 

  69. Sugimoto, C. R., Work, S., Larivière, V. & Haustein, S. Scholarly use of social media and altmetrics: a review of the literature. J. Assoc. Inf. Sci. Technol. 68, 2037–2062 (2017).

    Article  Google Scholar 

  70. Dunham, I. Human genes: time to follow the roads less traveled? PLoS Biol. 16, e3000034 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  71. Kustatscher, G. et al. Understudied proteins: opportunities and challenges for functional proteomics. Nat. Methods 19, 774–779 (2022).

    Article  CAS  PubMed  Google Scholar 

  72. Rosenthal, R. The file drawer problem and tolerance for null results. Psychol. Bull. 86, 638 (1979).

    Article  Google Scholar 

  73. Franco, A., Malhotra, N. & Simonovits, G. Publication bias in the social sciences: unlocking the file drawer. Science 345, 1502–1505 (2014).

    Article  CAS  PubMed  Google Scholar 

  74. Vera-Baceta, M.-A., Thelwall, M. & Kousha, K. Web of Science and Scopus language coverage. Scientometrics 121, 1803–1813 (2019).

    Article  Google Scholar 

  75. Waltman, L. A review of the literature on citation impact indicators. J. Informetr. 10, 365–391 (2016).

    Article  Google Scholar 

  76. Garfield, E. & Merton, R. K. Citation Indexing: Its Theory and Application in Science, Technology, and Humanities (Wiley, 1979).

  77. Kelly, B., Papanikolaou, D., Seru, A. & Taddy, M. Measuring Technological Innovation Over the Long Run Report No. 0898-2937 (National Bureau of Economic Research, 2018).

  78. Kogan, L., Papanikolaou, D., Seru, A. & Stoffman, N. Technological innovation, resource allocation, and growth. Q. J. Econ. 132, 665–712 (2017).

    Article  Google Scholar 

  79. Hall, B. H., Jaffe, A. & Trajtenberg, M. Market value and patent citations. RAND J. Econ. 36, 16–38 (2005).

    Google Scholar 

  80. Yan, E. & Ding, Y. Applying centrality measures to impact analysis: a coauthorship network analysis. J. Am. Soc. Inf. Sci. Technol. 60, 2107–2118 (2009).

    Article  Google Scholar 

  81. Radicchi, F., Fortunato, S., Markines, B. & Vespignani, A. Diffusion of scientific credits and the ranking of scientists. Phys. Rev. E 80, 056103 (2009).

    Article  Google Scholar 

  82. Bollen, J., Rodriquez, M. A. & Van de Sompel, H. Journal status. Scientometrics 69, 669–687 (2006).

    Article  CAS  Google Scholar 

  83. Bergstrom, C. T., West, J. D. & Wiseman, M. A. The eigenfactor™ metrics. J. Neurosci. 28, 11433–11434 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  84. Cronin, B. & Sugimoto, C. R. Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact (MIT Press, 2014).

  85. Hicks, D., Wouters, P., Waltman, L., De Rijcke, S. & Rafols, I. Bibliometrics: the Leiden Manifesto for research metrics. Nature 520, 429–431 (2015).

    Article  PubMed  Google Scholar 

  86. Catalini, C., Lacetera, N. & Oettl, A. The incidence and role of negative citations in science. Proc. Natl Acad. Sci. USA 112, 13823–13826 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  87. Alcacer, J. & Gittelman, M. Patent citations as a measure of knowledge flows: the influence of examiner citations. Rev. Econ. Stat. 88, 774–779 (2006).

    Article  Google Scholar 

  88. Ding, Y. et al. Content‐based citation analysis: the next generation of citation analysis. J. Assoc. Inf. Sci. Technol. 65, 1820–1833 (2014).

    Article  Google Scholar 

  89. Teufel, S., Siddharthan, A. & Tidhar, D. Automatic classification of citation function. In Proc. 2006 Conference on Empirical Methods in Natural Language Processing, 103–110 (Association for Computational Linguistics 2006)

  90. Seeber, M., Cattaneo, M., Meoli, M. & Malighetti, P. Self-citations as strategic response to the use of metrics for career decisions. Res. Policy 48, 478–491 (2019).

    Article  Google Scholar 

  91. Pendlebury, D. A. The use and misuse of journal metrics and other citation indicators. Arch. Immunol. Ther. Exp. 57, 1–11 (2009).

    Article  Google Scholar 

  92. Biagioli, M. Watch out for cheats in citation game. Nature 535, 201 (2016).

    Article  CAS  PubMed  Google Scholar 

  93. Jo, W. S., Liu, L. & Wang, D. See further upon the giants: quantifying intellectual lineage in science. Quant. Sci. Stud. 3, 319–330 (2022).

    Article  Google Scholar 

  94. Boyack, K. W., Klavans, R. & Börner, K. Mapping the backbone of science. Scientometrics 64, 351–374 (2005).

    Article  CAS  Google Scholar 

  95. Gates, A. J., Ke, Q., Varol, O. & Barabási, A.-L. Nature’s reach: narrow work has broad impact. Nature 575, 32–34 (2019).

    Article  CAS  PubMed  Google Scholar 

  96. Börner, K., Penumarthy, S., Meiss, M. & Ke, W. Mapping the diffusion of scholarly knowledge among major US research institutions. Scientometrics 68, 415–426 (2006).

    Article  Google Scholar 

  97. King, D. A. The scientific impact of nations. Nature 430, 311–316 (2004).

    Article  CAS  PubMed  Google Scholar 

  98. Pan, R. K., Kaski, K. & Fortunato, S. World citation and collaboration networks: uncovering the role of geography in science. Sci. Rep. 2, 902 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  99. Jaffe, A. B., Trajtenberg, M. & Henderson, R. Geographic localization of knowledge spillovers as evidenced by patent citations. Q. J. Econ. 108, 577–598 (1993).

    Article  Google Scholar 

  100. Funk, R. J. & Owen-Smith, J. A dynamic network measure of technological change. Manage. Sci. 63, 791–817 (2017).

    Article  Google Scholar 

  101. Yegros-Yegros, A., Rafols, I. & D’este, P. Does interdisciplinary research lead to higher citation impact? The different effect of proximal and distal interdisciplinarity. PLoS ONE 10, e0135095 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  102. Larivière, V., Haustein, S. & Börner, K. Long-distance interdisciplinarity leads to higher scientific impact. PLoS ONE 10, e0122565 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  103. Fleming, L., Greene, H., Li, G., Marx, M. & Yao, D. Government-funded research increasingly fuels innovation. Science 364, 1139–1141 (2019).

    Article  CAS  PubMed  Google Scholar 

  104. Bowen, A. & Casadevall, A. Increasing disparities between resource inputs and outcomes, as measured by certain health deliverables, in biomedical research. Proc. Natl Acad. Sci. USA 112, 11335–11340 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  105. Li, D., Azoulay, P. & Sampat, B. N. The applied value of public investments in biomedical research. Science 356, 78–81 (2017).

    Article  CAS  PubMed  Google Scholar 

  106. Lehman, H. C. Age and Achievement (Princeton Univ. Press, 2017).

  107. Simonton, D. K. Creative productivity: a predictive and explanatory model of career trajectories and landmarks. Psychol. Rev. 104, 66 (1997).

    Article  Google Scholar 

  108. Duch, J. et al. The possible role of resource requirements and academic career-choice risk on gender differences in publication rate and impact. PLoS ONE 7, e51332 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  109. Wang, Y., Jones, B. F. & Wang, D. Early-career setback and future career impact. Nat. Commun. 10, 4331 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  110. Bol, T., de Vaan, M. & van de Rijt, A. The Matthew effect in science funding. Proc. Natl Acad. Sci. USA 115, 4887–4890 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  111. Jones, B. F. Age and great invention. Rev. Econ. Stat. 92, 1–14 (2010).

    Article  Google Scholar 

  112. Newman, M. Networks (Oxford Univ. Press, 2018).

  113. Mazloumian, A., Eom, Y.-H., Helbing, D., Lozano, S. & Fortunato, S. How citation boosts promote scientific paradigm shifts and nobel prizes. PLoS ONE 6, e18975 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  114. Hirsch, J. E. An index to quantify an individual’s scientific research output. Proc. Natl Acad. Sci. USA 102, 16569–16572 (2005).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  115. Alonso, S., Cabrerizo, F. J., Herrera-Viedma, E. & Herrera, F. h-index: a review focused in its variants, computation and standardization for different scientific fields. J. Informetr. 3, 273–289 (2009).

    Article  Google Scholar 

  116. Egghe, L. An improvement of the h-index: the g-index. ISSI Newsl. 2, 8–9 (2006).

    Google Scholar 

  117. Kaur, J., Radicchi, F. & Menczer, F. Universality of scholarly impact metrics. J. Informetr. 7, 924–932 (2013).

    Article  Google Scholar 

  118. Majeti, D. et al. Scholar plot: design and evaluation of an information interface for faculty research performance. Front. Res. Metr. Anal. 4, 6 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  119. Sidiropoulos, A., Katsaros, D. & Manolopoulos, Y. Generalized Hirsch h-index for disclosing latent facts in citation networks. Scientometrics 72, 253–280 (2007).

    Article  CAS  Google Scholar 

  120. Jones, B. F. & Weinberg, B. A. Age dynamics in scientific creativity. Proc. Natl Acad. Sci. USA 108, 18910–18914 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  121. Dennis, W. Age and productivity among scientists. Science 123, 724–725 (1956).

    Article  CAS  PubMed  Google Scholar 

  122. Sanyal, D. K., Bhowmick, P. K. & Das, P. P. A review of author name disambiguation techniques for the PubMed bibliographic database. J. Inf. Sci. 47, 227–254 (2021).

    Article  Google Scholar 

  123. Haak, L. L., Fenner, M., Paglione, L., Pentz, E. & Ratner, H. ORCID: a system to uniquely identify researchers. Learn. Publ. 25, 259–264 (2012).

    Article  Google Scholar 

  124. Malmgren, R. D., Ottino, J. M. & Amaral, L. A. N. The role of mentorship in protégé performance. Nature 465, 662–667 (2010).

    Article  Google Scholar 

  125. Oettl, A. Reconceptualizing stars: scientist helpfulness and peer performance. Manage. Sci. 58, 1122–1140 (2012).

    Article  Google Scholar 

  126. Morgan, A. C. et al. The unequal impact of parenthood in academia. Sci. Adv. 7, eabd1996 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  127. Morgan, A. C. et al. Socioeconomic roots of academic faculty. Nat. Hum. Behav. 6, 1625–1633 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  128. San Francisco Declaration on Research Assessment (DORA) (American Society for Cell Biology, 2012).

  129. Falk‐Krzesinski, H. J. et al. Advancing the science of team science. Clin. Transl. Sci. 3, 263–266 (2010).

    Article  PubMed  PubMed Central  Google Scholar 

  130. Cooke, N. J. et al. Enhancing the Effectiveness of Team Science (National Academies Press, 2015).

  131. Börner, K. et al. A multi-level systems perspective for the science of team science. Sci. Transl. Med. 2, 49cm24 (2010).

    Article  PubMed  PubMed Central  Google Scholar 

  132. Leahey, E. From sole investigator to team scientist: trends in the practice and study of research collaboration. Annu. Rev. Sociol. 42, 81–100 (2016).

    Article  Google Scholar 

  133. AlShebli, B. K., Rahwan, T. & Woon, W. L. The preeminence of ethnic diversity in scientific collaboration. Nat. Commun. 9, 5163 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  134. Hsiehchen, D., Espinoza, M. & Hsieh, A. Multinational teams and diseconomies of scale in collaborative research. Sci. Adv. 1, e1500211 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  135. Koning, R., Samila, S. & Ferguson, J.-P. Who do we invent for? Patents by women focus more on women’s health, but few women get to invent. Science 372, 1345–1348 (2021).

    Article  CAS  PubMed  Google Scholar 

  136. Barabâsi, A.-L. et al. Evolution of the social network of scientific collaborations. Physica A 311, 590–614 (2002).

    Article  Google Scholar 

  137. Newman, M. E. Scientific collaboration networks. I. Network construction and fundamental results. Phys. Rev. E 64, 016131 (2001).

    Article  CAS  Google Scholar 

  138. Newman, M. E. Scientific collaboration networks. II. Shortest paths, weighted networks, and centrality. Phys. Rev. E 64, 016132 (2001).

    Article  CAS  Google Scholar 

  139. Palla, G., Barabási, A.-L. & Vicsek, T. Quantifying social group evolution. Nature 446, 664–667 (2007).

    Article  CAS  PubMed  Google Scholar 

  140. Ross, M. B. et al. Women are credited less in science than men. Nature 608, 135–145 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  141. Shen, H.-W. & Barabási, A.-L. Collective credit allocation in science. Proc. Natl Acad. Sci. USA 111, 12325–12330 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  142. Merton, R. K. Matthew effect in science. Science 159, 56–63 (1968).

    Article  CAS  PubMed  Google Scholar 

  143. Ni, C., Smith, E., Yuan, H., Larivière, V. & Sugimoto, C. R. The gendered nature of authorship. Sci. Adv. 7, eabe4639 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  144. Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N. & Malone, T. W. Evidence for a collective intelligence factor in the performance of human groups. Science 330, 686–688 (2010).

    Article  CAS  PubMed  Google Scholar 

  145. Feldon, D. F. et al. Postdocs’ lab engagement predicts trajectories of PhD students’ skill development. Proc. Natl Acad. Sci. USA 116, 20910–20916 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  146. Boudreau, K. J. et al. A field experiment on search costs and the formation of scientific collaborations. Rev. Econ. Stat. 99, 565–576 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  147. Holcombe, A. O. Contributorship, not authorship: use CRediT to indicate who did what. Publications 7, 48 (2019).

    Article  Google Scholar 

  148. Murray, D. et al. Unsupervised embedding of trajectories captures the latent structure of mobility. Preprint at https://doi.org/10.48550/arXiv.2012.02785 (2020).

  149. Deville, P. et al. Career on the move: geography, stratification, and scientific impact. Sci. Rep. 4, 4770 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  150. Edmunds, L. D. et al. Why do women choose or reject careers in academic medicine? A narrative review of empirical evidence. Lancet 388, 2948–2958 (2016).

    Article  PubMed  Google Scholar 

  151. Waldinger, F. Peer effects in science: evidence from the dismissal of scientists in Nazi Germany. Rev. Econ. Stud. 79, 838–861 (2012).

    Article  Google Scholar 

  152. Agrawal, A., McHale, J. & Oettl, A. How stars matter: recruiting and peer effects in evolutionary biology. Res. Policy 46, 853–867 (2017).

    Article  Google Scholar 

  153. Fiore, S. M. Interdisciplinarity as teamwork: how the science of teams can inform team science. Small Group Res. 39, 251–277 (2008).

    Article  Google Scholar 

  154. Hvide, H. K. & Jones, B. F. University innovation and the professor’s privilege. Am. Econ. Rev. 108, 1860–1898 (2018).

    Article  Google Scholar 

  155. Murray, F., Aghion, P., Dewatripont, M., Kolev, J. & Stern, S. Of mice and academics: examining the effect of openness on innovation. Am. Econ. J. Econ. Policy 8, 212–252 (2016).

    Article  Google Scholar 

  156. Radicchi, F., Fortunato, S. & Castellano, C. Universality of citation distributions: toward an objective measure of scientific impact. Proc. Natl Acad. Sci. USA 105, 17268–17272 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  157. Waltman, L., van Eck, N. J. & van Raan, A. F. Universality of citation distributions revisited. J. Am. Soc. Inf. Sci. Technol. 63, 72–77 (2012).

    Article  CAS  Google Scholar 

  158. Barabási, A.-L. & Albert, R. Emergence of scaling in random networks. Science 286, 509–512 (1999).

    Article  PubMed  Google Scholar 

  159. de Solla Price, D. A general theory of bibliometric and other cumulative advantage processes. J. Am. Soc. Inf. Sci. 27, 292–306 (1976).

    Article  Google Scholar 

  160. Cole, S. Age and scientific performance. Am. J. Sociol. 84, 958–977 (1979).

    Article  Google Scholar 

  161. Ke, Q., Ferrara, E., Radicchi, F. & Flammini, A. Defining and identifying sleeping beauties in science. Proc. Natl Acad. Sci. USA 112, 7426–7431 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  162. Bornmann, L., de Moya Anegón, F. & Leydesdorff, L. Do scientific advancements lean on the shoulders of giants? A bibliometric investigation of the Ortega hypothesis. PLoS ONE 5, e13327 (2010).

    Article  PubMed  PubMed Central  Google Scholar 

  163. Mukherjee, S., Romero, D. M., Jones, B. & Uzzi, B. The nearly universal link between the age of past knowledge and tomorrow’s breakthroughs in science and technology: the hotspot. Sci. Adv. 3, e1601315 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  164. Packalen, M. & Bhattacharya, J. NIH funding and the pursuit of edge science. Proc. Natl Acad. Sci. USA 117, 12011–12016 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  165. Zeng, A., Fan, Y., Di, Z., Wang, Y. & Havlin, S. Fresh teams are associated with original and multidisciplinary research. Nat. Hum. Behav. 5, 1314–1322 (2021).

    Article  PubMed  Google Scholar 

  166. Newman, M. E. The structure of scientific collaboration networks. Proc. Natl Acad. Sci. USA 98, 404–409 (2001).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  167. Larivière, V., Ni, C., Gingras, Y., Cronin, B. & Sugimoto, C. R. Bibliometrics: global gender disparities in science. Nature 504, 211–213 (2013).

    Article  PubMed  Google Scholar 

  168. West, J. D., Jacquet, J., King, M. M., Correll, S. J. & Bergstrom, C. T. The role of gender in scholarly authorship. PLoS ONE 8, e66212 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  169. Gao, J., Yin, Y., Myers, K. R., Lakhani, K. R. & Wang, D. Potentially long-lasting effects of the pandemic on scientists. Nat. Commun. 12, 6188 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  170. Jones, B. F., Wuchty, S. & Uzzi, B. Multi-university research teams: shifting impact, geography, and stratification in science. Science 322, 1259–1262 (2008).

    Article  CAS  PubMed  Google Scholar 

  171. Chu, J. S. & Evans, J. A. Slowed canonical progress in large fields of science. Proc. Natl Acad. Sci. USA 118, e2021636118 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  172. Wang, J., Veugelers, R. & Stephan, P. Bias against novelty in science: a cautionary tale for users of bibliometric indicators. Res. Policy 46, 1416–1436 (2017).

    Article  Google Scholar 

  173. Stringer, M. J., Sales-Pardo, M. & Amaral, L. A. Statistical validation of a global model for the distribution of the ultimate number of citations accrued by papers published in a scientific journal. J. Assoc. Inf. Sci. Technol. 61, 1377–1385 (2010).

    Article  Google Scholar 

  174. Bianconi, G. & Barabási, A.-L. Bose-Einstein condensation in complex networks. Phys. Rev. Lett. 86, 5632 (2001).

    Article  CAS  PubMed  Google Scholar 

  175. Bianconi, G. & Barabási, A.-L. Competition and multiscaling in evolving networks. Europhys. Lett. 54, 436 (2001).

    Article  CAS  Google Scholar 

  176. Yin, Y. & Wang, D. The time dimension of science: connecting the past to the future. J. Informetr. 11, 608–621 (2017).

    Article  Google Scholar 

  177. Pan, R. K., Petersen, A. M., Pammolli, F. & Fortunato, S. The memory of science: Inflation, myopia, and the knowledge network. J. Informetr. 12, 656–678 (2018).

    Article  Google Scholar 

  178. Yin, Y., Wang, Y., Evans, J. A. & Wang, D. Quantifying the dynamics of failure across science, startups and security. Nature 575, 190–194 (2019).

    Article  CAS  PubMed  Google Scholar 

  179. Candia, C. & Uzzi, B. Quantifying the selective forgetting and integration of ideas in science and technology. Am. Psychol. 76, 1067 (2021).

    Article  PubMed  Google Scholar 

  180. Milojević, S. Principles of scientific research team formation and evolution. Proc. Natl Acad. Sci. USA 111, 3984–3989 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  181. Guimera, R., Uzzi, B., Spiro, J. & Amaral, L. A. N. Team assembly mechanisms determine collaboration network structure and team performance. Science 308, 697–702 (2005).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  182. Newman, M. E. Coauthorship networks and patterns of scientific collaboration. Proc. Natl Acad. Sci. USA 101, 5200–5205 (2004).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  183. Newman, M. E. Clustering and preferential attachment in growing networks. Phys. Rev. E 64, 025102 (2001).

    Article  CAS  Google Scholar 

  184. Iacopini, I., Milojević, S. & Latora, V. Network dynamics of innovation processes. Phys. Rev. Lett. 120, 048301 (2018).

    Article  CAS  PubMed  Google Scholar 

  185. Kuhn, T., Perc, M. & Helbing, D. Inheritance patterns in citation networks reveal scientific memes. Phys. Rev. 4, 041036 (2014).

    Article  Google Scholar 

  186. Jia, T., Wang, D. & Szymanski, B. K. Quantifying patterns of research-interest evolution. Nat. Hum. Behav. 1, 0078 (2017).

    Article  Google Scholar 

  187. Zeng, A. et al. Increasing trend of scientists to switch between topics. Nat. Commun. https://doi.org/10.1038/s41467-019-11401-8 (2019).

  188. Siudem, G., Żogała-Siudem, B., Cena, A. & Gagolewski, M. Three dimensions of scientific impact. Proc. Natl Acad. Sci. USA 117, 13896–13900 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  189. Petersen, A. M. et al. Reputation and impact in academic careers. Proc. Natl Acad. Sci. USA 111, 15316–15321 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  190. Jin, C., Song, C., Bjelland, J., Canright, G. & Wang, D. Emergence of scaling in complex substitutive systems. Nat. Hum. Behav. 3, 837–846 (2019).

    Article  PubMed  Google Scholar 

  191. Hofman, J. M. et al. Integrating explanation and prediction in computational social science. Nature 595, 181–188 (2021).

    Article  CAS  PubMed  Google Scholar 

  192. Lazer, D. et al. Computational social science. Science 323, 721–723 (2009).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  193. Lazer, D. M. et al. Computational social science: obstacles and opportunities. Science 369, 1060–1062 (2020).

    Article  CAS  PubMed  Google Scholar 

  194. Albert, R. & Barabási, A.-L. Statistical mechanics of complex networks. Rev. Mod. Phys. 74, 47 (2002).

    Article  Google Scholar 

  195. Newman, M. E. The structure and function of complex networks. SIAM Rev. 45, 167–256 (2003).

    Article  Google Scholar 

  196. Song, C., Qu, Z., Blumm, N. & Barabási, A.-L. Limits of predictability in human mobility. Science 327, 1018–1021 (2010).

    Article  CAS  PubMed  Google Scholar 

  197. Alessandretti, L., Aslak, U. & Lehmann, S. The scales of human mobility. Nature 587, 402–407 (2020).

    Article  CAS  PubMed  Google Scholar 

  198. Pastor-Satorras, R. & Vespignani, A. Epidemic spreading in scale-free networks. Phys. Rev. Lett. 86, 3200 (2001).

    Article  CAS  PubMed  Google Scholar 

  199. Pastor-Satorras, R., Castellano, C., Van Mieghem, P. & Vespignani, A. Epidemic processes in complex networks. Rev. Mod. Phys. 87, 925 (2015).

    Article  Google Scholar 

  200. Goodfellow, I., Bengio, Y. & Courville, A. Deep Learning (MIT Press, 2016).

  201. Bishop, C. M. Pattern Recognition and Machine Learning (Springer, 2006).

  202. Dong, Y., Johnson, R. A. & Chawla, N. V. Will this paper increase your h-index? Scientific impact prediction. In Proc. 8th ACM International Conference on Web Search and Data Mining, 149–158 (ACM 2015)

  203. Xiao, S. et al. On modeling and predicting individual paper citation count over time. In IJCAI, 2676–2682 (IJCAI, 2016)

  204. Fortunato, S. Community detection in graphs. Phys. Rep. 486, 75–174 (2010).

    Article  Google Scholar 

  205. Chen, C. Science mapping: a systematic review of the literature. J. Data Inf. Sci. 2, 1–40 (2017).

    CAS  Google Scholar 

  206. Van Eck, N. J. & Waltman, L. Citation-based clustering of publications using CitNetExplorer and VOSviewer. Scientometrics 111, 1053–1070 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  207. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    Article  CAS  PubMed  Google Scholar 

  208. Senior, A. W. et al. Improved protein structure prediction using potentials from deep learning. Nature 577, 706–710 (2020).

    Article  CAS  PubMed  Google Scholar 

  209. Krenn, M. & Zeilinger, A. Predicting research trends with semantic and neural networks with an application in quantum physics. Proc. Natl Acad. Sci. USA 117, 1910–1916 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  210. Iten, R., Metger, T., Wilming, H., Del Rio, L. & Renner, R. Discovering physical concepts with neural networks. Phys. Rev. Lett. 124, 010508 (2020).

    Article  CAS  PubMed  Google Scholar 

  211. Guimerà, R. et al. A Bayesian machine scientist to aid in the solution of challenging scientific problems. Sci. Adv. 6, eaav6971 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  212. Segler, M. H., Preuss, M. & Waller, M. P. Planning chemical syntheses with deep neural networks and symbolic AI. Nature 555, 604–610 (2018).

    Article  CAS  PubMed  Google Scholar 

  213. Ryu, J. Y., Kim, H. U. & Lee, S. Y. Deep learning improves prediction of drug–drug and drug–food interactions. Proc. Natl Acad. Sci. USA 115, E4304–E4311 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  214. Kermany, D. S. et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172, 1122–1131.e9 (2018).

    Article  CAS  PubMed  Google Scholar 

  215. Peng, H., Ke, Q., Budak, C., Romero, D. M. & Ahn, Y.-Y. Neural embeddings of scholarly periodicals reveal complex disciplinary organizations. Sci. Adv. 7, eabb9004 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  216. Youyou, W., Yang, Y. & Uzzi, B. A discipline-wide investigation of the replicability of psychology papers over the past two decades. Proc. Natl Acad. Sci. USA 120, e2208863120 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  217. Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K. & Galstyan, A. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) 54, 1–35 (2021).

    Article  Google Scholar 

  218. Way, S. F., Morgan, A. C., Larremore, D. B. & Clauset, A. Productivity, prominence, and the effects of academic environment. Proc. Natl Acad. Sci. USA 116, 10729–10733 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  219. Li, W., Aste, T., Caccioli, F. & Livan, G. Early coauthorship with top scientists predicts success in academic careers. Nat. Commun. 10, 5170 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  220. Hendry, D. F., Pagan, A. R. & Sargan, J. D. Dynamic specification. Handb. Econ. 2, 1023–1100 (1984).

    Google Scholar 

  221. Jin, C., Ma, Y. & Uzzi, B. Scientific prizes and the extraordinary growth of scientific topics. Nat. Commun. 12, 5619 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  222. Azoulay, P., Ganguli, I. & Zivin, J. G. The mobility of elite life scientists: professional and personal determinants. Res. Policy 46, 573–590 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  223. Slavova, K., Fosfuri, A. & De Castro, J. O. Learning by hiring: the effects of scientists’ inbound mobility on research performance in academia. Organ. Sci. 27, 72–89 (2016).

    Article  Google Scholar 

  224. Sarsons, H. Recognition for group work: gender differences in academia. Am. Econ. Rev. 107, 141–145 (2017).

    Article  Google Scholar 

  225. Campbell, L. G., Mehtani, S., Dozier, M. E. & Rinehart, J. Gender-heterogeneous working groups produce higher quality science. PLoS ONE 8, e79147 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  226. Azoulay, P., Graff Zivin, J. S. & Wang, J. Superstar extinction. Q. J. Econ. 125, 549–589 (2010).

    Article  Google Scholar 

  227. Furman, J. L. & Stern, S. Climbing atop the shoulders of giants: the impact of institutions on cumulative research. Am. Econ. Rev. 101, 1933–1963 (2011).

    Article  Google Scholar 

  228. Williams, H. L. Intellectual property rights and innovation: evidence from the human genome. J. Polit. Econ. 121, 1–27 (2013).

    Article  Google Scholar 

  229. Rubin, A. & Rubin, E. Systematic Bias in the Progress of Research. J. Polit. Econ. 129, 2666–2719 (2021).

    Article  Google Scholar 

  230. Lu, S. F., Jin, G. Z., Uzzi, B. & Jones, B. The retraction penalty: evidence from the Web of Science. Sci. Rep. 3, 3146 (2013).

    Article  PubMed  PubMed Central  Google Scholar 

  231. Jin, G. Z., Jones, B., Lu, S. F. & Uzzi, B. The reverse Matthew effect: consequences of retraction in scientific teams. Rev. Econ. Stat. 101, 492–506 (2019).

    Article  Google Scholar 

  232. Azoulay, P., Bonatti, A. & Krieger, J. L. The career effects of scandal: evidence from scientific retractions. Res. Policy 46, 1552–1569 (2017).

    Article  Google Scholar 

  233. Goodman-Bacon, A. Difference-in-differences with variation in treatment timing. J. Econ. 225, 254–277 (2021).

    Article  Google Scholar 

  234. Callaway, B. & Sant’Anna, P. H. Difference-in-differences with multiple time periods. J. Econ. 225, 200–230 (2021).

    Article  Google Scholar 

  235. Hill, R. Searching for Superstars: Research Risk and Talent Discovery in Astronomy Working Paper (Massachusetts Institute of Technology, 2019).

  236. Bagues, M., Sylos-Labini, M. & Zinovyeva, N. Does the gender composition of scientific committees matter? Am. Econ. Rev. 107, 1207–1238 (2017).

    Article  Google Scholar 

  237. Sampat, B. & Williams, H. L. How do patents affect follow-on innovation? Evidence from the human genome. Am. Econ. Rev. 109, 203–236 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  238. Moretti, E. & Wilson, D. J. The effect of state taxes on the geographical location of top earners: evidence from star scientists. Am. Econ. Rev. 107, 1858–1903 (2017).

    Article  Google Scholar 

  239. Jacob, B. A. & Lefgren, L. The impact of research grant funding on scientific productivity. J. Public Econ. 95, 1168–1177 (2011).

    Article  PubMed  PubMed Central  Google Scholar 

  240. Li, D. Expertise versus bias in evaluation: evidence from the NIH. Am. Econ. J. Appl. Econ. 9, 60–92 (2017).

    Article  Google Scholar 

  241. Wang, Y., Jones, B. F. & Wang, D. Early-career setback and future career impact. Nat. Commun. 10, 4331 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  242. Pearl, J. Causal diagrams for empirical research. Biometrika 82, 669–688 (1995).

    Article  Google Scholar 

  243. Pearl, J. & Mackenzie, D. The Book of Why: The New Science of Cause and Effect (Basic Books, 2018).

  244. Traag, V. A. Inferring the causal effect of journals on citations. Quant. Sci. Stud. 2, 496–504 (2021).

    Article  Google Scholar 

  245. Traag, V. & Waltman, L. Causal foundations of bias, disparity and fairness. Preprint at https://doi.org/10.48550/arXiv.2207.13665 (2022).

  246. Imbens, G. W. Potential outcome and directed acyclic graph approaches to causality: relevance for empirical practice in economics. J. Econ. Lit. 58, 1129–1179 (2020).

    Article  Google Scholar 

  247. Heckman, J. J. & Pinto, R. Causality and Econometrics (National Bureau of Economic Research, 2022).

  248. Aggarwal, I., Woolley, A. W., Chabris, C. F. & Malone, T. W. The impact of cognitive style diversity on implicit learning in teams. Front. Psychol. 10, 112 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  249. Balietti, S., Goldstone, R. L. & Helbing, D. Peer review and competition in the Art Exhibition Game. Proc. Natl Acad. Sci. USA 113, 8414–8419 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  250. Paulus, F. M., Rademacher, L., Schäfer, T. A. J., Müller-Pinzler, L. & Krach, S. Journal impact factor shapes scientists’ reward signal in the prospect of publication. PLoS ONE 10, e0142537 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  251. Williams, W. M. & Ceci, S. J. National hiring experiments reveal 2:1 faculty preference for women on STEM tenure track. Proc. Natl Acad. Sci. USA 112, 5360–5365 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  252. Collaboration, O. S. Estimating the reproducibility of psychological science. Science 349, aac4716 (2015).

    Article  Google Scholar 

  253. Camerer, C. F. et al. Evaluating replicability of laboratory experiments in economics. Science 351, 1433–1436 (2016).

    Article  CAS  PubMed  Google Scholar 

  254. Camerer, C. F. et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat. Hum. Behav. 2, 637–644 (2018).

    Article  PubMed  Google Scholar 

  255. Duflo, E. & Banerjee, A. Handbook of Field Experiments (Elsevier, 2017).

  256. Tomkins, A., Zhang, M. & Heavlin, W. D. Reviewer bias in single versus double-blind peer review. Proc. Natl Acad. Sci. USA 114, 12708–12713 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  257. Blank, R. M. The effects of double-blind versus single-blind reviewing: experimental evidence from the American Economic Review. Am. Econ. Rev. 81, 1041–1067 (1991).

    Google Scholar 

  258. Boudreau, K. J., Guinan, E. C., Lakhani, K. R. & Riedl, C. Looking across and looking beyond the knowledge frontier: intellectual distance, novelty, and resource allocation in science. Manage. Sci. 62, 2765–2783 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  259. Lane, J. et al. When Do Experts Listen to Other Experts? The Role of Negative Information in Expert Evaluations for Novel Projects Working Paper #21-007 (Harvard Business School, 2020).

  260. Teplitskiy, M. et al. Do Experts Listen to Other Experts? Field Experimental Evidence from Scientific Peer Review (Harvard Business School, 2019).

  261. Moss-Racusin, C. A., Dovidio, J. F., Brescoll, V. L., Graham, M. J. & Handelsman, J. Science faculty’s subtle gender biases favor male students. Proc. Natl Acad. Sci. USA 109, 16474–16479 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  262. Forscher, P. S., Cox, W. T., Brauer, M. & Devine, P. G. Little race or gender bias in an experiment of initial review of NIH R01 grant proposals. Nat. Hum. Behav. 3, 257–264 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  263. Dennehy, T. C. & Dasgupta, N. Female peer mentors early in college increase women’s positive academic experiences and retention in engineering. Proc. Natl Acad. Sci. USA 114, 5964–5969 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  264. Azoulay, P. Turn the scientific method on ourselves. Nature 484, 31–32 (2012).

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgements

The authors thank all members of the Center for Science of Science and Innovation (CSSI) for invaluable comments. This work was supported by the Air Force Office of Scientific Research under award number FA9550-19-1-0354, National Science Foundation grant SBE 1829344, and the Alfred P. Sloan Foundation G-2019-12485.

Author information

Authors and Affiliations

  1. Center for Science of Science and Innovation, Northwestern University, Evanston, IL, USA

    Lu Liu, Benjamin F. Jones, Brian Uzzi & Dashun Wang

  2. Northwestern Institute on Complex Systems, Northwestern University, Evanston, IL, USA

    Lu Liu, Benjamin F. Jones, Brian Uzzi & Dashun Wang

  3. Kellogg School of Management, Northwestern University, Evanston, IL, USA

    Lu Liu, Benjamin F. Jones, Brian Uzzi & Dashun Wang

  4. College of Information Sciences and Technology, Pennsylvania State University, University Park, PA, USA

    Lu Liu

  5. National Bureau of Economic Research, Cambridge, MA, USA

    Benjamin F. Jones

  6. Brookings Institution, Washington, DC, USA

    Benjamin F. Jones

  7. McCormick School of Engineering, Northwestern University, Evanston, IL, USA

    Dashun Wang

Authors
  1. Lu Liu
    View author publications

    Search author on:PubMed Google Scholar

  2. Benjamin F. Jones
    View author publications

    Search author on:PubMed Google Scholar

  3. Brian Uzzi
    View author publications

    Search author on:PubMed Google Scholar

  4. Dashun Wang
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Dashun Wang.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Human Behaviour thanks Ludo Waltman, Erin Leahey and Sarah Bratt for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, L., Jones, B.F., Uzzi, B. et al. Data, measurement and empirical methods in the science of science. Nat Hum Behav 7, 1046–1058 (2023). https://doi.org/10.1038/s41562-023-01562-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Version of record:

  • Issue date:

  • DOI: https://doi.org/10.1038/s41562-023-01562-4

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing