Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Reproducibility in automated chemistry laboratories using computer science abstractions

Abstract

While abstraction is critical for the transferability of automated laboratory science in (bio)chemical and materials sciences, its improper implementation is a technical debt taken against the reproducibility of experimental results. Over the decades, computer science has developed guidelines and strategies for how abstractions are captured in programming languages—particularly concerning the substitutability of implementations of abstracted ideas and the clear definition of the contexts in which abstractions are used. However, few programming languages developed for automated experiments fully leverage the wisdom learned in computer science. To achieve collaborative sharing of scientific knowledge via automated laboratories, the way that experimental protocols are codified and interpreted by machine agents must use abstractions responsibly and with reproducibility, rather than solely transferability, at its core. This Review discusses how computer science principles of abstraction can be translated to create more reproducible automation as an enabler for the acceleration of collaborative research with self-driving laboratories.

This is a preview of subscription content, access via your institution

Access options

Buy this article

USD 39.95

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Comparisons of the repeatability and reproducibility of experimental results.
Fig. 2: Various implementations of the same abstraction leading to different results.
Fig. 3: Disparity between abstractly equivalent but hardware-dependent implementations for a Java sorting algorithm.
Fig. 4: Visualizations of the approaches described.
Fig. 5: Example excerpts of different languages for prescribing automated experiments.

Similar content being viewed by others

References

  1. Autoprotocol (Strateos Inc., 2021); https://autoprotocol.org/

  2. Pendleton, I. M. et al. Experiment Specification, Capture and Laboratory Automation Technology (ESCALATE): a software pipeline for automated chemical experimentation and data management. MRS Commun. 9, 846–859 (2019).

    Article  CAS  Google Scholar 

  3. Leonov, A. I. et al. An integrated self-optimizing programmable chemical synthesis and reaction engine. Nat. Commun. 15, 1240 (2024).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Documentation center. Emerald Cloud Lab https://www.emeraldcloudlab.com/documentation/ (Emerald Cloud Lab, Inc, 2023).

  5. Boiko, D. A., MacKnight, R., Kline, B. & Gomes, G. Autonomous chemical research with large language models. Nature 624, 570–578 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  6. Schäfer, B. A., Poetz, D. & Kramer, G. W. Documenting laboratory workflows using the Analytical Information Markup Language. JALA 9, 375–381 (2004).

    Google Scholar 

  7. Li, J., Tu, Y., Liu, R., Lu, Y. & Zhu, X. Toward “on-demand” materials synthesis and scientific discovery through intelligent robots. Adv. Sci. 7, 1901957 (2020).

    Article  CAS  Google Scholar 

  8. LabVIEW. LabVIEW Wiki (National Instruments, 2024) https://labviewwiki.org/wiki/LabVIEW

  9. Bartley, B. et al. Building an open representation for biological protocols. ACM J. Emerg. Technol. Comput. Syst. 19, 1–21 (2023).

    Article  Google Scholar 

  10. Ananthanarayanan, V. & Thies, W. Biocoder: a programming language for standardizing and automating biology protocols. J. Biol. Eng. 4, 13 (2010).

    Article  PubMed  PubMed Central  Google Scholar 

  11. Gupta, V., Irimia, J., Pau, I. & Rodríguez-Patón, A. BioBlocks: programming protocols in biology made easier. ACS Synth. Biol. 6, 1230–1232 (2017).

    Article  CAS  PubMed  Google Scholar 

  12. Wierenga, R. P., Golas, S. M., Ho, W., Coley, C. W. & Esvelt, K. M. PyLabRobot: an open-source, hardware-agnostic interface for liquid-handling robots and accessories. Device 1, 100111 (2023).

    Article  Google Scholar 

  13. Rauschen, R., Guy, M., Hein, J. E. & Cronin, L. Universal chemical programming language for robotic synthesis repeatability. Nat. Synth. 3, 488–496 (2024).

    Article  Google Scholar 

  14. Leong, C. J. et al. An object-oriented framework to enable workflow evolution across materials acceleration platforms. Matter 5, 3124–3134 (2022).

    Article  Google Scholar 

  15. Canty, R. B., Koscher, B. A., McDonald, M. A. & Jensen, K. F. Integrating autonomy into automated research platforms. Digit. Discov. 2, 1259–1268 (2023).

    Article  Google Scholar 

  16. Alexandron, G., Armoni, M., Gordon, M. & Harel, D. Scenario-based programming: reducing the cognitive load, fostering abstract thinking. In Companion Proc. 36th International Conference on Software Engineering 311–320 (Association for Computing Machinery, 2014); https://doi.org/10.1145/2591062.2591167

  17. Fidler, F. & Wilcox, J. in The Stanford Encyclopedia of Philosophy (ed. Zalta, E. N.) (Metaphysics Research Lab, Stanford University, 2021).

  18. Mandel, J. Repeatability and reproducibility. J. Qual. Technol. 4, 74–85 (1972).

    Article  Google Scholar 

  19. Feitelson, D. G. From repeatability to reproducibility and corroboration. ACM SIGOPS Oper. Syst. Rev. 49, 3–11 (2015).

    Article  Google Scholar 

  20. Pijper, B. et al. Addressing reproducibility challenges in high-throughput photochemistry. JACS Au https://doi.org/10.1021/jacsau.4c00312 (2024).

    Article  PubMed  PubMed Central  Google Scholar 

  21. Epps, R. W. et al. Artificial chemist: an autonomous quantum dot synthesis bot. Adv. Mater. 32, 2001626 (2020).

    Article  CAS  Google Scholar 

  22. Bateni, F. et al. Smart Dope: a self-driving fluidic lab for accelerated development of doped perovskite quantum dots. Adv. Energy Mater. 14, 2302303 (2024).

    Article  CAS  Google Scholar 

  23. Leeman, J. et al. Challenges in high-throughput inorganic materials prediction and autonomous synthesis. PRX Energy 3, 011002 (2024).

    Article  Google Scholar 

  24. Sayre, F. & Riegelman, A. The reproducibility crisis and academic libraries. Coll. Res. Libr. https://doi.org/10.5860/crl.79.1.2 (2018).

    Article  Google Scholar 

  25. Leins, D. A., Haase, S. B., Eslami, M., Schrier, J. & Freeman, J. T. Collaborative methods to enhance reproducibility and accelerate discovery. Digit. Discov. 2, 12–27 (2023).

    Article  Google Scholar 

  26. Liskov, B. & Zilles, S. Programming with abstract data types. ACM SIGPLAN Not. 9, 50–59 (1974).

    Article  Google Scholar 

  27. Parnas, D. L. On the criteria to be used in decomposing systems into modules. Commun. ACM 15, 1053–1058 (1972).

    Article  Google Scholar 

  28. Parnas, D. L., Shore, J. E. & Weiss, D. Abstract types defined as classes of variables. ACM SIGPLAN Not. 11, 149–154 (1976).

    Article  Google Scholar 

  29. Meyer, B. Applying ‘design by contract’. Computer 25, 40–51 (1992).

    Article  Google Scholar 

  30. Stevens, W. P., Myers, G. J. & Constantine, L. L. Structured design. IBM Syst. J. 13, 115–139 (1974).

    Article  Google Scholar 

  31. Taube-Schock, C., Walker, R. J. & Witten, I. H. Can we avoid high coupling? In ECOOP 2011 – Object-Oriented Programming (ed. Mezini, M.) 204–228 (Springer, 2011); https://doi.org/10.1007/978-3-642-22655-7_10

  32. Christensen, M. et al. Automation isn’t automatic. Chem. Sci. 12, 15473–15490 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  33. Koscher, B. A. et al. Autonomous, multiproperty-driven molecular discovery: from predictions to measurements and back. Science 382, eadi1407 (2023).

    Article  CAS  PubMed  Google Scholar 

  34. Karafiludis, S., Ryll, T. W., Buzanich, A. G., Emmerling, F. & Stawski, T. M. Phase stability studies on transition metal phosphates aided by an automated synthesis. CrystEngComm 25, 4333–4344 (2023).

    Article  CAS  Google Scholar 

  35. D’Agostino, C. et al. Understanding the solvent effect on the catalytic oxidation of 1,4-butanediol in methanol over Au/TiO2 catalyst: NMR diffusion and relaxation studies. Chem. Eur. J. 18, 14426–14433 (2012).

    Article  PubMed  Google Scholar 

  36. Pomberger, A. et al. Automated pH adjustment driven by robotic workflows and active machine learning. Chem. Eng. J. 451, 139099 (2023).

    Article  CAS  Google Scholar 

  37. Nian, R., Liu, J. & Huang, B. A review on reinforcement learning: introduction and applications in industrial process control. Comput. Chem. Eng. 139, 106886 (2020).

    Article  CAS  Google Scholar 

  38. Maffettone, P. M. et al. Gaming the beamlines—employing reinforcement learning to maximize scientific outcomes at large-scale user facilities. Mach. Learn. Sci. Technol. 2, 025025 (2021).

    Article  Google Scholar 

  39. Martens, J. A., Perez-Pariente, J., Sastre, E., Corma, A. & Jacobs, P. A. Isomerization and disproportionation of m-xylene: selectivities induced by the void structure of the zeolite framework. Appl. Catal. 45, 85–101 (1988).

    Article  CAS  Google Scholar 

  40. Molyneux, S. & Goss, R. J. M. Fully aqueous and air-compatible cross-coupling of primary alkyl halides with aryl boronic species: a possible and facile method. ACS Catal. 13, 6365–6374 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  41. Jessop-Fabre, M. M. & Sonnenschein, N. Improving reproducibility in synthetic biology. Front. Bioeng. Biotechnol. 7, 18 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  42. Bai, J. et al. From platform to knowledge graph: evolution of laboratory automation. JACS Au 2, 292–309 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  43. Liskov, B. H. & Wing, J. M. A behavioral notion of subtyping. ACM Trans. Program. Lang. Syst. 16, 1811–1841 (1994).

    Article  Google Scholar 

  44. Hähnle, R., Kamburjan, E. & Scaletta, M. in Active Object Languages: Current Research Trends (eds de Boer, F. et al.) 289–322 (Springer, 2024); https://doi.org/10.1007/978-3-031-51060-1_11

  45. Giordano, G. et al. On the adoption and effects of source code reuse on defect proneness and maintenance effort. Empir. Softw. Eng. 29, 20 (2023).

    Article  Google Scholar 

  46. Bloch, J. Extra, extra – read all about it: nearly all binary searches and mergesorts are broken. Google Research https://blog.research.google/2006/06/extra-extra-read-all-about-it-nearly.html (2006).

  47. Hohpe, G. & Woolf, B. Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions (Addison-Wesley, 2003).

  48. Melloul, L. & Fox, A. Reusable functional composition patterns for Web services. In Proc. IEEE International Conference on Web Services (IEEE, 2004); https://ieeexplore.ieee.org/abstract/document/1314775

  49. Tkaczyk, R. et al. Cataloging design patterns for internet of things artifact integration. In 2018 IEEE International Conference on Communications Workshops (ICC Workshops) 1–6 (IEEE, 2018); https://doi.org/10.1109/ICCW.2018.8403758

  50. Ramadas, A., Domingues, G., Dias, J. P., Aguiar, A. & Ferreira, H. S. Patterns for things that fail. In Proc. 24th Conference on Pattern Languages of Programs 1–10 (The Hillside Group, 2017).

  51. Nesnas, I. A. D. in Software Engineering for Experimental Robotics (ed. Brugali, D.) 31–70 (Springer, 2007); https://doi.org/10.1007/978-3-540-68951-5_3

  52. Rees-Hill, J. A. Error Handling Approaches in Programming Languages (Oberlin College, 2022).

  53. Erwig, M. & Ren, D. Monadification of functional programs. Sci. Comput. Program. 52, 101–129 (2004).

    Article  Google Scholar 

  54. Salvaneschi, G., Ghezzi, C. & Pradella, M. Context-oriented programming: a software engineering perspective. J. Syst. Softw. 85, 1801–1817 (2012).

    Article  Google Scholar 

  55. Cardozo, N. & Mens, K. Programming language implementations for context-oriented self-adaptive systems. Inf. Softw. Technol. 143, 106789 (2022).

    Article  Google Scholar 

  56. Carbin, M., Misailovic, S. & Rinard, M. C. Verifying quantitative reliability for programs that execute on unreliable hardware. In Proc. 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages & Applications 33–52 (Association for Computing Machinery, 2013); https://doi.org/10.1145/2509136.2509546

  57. Armstrong, J. Making Reliable Distributed Systems in the Presence of Software Errors. PhD thesis, Royal Institute of Technology, Stockholm (2003).

  58. The Python Language Reference (Python Software Foundation, 2024); http://python.org

  59. Fakhruldeen, H., Pizzuto, G., Glowacki, J. & Cooper, A. I. ARChemist: Autonomous Robotic Chemistry system architecture. In Proc. 2022 International Conference on Robotics and Automation (ICRA) 6013–6019 (IEEE, 2022); https://doi.org/10.1109/ICRA46639.2022.9811996

  60. Darvish, K. et al. ORGANA: a robotic assistant for automated chemistry experimentation and characterization. Preprint at https://arxiv.org/abs/2401.06949 (2024).

  61. Higgins, S. G., Nogiwa-Valdez, A. A. & Stevens, M. M. Considerations for implementing electronic laboratory notebooks in an academic research environment. Nat. Protoc. 17, 179–189 (2022).

    Article  CAS  PubMed  Google Scholar 

  62. Statt, M. J. et al. ESAMP: event-sourced architecture for materials provenance management and application to accelerated materials discovery. Digit. Discov. 2, 1078–1088 (2023).

    Article  CAS  Google Scholar 

  63. Duke, R., McCoy, R., Risko, C. & Bursten, J. R. S. Promises and perils of big data: philosophical constraints on chemical ontologies. J. Am. Chem. Soc. https://doi.org/10.1021/jacs.3c11399 (2024).

    Article  PubMed  Google Scholar 

  64. Statt, M. J., Rohr, B. A., Guevarra, D., Suram, S. K. & Gregoire, J. M. Event-driven data management with cloud computing for extensible materials acceleration platforms. Digit. Discov. 3, 238–242 (2024).

    Article  Google Scholar 

  65. Jung, E., Cho, I. & Kang, S. M. An agent modeling for overcoming the heterogeneity in the IoT with design patterns. In Mobile, Ubiquitous, and Intelligent Computing: MUSIC 2013 (eds Park, J. J. et al.) 69–74 (Springer, 2014); https://doi.org/10.1007/978-3-642-40675-1_11

  66. Green, D. V. S. et al. BRADSHAW: a system for automated molecular design. J. Comput. Aided Mol. Des. 34, 747–765 (2020).

    Article  CAS  PubMed  Google Scholar 

  67. Walsh, D. J. et al. Community Resource for Innovation in Polymer Technology (CRIPT): a scalable polymer material data structure. ACS Cent. Sci. 9, 330–338 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  68. XDL Documentation (Cronin Group, University of Glasgow, 2022); https://croningroup.gitlab.io/chemputer/xdl/

  69. John, W. et al. The future of cloud computing: highly distributed with heterogeneous hardware. Ericsson Technology Review (12 May 2020).

  70. Carbin, M. & Misailovic, S. in Foundations of Probabilistic Programming (eds Silva, A. et al.) 533–568 (Cambridge Univ. Press, 2020); https://doi.org/10.1017/9781108770750.016

  71. Craven, M., Keenan, G., Khan, A., Lee M. & Wilbraham L. ChemIDE (Cronin Group, University of Glasgow, 2021); https://croningroup.gitlab.io/chemputer/xdlapp/

  72. Delgado-Licona, F. & Abolhasani, M. Research acceleration in self-driving labs: technological roadmap toward accelerated materials and molecular discovery. Adv. Intell. Syst. 5, 2200331 (2023).

    Article  Google Scholar 

  73. Bennett, J. A. et al. Autonomous reaction Pareto-front mapping with a self-driving catalysis laboratory. Nat. Chem. Eng. 1, 240–250 (2024).

    Article  Google Scholar 

  74. Fitzpatrick, D. E., Battilocchio, C. & Ley, S. V. A novel internet-based reaction monitoring, control and autonomous self-optimization platform for chemical synthesis. Org. Process Res. Dev. 20, 386–394 (2016).

    Article  CAS  Google Scholar 

  75. Li, J. et al. Autonomous discovery of optically active chiral inorganic perovskite nanocrystals through an intelligent cloud lab. Nat. Commun. 11, 2046 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  76. Gromski, P. S., Granda, J. M. & Cronin, L. Universal chemical synthesis and discovery with ‘the Chemputer’. Trends Chem. 2, 4–12 (2020).

    Article  CAS  Google Scholar 

  77. Rahmanian, F. et al. Enabling modular autonomous feedback-loops in materials science through hierarchical experimental laboratory automation and orchestration. Adv. Mater. Interfaces 9, 2101987 (2022).

    Article  Google Scholar 

  78. Sim, M. et al. ChemOS 2.0: an orchestration architecture for chemical self-driving laboratories. Matter https://doi.org/10.1016/j.matt.2024.04.022 (2024).

    Article  Google Scholar 

  79. Ramírez, S. FastAPI (MIT, 2018); https://fastapi.tiangolo.com/

  80. Consortium for Standardization in Lab Automation Standards. SiLA Rapid Integration https://sila-standard.com/standards/ (SILA, 2017).

  81. Zhang, L., Merrifield, R., Deguet, A. & Yang, G.-Z. Powering the world’s robots—10 years of ROS. Sci. Robot. 2, eaar1868 (2017).

    Article  PubMed  Google Scholar 

  82. Munroe, R. Standards. xkcd (2011); https://xkcd.com/927/

  83. Volk, A. A. & Abolhasani, M. Performance metrics to unleash the power of self-driving labs in chemistry and materials science. Nat. Commun. 15, 1378 (2024).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  84. Volk, A. A. et al. AlphaFlow: autonomous discovery and optimization of multi-step chemistry using a self-driven fluidic lab guided by reinforcement learning. Nat. Commun. 14, 1403 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  85. Snapp, K. L. & Brown, K. A. Driving school for self-driving labs. Digit. Discov. 2, 1620–1629 (2023).

    Article  CAS  Google Scholar 

  86. Abolhasani, M. & Kumacheva, E. The rise of self-driving labs in chemical and materials sciences. Nat. Synth. 2, 483–492 (2023).

    Article  Google Scholar 

  87. Epps, R. W., Volk, A. A., Ibrahim, M. Y. S. & Abolhasani, M. Universal self-driving laboratory for accelerated discovery of materials and molecules. Chem 7, 2541–2545 (2021).

    Article  CAS  Google Scholar 

  88. Yang, Z. et al. Exploring and unleashing the power of large language models in automated code translation. In Proc. ACM on Software Engineering 585–1608 (ACM, 2024).

  89. Bran, A. M. et al. Augmenting large language models with chemistry tools. Nat. Mach. Intell. 6, 525–535 (2024).

    Article  Google Scholar 

  90. Yoshikawa, N. et al. Large language models for chemistry robotics. Auton. Robot. 47, 1057–1086 (2023).

    Article  Google Scholar 

  91. Lunt, A. M. et al. Modular, multi-robot integration of laboratories: an autonomous workflow for solid-state chemistry. Chem. Sci. 15, 2456–2463 (2024).

    Article  CAS  PubMed  Google Scholar 

  92. Lunt, A. sgalunt/Thesis_Amy_Lunt (GitHub, 2023); https://github.com/sgalunt/Thesis_Amy_Lunt/blob/main/Appendix%204%20ARChemist%20code/Recipe%20Files/yumi_recipe.yaml

  93. Clarke, E. tests/files/orgsyn_v83p0184a.xdl (GitLab, 2021); https://gitlab.com/croningroup/chemputer/xdl/-/blob/master/tests/files/orgsyn_v83p0184a.xdl

  94. autoprotocol-python (GitHub, 2023); https://github.com/autoprotocol/autoprotocol-python/tree/master

  95. Laboratory of Artificial Intelligence. BioBlocks. GitHub https://github.com/liaupm/BioBlocks (2020).

  96. Felleisen, M. On the expressive power of programming languages. In ESOP ’90 (ed. Jones, N.) 134–151 (Springer, 1990); https://doi.org/10.1007/3-540-52592-0_60

  97. Cunningham, K., Ericson, B. J., Agrawal Bejarano, R. & Guzdial, M. Avoiding the Turing tarpit: learning conversational programming by starting from code’s purpose. In Proc. 2021 CHI Conference on Human Factors in Computing Systems 1–15 (Association for Computing Machinery, 2021); https://doi.org/10.1145/3411764.3445571

  98. Meyer, B. Object-Oriented Software Construction (Pearson Education, 2023).

  99. Fahland, D. et al. Declarative versus Imperative Process Modeling Languages: the issue of understandability. In Enterprise, Business-Process and Information Systems Modeling (eds Halpin, T. et al.) 353–366 (Springer, 2009); https://doi.org/10.1007/978-3-642-01862-6_29

  100. Gongora, A. E. et al. A Bayesian experimental autonomous researcher for mechanical design. Sci. Adv. 6, eaaz1708 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  101. MacLeod, B. P., Parlane, F. G. L., Brown, A. K., Hein, J. E. & Berlinguette, C. P. Flexible automation accelerates materials discovery. Nat. Mater. 21, 722–726 (2022).

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgements

We would like to thank H. Moran (North Carolina State University) for discussions around making the computer science jargon more accessible. M.A. gratefully acknowledges financial support from the University of North Carolina Research Opportunities Initiative (UNC-ROI) and National Science Foundation (award nos. 1940959 and 2208406).

Author information

Authors and Affiliations

Authors

Contributions

R.B.C. conceived and draughted the work. M.A. and R.B.C. reviewed and edited the manuscript.

Corresponding author

Correspondence to Milad Abolhasani.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Synthesis thanks Xiaonan Wang and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling Editor: Peter Seavill, in collaboration with the Nature Synthesis team.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Sections 1–3 and Discussion, including descriptions and comparisons.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Canty, R.B., Abolhasani, M. Reproducibility in automated chemistry laboratories using computer science abstractions. Nat. Synth 3, 1327–1339 (2024). https://doi.org/10.1038/s44160-024-00649-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Version of record:

  • Issue date:

  • DOI: https://doi.org/10.1038/s44160-024-00649-8

This article is cited by

Search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics