Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Resource
  • Published:

An open-source platform for multimodal digital trace data collection from smartphones

Nature Health volume 1, pages 437–448 (2026) Cite this article

Subjects

Abstract

Smartphone-based digital trace data can offer powerful insights for identifying behavioural patterns and health risks. However, existing tools for comprehensive data collection lack scalability, customizability, transparency and accessibility. To address these gaps, we developed an open-source platform that enables in situ capture of multimodal digital traces from smartphones (for example, moment-by-moment capture of screenshots, application usage logs, interaction histories and phone sensor readings). The Stanford Screenomics Data Collection application allows researchers to tailor data types and quality, data transfer methods and upload cadence. The Dashboard application supports real-time monitoring of participants’ data provision, identification of data issues and automated reactive communications to participants. The platform’s back end uses a NoSQL database for secure, and Health Insurance Portability and Accountability Act-compliant storage. Using illustrative 24-h digital trace data, we demonstrate how the platform expands the range of possible digital phenotyping studies.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to the full article PDF.

USD 39.95

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: A chord diagram illustrating connections among multimodal digital trace data, traditional health measures and research domains.
The alternative text for this image may have been generated using AI.
Fig. 2: General platform architecture and overall data flow.
The alternative text for this image may have been generated using AI.

Similar content being viewed by others

Data availability

The illustrative 24-h digital trace text-based data collected using the Stanford Screenomics platform are available in the Chapter 5 Data Management section of the GitHub repository at https://github.com/StanfordScreenomics/Platform/. The original screenshots are not available to underscore the privacy concerns surrounding their collection and use of screenshot data.

Code availability

The source code for the Stanford Screenomics platform developed by I.K. and J.B. is available for public access at https://www.github.com/StanfordScreenomics/Platform/. The repository will be actively maintained, with periodic updates for bug fixes, compatibility with new Android versions and improvements to documentation. Researchers are encouraged to submit issues or feature requests via the GitHub repository, and contributions from the community will be reviewed and incorporated as appropriate.

References

  1. Navarro, J. L. & Tudge, J. R. Technologizing Bronfenbrenner: neo-ecological theory. Curr. Psychol. 42, 19338–19354 (2023).

    Article  Google Scholar 

  2. Allen, K. A., Ryan, T., Gray, D. L., McInerney, D. M. & Waters, L. Social media use and social connectedness in adolescents: the positives and the potential pitfalls. Aust. Educ. Dev. Psychol. 31, 18–31 (2014).

    Article  Google Scholar 

  3. Pentland, A. Social Physics: how Good Ideas Spread-the Lessons From a New Science (Penguin, 2014).

  4. Audy Martínek, P., Caliandro, A. & Denegri-Knott, J. Digital practices tracing: studying consumer lurking in digital environments. J. Mark. Manag. 39, 244–274 (2023).

    Google Scholar 

  5. Kunst, K. & Vatrapu, R. Understanding electronic word of behavior: conceptualization of the observable digital traces of consumers’ behaviors. Electron. Mark. 29, 323–336 (2019).

    Article  Google Scholar 

  6. Chan, M., Chen, H. -T. & Lee, F. L. Examining the roles of mobile and social media in political participation: a cross-national analysis of three Asian societies using a communication mediation approach. New Media Soc. 19, 2003–2021 (2017).

    Article  Google Scholar 

  7. Kosinski, M., Stillwell, D. & Graepel, T. Private traits and attributes are predictable from digital records of human behavior. Proc. Natl Acad. Sci. USA 110, 5802–5805 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Du, J., Hew, K. F. & Liu, L. What can online traces tell us about students’ self-regulated learning? A systematic review of online trace data analysis. Comput. Educ. 201, 104828 (2023).

    Article  Google Scholar 

  9. Lämsä, J. et al. Measuring secondary education students’ self-regulated learning processes with digital trace data. Learn. Individ. Differ. 118, 102625 (2025).

    Article  Google Scholar 

  10. Jensen, M., George, M. J., Russell, M. R. & Odgers, C. L. Young adolescents’ digital technology use and mental health symptoms: little evidence of longitudinal or daily linkages. Clin. Psychol. Sci. 7, 1416–1433 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  11. Orben, A. & Przybylski, A. K. The association between adolescent well-being and digital technology use. Nat. Human Behav. 3, 173–182 (2019).

    Article  Google Scholar 

  12. Amir, S., Dredze, M. & Ayers, J. W. Mental health surveillance over social media with digital cohorts. In Proc. 6th Workshop on Computational Linguistics and Clinical Psychology, 114–120 (2019).

  13. Kim, I. et al. E-Cigarette–related health beliefs expressed on Twitter within the US. AJPM Focus 2, 100067 (2023).

    Article  PubMed  PubMed Central  Google Scholar 

  14. Eagle, N. & Pentland, A. Reality mining: sensing complex social systems. Pers. Ubiquit. Comput. 10, 255–268 (2006).

    Article  Google Scholar 

  15. Gordon, A. M. & Mendes, W. B. A large-scale study of stress, emotions, and blood pressure in daily life using a digital platform. Proc. Natl Acad. Sci. USA 118, e2105573118 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  16. Lathia, N., Sandstrom, G. M., Mascolo, C. & Rentfrow, P. J. Happier people live more active lives: using smartphones to link happiness and physical activity. PLoS ONE 12, e0160589 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  17. Lee, K. et al. Using digital phenotyping to understand health-related outcomes: a scoping review. Int. J. Med. Inform. 174, 105061 (2023).

    Article  PubMed  Google Scholar 

  18. Ohme, J. et al. Digital trace data collection for social media effects research: APIs, data donation, and (screen) tracking. Commun. Methods Meas. 18, 124–141 (2024).

    Article  Google Scholar 

  19. Kleinman, R. A. & Merkel, C. Digital contact tracing for COVID-19. CMAJ 192, E653–E656 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  20. Mejova, Y. Digital epidemiology. In Handbook of Computational Social Science for Policy (eds Bertoni, E. et al.) 279–303 (Springer, 2022).

  21. Amez, S. & Baert, S. Smartphone use and academic performance: a literature review. Int. J. Educ. Res. 103, 101618 (2020).

    Article  Google Scholar 

  22. Nam, R. J. & Cha, C. B. Examining highly novel positive future thinking in suicidal and nonsuicidal adolescents. Arch. Suicide Res. 28, 1200–1214 (2024).

    Article  PubMed  Google Scholar 

  23. Singh, M. K. K. & Samah, N. A. Impact of smartphone: a review on positive and negative effects on students. Asian Social Sci. 14, 83 (2018).

    Article  Google Scholar 

  24. Twenge, J. M. Have smartphones destroyed a generation? The Atlantic https://www.theatlantic.com/magazine/archive/2017/09/has-the-smartphone-destroyed-a-generation/534198/ (2017).

  25. Faust, A. M., Auerbeck, A., Lee, A. M., Kim, I. & Conroy, D. E. Passive sensing of smartphone use, physical activity and sedentary behavior among adolescents and young adults during the COVID-19 pandemic. J. Behav. Med. 47, 770–781 (2024).

  26. Yee, A. Z. et al. ScreenLife Capture: an open-source and user-friendly framework for collecting screenomes from Android smartphones. Behav. Res. Methods 55, 4068–4085 (2023).

    Article  PubMed  Google Scholar 

  27. Ram, N. et al. Screenomics: a new approach for observing and studying individuals’ digital lives. J. Adolesc. Res. 35, 16–50 (2020).

    Article  PubMed  Google Scholar 

  28. Reeves, B. et al. Screenomics: a framework to capture and analyze personal life experiences and the ways that technology shapes them. Hum. Comput. Interact. 36, 150–201 (2021).

    Article  PubMed  Google Scholar 

  29. Cho, M. -J., Reeves, B., Robinson, T. N. & Ram, N. Media production on smartphones: analysis of the timing, content, and context of message production using real-world smartphone use data. Cyberpsychol. Behav. Soc. Netw. 26, 371–379 (2023).

    Article  PubMed  Google Scholar 

  30. Lee, J., Hamilton, J. T., Ram, N., Roehrick, K. & Reeves, B. The psychology of poverty and life online: natural experiments on the effects of smartphone payday loan ads on psychological stress. Inform. Commun. Soc. 26, 2775–2796 (2023).

    Article  Google Scholar 

  31. Sun, X. et al. Connectedness and independence of young adults and parents in the digital world: observing smartphone interactions at multiple timescales using Screenomics. J. Soc. Pers. Relat. 40, 1126–1150 (2023).

    Article  Google Scholar 

  32. Jacobucci, R., Ammerman, B. & Ram, N. Examining passively collected smartphone-based data in the days prior to psychiatric hospitalization for a suicidal crisis: comparative case analysis. JMIR Form. Res. 8, e55999 (2024).

    Article  PubMed  PubMed Central  Google Scholar 

  33. Brinberg, M. et al. The idiosyncrasies of everyday digital lives: using the Human Screenome Project to study user behavior on smartphones. Comput. Hum. Behav. 114, 106570 (2021).

    Article  Google Scholar 

  34. Cerit, M. et al. Person-Specific analyses of smartphone use and mental health: intensive longitudinal study. JMIR Form. Res. 9, e59875 (2025).

    Article  PubMed  PubMed Central  Google Scholar 

  35. Ram, N., Haber, N., Robinson, T. N. & Reeves, B. Binding the person-specific approach to modern AI in the human screenome project: moving past generalizability to transferability. Multivariate Behav. Res. 59, 1211–1219 (2024).

    Article  PubMed  Google Scholar 

  36. Rodrigues, A., Montague, K. & Guerreiro, T. Data donors: sharing knowledge for mobile accessibility. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, 1–6 (2018).

  37. Antal, M., Bokor, Z. & Szabó, L. Z. Information revealed from scrolling interactions on mobile devices. Pattern Recogn. Lett. 56, 7–13 (2015).

    Article  Google Scholar 

  38. Zingaro, D., Savino, G. -L., Reichenbacher, T., Schöning, J. & Fabrikant, S. I. Tapping into mobile app use: what touch event data can reveal about active and passive in-app usage behaviour. Available at SSRN 4768783 (2024).

  39. Estrin, D. & Sim, I. Open mHealth architecture: an engine for health care innovation. Science 330, 759–760 (2010).

    Article  CAS  PubMed  Google Scholar 

  40. Ferreira, D., Kostakos, V. & Dey, A. K. AWARE: mobile context instrumentation framework. Front. ICT 2, 6 (2015).

    Article  Google Scholar 

  41. Jardine, J., Fisher, J. & Carrick, B. Apple’s ResearchKit: smart data collection for the smartphone era? J. R. Soc. Med. 108, 294–296 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  42. Lakshminarasimhappa, M. C. Web-based and smart mobile app for data collection: Kobo Toolbox/Kobo collect. J. Indian Libr. Assoc. 57, 72–79 (2022).

    Google Scholar 

  43. Menaspà, P. Effortless activity tracking with Google Fit. Br. J. Sports Med. 49, 1598–1598 (2015).

    Article  PubMed  Google Scholar 

  44. Onnela, J. -P. et al. Beiwe: a data collection platform for high-throughput digital phenotyping. J. Open Source Softw. 6, 3417 (2021).

    Article  Google Scholar 

  45. Wang, R. et al. StudentLife: using smartphones to assess mental health and academic performance of college students. In Mobile Health (eds Rehg, J. M. et al.) 7–33 (Springer, 2017).

  46. Freelon, D. On the interpretation of digital trace data in communication and social computing research. J. Broadcasting Electron. Media 58, 59–75 (2014).

    Article  Google Scholar 

  47. Huckvale, K., Venkatesh, S. & Christensen, H. Toward clinical digital phenotyping: a timely opportunity to consider purpose, quality, and safety. NPJ Digit. Med. 2, 88 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  48. Insel, T. R. Digital phenotyping: technology for a new science of behavior. JAMA 318, 1215–1216 (2017).

    Article  PubMed  Google Scholar 

  49. Jungherr, A., Schoen, H., Posegga, O. & Jürgens, P. Digital trace data in the study of public opinion: an indicator of attention toward politics rather than political support. Soc. Sci. Comput. Rev. 35, 336–356 (2017).

    Article  Google Scholar 

  50. Liang, Y., Zheng, X. & Zeng, D. D. A survey on big data-driven digital phenotyping of mental health. Inf. Fusion 52, 290–307 (2019).

    Article  Google Scholar 

  51. Thylstrup, N. B. Data out of place: toxic traces and the politics of recycling. Big Data Soc. 6, 2053951719875479 (2019).

    Article  Google Scholar 

  52. Cheng, X., Fang, L., Yang, L. & Cui, S. Mobile big data: the fuel for data-driven wireless. IEEE Internet Things J. 4, 1489–1516 (2017).

    Article  Google Scholar 

  53. Maeckelberghe, E., Zdunek, K., Marceglia, S., Farsides, B. & Rigby, M. The ethical challenges of personalized digital health. Front. Med. 10, 1123863 (2023).

    Article  Google Scholar 

  54. Nebeker, C., Bartlett Ellis, R. J. & Torous, J. Development of a decision-making checklist tool to support technology selection in digital health research. Transl. Behav. Med. 10, 1004–1015 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  55. Nebeker, C., Gholami, M., Kareem, D. & Kim, E. Applying a digital health checklist and readability tools to improve informed consent for digital health research. Front. Digit. Health 3, 690901 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  56. Dunseath, S., Weibel, N., Bloss, C. S. & Nebeker, C. NIH support of mobile, imaging, pervasive sensing, social media and location tracking (MISST) research: laying the foundation to examine research ethics in the digital age. NPJ Digit. Med. 1, 20171 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  57. Nebeker, C., Torous, J. & Bartlett Ellis, R. J. Building the case for actionable ethics in digital health research supported by artificial intelligence. BMC Med. 17, 137 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  58. Filkins, B. L. et al. Privacy and security in the era of digital health: what should translational researchers know and do about it? Am. J. Transl. Res. 8, 1560–1580 (2016).

    PubMed  PubMed Central  Google Scholar 

  59. Watson, H. J. & Nations, C. Addressing the growing need for algorithmic transparency. Communic. Assoc. Inf. Syst. 45, 26 (2019).

    Google Scholar 

  60. Wrzus, C. & Schoedel, R. Transparency and reproducibility in mobile sensing research. In Mobile Sensing in Psychology: Methods and Applications (eds Mehl, M. R. et al.) 53–77 (Guilford Press, 2023).

Download references

Acknowledgements

This research was supported in part by the National Heart, Lung, and Blood Institute of the National Institutes of Health under award number R01 HL169601 (to T.N.R., B.R., N.H. and N.R.). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the paper.

Author information

Authors and Affiliations

  1. Department of Psychology, Stanford University, Stanford, CA, USA

    Ian Kim, Jack Boffa & Nilàm Ram

  2. Department of Pediatrics, Stanford University, Stanford, CA, USA

    Ian Kim, Nathan Kline & Thomas N. Robinson

  3. Research Center for Humanities and Social Sciences, Academia Sinica, Taipei City, Taiwan

    Mujung Cho

  4. School of Kinesiology, University of Michigan, Ann Arbor, MI, USA

    David E. Conroy

  5. Graduate School of Education, Stanford University, Stanford, CA, USA

    Nick Haber

  6. Department of Medicine, Stanford University, Stanford, CA, USA

    Thomas N. Robinson

  7. Department of Communication, Stanford University, Stanford, CA, USA

    Byron Reeves & Nilàm Ram

Authors
  1. Ian Kim
    View author publications

    Search author on:PubMed Google Scholar

  2. Jack Boffa
    View author publications

    Search author on:PubMed Google Scholar

  3. Mujung Cho
    View author publications

    Search author on:PubMed Google Scholar

  4. David E. Conroy
    View author publications

    Search author on:PubMed Google Scholar

  5. Nathan Kline
    View author publications

    Search author on:PubMed Google Scholar

  6. Nick Haber
    View author publications

    Search author on:PubMed Google Scholar

  7. Thomas N. Robinson
    View author publications

    Search author on:PubMed Google Scholar

  8. Byron Reeves
    View author publications

    Search author on:PubMed Google Scholar

  9. Nilàm Ram
    View author publications

    Search author on:PubMed Google Scholar

Contributions

I.K. conceived and designed the work, conducted data acquisition and analysis, interpreted data, drafted the paper and performed critical revisions. I.K. and J.B. developed the software, with N.K. assisting in its testing. M.C. contributed to the conception and design of the work. D.E.C. contributed to data interpretation and critical revisions. T.N.R., B.R., N.H. and N.R. contributed to the conception and design, data interpretation and critical revisions, and secured funding. All authors reviewed and approved the final paper and consent to its publication.

Corresponding author

Correspondence to Ian Kim.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Health thanks Jakob Ohme and Andrew Yee for their contribution to the peer review of this work. Primary Handling Editor: Lorenzo Righetto, in collaboration with the Nature Health team.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, I., Boffa, J., Cho, M. et al. An open-source platform for multimodal digital trace data collection from smartphones. Nat. Health 1, 437–448 (2026). https://doi.org/10.1038/s44360-026-00072-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Version of record:

  • Issue date:

  • DOI: https://doi.org/10.1038/s44360-026-00072-7

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing