Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Correspondence
  • Published:

Multimodal AI needs active human interaction

Nature Human Behaviour volume 8, pages 1825–1826 (2024)Cite this article

Subjects

Consider that you are comfortably seated on your sofa, reading this article in Nature Human Behaviour. Your AI assistant says “I see you are reading about multimodal AI. But you asked me to remind you that we need to finalize visuals for your presentation tomorrow morning. I’ve gone through the audience feedback of your last presentation and have mocked up new illustrations to introduce your idea.”

Current multimodal AI models have the ingredients for this sort of interaction. Many real-life tasks, such as driving and medical diagnosis1, are difficult to solve solely through verbal communication and require multimodal information. Recent commercial general-purpose AI is equipped with modalities of vision and audition (for example, GPT-4o, Gemini 1.5 and Claude 3). Techniques such as retrieval-augmented generation are being developed to enable large language models (LLMs) to use multimodal databases2. Portable multimodal AI devices (for example, the handheld rabbit r1, the wearable ai pin, and Ray-Ban Meta smart glasses) are being developed to provide assistance in the physical world. This multimodal trend greatly increases the range of problems that can be solved or assisted by AI tools. It also opens up human–AI communication channels in real time through voice and facial expression.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to the full article PDF.

USD 39.95

Prices may be subject to local taxes which are calculated during checkout

References

  1. Acosta, J. N., Falcone, G. J., Rajpurkar, P. & Topol, E. J. Nat. Med. 28, 1773–1784 (2022).

    Article  CAS  PubMed  Google Scholar 

  2. Surla, A., Bodhankar, A. & Varshney, T. An easy introduction to multimodal retrieval augmented generation. NVIDIA, https://developer.nvidia.com/blog/an-easy-introduction-to-multimodal-retrieval-augmented-generation (20 March 2024).

  3. Mitchell, M., Palmarini, A. B. & Moskvichev, A. Comparing humans, GPT-4, and GPT-4V on abstraction and reasoning tasks. openreview.net, https://openreview.net/forum?id=3rGT5OkzpC (2024).

  4. Biron, C. L. FEATURE-AI’s ‘insane’ translation mistakes endanger US asylum cases. Reuters, https://www.reuters.com/article/idUSL8N3AO23S (18 September 2023).

  5. Gitclear. Coding on copilot: 2023 data suggests downward pressure on code quality. Gitclear, https://www.gitclear.com/coding_on_copilot_data_shows_ais_downward_pressure_on_code_quality (2023).

  6. Cohn, M. et al. Believing anthropomorphism: examining the role of anthropomorphic cues on trust in large language models. In CHI EA '24: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (eds Mueller, F. F. et al) article no. 54 (ACM, 2024).

  7. Bengio, Y. et al. Science 384, 842–845 (2024).

    Article  CAS  PubMed  Google Scholar 

  8. Tao, T. Machine assisted proof. terrytao.wordpress.com, https://terrytao.wordpress.com/wp-content/uploads/2024/03/machine-assisted-proof-notices.pdf (2024).

Download references

Acknowledgements

We thank S. Ron for helpful discussions.

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Miami, Coral Gables, FL, USA

    Xu Pan & Odelia Schwartz

Authors
  1. Xu Pan
    View author publications

    Search author on:PubMed Google Scholar

  2. Odelia Schwartz
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Odelia Schwartz.

Ethics declarations

Competing interests

The authors declare no competing interests.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pan, X., Schwartz, O. Multimodal AI needs active human interaction. Nat Hum Behav 8, 1825–1826 (2024). https://doi.org/10.1038/s41562-024-01932-6

Download citation

  • Published:

  • Version of record:

  • Issue date:

  • DOI: https://doi.org/10.1038/s41562-024-01932-6

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing