In their recent paper1, Maples et al. surveyed 1006 users of the Replika app2 who were students and who had been using the app for at least a month. Among their results, they reported that participants were relatively lonely and used Replika for diverse purposes, including “as a friend, a therapist, and an intellectual mirror”. They emphasized that “3% [of participants] reported that Replika halted their suicidal ideation”1. While Generative AI has exciting potential in many fields, and we don’t disagree that “intelligent social agents” may warrant further study as cognitive and therapeutic aids1,3, we think these results should be treated with caution, as important context about how Replika, in particular, has been marketed (and used) was missing. (Maples et al.3 seems to have been an earlier exploration in the same vein as the paper under discussion. Replika is depicted similarly by the authors in both works.) We supplement that context here with information about Replika’s sexual component, and discuss the broader threat of industry interests to scientific integrity.

According to Replika’s website, and notably from the pull quotes therein from users and from publications like Forbes, The New York Times, and Wired, the services it offers are congruent with the use cases described in Maples et al. You can find “the perfect companion in Replika”, a “friend, partner, or mentor”. Although there are indications—a red heart, the phrase “AI soulmate”—of other, more romantic use cases, the emphasis is on the AI as a tool for emotional support and self-improvement2. (The name of the AI companion is Replika and the name of the company behind it is Luka, Inc., but for simplicity, we will refer to both – in their capacity of providing and marketing Replika – as Replika.) On the other hand, advertisements for Replika on social media have included elements like “NSFW pics from her”, “can you make AI fall in love with you?”, “...you can role-play and get NSFW pics from her”, “Role-play with Replika” followed by “Get hot pics from her”, and “wanna join me tonight?” followed by a winking emoji4,5. These ads used meme formats and featured female Replika AI personas.

Although the exact timeline of Replika’s ads, services, and features is not clear, the sexual component seems to have existed in some form since at least 2020. Communication by Replika acknowledged in 2020 that sexual role-playing had moved behind a paywall6. Coverage in the Washington Post in 2023 described a perceived change in Replika that users felt interrupted their established romantic and sexual relationships, some of which had been ongoing across multiple years7. Reddit discussions of sexual content in Replika date from 2020 onward (e.g., ref. 8). Reddit users posted plentiful purported screenshots of ads in this genre circa 20239, including what claim to be screenshots of suggestive interactions with free Replika accounts. (One user in the Washington Post article described the sexual relationship with her Replika companion as lasting from 2021 to 2023. For further discussion and links to additional sources, see refs. 4,7,9,10.) Using this evidence, we surmise that the erotic role play (ERP)-related use case is a significant one, known to Replika, and intended to drive paid memberships. Further, we think it is at least plausible that this was the case during significant portions of the project culminating in the paper by Maples et al. However, if Replika were encountered solely through Maples et al.1, just as through its own website2, its non-trivial use case related to ERP would be obscured.

ERP-related usage of Replika is attested in the academic literature previous to the publication of Maples et al. That users of Replika may seek it out for reasons related to companionship—and specifically, to love, sex, and intimacy—was reported in ref. 11. Users in that study were mostly single, White, American men; loneliness was also reported as a thematic reason they started using Replika. McStay, 2022 stated that “[t]he paid-for version of Replika unlocks romantic and erotic dimensions” and discussed the “projection of female gender stereotypes... onto synthetic agents” amongst some of its users12. We bring these related works up to show that other researchers found this kind of use relevant to report, and to show that there is peer-reviewed evidence that aligns with our more subjective, informal interpretation of the ads run by Replika.

That many of its users come to Replika specifically because it offers romantic and sexual companionship suggests possible differences between its user base and that of otherwise comparable technology (LLM-based chatbots) like ChatGPT13. (That the target of some of their social media ads seems to be very online, younger men also raises the concern of “trolling”4.) However, this received little discussion in Maples et al.1. There is a mention of Replika’s “sexual conversations”: “Two participants reported discomfort with Replika’s sexual conversations, which highlights the importance of ethical considerations and boundaries in AI chatbot interactions”1. (For more on this topic, see ref. 10.) There is also the passage “[d]uring data collection in late 2021, Replika was not programmed to initiate therapeutic or intimate relationships”, and the statement that “[w]e found no evidence that they [participants] differed from a typical student population beyond this high loneliness score. It is not clear whether this increased loneliness was the cause of their initial interest in Replika”1. (It is not specified in Maples et al. whether survey participants were using free or paid accounts. They were “recruited randomly via email from a list of app users”1.)

Given the gravity of some of the interpretations of their findings—the paper heavily emphasizes the potential use of ISAs like Replika in suicide mitigation, as has later press coverage—we think it is important to consider who uses Replika and why, beyond the image that Replika presents of itself. Replika, for whatever reason, seems to want to minimize its sexual component—perhaps because of stigma, perhaps because selling itself as therapeutic or motivational is more prestigious or lucrative, perhaps something else altogether. (For an example of why this matters, imagine that a university administrator read this paper1, and then offered all students a school-bought Replika account as a form of mental health support; it is plausible that an immediate consequence would be students being made uncomfortable when their Replika AI repeatedly tried to move the conversation in a sexual direction10. It’s not hard to imagine additional scenarios.) Since publication, the work of Maples et al. has garnered press coverage14,15,16,17, including coverage created explicitly in collaboration with Replika18. We think it is worth pointing out that, in our estimation, much of that coverage could be described as positive, both for Replika in particular and for the use of AI chatbots in general (including, notably, for their use in mental health and educational settings); for example, this headline from Forbes, which runs, “AI Chatbot Helps Prevent Suicide Among Students, Stanford Study Finds”14.

Why wouldn’t the ERP-related use case be mentioned at all in the paper, since it seems to be a significant aspect of Replika’s services? If Replika provided additional evidence as to why the ERP-related users are negligible or wouldn’t be included in the samples accessible to the researchers, we think that would have been worth stating in the paper. What appears to us to be the unquestioning acceptance of the company line (as to who uses Replika and why, beyond the image that Replika presents of itself, and what Replika is designed to do) by Maples et al.1 is concerning not just with respect to the interpretation of their results, but also because it seems emblematic of a broader problem within science, specifically within the burgeoning field of Generative AI. There is no clear line demarcating science and industry, especially as companies (Google, Meta, etc.) provide funding and resources (including access to AI models) to researchers, and write papers alongside them19,20,21,22. (As in cyberneticist Stafford Beer’s famous saying, “The purpose of a system is what it does”, so there can be no permanent extrication of science from money or power.) It is the responsibility of all scientists to interrogate the interests that underpin resources or access provided to them, and, when pertinent, to communicate that process. When information flows from another party into the academic work product, that should be clearly stated. (As one example, the aforementioned passage, “[d]uring data collection in late 2021, Replika was not programmed to initiate therapeutic or intimate relationships” does not make it clear how the authors ascertained the scope of Replika’s programming. As another example, there is the passage: “It is critical to note that at the time, Replika was not focused on providing therapy as a key service, and included these conversational pathways out of an abundance of caution for user mental health.”1 Given Replika’s obvious financial stake, we don’t think it’s reasonable to seemingly take the motivation of “an abundance of caution for user mental health” at face value.) Otherwise, scientific research (and institutions, and events) can be used to launder—legitimize, and sanitize—standards, datasets, processes, results, mistakes, falsehoods, and even personal reputations20,23.

Likewise, it is the responsibility of all scientists to interrogate and communicate their own interests when those are intertwined with their research. This imperative is part of scientific practice even though specific policies regarding ethics, disclosures, and conflicts of interest vary (over time, across fields, and across journals)24,25. Despite the variety, discussions exist in the literature that prompt scientists towards a generous and expansive interpretation of conflict of interest, noting, for example, that competing interests can be anything that could “possibly corrupt the motivation”25 of the people behind the research, that they can be “potential or actual, perceived or real, insignificant or harmful”, and that they need only “represent the potential for biased judgment”, not its certainty26. We note that while the actors explicitly called out in the literature are often the scientists, peer reviewers, or publishers, when data or certain kinds of tools or services are relied on in the final work product but supplied by a third party, the interests of that third party ought to be considered as well. (Bethanie Maples, lead author of ref. 1, is the founder/CEO of Atypical AI: “The Generative AI Platform for Education”27, founded in 202328,29. Since Maples et al.1 could persuade people of the value of Generative AI, specifically in contexts related to education and mental health treatment, we think that Maples’ affiliation with an education-related Generative AI company would have been worth disclosing alongside her Stanford affiliation/as a competing interest.) We contacted Maples, the corresponding author, about this article on February 11, 2024. As of this writing, we have received no response.