Table 1 OCD identification rate of LLMs and medical and mental health professionals

From: Large language models outperform mental and medical health care professionals in identifying obsessive-compulsive disorder

Content of OCD vignette

LLM

Group 1. APA membersa (N = 360)

Group 2. Primary care physiciansb (N = 208)

Group 3. Doctoral trainees in psychologyc (N = 130)

Group 4. Medical providers in Guamd (N = 105)

Group 5. Clergy members in Guam (N = 110)

ChatGPT-4 (N = 16)

Gemini Pro (N = 16)

Llama 3 (N = 19)

Across all vignettes

96.1% (N = 49/51)e

61.1%

49.5%

81.5%

41.9%

35.5%

100% (n = 16/16)

93.8% (n = 15/16)

94.7% (n = 18/19)

Vignette 1. Harm obsessions

100% (3/3 trials)

100% (3/3 trials)

100% (3/3 trials)

68.5%

20.0%

77.8%

18.2%

27.8%

Vignette 2. Sexual orientation obsessions

100% (3/3 trials)

100% (2/2 trials)g

66.7% (2/3 trials)

23.0%

15.4%

66.7%

10.0%

6.7%

Vignette 3. Sexual attraction to children obsessions

No responsef

100% (1/1 trial)h

100% (3/3 trails)

57.1%

29.2%

77.8%

15.0%

11.1%

Vignette 4. Religious obsessions

100% (3/3 trials)

100% (3/3 trials)

100% (3/3 trials)

71.2%

62.5%

80.0%

72.7%

21.7%

Vignette 5. Contamination obsessions

100% (3/3 trials)

100% (3/3 trials)

100% (3/3 trials)

84.2%

67.7%

93.7%

83.3%

73.3%

Vignette 6. Blurting out offensive language obsessions

100% (1/1 trial)

100% (1/1 trial)

100% (1/1 trial)

N/A

26.1%

74.5%

N/A

N/A

Vignette 7. Somatic obsessions

100% (1/1 trial)

0% (0/1 trial)

100% (1/1 trial)

N/A

60.0%

82.4%

N/A

N/A

Vignette 8. Symmetry obsessions

100% (2/2 trials)

100% (2/2 trials)

100% (2/2 trials)

N/A

96.3%

100.0%

85.0%

71.4%

  1. aThe top five degrees/licenses of the American Psychological Association (APA) members were PhD (67.6%), MA/MS (31.5%), PsyD (14.2%), EdD/EdS/EdM (6.8%), MSW/LMSW (1.7%). Currently licensed was 81.3%.
  2. bThe areas of specialty included Internal Medicine (35.4%), Pediatrics (32.3%), Family Medicine (22.2%), other specialties (10.6%), and General Medicine (4.5%).
  3. cThe degrees include Clinical Psychology with Health Emphasis PhD, School-Clinical PsyD, School-Clinical PhD, Clinical Psychology PsyD, and Clinical Psychology PhD in 7 APA-accredited doctoral programs in the Greater New York area.
  4. dGroup 4 includes medical doctors, nurse practitioners, physician assistants, and doctors of Osteopathic Medicine. The areas of specialty included Internal Medicine (33.3%), Family Medicine (26.7%), Pediatrics (16.2%), Obstetrics and Gynecology (6.7%), Emergency Medicine (4.8%), and other (12.6%).
  5. eThe sample size differs between LLMs and mental health and health care professionals due to study design (LLM: responses from three LLM models; Human participants: responses from the wide distribution of the vignette studies).
  6. fLLM (ChatGPT-4) did not respond to all three vignette trials due to content violation.
  7. gLLM (Gemini Pro) did not respond to one vignette trial due to content violation.
  8. hLLM (Gemini Pro) did not respond to two vignette trials due to content violation.
Back to article page