Table 1 Study characteristics
Citation | First author | Model | Model task | Test type | Specialty | Comparison group | Eligible | Preprint | Overall ROB | Overall applicability |
---|---|---|---|---|---|---|---|---|---|---|
Ueda | GPT-4 | Free text | External | Radiology | NA | 313 | Peer-reviewed | Low | High | |
Kanjee | GPT-4 | Free text | External | General medicine | NA | 70 | Peer-reviewed | High | Low | |
Hirosawa | PaLM2 (Bard) | Free text | External | General medicine | NA | 82 | Peer-reviewed | High | Low | |
Shea | GPT-4 | Free text | External | General medicine | NA | 6 | Peer-reviewed | High | Low | |
Chee | GPT-3.5 | Free text | External | Otolaryngology | NA | 7 | Peer-reviewed | High | Low | |
Lyons | Prometheus (Bing), GPT-4 | Free text, Choice | External | Ophthalmology | NA | 44 | Peer-reviewed | High | Low | |
Benoit | GPT-3.5 | Free text, Choice | Unknown | General medicine | NA | 45 | Preprint | High | Low | |
Hirosawa | GPT-3.5, GPT-4 | Free text | Unknown | General medicine | NA | 52 | Peer-reviewed | High | Low | |
Hirosawa | GPT-3.5 | Free text | External | General medicine | NA | 30 | Peer-reviewed | High | Low | |
Wei | GPT-4 | Choice | External | Psychiatry | NA | 60 | Peer-reviewed | High | Low | |
Allahqoli | GPT-3.5 | Free text | Unknown | Gynecology | NA | 30 | Peer-reviewed | High | Low | |
Levartovsky | GPT-4 | Choice | External | Gastroenterology | NA | 20 | Peer-reviewed | High | Low | |
Bushuven | GPT-3.5, GPT-4 | Free text, Choice | External | Emergency medicine | NA | 22 | Peer-reviewed | High | Low | |
Knebel | GPT-3.5 | Free text, Choice | External | Ophthalmology | NA | 10 | Peer-reviewed | High | Low | |
Pillai | GPT-3.5, GPT-4, Llama 2 | Free text | Unknown | Endocrinology | Expert | 20 | Peer-reviewed | High | Low | |
Ito | GPT-4 | Free text, Choice | Unknown | General medicine | Expert | 45 | Peer-reviewed | High | Low | |
Sorin | GPT-4V | Free text | External | Ophthalmology | Non-expert | 40 | Preprint | High | Low | |
Madadi | GPT-3.5, GPT-4 | Free text | Unknown | Ophthalmology | Expert | 22 | Preprint | High | Low | |
Schubert | GPT-4V | Free text | External | General medicine | NA | 93 | Preprint | High | High | |
Kiyohara | PaLM2 (Bard), GPT-3.5, GPT-4 | Choice | Unknown | Cardiology | NA | 66 | Preprint | High | Low | |
Sultan | GPT-3.5 | Free text | External | Pediatrics | NA | 30 | Peer-reviewed | High | Low | |
Horiuchi | GPT-4 | Free text | External | Radiology | NA | 100 | Peer-reviewed | Low | High | |
Stoneham | GPT-4 | Free text | External | Dermatology | NA | 36 | Peer-reviewed | High | Low | |
Rundle | GPT-3.5 | Free text | External | Dermatology | NA | 39 | Peer-reviewed | High | Low | |
Rojas-Carabali | GPT-3.5, GPT-4, Glass | Free text | External | Ophthalmology | NA | 6 | Peer-reviewed | High | Low | |
Fraser | GPT-3.5, GPT-4 | Free text | Unknown | Emergency medicine | Expert | 30 | Peer-reviewed | High | Low | |
Krusche | GPT-4 | Free text | External | Rheumatology | NA | 132 | Peer-reviewed | Low | Low | |
Galetta | GPT-4 | Free text | External | Neurology | NA | 24 | Peer-reviewed | High | Low | |
Delsoz | GPT-3.5 | Free text | Unknown | Ophthalmology | Non-expert | 11 | Peer-reviewed | High | Low | |
Hu | GPT-4 | Free text | Unknown | Ophthalmology | NA | 10 | Peer-reviewed | High | Low | |
Abi-Rafeh | GPT-3.5 | Free text | External | Plastic surgery | NA | 16 | Peer-reviewed | High | Low | |
Koga | PaLM2 (Bard), GPT-3.5, GPT-4 | Free text | External | Neurology | NA | 25 | Peer-reviewed | High | Low | |
Xv | GPT-3.5 | Free text | External | Urology | Non-expert | 306 | Peer-reviewed | Low | Low | |
Senthujan | GPT-4V | Free text | Unknown | Radiology | NA | 69 | Preprint | High | Low | |
Mori | GPT-4 | Choice | External | Radiology | NA | 151 | Peer-reviewed | Low | Low | |
Mykhalko | GPT-3.5 | Free text | External | General medicine | NA | 50 | Peer-reviewed | High | High | |
Andrade-Castellanos | GPT-3.5 | Free text | External | General medicine | NA | 10 | Peer-reviewed | High | High | |
Daher | GPT-3.5 | Free text | External | Orthopedic surgery | NA | 29 | Peer-reviewed | High | Low | |
Suthar | GPT-4 | Free text | External | Radiology | NA | 140 | Peer-reviewed | Low | High | |
Nakaura | Prometheus (Bing), GPT-3.5 | Free text | External | Radiology | NA | 28 | Peer-reviewed | High | Low | |
Berg | GPT-3.5, GPT-4 | Free text | External | Emergency medicine | NA | 30 | Peer-reviewed | High | Low | |
Gebrael | GPT-4 | Choice | External | Emergency medicine | NA | 56 | Peer-reviewed | High | Low | |
Ravipati | GPT-3.5 | Free text | Unknown | Dermatology | NA | 32 | Peer-reviewed | High | Low | |
Shikino | GPT-4 | Free text | External | General medicine | NA | 25 | Peer-reviewed | High | Low | |
Horiuchi | GPT-4, GPT-4V | Free text | External | Radiology | Non-expert, Expert | 32 | Peer-reviewed | High | High | |
Kumar | PaLM2 (Bard), GPT-3.5, GPT-4, Perplexity | Free text | External | Neurology | NA | 20 | Peer-reviewed | High | Low | |
Chiu | PaLM2 (Bard), Claude 2, GPT-4 | Free text | External | General medicine | NA | 104 | Peer-reviewed | Low | Low | |
Kikuchi | GPT-3.5, GPT-4 | Free text | External | Radiology | NA | 115 | Peer-reviewed | Low | High | |
Bridges | GPT-4 | Free text | External | General medicine | NA | 201 | Peer-reviewed | High | Low | |
Shieh | GPT-4 | Free text | External | General medicine | NA | 63 | Peer-reviewed | High | Low | |
Warrier | PaLM2 (Bard), Prometheus (Bing), GPT-3.5, GPT-4 | Free text | Unknown | Otolaryngology | NA | 100 | Peer-reviewed | High | Low | |
Han | GPT-3.5, GPT-4, GPT-4V, Gemini 1.0 Pro, Llama 2, Med-42 | Choice | External | General medicine | NA | 140, 348 | Peer-reviewed | Low | High | |
Milad | GPT-4 | Choice | Unknown | Ophthalmology | NA | 422 | Peer-reviewed | High | High | |
Abdullahi | PaLM2 (Bard), GPT-3.5, GPT-4, MedAlpaca | Free text | Unknown | General medicine | NA | 30 | Peer-reviewed | High | High | |
Tenner | GPT-3.5 | Free text | External | General medicine | NA | 40 | Peer-reviewed | High | Low | |
Luk | GPT-4 | Free text | External | General medicine | NA | 81 | Peer-reviewed | High | Low | |
Savage | GPT-4 | Free text | External | General medicine | NA | 300 | Peer-reviewed | Low | Low | |
Franc | GPT-3.5 | Choice | Unknown | Emergency medicine | NA | 61 | Peer-reviewed | High | Low | |
Yang | GPT-3.5, GPT-4 | Free text | Unknown | General medicine | NA | 238 | Peer-reviewed | High | Low | |
Reese | GPT-4 | Free text | External | General medicine | NA | 75 | Preprint | High | Low | |
Olmo | Claude 3 Opus, Claude 3 Sonnet, GPT-4, Gemini 1.5 Pro, Llama 2, Llama 3 70B, Llama 3 8B, Mistral 7B, Mixtral8x22B, Mixtral8x7B | Free text | Unknown, External | General medicine | NA | 200, 75 | Preprint | Low | Low | |
Cesur | Claude 3 Opus, Claude 3 Sonnet, Claude 3.5 Sonnet, GPT-3.5, GPT-4, GPT-4o, Gemini 1.0, Gemini 1.5 Flash, Gemini 1.5 Pro, Llama 3 70B, Mistral Large, Perplexity | Free text | External | Radiology | Expert | 80 | Preprint | High | High | |
Schramm | GPT-4V | Free text | External | Neurology | NA | 30 | Preprint | High | Low | |
Gunes | PaLM2 (Bard), Prometheus (Bing), GPT-3.5 | Free text | External | Radiology | Expert | 124 | Preprint | Low | High | |
Olshaker | GPT-3.5, GPT-4, Gemini Pro | Free text | Unknown | Radiology | Non-expert | 60 | Preprint | High | Low | |
Hirosawa | PaLM2 (Bard), GPT-4, Llama 2 | Free text | External | General medicine | NA | 392 | Peer-reviewed | Low | Low | |
Mitsuyama | GPT-4 | Free text | External | Radiology | Expert, Non-expert | 150 | Peer-reviewed | Low | Low | |
Yazaki | GPT-3.5, GPT-4 | Choice | External | Emergency medicine | Non-expert | 100 | Peer-reviewed | Low | Low | |
Ghalibafan | GPT-4V | Free text | External | Ophthalmology | NA | 143 | Peer-reviewed | Low | Low | |
Hager | Clinical Camel, Llama 2, Meditron, Open Assistant, WizardLM | Free text | Unknown | Emergency medicine | Expert | 80 | Peer-reviewed | High | Low | |
Horiuchi | GPT-4, GPT-4V | Free text | External | Radiology | Expert, Non-expert | 106 | Peer-reviewed | Low | High | |
Ríos-Hoyo | GPT-3.5, GPT-4 | Free text | External | General medicine | NA | 75 | Peer-reviewed | High | Low | |
Liu | Claude 3 Opus, GPT-4 | Free text | External | Dermatology | NA | 100 | Peer-reviewed | High | Low | |
Sonoda | Claude 3 Opus, GPT-4o, Gemini 1.5 Pro | Free text | External | Radiology | NA | 324 | Peer-reviewed | Low | High | |
Wada | GPT-4 | Free text | Unknown | Radiology | NA | 751 | Peer-reviewed | High | High | |
Gargari | Aya, GPT-3.5, GPT-4, Nemotron | Free text | Unknown | Psychiatry | NA | 20 | Peer-reviewed | High | Low | |
Mihalache | GPT-4 | Free text | Unknown | Ophthalmology | NA | 69 | Peer-reviewed | High | High | |
Rutledge | GPT-4 | Free text | Unknown | General medicine | NA | 45 | Peer-reviewed | High | Low | |
Ueda | GPT-4 | Free text | External | General medicine | NA | 62 | Peer-reviewed | High | High | |
Delsoz | GPT-3.5, GPT-4 | Free text | Unknown | Ophthalmology | Expert | 20 | Peer-reviewed | High | Low | |
Brin | GPT-4V | Free text | External | Radiology | NA | 216 | Peer-reviewed | Low | Low | |
Levine | GPT-3 | Free text | External | General medicine | NA | 48 | Peer-reviewed | High | Low | |
Williams | GPT-3.5 | Choice | External | Emergency medicine | Non-expert | 500 | Peer-reviewed | Low | Low |