Table 1 Study characteristics

From: A systematic review and meta-analysis of diagnostic performance comparison between generative AI and physicians

Citation

First author

Model

Model task

Test type

Specialty

Comparison group

Eligible

Preprint

Overall ROB

Overall applicability

11

Ueda

GPT-4

Free text

External

Radiology

NA

313

Peer-reviewed

Low

High

12

Kanjee

GPT-4

Free text

External

General medicine

NA

70

Peer-reviewed

High

Low

13

Hirosawa

PaLM2 (Bard)

Free text

External

General medicine

NA

82

Peer-reviewed

High

Low

14

Shea

GPT-4

Free text

External

General medicine

NA

6

Peer-reviewed

High

Low

15

Chee

GPT-3.5

Free text

External

Otolaryngology

NA

7

Peer-reviewed

High

Low

16

Lyons

Prometheus (Bing),

GPT-4

Free text, Choice

External

Ophthalmology

NA

44

Peer-reviewed

High

Low

17

Benoit

GPT-3.5

Free text, Choice

Unknown

General medicine

NA

45

Preprint

High

Low

18

Hirosawa

GPT-3.5, GPT-4

Free text

Unknown

General medicine

NA

52

Peer-reviewed

High

Low

19

Hirosawa

GPT-3.5

Free text

External

General medicine

NA

30

Peer-reviewed

High

Low

20

Wei

GPT-4

Choice

External

Psychiatry

NA

60

Peer-reviewed

High

Low

21

Allahqoli

GPT-3.5

Free text

Unknown

Gynecology

NA

30

Peer-reviewed

High

Low

22

Levartovsky

GPT-4

Choice

External

Gastroenterology

NA

20

Peer-reviewed

High

Low

23

Bushuven

GPT-3.5, GPT-4

Free text, Choice

External

Emergency medicine

NA

22

Peer-reviewed

High

Low

24

Knebel

GPT-3.5

Free text, Choice

External

Ophthalmology

NA

10

Peer-reviewed

High

Low

25

Pillai

GPT-3.5, GPT-4, Llama 2

Free text

Unknown

Endocrinology

Expert

20

Peer-reviewed

High

Low

26

Ito

GPT-4

Free text, Choice

Unknown

General medicine

Expert

45

Peer-reviewed

High

Low

27

Sorin

GPT-4V

Free text

External

Ophthalmology

Non-expert

40

Preprint

High

Low

28

Madadi

GPT-3.5, GPT-4

Free text

Unknown

Ophthalmology

Expert

22

Preprint

High

Low

29

Schubert

GPT-4V

Free text

External

General medicine

NA

93

Preprint

High

High

30

Kiyohara

PaLM2 (Bard),

GPT-3.5, GPT-4

Choice

Unknown

Cardiology

NA

66

Preprint

High

Low

31

Sultan

GPT-3.5

Free text

External

Pediatrics

NA

30

Peer-reviewed

High

Low

32

Horiuchi

GPT-4

Free text

External

Radiology

NA

100

Peer-reviewed

Low

High

33

Stoneham

GPT-4

Free text

External

Dermatology

NA

36

Peer-reviewed

High

Low

34

Rundle

GPT-3.5

Free text

External

Dermatology

NA

39

Peer-reviewed

High

Low

35

Rojas-Carabali

GPT-3.5, GPT-4, Glass

Free text

External

Ophthalmology

NA

6

Peer-reviewed

High

Low

36

Fraser

GPT-3.5, GPT-4

Free text

Unknown

Emergency medicine

Expert

30

Peer-reviewed

High

Low

37

Krusche

GPT-4

Free text

External

Rheumatology

NA

132

Peer-reviewed

Low

Low

38

Galetta

GPT-4

Free text

External

Neurology

NA

24

Peer-reviewed

High

Low

39

Delsoz

GPT-3.5

Free text

Unknown

Ophthalmology

Non-expert

11

Peer-reviewed

High

Low

40

Hu

GPT-4

Free text

Unknown

Ophthalmology

NA

10

Peer-reviewed

High

Low

41

Abi-Rafeh

GPT-3.5

Free text

External

Plastic surgery

NA

16

Peer-reviewed

High

Low

42

Koga

PaLM2 (Bard),

GPT-3.5, GPT-4

Free text

External

Neurology

NA

25

Peer-reviewed

High

Low

43

Xv

GPT-3.5

Free text

External

Urology

Non-expert

306

Peer-reviewed

Low

Low

44

Senthujan

GPT-4V

Free text

Unknown

Radiology

NA

69

Preprint

High

Low

45

Mori

GPT-4

Choice

External

Radiology

NA

151

Peer-reviewed

Low

Low

46

Mykhalko

GPT-3.5

Free text

External

General medicine

NA

50

Peer-reviewed

High

High

47

Andrade-Castellanos

GPT-3.5

Free text

External

General medicine

NA

10

Peer-reviewed

High

High

48

Daher

GPT-3.5

Free text

External

Orthopedic surgery

NA

29

Peer-reviewed

High

Low

49

Suthar

GPT-4

Free text

External

Radiology

NA

140

Peer-reviewed

Low

High

50

Nakaura

Prometheus (Bing),

GPT-3.5

Free text

External

Radiology

NA

28

Peer-reviewed

High

Low

51

Berg

GPT-3.5, GPT-4

Free text

External

Emergency medicine

NA

30

Peer-reviewed

High

Low

52

Gebrael

GPT-4

Choice

External

Emergency medicine

NA

56

Peer-reviewed

High

Low

53

Ravipati

GPT-3.5

Free text

Unknown

Dermatology

NA

32

Peer-reviewed

High

Low

54

Shikino

GPT-4

Free text

External

General medicine

NA

25

Peer-reviewed

High

Low

55

Horiuchi

GPT-4, GPT-4V

Free text

External

Radiology

Non-expert, Expert

32

Peer-reviewed

High

High

56

Kumar

PaLM2 (Bard),

GPT-3.5, GPT-4, Perplexity

Free text

External

Neurology

NA

20

Peer-reviewed

High

Low

57

Chiu

PaLM2 (Bard), Claude 2, GPT-4

Free text

External

General medicine

NA

104

Peer-reviewed

Low

Low

58

Kikuchi

GPT-3.5, GPT-4

Free text

External

Radiology

NA

115

Peer-reviewed

Low

High

59

Bridges

GPT-4

Free text

External

General medicine

NA

201

Peer-reviewed

High

Low

60

Shieh

GPT-4

Free text

External

General medicine

NA

63

Peer-reviewed

High

Low

61

Warrier

PaLM2 (Bard),

Prometheus (Bing), GPT-3.5, GPT-4

Free text

Unknown

Otolaryngology

NA

100

Peer-reviewed

High

Low

62

Han

GPT-3.5, GPT-4,

GPT-4V, Gemini 1.0 Pro, Llama 2, Med-42

Choice

External

General medicine

NA

140, 348

Peer-reviewed

Low

High

63

Milad

GPT-4

Choice

Unknown

Ophthalmology

NA

422

Peer-reviewed

High

High

64

Abdullahi

PaLM2 (Bard),

GPT-3.5, GPT-4, MedAlpaca

Free text

Unknown

General medicine

NA

30

Peer-reviewed

High

High

65

Tenner

GPT-3.5

Free text

External

General medicine

NA

40

Peer-reviewed

High

Low

66

Luk

GPT-4

Free text

External

General medicine

NA

81

Peer-reviewed

High

Low

67

Savage

GPT-4

Free text

External

General medicine

NA

300

Peer-reviewed

Low

Low

68

Franc

GPT-3.5

Choice

Unknown

Emergency medicine

NA

61

Peer-reviewed

High

Low

69

Yang

GPT-3.5, GPT-4

Free text

Unknown

General medicine

NA

238

Peer-reviewed

High

Low

70

Reese

GPT-4

Free text

External

General medicine

NA

75

Preprint

High

Low

71

Olmo

Claude 3 Opus, Claude 3 Sonnet, GPT-4, Gemini 1.5 Pro, Llama 2, Llama 3 70B, Llama 3 8B, Mistral 7B, Mixtral8x22B, Mixtral8x7B

Free text

Unknown, External

General medicine

NA

200, 75

Preprint

Low

Low

72

Cesur

Claude 3 Opus, Claude 3 Sonnet, Claude 3.5 Sonnet, GPT-3.5, GPT-4, GPT-4o, Gemini 1.0, Gemini 1.5 Flash, Gemini 1.5 Pro, Llama 3 70B, Mistral Large, Perplexity

Free text

External

Radiology

Expert

80

Preprint

High

High

73

Schramm

GPT-4V

Free text

External

Neurology

NA

30

Preprint

High

Low

74

Gunes

PaLM2 (Bard),

Prometheus (Bing), GPT-3.5

Free text

External

Radiology

Expert

124

Preprint

Low

High

75

Olshaker

GPT-3.5, GPT-4,

Gemini Pro

Free text

Unknown

Radiology

Non-expert

60

Preprint

High

Low

76

Hirosawa

PaLM2 (Bard), GPT-4, Llama 2

Free text

External

General medicine

NA

392

Peer-reviewed

Low

Low

77

Mitsuyama

GPT-4

Free text

External

Radiology

Expert, Non-expert

150

Peer-reviewed

Low

Low

78

Yazaki

GPT-3.5, GPT-4

Choice

External

Emergency medicine

Non-expert

100

Peer-reviewed

Low

Low

79

Ghalibafan

GPT-4V

Free text

External

Ophthalmology

NA

143

Peer-reviewed

Low

Low

80

Hager

Clinical Camel, Llama 2, Meditron, Open Assistant, WizardLM

Free text

Unknown

Emergency medicine

Expert

80

Peer-reviewed

High

Low

81

Horiuchi

GPT-4, GPT-4V

Free text

External

Radiology

Expert, Non-expert

106

Peer-reviewed

Low

High

82

Ríos-Hoyo

GPT-3.5, GPT-4

Free text

External

General medicine

NA

75

Peer-reviewed

High

Low

83

Liu

Claude 3 Opus, GPT-4

Free text

External

Dermatology

NA

100

Peer-reviewed

High

Low

84

Sonoda

Claude 3 Opus, GPT-4o, Gemini 1.5 Pro

Free text

External

Radiology

NA

324

Peer-reviewed

Low

High

85

Wada

GPT-4

Free text

Unknown

Radiology

NA

751

Peer-reviewed

High

High

86

Gargari

Aya, GPT-3.5, GPT-4, Nemotron

Free text

Unknown

Psychiatry

NA

20

Peer-reviewed

High

Low

87

Mihalache

GPT-4

Free text

Unknown

Ophthalmology

NA

69

Peer-reviewed

High

High

88

Rutledge

GPT-4

Free text

Unknown

General medicine

NA

45

Peer-reviewed

High

Low

89

Ueda

GPT-4

Free text

External

General medicine

NA

62

Peer-reviewed

High

High

90

Delsoz

GPT-3.5, GPT-4

Free text

Unknown

Ophthalmology

Expert

20

Peer-reviewed

High

Low

91

Brin

GPT-4V

Free text

External

Radiology

NA

216

Peer-reviewed

Low

Low

92

Levine

GPT-3

Free text

External

General medicine

NA

48

Peer-reviewed

High

Low

93

Williams

GPT-3.5

Choice

External

Emergency medicine

Non-expert

500

Peer-reviewed

Low

Low

  1. ROB risk of bias.