Table 2 Human-authored vs LLM-generated exam question on the same topic, with the prompt used to generate the LLM item (Faculty of Medicine and University Hospital Cologne, University of Cologne; summer 2025)

From: Psychometric properties and detectability of GPT-4o–generated multiple-choice questions compared with human-authored items across imaging specialties

Factor

Examplea

Human-authored question

Which of the following best describes oncologic follow-up after radiation therapy?

(A) Regular follow-up visits for early detection of tumor recurrence and management of treatment-related adverse effects, as well as contributions to treatment quality assurance.

(B) Imaging studies to document tumor size.

(C) Evaluation of a reduced-dose radiotherapy regimen to prevent future recurrence.

(D) A one-time end-of-therapy examination performed for the statistical reporting of treatment results.

(E) Follow-up after radiotherapy includes documenting acute side effects occurring during the course of radiation treatment.

Correct answer: A.

Prompt used to generate LLM questionb,c

Create two multiple-choice questions, each with five answer options and one unambiguously correct answer. Each question should be answerable by both medical students and specialist physicians. The topic of the question is [learning objective]. The question should address [specification], although these terms do not need to appear in every answer option.

Ensure that the answer options are not mutually exclusive – that is, different options must not contain directly contradictory statements. Provide the correct answer at the end of each question.

In addition, avoid absolute wording and hidden cues, such as “none,” “always,” “exclusively,” or “all.” Ideally, keep the answer options approximately similar in length to avoid unintended cues.

Learning objective specified in prompt

Understanding the tasks and objectives of oncologic follow-up after radiotherapy.

Specification indicated in prompt

Follow-up care: content, objectives, and relevance to patient care and quality assurance.

LLM-generated question

What are the key objectives of follow-up care after radiation therapy?

(A) Verifying that radiation therapy was performed in accordance with technical standards

(B) Early detection of tumor recurrence and management of treatment-related adverse effects

(C) Collecting epidemiological data on the long-term effects of radiation exposure in the population

(D) Documenting the delivered radiation dose to meet statutory reporting requirements

(E) Coordinating billing procedures within oncology follow-up programs

Correct answer: B.

  1. LLM large language model, MCQ multiple-choice question.
  2. aThe human-authored items, LLM-generated items, and associated prompts were originally created in German. For Table 2, the examples were translated into English using DeepL (version 25.7.12287188; DeepL SE, Cologne, Germany). A native English-speaking expert subsequently reviewed the translations for accuracy and fluency without altering item content.
  3. bPrompts were entered into ChatGPT (GPT-4o-Plus; OpenAI, San Francisco, CA, model release April 25, 2025).
  4. cThe prompt instructs the model to produce two questions; for display purposes, one representative LLM-generated question is shown here.
Back to article page