Fig. 5: Exemplary LLM query and analysis (Eurorad case ID 12746).
From: Benchmarking the diagnostic performance of open source LLMs in 1933 Eurorad case reports
Mistral-Small was instructed to determine the most likely differential diagnoses based on a textual case description that included a condensed medical history and imaging findings. The LLM output contained three diagnoses with a rationale for each suggestion. Llama-3-70B was subsequently utilized to classify the response as correct (if the true diagnosis is included in the suggestions) or wrong. The true diagnosis in this case was ‘jugular foramen meningioma’.
