Table 1 The LLM-RAG model prompt, an example clinical scenario (scenario 1), the expert panel’s proposed correct answer, and various LLM-RAG responses
Prompt for LLM-RAG model: | |
|---|---|
You are the anesthesiologist seeing this patient in the preoperative clinic 2 weeks before the date of operation. The patients have already taken their routine preoperative investigations and the findings are listed within the clinical summary. Your role is to evaluate the clinical summary and give the preoperative anesthesia instructions for the following patient targeted to your fellow medical colleagues. You are to follow strictly the department’s guidelines. Your instructions should consist of the following components: 1. Should the patient be seen by a Doctor or a Nurse—Doctor/Nurse 2. Fasting instructions - list instructions based on the number of hours before the time of the listed surgery 3. Suitability for preoperative carbohydrate loading—yes/no. 4. Medication instructions—name each medication and give the instructions for the day of the operation and days leading up to the operation as required. 5. Any instructions for the healthcare team—for example, preoperative blood group matching, arranging for preoperative dialysis, or standby post-operative high dependency/ICU beds. 6. Any preoperative optimization required for the patient—list what needs to be optimized. 7. Any need to delay the operation for further medical workup and preoperative optimization? 8. Any specific department protocols to follow for this patient - name as many as necessary, and give short reasoning for using these protocols. Your instructions are the final instructions, do not give uncertain answers. If the medical condition is already optimized, there is no need to offer further optimization. If there are no relevant instructions in any of the above categories, leave it blank and write NA. | |
Clinical scenario 1 | |
38/Chinese/Female Allergy to aspirin, paracetamol, penicillin - rashes and itchiness ExSmoker—smoked 10 years ago/Occasional Drinker LMP: last month Wt 94.7 Ht 166.3 BMI 34.2 BP 127/81 HR 88 SpO2 100% on RA Coming in for BILATERAL REVISION FESS, REVISION SEPTOPLASTY, ADENOIDECTOMY, AND BILATERAL INFERIOR TURBINOPLASTIES/SEVERE OSA ON CPAP === PAST MEDICAL HISTORY === 1. Severe OSA on CPAP—AHI 58—CPAP settings: AutoCPAP (4–15) cmH2O, without humidifier/Chinstrap 2. Right persistent Sinusitis 3. Allergic rhinitis 4. Adenoid hypertrophy 5. High BMI 6. Asthma—f/u GP, last seen 3 months ago for attack—on PRN ventolin—Does not use ventolin at all—No previous admissions/ intubations for asthma 7. Diabetes—HbA1C 9.4%, Last seen outpatient doctor >1 year ago. No history of HTN/ HLD/ IHD/ CVA === SURGICAL HISTORY === Tonsillectomy > 10 years ago mild PONV ===Investigations=== Hb 13.0 TW 4 Plt 392 INR PT APTT normal Na 134 K3.4 Cr 77 Glu 13 ECG NSR CXR NAD === MEDICATIONS=== Ventolin PRN LMP; Last menstrual period, Wt; Weight | |
Correct Answers: | |
Fasting instructions: | No food by mouth at least 6 h before the operation, and clear water up to 2 h before the operation. |
Preoperative carbohydrate loading: | No |
Medication instructions | Bring ventolin on the morning of the operation |
Instructions to the healthcare team (Perioperative instructions) | 1. List as morning case. 2. Do a hypocount on the morning of the operation. 3. Bring CPAP 4. Book Post operative HD |
Preoperative optimization required | 1. Refer to the internal medicine perioperative team (IMPT) for diabetes optimization |
Need to delay operation | Yes—Due to the need to improve diabetes control. |
Example: GPT4_international response | |
1. The patient should be seen by a Doctor due to her complex medical history including severe obstructive sleep apnea (OSA) on CPAP, high BMI, asthma, and diabetes. 2. Fasting instructions: The patient should fast for at least 6 h for solids and 2 h for clear fluids before the time of surgery. 3. The patient is not suitable for preoperative carbohydrate loading due to her diabetes. 4. Medication instructions: - Ventolin: Continue as needed. - Diabetes medications: The patient should consult with her endocrinologist for specific instructions regarding her diabetes medications in the days leading up to the operation. On the day of the operation, these medications are typically held. - CPAP: The patient should continue using her CPAP machine as prescribed up until the time of surgery. 5. Instructions for the healthcare team: - Arrange for post-operative high dependency/ICU beds due to the patient’s severe OSA. - Ensure the availability of the patient’s CPAP machine in the recovery area. - Monitor blood glucose levels closely in the perioperative period. 6. Preoperative optimization: - Optimize the patient’s blood glucose levels. Consultation with an endocrinologist may be necessary. - Ensure the patient’s asthma is well-controlled. 7. There is a need to delay the operation for further medical workup and preoperative optimization, particularly for her poorly controlled diabetes. | |
Example: Llama2-70b_local response (With hallucinations) | |
“Doctor/Nurse—Doctor Fasting instructions—6 h for solids, 2 h for clear fluids Suitability for preoperative carbohydrate loading—yes (wrong answer) Medication instructions—Ventolin PRN, continue as prescribed Any instructions for the healthcare team—None (wrong answer) Any preoperative optimization required for the patient—optimization of medical comorbidities, including iron-deficiency anemia (The patient does not have iron-deficiency anemia) Any need to delay the operation for further medical workup and preoperative optimization—No (wrong answer) | |
