Fig. 4: Anatomy of an evaluator prompt.
From: Current and future state of evaluation of large language models for medical summarization tasks
An evaluator prompt consists of three sections: prompt, information, and evaluation. All three components are essential for an LLM serving as an evaluator. The Evaluator Prompt needs to instruct the LLM on the task (Prompt), provide the LLM will all the necessary information to make an evaluation (Information), and all the information that defines the guidelines and formatting of the evaluation (Evaluation).
