Table 2 Relevance of multiple labels

From: Iterative refinement and goal articulation to optimize large language models for clinical information extraction

Report text

A. Right kidney and adrenal gland, radical nephrectomy:

- Renal cell carcinoma, clear-cell type

- Adrenal gland, negative for malignancy

Discordant labels

A_anatomical-site: Kidney, right; Adrenal gland

A_anatomical-site: Kidney, right

Context

- The original instructions required listing all anatomical sites in the specimen, as some specimens have multiple anatomical sites.

- In the above report, the adrenal gland and kidney are anatomical sites in the same subpart; however only the kidney is positive for RCC.

- Ambiguity arose over whether to include both sites in the label for such contexts.

Addressing

Action

- It was decided that for our purposes, we wanted the “anatomical site” field to continue to capture the primary organs/tissues removed for a specimen with no carve-outs for histology. As such, in this case, we would rely on the diagnosis and histology fields to guide our understanding that this was NOT a case of adrenal metastasis.

Continued Error Severity Examples

- Major: An anatomical site of only “Adrenal gland”, omitting the more important site.

- Minor: An anatomical site of only “Right kidney”. Although the adrenal gland is missing, because it is only benign tissue and not an RCC metastasis, its omission does not substantially affect the planned downstream analysis.