Table 3 Specification issues, granularity
Report text | IHC was performed on A2. Tumor cells are diffusely positive for CA-IX in a membranous pattern | |
Discordant labels | A_block_A2_IHC_CAIX: Positive, diffuse membranous | A_block_A2_IHC_CAIX: Positive, diffuse |
Context | - In our original schema, we attempted to provide a list of all possible IHC results to choose from. - After review, we found this to be entirely impractical as the space of possible test results became enormous. - We needed to precisely define the granularity of test results that we were interested in. | |
Addressing Action | - We shifted to a more modular schema comprising four dimensions—status, intensity, extent, and pattern—each with its own controlled vocabulary (see Fig. 1A for an example). - Under this new approach, the LLM is instructed to sequentially append any applicable modifiers (intensity, then extent, then pattern) to the primary status label, omitting those not present. | |
Continued error severity examples | - Major: Returning only “Positive” as in RCC, we are very interested in detailed CA-IX staining patterns. - Minor: If in the example report, CA-IX had additional describers/modifiers that we are not interested in, thus are not in the schema, and are then returned by the LLM. These additional modifiers would not be factually incorrect, but would be beyond the standardized level of detail that we desire. | |
