Fig. 4: Tissue annotation using marker genes.

a, Schematic illustration of tissue annotation using H&E image and reference marker genes. The annotation result is decided by choosing the candidate texts with the highest similarity score to the input image query. For Loki, we used the text content of marker gene symbols of each tissue type. For the PLIP model, we used the text content of natural language description of each tissue type. b, Examples of similarity scores of images and texts calculated by Loki and OpenAI CLIP model, respectively. c, Comparison of zero-shot performances, represented by weighted F1 scores, across four datasets using Loki and OpenAI CLIP, respectively. Number of test samples for each dataset: CRC7K (n = 6,333); WSSS4LUAD (n = 10,091); LC25000 (n = 15,000); and PatchCamelyon (n = 32,768). d, Comparison of zero-shot performances, represented by weighted F1 scores, across four datasets using Loki, PLIP and incorporating Loki and PLIP models by average similarity (shown in a; Methods), respectively. e, Comparison of zero-shot performances, represented by weighted F1 scores of each tissue type in the CRC7K dataset using OpenAI CLIP model, Loki, PLIP model and incorporating Loki and PLIP models, respectively. f, Confusion matrix of the CRC7K dataset using Loki (left), PLIP model (middle) and incorporating Loki and PLIP models (right), respectively. The ground-truth labels are presented in rows and the predicted labels are presented in columns. ADI, adipose tissue; NOR, normal colon mucosa; TUM, colorectal carcinoma epithelium; LYM, lymphocytes; MUC, mucus; DEB, debris; MUS, smooth muscle; STR, cancer-associated stroma.