Fig. 6: Schematic representation of CLOOME.
From: CLOOME: contrastive learning unlocks bioimaging databases for queries with chemical structures
Contrastive pre-training of embeddings of the two modalities, microscopy image and chemical structure, of a molecule using the CLOOB11 or CLIP10 approach. The training dataset consists of N pairs of microscopy images of molecule-perturbed cells and chemical structures of molecules {(x1, z1), …, (xN, zN)}. We assume that an adaptive image encoder hx(.) and an adaptive structure-encoder hz(.) are available that map the microscopy images and chemical structures to their embeddings xn = hx(xn) and zn = hz(zn), respectively. In the CLIP approach, these embeddings are directly used to compute the InfoNCE loss. In the CLOOB approach, image and structure embeddings are retrieved from stored image embeddings U and structure embeddings V, such that Ux denotes image-retrieved image embeddings, Uz structure-retrieved image embeddings, Vx image-retrieved structure embeddings and Vz structure-retrieved structure embeddings. In this case, the InfoLOOB loss is computed with the latter embeddings.
