Fig. 4: Overview of the four different use-cases of CLOOME evaluated in this study.
From: CLOOME: contrastive learning unlocks bioimaging databases for queries with chemical structures
An adaptive image encoder hx(.) and an adaptive structure-encoder hz(.) map the microscopy images and chemical structures to their embeddings xn = hx(xn) and zn = hz(zn), respectively. a Multi-modal retrieval task using CLOOME image and molecule embeddings. The resulting embeddings can be used to rank chemical structures that induce similar phenotypic effects, and vice versa. b Using the CLOOME embeddings for activity prediction. A logistic regression model is trained for activity prediction tasks. c Zero-shot image-to-image classification task using CLOOME image embeddings for molecule prediction. A set of representative images, one for each molecule, are used to infer which compound was applied in a query image. d Zero-shot image-to-image classification task using CLOOME image embeddings for mechanism of action (MoA) prediction. In this case, the set of representative images depict each MoA. A query image is classified into the most likely MoA category based on its similarity with a corresponding representative image. Icons representing different MoAs were created with BioRender.com.
