Fig. 1: Multimodal fusion with relational learning for molecular property prediction (MMFRL).
From: Multimodal fusion with relational learning for molecular property prediction
This figure shows our proposed idea about how to transfer the knowledge from other modalities and use fusion to improve the performance further. Unlike the general contrastive learning framework shown in Supplementary Information Figure 1, MMFRL doesn’t need to define positive or negative pairs and is capable of learning continuous ordering from target similarity. In Early fusion, a single Initialized GNN is created by combining all modality information during pretraining. In Intermediate and Late Fusion, each modality has its own initialized GNN.
