Fig. 2: Visualization of self-attention in one compound.
From: Compositionally restricted attention-based network for materials property predictions
Displayed are the four attention heads (a–d) from the first layer of a CrabNet model trained on mp_bulk_modulus and evaluated on the composition Al2O3. Each row represents an element in the system. Each column represents an element being attended to. Each element’s fractional amount is shown on the x-axis. The values in the attention matrix are scores representing element-element interactions for the compound. As an example, in head a, Al0.4 and O0.6 are attending strongly to each other, with attention scores of 1.00 between these two elements.
