Fig. 5: Attention heads in the model recognise neighbouring atoms in the crystal.

A Examples of pairwise attentions in the model identifying close contacts in the crystal structure. Attention is calculated between the reference atom and the remaining atoms. In each example, we have confirmed that the reference atom and the highest attention atom are within 3.5 Å in the crystal structure. B The mean attentions between neighbouring pairs and non-neighbouring pairs for each attention head in the model across four different crystal structures. Some heads in the model focused on neighbouring pairs up to 50% more compared to the uniform baseline; meanwhile, no head focuses on non-neighbouring pairs. This result arises without the model ever seeing a crystal structure during training. We defined pairs as neighbours if they were within 3.5 Å in the crystal. C An excerpt from one of the X-ray structures, highlighting the non-trivial packing identified by the model.