Fig. 13: Illustration of an attention block from CrabNet⁹⁰.

a The input position-aware features (i.e., EDM) are first mapped to query (Q), key (K), and value (V) matrices by the corresponding fully connected layers. b Q and K are multiplied to give the interaction matrix. This matrix multiplication operation is inspired by the fact that dot product shows the distance between vectors. c The interaction matrix is scaled and normalized to give the self-attention matrix, which is then combined with V to generate the refined feature representation Z. Z is further refined by passing through a shallow feed-forward network, which is not shown here. Figure reprinted from ref. ⁹⁰ under the CC BY 4.0 license¹⁵⁶.