Fig. 3: The Nucleotide Transformer models acquired knowledge about genomic elements.
From: Nucleotide Transformer: building and evaluating robust foundation models for human genomics
a, t-SNE projections of embeddings of five genomic elements from layer 1, 5 and 21 based on the Multispecies 2.5B model. b, Accuracy estimates based on probing to classify five genomic elements across layers. c, Schematic describing the evaluation of attention levels at a given genomic element. d, Attention percentages per head and layer across the Multispecies 2.5B model computed on 5′ UTR, exon, enhancer and promoter regions. Barplot on the right of each tile plot shows the maximum attention percentage across all heads for a given layer.
