Fig. 3: Validation of embeddings.
From: Semantic embeddings reveal and address taxonomic incommensurability in psychological measurement
a, The correlation between the empirical (α) and the predicted \(\hat{\alpha}\) internal consistency concerning in-sample (blue) and out-of-sample (yellow) predictions from our fine-tuned model for 449 scales. b, The structural fidelity (defined as the z-score) is validated, contrasting the similarity of items within a given scale to items of others scales within the same inventory for all inventories with at least 3 scales (N = 433). In addition, it contrasts the structural fidelity that embeddings achieve for five empirical datasets (NEO, HEXACO, FFM, BIG5 and 16PF) relative to the observed structural fidelity (represented as points) obtained from large-scale empirical assessment. c,d, The mean error (c) and correlation (d) between the predicted and observed scale correlations for the same five datasets is shown. e, A visual comparison of the observed correlation among the 30 scales of the IPIP-NEO-300 in 1 empirical dataset and the in-sample and out-of-sample predictions based on the embeddings. The correlations are ordered along to the personality factors Openness (O), Conscientiousness (C), Extraversion (E), Agreeableness (A) and Neuroticism (N). f, Illustration of our construct label evaluation. The points show the average alignment score as defined in the text for embeddings of the labels of distinct constructs present in IPIP, contextualized labels, APA definitions, curated APA definitions and definitions generated by GPT-4.
