Expert assignment system based on natural language processing for Marie Sklodowska-Curie actions

Abstract

Assigning experts to project proposals is a critical process in research evaluation. Traditional Information Retrieval (IR) methods, such as the Single Evaluation Platform (SEP) used by the European Research Executive Agency, automatically assign experts based on keyword matching, but these assignments are subsequently reviewed and corrected by Vice Chairs (VCs) to ensure suitability. To address the limitations of keyword-based systems and enhance semantic relevance, we developed a novel expert assignment system leveraging Natural Language Processing with Large Language Models (LLMs). Our approach integrates dynamic retrieval of expert publications via ORCID with GALACTICA, a specialized scientific LLM, to compute fine-grained semantic similarity between publications and proposal abstracts. Using a dataset of 48 experts and 181 proposals, we evaluated three similarity aggregation strategies: Sum, Product, and Maximum. The Maximum similarity approach most closely replicated VCs-reviewed assignments, achieving an AUC of 0.82, significantly outperforming the traditional SEP system (AUC = 0.75), Sum (AUC = 0.69), and Product (AUC = 0.57). These results demonstrate that focusing on the single most relevant match effectively captures human decision-making, highlighting the potential of LLM-based semantic matching to provide a more accurate and scalable alternative to existing IR systems. Furthermore, unlike SEP’s discrete affinity scores, our aggregation strategies produce highly discriminative, fine-grained ratings, allowing for more nuanced differentiation among candidate experts.