Fig. 2: Metrics Reloaded recommendation framework from a user perspective.

In step 1 – problem fingerprinting, the given biomedical image analysis problem is mapped to the appropriate image problem category, namely ImLC, SemS, ObD or InS; Fig. 4). The problem category and further characteristics of the given biomedical problem relevant for metric selection are then captured in a problem fingerprint (Fig. 3). In step 2 – metric selection, the user follows the respective colored path of the chosen problem category (ImLC →, SemS →, ObD → or InS →) to select a suitable pool of metrics from the Metrics Reloaded pools shown in green. When a tree branches, the fingerprint items determine which exact path to take. Finally, in step 3 – metric application, the user is supported in applying the metrics to a given dataset. During the traversal of the decision tree, the user goes through subprocesses, indicated by the plus sign, which are provided in Extended Data Figs. 1–9 and represent relevant steps in the metric selection process. Ambiguities related to metric selection are resolved via decision guides (Supplementary Note 2.7) that help users make an educated decision when multiple options are possible. A comprehensive textual description of the recommendations for all four problem categories as well as for the selection of corresponding calibration metrics (if any) is provided in Supplementary Notes 2.2–2.6. An overview of the symbols used in the process diagram is provided in Fig. SN 5.1. Condensed versions of the mappings for every category can be found in Supplementary Note 2.2 for ImLC, Supplementary Note 2.3 for SemS, Supplementary Note 2.4 for ObD and Supplementary Note 2.5 for InS.