Table 3 Description of the considered validation metrics present in the benchmark. \(\Omega ^\pm \triangleq \{(i,j), A_{i,j} = \pm 1 \mid i \le N_S, j \le N_P\}\) is the set of all positive (\(\Omega ^+\)) or negative (\(\Omega ^-\)) drug-disease associations, whereas \(\Omega ^+_j \triangleq \{i \mid A_{i,j} = +1\}\) is the set of drugs involved in positive associations with disease j and \(\widetilde{\Omega }_j \triangleq \{(i,i') \mid A_{i,j} > A_{i',j}\}\) for any \(j \le N_P\) is the set of correctly ordered pairs of drugs for the score ranking in disease j. In the benchmark, \(t=0\) and \(\mathbbm {1}(C)\) is equal to 1 if C is satisfied, 0 otherwise. \(\sigma _{V}\) is the permutation that sorts all coefficients of any vector V of length n in decreasing order, that is, \(V_{\sigma _V(1)} \ge V_{\sigma _V(2)} \ge \dots \ge V_{\sigma _V(n)}\). The true positive rate is formally defined as \(\texttt {TPR}(t; \hat{R}, A) = \sum _{(i,j),A_{i,j}=+1} \mathbbm {1}(\hat{R}_{i,j}>t)/\sum _{(i,j)} \mathbbm {1}(\hat{R}_{i,j}>t)\) and \(\texttt {FPR}(t; \hat{R}, A) = \sum _{(i,j),A_{i,j}=-1} \mathbbm {1}(\hat{R}_{i,j}>t)/\sum _{(i,j)} \mathbbm {1}(\hat{R}_{i,j}\le t)\) is the false positive rate. Finally, \(N^{+,j}_S\) is defined as \(\min (N_S,|\Omega ^+_j|)\).
From: Comprehensive evaluation of pure and hybrid collaborative filtering in drug repurposing
Type | Metric | Notation | Formula |
|---|---|---|---|
Global | Accuracy | \(\texttt {Acc}(\hat{R},A;t)\) | \((|\Omega ^-|+|\Omega ^+|)^{-1} \sum _{(i,j) \in \Omega ^- \cup \Omega ^+} \mathbbm {1}((\hat{R}_{i,j}-t)A_{i,j} > 0)\) |
Area Under the Curve | \(\texttt {AUC}(\hat{R},A)\) | \(\int _0^1 \text {TPR}(\text {FPR}^{-1}(x; \hat{R}, A); \hat{R}, A)dx\) | |
Local | Average AUC | \(\texttt {AUC}_d(\hat{R},A)\) | \(N_P^{-1} \sum _{j \le N_P} \texttt {AUC}(\hat{R}[\cdot ,j], A[\cdot ,j])\) |
Average NS-AUC60 | \(\texttt {NS-AUC}(\hat{R},A)\) | \(|N_P|^{-1} \sum _{j \le N_P} |\widetilde{\Omega }_j|^{-1} \sum _{(i,i') \in \widetilde{\Omega }_j} \mathbbm {1}(\hat{R}_{i,j}>\hat{R}_{i',j})\) | |
Average NDCG@\(N_S\) | \(\texttt {NDCG}(\hat{R},A)\) | \(N_P^{-1} \sum _{j \le N_P} \left( \sum _{i=1}^{N^{+,j}_S} \frac{A_{\sigma _{\hat{R}_{\cdot ,j}}(i), j}}{\log _2(i+1)} \right) /\left( \sum _{i=1}^{N^{+,j}_S} \frac{1}{\log _2(i+1)} \right)\) |