Supplementary Figure 5: Evaluation of the performance of NicheNet in predicting the transcriptional response: comparison to randomized networks, NicheNet models with fewer data sources, and NicheNet without parameter optimization.

For 111 ligand treatment expression datasets, we calculated several classification evaluation metrics to assess how well NicheNet predicts the set of genes that are differentially expressed in cells upon stimulation with a particular ligand. The target gene prediction performance was compared between: 1) 100 models constructed from 100 randomized networks; 2) 280 incomplete unoptimized NicheNet one-vs-one-vs-one models for which ligand-target regulatory potential scores were calculated after using only one comprehensive ligand-receptor, one signaling, and one gene regulatory database; 3) NicheNet containing all data sources, but without parameter optimization; 4) NicheNet containing all data sources and with optimized parameters. For 1) and 2), we calculated the median performance for each ligand treatment dataset over respectively all 100 and 280 models. Each dot indicates the performance for one dataset, and the black line indicates the median performance over all datasets. Several classification evaluation metrics were calculated: AUC-iRegulon (corrected): area under the cumulative recovery curve (considering the top 3% of the ranking), corrected for random prediction; AUPR (corrected): area under the precision-recall curve, corrected for random prediction; AUROC: area under the receiver operating characteristic curve; Mean-rank gene-set enrichment: negative natural logarithm of the mean-rank gene-set enrichment p-value. Red dashed line: performance of random guessing.