Introduction

Human beings have more than 20,000 genes, which form a redundant and highly fault-tolerant system. Among these genes, some are vital for the survival and reproduction of us, but others are not. These two groups of genes are named as essential genes and nonessential genes. Essential genes are a group of fundamental genes necessary for a specific organism to survive in a particular environment. Cell essential genes refer to a subset of genes that are indispensable for the viability of individual human cell types1,2 as opposed to genes required for the survival of a multicellular organism. Here we focus on predicting cell essential genes. These cell essential genes encode conservative functional elements which mainly contribute to DNA replication, gene translation, gene transcription, and substance transportation. The identification and analysis of essential genes is very important for understanding the minimal requirements of basic life, and it’s vital for drug-target identification, synthetic biology, and cancer research.

There are two ways to identify essential genes: wet lab experimental methods and computational approaches. For example, gene direct deletion and transposon-based randomized mutagenesis have been used to identify essential genes for bacteria and yeast in the genome scale3; microinject KO and nuclear transfer techniques have been used to identify essential genes in mice4; the CRISPR/Cas9 genome editing system has been used to identify essential genes from human cell lines1,2,5. Experimental methods are often costly, time-consuming, and laborious. The accumulation of essential gene datasets, gene and protein sequence data, as well as multiple functional data, enables researchers to explore the relationships between gene essentiality and different omics data and to develop effective models to predict gene essentiality. These computational methods can greatly reduce the cost and time involved in finding essential genes which further boosts our understanding of basic life and human diseases and helps to find new drug targets and develop drugs efficiently.

Computational methods can be classified into two categories. One focuses on designing a centrality measure to rank proteins/genes, while the other focuses on integrating multiple features using machine learning to predict gene essentiality. The most widely used centrality measures include degree centrality, betweenness centrality, closeness centrality, and eigenvector centrality, to name a few. These centrality measures have been found to have a relationship with gene essentiality in multiple model organisms and humans6,7. However, they can only differentiate a subset of essential genes from nonessential genes. A cause can be the incompleteness and false positive/false negative interactions in the protein-protein interaction (PPI) networks. Another reason might be the fact that gene essentiality relates to multiple biological factors rather than just topological characteristics. Due to these motivations, researchers have proposed more complicated centrality measures by integrating network topological properties with other biological information. For example, Zhang et al. proposed a method, CoEWC, to capture the common properties of essential genes in both date hubs and party hubs by integrating network topological properties with gene expression profiles, which showed significant improvement in prediction ability compared to those only based on the PPI networks6. An ensemble framework was also proposed based on gene expression data and the PPI network, which can greatly enhance the prediction power of commonly used centrality measures8. Luo et al. proposed a method, LIDC, to predict essential proteins by combining local interaction density with protein complexes9. Zhang et al. proposed OGN by integrating network topological properties, gene expression profiles, and orthologous information10. GOS was proposed by Li et al. by integrating gene expression, orthologous information, subcellular localization, and PPI networks11. These proposed integrated centrality measures have improved prediction power over the ones that are only based on PPI network topological properties. However, they still have limited prediction accuracy since gene essentiality relates to many complicated factors that are impossible to represent by a scalar score. At this point, machine learning is a good choice to fully utilize multiple features for predicting essential genes.

Many machine learning models and deep learning frameworks have been proposed and successfully applied in different fields. For example, a density-based neural network was used for pavement distress image classification12; a convolutional neural network (CNN) was used for drug-target prediction13; active learning and transductive k-nearest neighbors were used for text categorization14,15; Support Vector Machines (SVM)16 and graph attention networks (GAT)17 were used to predict essential genes. In the research field of essential gene prediction, many machine learning-based prediction methods have been proposed to integrate features from multiple omics data18. As shown in the review article by Zhang et al.18, traditional machine learning methods were used to predict gene essentiality, and most of them were evaluated on data from model organisms. In these methods, topological features together with features from sequence and other functional genomics data were extracted manually and then used to train the models. The approach to extracting informative features is very important and challenging, as it requires ample domain knowledge as well as the understanding of the relationship between gene essentiality and each omics data. Usually, we only know that an omics data would contribute to gene essentiality, but we are not sure which attributes of it have such an effect and how to represent it. This puts a limitation on traditional machine learning methods to obtain good prediction accuracy as they require manual feature engineering.

In recent years, deep learning techniques have been used to automatically extract features and to train a more powerful classification model for predicting essential genes. Grover et al. proposed a network embedding method based on deep learning, node2vec, to learn a low-dimensional representation for each node19. This method has been used to extract topological features from PPI networks for predicting essential genes, and these features are more informative than those obtained via some popular centrality measures7,20,21,22. CNN was used to extract local patterns from time-serial gene expression profiles from S. cerevisiae20 and Zeng et al. also used bidirectional long short-term memory (LSTM) cells to extract features from the same gene expression profiles21. The automatically learned features are combined with other manually extracted ones to train a deep learning model for human essential gene prediction7. A six-hidden-layer neural network was designed to predict essential genes in microbes by only using manually extracted features from sequence data23. Li et al. proposed a deep learning method for predicting cell line-specific essential genes based on sequence data24, which integrates a CNN, bidirectional LSTM, and a multi-head self-attention mechanism together, expecting to learn short- and long-range information from protein sequence and provide residue-level model interpretability. Since sequence data is not cell line-specific, the model can only be trained and tested for each cell line separately. Yue et al. proposed a deep learning model for predicting essential proteins by integrating the PPI network, subcellular localization, and gene expression profiles together, which is evaluated on the data of Saccharomyces cerevisiae25.

Recently, human essential genes have been identified in several human cancer cell lines by utilizing CRISPR-Cas9 and gene-trap technology1,2,26. These identified essential genes provide a clear definition of the requirements for sustaining the basic cell activities of individual human tumor cell types, and can be regarded as targets for cancer treatment. These essential gene datasets together with other available biological data sources enable us to test an important and interesting assumption that human gene essentiality can be accurately predicted using computational methods. In this paper, we propose a Deep learning-based Essential Protein prediction method using a novel snapshot ensemble mechanism, DeEPsnap, to predict human essential genes. DeEPsnap integrates features from five omics data, including features derived from nucleotide sequence and protein sequence data, features learned from the PPI network, features encoded using gene ontology (GO) enrichment scores, features from protein complexes, and features from protein domain data. The proposed snapshot ensemble mechanism is inspired by the work of Huang et al.27, which shows that cyclic learning rates can be effective for training CNNs. In this paper, we propose a new cyclic learning method for our essential gene prediction problem. We also show that DeEPsnap can predict human essential genes with high accuracy. The main contributions of this paper include: (1) extract useful features from multi-omics data and integrate them for predicting human gene essentiality. (2) propose a new snapshot ensemble mechanism to improve the prediction performance. (3) showcase the usefulness and effectiveness of features extracted from five omics data to gene essentiality prediction and the contributions of each.

Snapshot ensemble deep learning model

DeEPsnap consists of three main modules: input data module, feature extraction and feature learning module, as well as classification module. The flowchart of DeEPsnap is shown in Fig. 1. Using node2vec to learn the network embedding features from PPI data is very time-consuming. In order to improve the training efficiency, features from each omics data were first extracted or learned before training the classification model. According to the definition of end-to-end learning in AI, DeEPsnap is not an end-to-end learning model.

Fig. 1
figure 1

The Flowchart of DeEPsnap.

Full size image

Input data module

In this paper, we mainly consider five omics data to explore their relevance and efficiency for predicting gene essentiality. As shown in Fig. 1, the input data module includes five biological data sources, which are sequence data (DNA and protein sequence), GO, PPI network, protein complex, and protein domain.

Feature extraction and feature learning module

The feature extraction and feature learning module uses different methods to extract or learn features from multiple omics data, as detailed below. Then the features are concatenated together as the input to the classification module. In this paper, we considered five types of features. More types of features can be easily integrated into our model to potentially further improve the prediction accuracy.

GO enrichment score encoded features

Features encoded with GO enrichment scores are calculated as follows. We choose the first 100 GO terms from the cellular component (CC) subcategory to encode the genes, where CC terms are ranked in descending order based on the number of essential genes involved in the terms. For each gene, we first obtain its direct neighbors from the PPI network to form a gene set consisting of itself and its neighbors, then perform GO enrichment analysis for this gene set against the 100 GO terms using a hypergeometric test. The enrichment score is calculated as \(-log_{10}\) (p-value) for each GO term. In this way, we get a 100-dimension feature vector for each gene. The GO enrichment score encoded features capture information from both the PPI network and subcellular localization of genes.

Features derived from the protein complex data

From protein complexes, we extracted two features for each gene. The first feature is the number of protein complexes the gene is involved in. The second feature is calculated as follows: For a gene, we first get the gene set N consisting of its direct neighbors in a PPI network. Suppose there are M neighbors of this gene involved in a protein complex. We calculate a score s = \(\frac{|M|}{|N|}\) as the ratio of its neighbors being involved in a protein complex. The second feature is the sum of the ratios across all the protein complexes. Therefore, the second feature also considers the information from the PPI network in addition to that from the protein complex data.

Features learned from the PPI network

Network features are learned based on a network embedding method: node2vec19. Each gene is represented by a 64-dimension feature vector learned from the PPI network. Previous studies showed that this low-dimension representation learned using node2vec is superior to the features calculated by popular centrality measures7,20,21.

Features derived from the sequence data

Sequence features consist of codon frequency, maximum relative synonymous codon usage (RSCUmax), codon adaptation index (CAI), gene length, GC content, amino acid frequency, and protein sequence length. There are 89 sequence features in total. For more details about how these features are calculated, please refer to Deephe7.

Features derived from the protein domain data

From the protein domain data, we extract three features for each gene/protein. The first feature is the number of domain types the protein has. The second feature is the number of unique domain types the protein has. The third feature is the sum of its inverse domain frequency (IDF). The frequency f of a domain is the number of proteins that have this domain. Its IDF is \(\frac{1}{f}\). Suppose a protein u has n domains, then the third feature, SIDF (sum of inverse domain frequency), of u is calculated as in Eq. (1).

$$\begin{aligned} SIDF(u)&= \sum _{i=1}^{n} \frac{1}{f_i} \end{aligned}$$
(1)

Classification module

Baseline model

The classification module of DeEPsnap is based on a snapshot ensemble deep neural network. The baseline model here is a multilayer perceptron enhanced by several deep-learning techniques, and we name it DNN. It includes one input layer, three hidden layers, and one output layer. We use the rectified linear unit (ReLU) as the activation function for all the hidden layers, while the output layer uses the sigmoid activation function to perform discrete classification. The loss function in DeEPsnap is binary cross-entropy, as shown in Eq. (2), where N is the number of samples, \(y_i\) is the label (in our case, either 0 or 1), and \(p_i\) is the predicted probability of the label being 1. Each hidden layer is followed by a dropout layer to make the network less sensitive to noise in the training data and increase its generalization ability. To address the imbalanced learning issue inherent in the essential gene prediction problem, we utilize class weight to train a weighted neural network, which gives larger penalties when the model misclassifies an instance from the minority class. The use of class weight encourages the model to pay more attention to instances from the minority class than those from the majority class, leading to a more balanced and effective classifier.

$$\begin{aligned} loss&= \frac{1}{N} \sum _{i=1}^{N} - (y_i \times ln(p_i) + (1 - y_i) \times ln(1 - p_i)) \end{aligned}$$
(2)

Cyclic sine annealing

In machine learning, ensemble methods utilize multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone. Likewise, ensembles of deep neural networks are known to be much more robust and accurate than individual networks. However, training multiple deep neural networks is computationally expensive. In this paper, we propose a snapshot ensemble mechanism to obtain multiple trained models without incurring extra training costs. The snapshot ensemble mechanism creates an ensemble of accurate and diverse models from a single training process. We expect such an optimization process which will visit several local minima before converging to a final solution. Model snapshots at these various minima are taken and their predictions are averaged at the test time.

Fig. 2
figure 2

Learning rate cycles for the snapshot ensemble models.

Full size image
Table 1 Parameters of DeEPsnap.
Full size table

To converge to multiple local minima, we propose a cyclic annealing scheduler based on the sine function. The learning rate is decreasing at a very fast pace, encouraging the model to converge towards its first local minimum after as few as 5 epochs in the DeEPsnap training process. The optimization is then continued at a larger learning rate in order to perturb the model and dislodge it from the current local minimum. This process is repeated several times to obtain multiple convergences. Formally, the learning rate lr(t) has the form shown in Eq. (3), where t is the current training epoch number (0-based), C is the total number of epochs to be included in an annealing cycle, \(lr_0\) is the initial learning rate at the beginning of each cycle, while \(\%\) represents the modulo operation. If the number of the total training epochs is set as T, then the training process is split into \(M = \left\lfloor \frac{T}{C} \right\rfloor\) cycles. If T is not divisible by C, the remaining epochs will still be trained but will not contribute to the final ensemble model. Each cycle starts with a larger learning rate for exploration and faster convergence, which is annealed gradually to a smaller learning rate for more accurate positioning of the local minima. Finally, the M snapshot models will be saved at the end of their learning rate annealing cycles and used to make predictions during the testing stage. Figure 2 illustrates the learning rate cycles in DeEPsnap for the first 3 of the M snapshot models. In each learning rate cycle, the learning rate starts from 0.001 and anneals towards 0, but never reaches 0. The snapshot models, the time points at which they are saved are denoted by the dotted lines, form an ensemble at the end of training. The parameters of DeEPsnap in our experiments are shown in Table 1.

$$\begin{aligned} lr(t) = (sin((1 + \frac{t \% C}{2C}) \pi ) + 1) \times lr_0 \end{aligned}$$
(3)

Results and discussion

Datasets

Human essential genes are downloaded from the DEG database28. There are 16 human essential gene datasets. We chose the genes that are contained in at least 5 of the datasets to be included in our essential gene dataset. Excluding the genes annotated as essential genes in any of the datasets in DEG, all other genes are considered nonessential genes.

The DNA sequence data and protein sequence data are downloaded from Ensembl29 (release 97, July 2019). We downloaded the PPI data from BioGRID30 (release 3.5.181, February 2020). Only physical interactions between human genes are used. After filtering out self-interactions and several small subgraphs, we obtain a PPI network with 17,762 nodes and 355,647 edges. This interaction network is used to learn embedding features for each gene. It also aids in computing some features from GO, protein complex, and protein domain.

GO data are downloaded from the Gene Ontology website31,32 and protein complex data are downloaded from CORUM33. Protein domain data are from the Pfam database34, and we collect this data via the Ensembl BioMart29. The genes having sequence features, network embedding features as well as GO enrichment scores are used for the following classification performance evaluation. In total, 2009 essential genes and 8414 nonessential genes are used for the following analysis.

Evaluation metrics

We use multiple metrics to evaluate the performance of DeEPsnap. The first metric is the area under the receiver operating characteristic (ROC) curve (AUROC). ROC plots represent the trade-off between sensitivity and specificity for all possible thresholds. The second metric is the area under the precision-recall curve (AUPRC). Precision-recall (PR) curves summarize the trade-off between the true positive rate and the positive predictive value using different probability thresholds. ROC curves are appropriate for balanced classification problems while PR curves are more appropriate for imbalanced datasets. Since essential gene prediction here is an imbalanced classification problem, the AUPRC metric is considered more important than AUROC. We further looked at the Matthews correlation coefficient (MCC) and F1 measure. In addition to these four comprehensive metrics, we also give the comparison results in terms of accuracy. The definitions of MCC, accuracy, and F1 are presented in Eqs. (4)–(6), where TP, TN, FP, and FN are the number of true positives, true negatives, false positives, and false negatives, respectively.

$$\begin{aligned} MCC&= \frac{TP \times TN - FP \times FN}{\sqrt{(TP + FP) \times (TP + FN) \times (TN + FP) \times (TN +FN)}} \end{aligned}$$
(4)
$$\begin{aligned} Accuracy&= \frac{TP + TN}{TP + FN + TN + FP} \end{aligned}$$
(5)
$$\begin{aligned} F1&= \frac{2 \times precision \times recall}{precision + recall} = \frac{TP}{TP + \frac{FP + FN}{2}} \end{aligned}$$
(6)

Performance evaluation

In the following experiments, we use the parameters for DeEPsnap as shown in Table 1. In order to cope with the imbalanced learning issue, we set the class weight to 4.5 for essential genes and 1 for nonessential genes. The stratified randomized 10-fold cross-validation is used to evaluate the performance of DeEPsnap. At each fold, \(10\%\) of the data are held out for testing, and the other \(90\%\) are used for training.

Fig. 3
figure 3

The ROC curves of DeEPsnap.

Full size image

Figure 3 presents the ROC curve of the DeEPsnap across the 10-fold cross-validation. From Fig. 3, we can see that DeEPsnap reaches its best performance at folds 3, 4, 5, and 7, with AUROC = 0.97. The average AUROC of DeEPsnap across the 10-fold cross-validation is \(96.16\%\) with a standard deviation STD = \(0.59\%\), and the average AUPRC is \(93.83\%\) with STD = \(0.83\%\). In addition, the performance of DeEPsnap is quite stable since the difference is less than \(2.12\%\) between its best and worst AUROC scores across the 10-fold cross-validation. The worst AUPRC score is still above \(92.39\%\), indicating that DeEPsnap is very effective for predicting human essential genes. In addition to good scores of AUROC and AUPRC, its average accuracy, MCC, and F1 scores are \(92.36\%\), \(75.92\%\), and \(80.62\%\) respectively (Table 2).

Table 2 Performance comparison of DeEPsnap, DNN, DeepHE, SVM, RF, Adaboost, and GAT.
Full size table

Performance comparison with other machine learning models

To demonstrate the superiority of DeEPsnap, we also compare it with three popular traditional machine learning models-SVM, Random Forest (RF), and Adaboost-as well as a recent deep learning-based essential gene prediction model DeepHE, and two deep learning models (GAT and DNN). For a fair comparison, the three traditional machine learning models and DNN use the same input features that we created for DeEPsnap, so the only difference here is the classification method. GAT is based on the implementation and parameter settings in EPGAT17, while we use an early-stopping mechanism (patience = 200 epochs, \(10\%\) of the training data are held for validation) to save the best model for the testing stage and set the maximum epochs = 3000. Since GAT is a graph neural network that can capture the structure information of input graphs, we use the PPI network as the input graph and the features extracted from the other four omics data as node features. DNN is the baseline model used in DeEPsnap, while it uses a constant learning rate of 0.001 and an early stopping mechanism with patience of 15 epochs (\(10\%\) of the training data are used as validation data). For DeepHE, we use its original features and model structures7.

Table 2 shows the performance comparison between DeEPsnap and the other compared models across 10-fold cross-validation. From Table 2, we can see that the AUROC scores of all the compared models are above \(91\%\), which indicates that the features extracted/learned from the five omics data are very effective for predicting human gene essentiality. DeEPsnap outperforms all the compared models, especially DNN, for all the five measures, which tells us that the proposed snapshot ensemble mechanism is a useful technique for improving model performance without incurring extra training costs. The AUPRC scores of the three DNN-based models (DeEPsnap, DeepHE, DNN) are higher than those of the other compared models, which indicates that multilayer perceptron enhanced with deep learning techniques coupled with class weight can be more effective for coping with imbalanced learning problems. The comparison experiments also show that the training times of all the compared models, except GAT, are comparable-less than 16 minutes for a 10-fold cross-validation on a laptop using CPUs. However, GAT needs more than 9 hours in the same setting, which is 35 times longer than the time needed for DeEPsnap. While DeEPsnap only needs 50 epochs in training for each fold, GAT needs about 2500 epochs (the number of epochs needed for each fold varies from 1500 to 3000 epochs across the 10-fold cross-validation).

Ablation study

In order to comprehend the contribution of each type of feature used in DeEPsnap, we also evaluate it by removing one type of feature each time. The experiments use the same settings except for the input features. Table 3 gives the performance comparison results of DeEPsnap with different types of input features, which reveals that DeEPsnap with the integration of all the five types of features performs best which further confirms the contribution and complementarity of these five types of features. By eliminating one type of feature at a time, we can assess how each feature type contributes to the overall performance and how well it complements the other feature types.

Table 3 Performance comparison of DeEPsnap with different types of features. S: sequence features; N: network embedding features; G: gene ontology features; C: protein complex features; D: protein domain features.
Full size table

From Table 3, we can see that the combination of N + G + C + D performs worse which might be due to the fact that three of the feature types (i.e. N, G, C) in this combination all utilize the PPI network topological information so that they have less complement effect. The other four combinations with sequence features perform only slightly worse than using all five types of features, which reveals that sequence features are highly complementary to the other types of features. As shown in Deephe7, when only using one type of feature, a deep learning model using network embedding features performs better than that using sequence features. Therefore, the above phenomenon doesn’t indicate that sequence features are superior to network features, but that sequence features are more complementary to the other three types of features.

Enrichment analysis for essential genes

The list of essential genes was submitted to Enrichr35 for analysis for GO Biological processes31,32 and Reactome pathway36 enrichment. The genes were highly statistically enriched for essential macromolecular biosynthetic processes, as shown in Fig. 4. These processes include translation, gene expression, ribosome biogenesis, mRNA processing, and more. The pathways enriched were similar to the GO term analysis with the exception that there is higher enrichment for cell cycle genes which is clearly an essential process. From this analysis, we conclude that our algorithm is selecting the types of genes and processes that would be expected for essential genes. This further confirms the fidelity of our method.

Fig. 4
figure 4

Biological process and pathway enrichment of the essential genes. (a) Bar plot of the top enriched BP terms; (b) Bar plot of the top enriched pathways in Reactome.

Full size image

Conclusions and future directions

In this paper, we propose a snapshot ensemble deep learning method, DeEPsnap, to predict human essential genes. DeEPsnap integrates five types of features extracted/learned from sequence and functional genomics data. It utilizes multiple deep-learning techniques and a cyclic annealing mechanism to train an ensemble of cost-sensitive classifiers to enhance the prediction accuracy of gene essentiality. Our 10-fold cross-validation experiments demonstrate: (1) the proposed snapshot ensemble deep learning method, DeEPsnap, is superior to the traditional machine learning models and some deep learning models, which is more effective for predicting human essential genes; (2) the extracted features from the five omics data are effective and complementary to gene essentiality prediction; (3) the proposed snapshot ensemble mechanism is promising for improving a model’s prediction performance without incurring extra cost.

In the future, we are interested in how we can use deep learning to automatically learn features from different types of biological data as well as to explore the appropriate knowledge representations for different omics data. For example, learning a low-dimensional representation using all GO terms to encode genes instead of the subset of selected GO terms. It’s especially interesting to explore novel feature learning methods to extract more informative representation features from protein complex and protein domain data. In addition, exploring and integrating more biological data into the learning and classification model is another interesting direction, such as epigenomics data and gene expression profiles. Predicting cancer cell line-specific and tissue-specific essential genes by designing effective deep learning models and integrating cell line-specific and tissue-specific information would be very interesting, especially the prediction of essential genes across cell lines and tissues via useful transfer learning techniques. We are also interested in testing whether data editing37 and clustering-aided techniques38 are useful for imbalanced learning problems. Using cross-attention mechanisms to explore the interactions between different omics data so as to learn more informative features would be another interesting research direction.