Topological identification and interpretation for single-cell epigenetic regulation elucidation in multi-tasks using scAGDE

Abstract

Single-cell ATAC-seq technology advances our understanding of single-cell heterogeneity in gene regulation by enabling exploration of epigenetic landscapes and regulatory elements. However, low sequencing depth per cell leads to data sparsity and high dimensionality, limiting the characterization of gene regulatory elements. Here, we develop scAGDE, a single-cell chromatin accessibility model-based deep graph representation learning method that simultaneously learns representation and clustering through explicit modeling of data generation. Our evaluations demonstrated that scAGDE outperforms existing methods in cell segregation, key marker identification, and visualization across diverse datasets while mitigating dropout events and unveiling hidden chromatin-accessible regions. We find that scAGDE preferentially identifies enhancer-like regions and elucidates complex regulatory landscapes, pinpointing putative enhancers regulating the constitutive expression of CTLA4 and the transcriptional dynamics of CD8A in immune cells. When applied to human brain tissue, scAGDE successfully annotated cis-regulatory element-specified cell types and revealed functional diversity and regulatory mechanisms of glutamatergic neurons.

Introduction

Gene expression is regulated by millions of active DNA regulatory elements exposed in accessible chromatin regions of the genome¹. The technique of Single-cell Assay for Transposase-Accessible Chromatin using sequencing (scATAC-seq) enables the exploration of the chromatin accessibility landscape via the hyperactive Tn5 transposase². Consequently, scATAC-seq has provided notable insights into cellular heterogeneity by profiling chromatin accessibility beyond transcriptome measurements, shedding light on epigenetic regulatory mechanisms at the single-cell level³. However, during whole-genome profiling, the capture rate of open chromatin sites is relatively low, leading to high dimensionality and sparsity in scATAC-seq data, which poses a significant analytical challenge. This sparsity obscures important genomic information and complicates downstream analyses, including peak calling and motif discovery. Furthermore, the inherent limitations in DNA copy numbers exacerbate the binary nature of the data, making it difficult to distinguish signal from noise. These issues collectively create substantial challenges for computational analysis, and the rapidly expanding scATAC-seq datasets⁴ highlight the urgent need for better and more robust analytical tools.

Accurate identification of cell types from single cell resolution ATAC-seq data is fundamental in understanding gene regulation and epigenetic heterogeneity within complex tissues. To date, most scATAC-seq studies have directly applied clustering algorithms that were developed for single-cell RNA sequencing (scRNA-seq) data to identify cell types^5,6,7. However, their performance may vary when directly applied to scATAC-seq data, due to distinct differences in data sparsity and distribution characteristics. In recent years, computational methods have been aimed at transforming data into a low-dimensional space while preserving the relevant structure of the data; for example, scABC⁸, cisTopic⁹, SnapATAC¹⁰, chromVAR¹¹, and Cicero¹² all compute a low-dimensional embedding for each individual cell. However, the performance of these methods is surpassed by subsequent deep learning-based models, including SCALE¹³, SAILER¹⁴, and PeakVI¹⁵, which have demonstrated their ability to perform nonlinear transformations and create low-dimensional embeddings by automatically, effectively extracting key information from the scATAC-seq data. Although these models are continually complicating the variational autoencoder (VAE) architecture to improve the ability to model scATAC-seq data, they have always been based on an autoencoder approach that inevitably ignores complex cell relationships of the global cell population.

Beyond cell type identification, scATAC-seq research aims to reveal gene regulatory networks and epigenetic heterogeneity by providing deep insights into single-cell chromatin accessibility data through precise peak selection and imputation. However, most existing work ignores this or directly employs methods designed for gene selection in scRNA-seq data analysis, such as Scanpy⁶. To better accommodate scATAC-seq data, EpiScanpy¹⁶ was introduced as a toolkit for the analysis of single cell epigenomic data, which performs peak selection based on fixed predefined statistical indicators that do not vary depending on the dataset. Another critical task is imputation, which aims to eliminate noise signals and recover true chromatin accessibility landscapes, thus enhancing downstream analyses, including the identification of regulatory elements and transcription factor motifs. Currently, SCALE is one of the few methods specifically designed to address the unique imputation challenges in scATAC-seq data¹³. However, SCALE and similar methods may not fully capture the intricate cell-cell interactions and topological structure present in chromatin accessibility landscapes. Emerging graph neural networks (GNNs) have demonstrated a natural ability to capture structural information from graphs by propagating neighbor information¹⁷. These networks typically integrate cell graphs with deep autoencoders to uncover compact latent representations, thereby exploring both rich content and structural information. Compared to traditional autoencoders used in scATAC-seq analysis^18,19, the incorporation of cell graphs and GNNs enables the learning of low-dimensional representations of graph topology, which enhances the modeling of cell-cell relationships, which are driven by complex mechanisms such as underlying cellular interactions or communication in the context of their microenvironments, offering a more comprehensive understanding of the structural and functional context of the data.

To this end, we propose scAGDE, a single-cell chromatin accessibility model-based deep graph embedded learning method designed to efficiently process sparse single-cell ATAC-seq data. scAGDE distinguishes itself from previous tools in four key aspects. First, scAGDE reconstructs both the chromatin accessibility profiles and the neighboring graph from the same low-dimensional cell representation. This design allows scAGDE to retain the chromatin accessibility profiles of the analyzed cells and their neighboring cells, as well as the cell-cell interaction relationships between them. In contrast, other methods do not construct graphs and merely analyze chromatin accessibility profiles, lacking the ability to learn relationships between neighboring cells. Second, scAGDE uses a chromatin accessibility-based autoencoder to measure the importance of the peaks and select the key peaks, enabling scAGDE to focus on highlighting the chromatin accessibility profiles of each analyzed cell or the most critical peak regions. Third, scAGDE employs a dual-decoder component that can reconstruct the cell topology and estimate the distribution of data. This ensures that scAGDE accurately models scATAC-seq data while retaining the relationship of each cell with its neighboring cell profiles in representation. Lastly, scAGDE defines a dual cluster optimization objective to guide the preservation of the information about cell heterogeneity in its representation. This explicit cluster optimization task is overlooked by other methods and prompts our model to focus on distinguishing cell heterogeneity. Experimental tests have demonstrated that scAGDE outperforms existing scATAC-seq analysis methods across multiple synthetic datasets derived from ATAC-seq data of bone marrow and several real-world datasets characterized by different degrees of sparsity, sequencing platforms, and species diversity. Moreover, scAGDE effectively supports dimensionality reduction, visualization, and dropout event correction. Specifically, through the imputation of a mouse forebrain dataset, scAGDE identified potential accessible peaks that contain insightful regulatory elements, including crucial transcription factor binding motifs. Extended analysis in a human brain dataset revealed that scAGDE could successfully annotate cis-regulatory element (CRE)-specified cell types while uncovering functional diversity within glutamatergic neurons.

Results

Methodology overview of scAGDE

The primary objective of scAGDE is to revolutionize the analysis of high-dimensional sparse scATAC-seq data by learning low-dimensional topological embedding representations. The summary of scAGDE is shown in Fig. 1a, which mainly consists of processing scATAC-seq data using a chromatin accessibility-based autoencoder and the graph embedding learning procedure. The proposed chromatin accessibility-based autoencoder is used to comprehensively learn the latent representation of the raw data matrix. This step involves selecting significant peaks as features for cell characterization and constructing a cell graph based on this information. In the next step, scAGDE integrates a Graph Convolutional Network (GCN) as an encoder, which extracts crucial information and considers cell-cell relationships in the cell graph, and then applies a Bernoulli-based decoder to model the probability of chromatin opening events. Additionally, scAGDE infers clustering assignments using a cluster layer as soft labels from latent embedding representations by optimizing two loss functions and incorporates a self-supervised learning mechanism to refine the representation learning process. The scAGDE produces the final topological embeddings, providing efficient and accurate representations of scATAC-seq data.

Fig. 1: Overview of the scAGDE framework.

a A summary graphical illustration of the scAGDE workflow. scAGDE takes the binary cell-by-peak matrix as input first into a chromatin accessibility-based autoencoder and then performs the graph embedding learning. b The chromatin accessibility-based autoencoder maps data into a latent space, where each individual cell connects its nearest cell as neighbours to construct a cell graph. The variation in the encoder’s weights can be translated into the importance score of peaks for the peak selection procedure. c The well-prepared cell graph and filtered data are simultaneously handled by a two-layer GCN encoder (i) and mapped into the latent space (ii). On the one hand, the latent embedding serves as input to the dual decoders (iii), which include a graph decoder module to reconstruct cell graph from embedding, and a Bernoulli-based decoder module to estimate the probability of a peak being accessible, which are estimates of the true chromatin landscape in each cell. On the other hand, the dual clustering optimizations are introduced into scAGDE (iv), where a network of cluster layer, which is initialized by K-means results on the embedding, infers soft clustering labels. The target distribution and one-hot pseudo label are sequentially calculated and used for label prediction loss and distribution alignment loss. d scAGDE facilitates critical downstream applications of clustering, visualization, imputation, enrichment analysis and discovery of regulators.

Specifically, the chromatin accessibility-based autoencoder takes a binary preprocessed scATAC-seq data matrix as input to learn the preliminary compressed representations, while retaining essential chromatin accessibility information and minimizing noise in an unsupervised manner. The encoder module is expected to identify key accessible patterns within the accessibility landscapes, and thus is applied to extract crucial information for cell topology inference and informative peak selection, remarkably enhancing subsequent analyses. In particular, we define a cell neighborhood graph to structure the cell topological information, where cells constitute the nodes in the graph, and the edges between cells indicate the neighborhood relationships. To construct a cell graph, we search for a certain number of nearest cells as neighbors for each cell in the low-dimensional space obtained from the encoder outputs (Fig. 1b). Additionally, to balance the benefits of graph learning with resource overhead in scAGDE, we employ a weight distribution-based strategy on the encoder module for peak selection.

To extract essential chromatin accessibility information while simultaneously capturing cell-cell topology from both the cell graph and the filtered count matrix, scAGDE employs a GCN encoder module (Fig. 1c). Guided by the cell graph, this encoder module encodes each cell’s filtered accessibility profiles, while aggregating features from its neighbors. This approach significantly advances previous autoencoder-based methods, which often overlook the inter-cell relationships. Eventually, the GCN encoder module outputs the embeddings for cells in the low-dimensional latent space, where each cell is characterized by restricted features. Subsequently, the embeddings are decoded via dual decoder modules to reconstruct topological and accessibility information. First, a graph decoder module recovers a cell graph by similarity measurement on cell embeddings, minimizing differences between the corrupted and input cell graph, thus preserving the inter-cellular topology as accurately as possible. Second, scAGDE designs a Bernoulli-based decoder to represent the global probabilistic structure of the dataset by simulating the probability of chromatin site opening events. This choice of Bernoulli distribution aligns with the binary nature of chromatin site accessibility.

To self-optimize the clustering task, scAGDE incorporates the sequential clustering optimization (Fig. 1c). The first optimization uses KL divergence to align clustering assignments with an auxiliary target distribution, both derived from the cell embeddings. Here, the assignments act as soft labels that guide the iterative refinement of the clustering assignments. Clustering centers are initialized through pretraining and K-means to mitigate potential deviations in clustering distribution due to unreliable assignments. The second optimization uses cross-entropy loss to align soft labels with pseudo labels, which are determined by high-confidence assignments within the soft labels. By optimizing a comprehensive set of losses, scAGDE can jointly evaluate clustering label assignments and feature learning of the topological structure by training the entire graph autoencoder framework. This comprehensive approach allows scAGDE to accurately balance clustering and feature learning, ensuring reliable performance in scATAC-seq data analysis.

We comprehensively demonstrated the effectiveness of scAGDE on multiple synthetic datasets considering five sequencing depths, five noise levels, and seven dropout rates, as well as on several single cell experimental datasets differing by sequencing platform and other data characteristics. Comparative analysis with other scATAC-seq and scRNA-seq methodologies elucidates scAGDE’s superior performance on scATAC-seq data. In addition, scAGDE demonstrated exceptional ability in dimensionality reduction, visualization, dropout correction, and cell-type-specific enhancer discovery (Fig. 1d). It is notable that scAGDE’s imputation capacity has uncovered a wide spectrum of chromatin accessibility, enhancing the identification of critical regulatory elements associated with cellular identity. Furthermore, scAGDE effectively characterized single-cell heterogeneity and revealed functional diversity within a human brain dataset.

scAGDE achieves state-of-the-art performance on simulated scATAC-seq scenarios

We evaluated the clustering performance of scAGDE on simulated single cell ATAC-seq datasets, having various dataset characteristics. To achieve this, we sampled synthetic datasets from extensive bulk Bone Marrow ATAC-seq data according to reference²⁰. Simulation datasets were varied in terms of read depth, noise level, and dropout rate to simulate different biological scenarios and generated with six annotated cell populations, including hematopoietic stem cells (HSC), common myeloid progenitor cells (CMP), erythroid cells (Ery), natural killer cells (NK), and CD4 and CD8 cells. To evaluate the consistency of generated clustering results and the true labels, we employed several metrics: the normalized mutual information (NMI), adjusted rand index (ARI) and F1 score. For the evaluation metrics, a score closer to 1 indicates superior clustering performance. On this basis, we conducted a comparative analysis of scAGDE to eleven other scATAC-seq data clustering methods, including ArchR²¹, SnapATAC2²², SCALE¹³, SIMBA²³, cisTopic⁹, Signac²⁴, PeakVI¹⁵, BAVARIA¹⁸, SnapATAC¹⁰, SAILER¹⁴, and scABC⁸. This comparison aimed to comprehensively evaluate clustering performance in various conditions and explore the potential of scAGDE to analyze scATAC-seq data.

We first assessed the performance of the clustering methods on single-cell ATAC-seq datasets with adjusted read depth, represented by the number of fragments per cell (Fig. 2a). We systematically varied the fragment count, starting from a low depth represented by 250 or 500 fragments, a moderate depth of 1500 or 2500 fragments, and a high depth of 5000 fragments. The NMI values of the clustering performance are depicted in Fig. 2a and additional results for ARI and F1 values, as well as visualizations of the clustering assignments are provided in Supplementary Figs. 1a and 2. From the results we see several observations that firstly, the performance of all the computational methods decreased as the read depth decreased, suggesting that read depth has a critical effect on clustering accuracy. Secondly, we observe that the deep learning methods were more susceptible to read depth. At low read depths (250 or 500 fragments), SCALE struggled to differentiate between Ery cells and other cell populations, while PeakVI, SAILER, and BAVARIA failed to differentiate entirely. In contrast, ArchR, SnapATAC2, cisTopic, Signac and SnapATAC had ARI values above 0.5 at 500 fragments, successfully distinguishing both Ery and HSC cells. ArchR improved on Signac using an iterative LSI algorithm and consistently outperformed Signac. SnapATAC2 exhibited comparable performance to scAGDE yet did not obtain an ARI value near 1.0 until 2500 fragments. SIMBA worked well but not for low sequencing depths. Of note, scAGDE consistently surpassed the other methods across all read depths, with metric values closer to 1.00. Specifically, scAGDE demonstrated robustness to variation in read depth, providing accurate clustering even at low fragment counts. Remarkably, when the number of fragments per cell was increased to 1500, scAGDE approached a clustering accuracy of almost 1.00, a level of performance that other methods only reached at high fragments coverage. This suggests that scAGDE can effectively mine information contained in chromatin accessibility profiles, thereby revealing cellular heterogeneity via accurate and stable clustering results.

Fig. 2: scAGDE is the most of the compared clustering methods on varying simulated scATAC-seq scenarios.

a Comparison of ARI values obtained for each method across five simulated datasets of varying sequencing depth. A value approach 1 indicates optimal clustering. The sequencing depth is represented by fragment numbers per cell, which covers a low depth of 250 to 500, moderate depth of 1500 to 2500 and high depth of 5000. b Table displaying NMI average results between scAGDE and compared methods under five noise levels. c UMAP visualization annotated by the true cell-type labels (first column, True-label) and obtained clustering labels from each method (subsequent columns). The first to last rows, respectively show clustering on simulated datasets with low (10%) to high (40%) noise levels. d Table displaying NMI values for each method across seven simulation datasets with varying dropout rates (top, 10–70%). The violin plots (bottom) aligned with the top horizontal axis displaying the increasing sparsity distribution of corresponding datasets (n = 1200 cells in each group; center black dot, median; box limits, upper and lower quartiles; whiskers, the 95% confidence intervals; violin plot edges represent minima and maxima). Source data are provided as a Source Data file.

Next, we evaluated the robustness of the computational methods under various levels of added noise. We increased the proportion of fragments sampled from the random distribution from zero to 0.4 in steps of 0.1. From Fig. 2b and Supplementary Fig. 1b, scAGDE demonstrated superior performance, maintaining stability across different noise levels. It is observed that NMI values of all the methods decreased with an increase in noise and in a noise-free environment, scAGDE achieved perfect cell assignment. ArchR, SnapATAC2, SIMBA, cisTopic and SnapATAC also performed well, reaching NMI values close to 1.0. Even with increased noise level, the performance of scAGDE remained robust, with an NMI value close to 1.0 and only a slight decrease even at the highest noise level, in contrast to the seven compared methods that had NMI values below 0.8. A couple of up-to-date methods such as ArchR, SnapATAC2 and SIMBA also showed decent robustness to noise, but it was slightly less resilient to dropout events compared to SnapATAC2 and scAGDE. Additionally, we visualized the raw data by projecting it onto a 2D space. We then annotated the visualization with the clustering results derived from the computational methods evaluated (Fig. 2c and Supplementary Fig. 3). From the results, we find that the introduction of noise obscured cellular heterogeneity, causing closely related cell types (such as HSC and CMP, CD4 and CD8) to become less distinguishable, as evidenced by the overlapping of these cell populations in the visualization plots. Nonetheless, scAGDE successfully resolved even the challenging cell populations and had a minimum misclassification rate, demonstrating its remarkable adaptability to noise.

Finally, we examined the influence of dropout events on the performance of our scAGDE model. Like scRNA-seq, scATAC-seq is also plagued by “dropout” issues, resulting in a sparse and high-dimensional count matrix that complicates downstream analysis, as noted in reference²⁵. To address this, we simulated seven datasets with increasing dropout rates, defined as the proportion of peaks omitted from the read counts relative to all accessible peaks in each cell. The experimental results are summarized in Fig. 2d and Supplementary Fig. 1c of the clustering performance. The visualization of the overwhelming overlap of cell populations is listed in Supplementary Fig. 4. From the figures, we can observe that the performances of scABC, SAILER and BAVARIA were significantly affected by the increased dropout rates and there is a large number of methods had degraded performance sharply at dropout rates of 40% or higher, such as scABC, SAILER, SIMBA and ArchR. Most impressively, scAGDE proved to be resilient against increasing dropout frequencies as scAGDE maintained an NMI value close to 1.0 up to the 60% dropout rate, and remained above 0.85 even as dropout rates increased further. SnapATAC2 performed comparably to scAGDE, but degraded more severely at the highest dropout rate.

In summary, we can conclude that these simulation experiments highlight the superiority and adaptability of scAGDE with varying sequencing depth, diverse noise levels, and increasing rates of dropout events, demonstrating the potential of scAGDE for effective application to real-world scATAC-seq datasets under a variety of experimental conditions.

scAGDE provides promising performance than other scATAC-seq clustering methods across multiple real-world scATAC-seq datasets

We extended our experiments to real-world single-cell ATAC-seq datasets to investigate the clustering capabilities of scAGDE under real experimental conditions, compared to the other 11 computational methods. Specifically, we collected nine datasets of typical size and two datasets with considerably larger scales generated in different sequencing platforms. These datasets presented cell counts spanning from 300 to nearly 200,000 and peak numbers ranging from 7000 to 70,000, reflecting various degrees of sparsity, as detailed in Supplementary Note 20 and Supplementary Figs. 41 and 42. According to the prior benchmarking of single-cell genomics²⁶, we assess the bio-conservation quality of the cell embeddings through a suite of metrics, including the ARI, NMI, F1, cell-type average silhouette width (ASW), with larger values of these metrics indicating better performance. To derive an overview of the comparison, we first for each method, averaged the value of each metric over all datasets. Then we normalized each metric using min-max scaling to aggregate these individual metrics into a unified score. In addition, an overall score was computed by taking the mean of all metrics, with a score of 1 indicating optimal performance. Fig. 3a, b represent the bio-conservation metrics and the ARI values of each method across all datasets, respectively. Our results reveal that scAGDE outperformed and maintained a large lead with other methods. Specifically, scAGDE reached the highest ARI on eight out of the eleven datasets and comparably high performance as SCALE and SnapATAC2 on the GM12878vsHEK, GM12878vsHL and InSilico datasets. The superiority of scAGDE on each dataset can be further seen in Supplementary Figs. 5 and 6. After scAGDE, SnapATAC2, ArchR and SCALE had the next highest overall scores, but were significantly less effective than scAGDE on datasets with more than 20,000 cells, such as the DropBlood, Lung and Liver datasets. Methods like BAVARIA, PeakVI, and SAILER generally had lower performance than SCALE, especially on sparser datasets. For instance, on the Leukemia and Forebrain datasets, their ARI values were significantly lower than those of scAGDE. Additionally, PeakVI failed completely on the two notably sparse datasets (GM12878vsHEK and GM12878vsHL). Given that ArchR also features its own unique peak selection, we compared it with scAGDE’s peak selection. When exchanging peak selection with each other, both performance degraded, but scAGDE still performed better (Supplementary Note 3 and Supplementary Fig. 12). The comparison of all algorithms using scAGDE’s peak selection strategy, alongside a cross-comparison with ArchR’s peak selection strategy, showed that scAGDE’s advantage lies in its model design itself, rather than peak selection alone (Supplementary Note 4 and Supplementary Fig. 13).

Fig. 3: Benchmarking analysis of clustering between scAGDE and other methods on real datasets.

a The normalized scores (0–1 range) of six metrics used to evaluate each method’s bio-conservation across all datasets, with score of 1 indicating optimal performance. The overall score is the average of all metric values. b Each method’s ARI scores on all datasets. c UMAP visualization annotated by true cell type labels and obtained clustering labels on the Splenocyte dataset. The two dotted ellipses in each box indicate four T cell types (large dotted circle) and dendritic cells and macrophages (smaller dotted circle). d The agreement between the clustering labels and true cell type labels. Matrices with a clear diagonal indicate better performance. Source data are provided as a Source Data file.

Despite the density of the Splenocyte dataset, all methods except scAGDE and SIMBA produced suboptimal results on this dataset, with ARI values of 0.6 at the most. scAGDE, however, achieved an ARI of 0.84, illustrating its adeptness in clustering complex datasets. The visual clustering representation in Fig. 3c shows that scAGDE and SIMBA successfully segregated the four T cell types into distinct clusters and the accurate alignment of known B cell labels. The “reference” plot displays the ground truth and compared to this, scAGDE was able to differentiate two rare cell populations, Dendritic and Macrophage cells, which other methods such as SnapATAC2, SnapATAC, SCALE, SAILER and PeakVI failed to do accurately, often confusing the three B cell types into multiple overlapping groups. This indicates that scAGDE surpassed the other methods in distinguishing cell subpopulations across various datasets (Supplementary Figs. 7–10). In addition, scAGDE effectively separated three subpopulations of excitatory neuron cells (EX1, EX2, and EX3) with minimal overlap, as illustrated in Fig. 3d. To explore the reasons for SnapATAC’s poor performance on the Splenocyte dataset, we conducted exhaustive algorithmic component analysis and ablation experiments (Supplementary Note 5 and Supplementary Fig. 14).

We also compared the runtime and memory usage of scAGDE on the datasets to the other methods (Supplementary Fig. 11). It is noteworthy that scAGDE divides the data into multiple batches for learning when handling datasets with more than 50,000 cells. As depicted in Supplementary Fig. 11a, the runtime for most methods correlated with the overall data scale, which is determined by the number of cells and peaks. SnapATAC, ArchR and Signac emerged as the three fastest methods across all datasets, particularly at lower scales. In contrast, the performance of scAGDE is in line with the other deep learning methods, balancing speed and accuracy. Although scAGDE is inherently slower than the non-deep learning methods, its runtime remains in a competitive range. We also evaluated the runtime of deep learning methods on one core of a 2.20 GHz Intel Xeon Gold 5220 CPU, which is an alternative solution when a GPU is not available (Supplementary Fig. 11c). The results reveal that scAGDE takes as long as SCALE when clustering datasets with  <15K cells and requires comparable runtime with PeakVI on large-scale datasets such as Lung and Liver. For the memory usage, we found scAGDE maintains reasonable memory requirements (Supplementary Fig. 11b), and there is not much difference between scAGDE and the deep learning methods when clustering large-scale datasets although scAGDE needs to process the entire cell graph due to its graph learning framework. In general, scAGDE’s architecture proved advantageous by optimizing the balance of speed and accuracy. This is probably attributable to the fewer model parameters (Supplementary Fig. 11d) and its peak importance filtering operation, a notable feature of scAGDE, simplified processing by focusing computational resources on the most informative parts of the data, thereby reducing unnecessary computation. Therefore, we can conclude that the balance between speed and analytical depth affirms scAGDE’s suitability for expansive single-cell ATAC-seq data analysis. To further demonstrate scAGDE’s scalability, we applied it to the human fetal atlas dataset from Domcke et al.²⁷, which contains approximately 800,000 single cells from 59 human fetal samples, spanning 15 organs and ranging from 89 to 125 days in estimated postconceptual age. The results and analysis on this dataset are detailed in Supplementary Note 6 and Supplementary Fig. 15. Moreover, we investigated whether the methods developed for the scRNA-seq data could be directly applied to the scATAC-seq data analysis and the results demonstrated the necessity for specialized and tailored approaches to scATAC-seq data (Supplementary Note 7 and Supplementary Figs. 16–18).

scAGDE performs efficient dimensionality reduction correction and significantly improves visualization in scATAC-seq data

Dimensionality reduction and visualization in high-throughput single-cell ATAC-seq data are crucial for identifying potential cell subpopulations and investigating functional differences. To demonstrate the effectiveness of scAGDE, we compared it with widely-used dimensionality reduction tools: PCA²⁸, t-SNE²⁹, and UMAP³⁰ and ArchR’s iterative LSI which is a specialized algorithm for sparse scATAC-seq data. We chose these tools due to their prevalent use and distinct methodological approaches to data analysis. We employed the ASW, the Calinski-Harabasz index (CHI), and the Davies-Bouldin index (DBI), calculated using ground truth cell-type labels and the resulting low-dimensional data matrices. The metrics used, namely ASW, CHI, and DBI, are essential in evaluating the quality of clustering. The ASW ranges from −1.0 to 1.0, where higher values indicate a better definition of clusters. Both the CHI and the DBI are positive metrics, with higher CHI values signifying more distinct inter-cluster separation and lower DBI values indicating tighter intra-cluster aggregation. These metrics collectively provide a comprehensive assessment of clustering performance across various computational methodologies.

Our results showed that scAGDE consistently achieved the highest average ASW and CHI values and the lowest DBI value, indicating its superior performance compared to PCA, t-SNE, UMAP, and LSI (Fig. 4b). As depicted in Fig. 4a, scAGDE effectively segregated cells into distinct groups, demonstrating a high degree of consistency with known cell types in the visualization space. In particular, on the GM12878vsHEK dataset, scAGDE and LSI divided cells from different cell lines into two compact and separated groups while the other methods dramatically expanded the two cell populations, with some of the cells even closer together in the results of PCA. On the Blood2K dataset which profiled  ~2000 hematopoietic cells, scAGDE demonstrated its effectiveness by providing a clear and distinct separation between various cell types. Notably, it accurately depicted the complex dynamics of hematopoiesis, which was evidenced by the adjacent but distinct distribution patterns of HSCs, multipotent progenitor cells (MPPs), and CMPs, as well as the discernible separation between CMPs and lymphoid-primed multipotent progenitor cells (LMPPs)³¹. Although LSI showed a similar continuous distribution, unlike scAGDE, it could not separate MPP and HSC cells, and GMP and LMPP cells. The visualization and ASW results across all other datasets are presented in Supplementary Figs. 19 and 20.

Fig. 4: Performance in dimensionality reduction and visualization capability of scAGDE.

a Visualization of scAGDE and four widely-used dimensionality reduction tools (PCA, t-SNE, UMAP, and LSI) on the GM12878vsHEK and Blood2K datasets. The ASW values are displayed at the top-left edge of the boxes, which ranges from −1.0 to 1.0 with a value approaching 1.0 indicating better performance. b Performance comparison by ASW, CHI and DBI across 11 datasets as boxplots (n = 11 datasets in each group; center line, median; box limits, upper and lower quartiles; whiskers, 1.5 × interquartile range). c, d The UMAP results of scAGDE and two single-cell ATAC-seq analysis packages (Signac and EpiScanpy) labeled with true cell-type labels (c), and the evaluation using benchmark metrics of clustering (ARI, NMI, F1, and V-Measure) and visualization (ASW, CHI, and DBI) (d). The CHI values shown are in logarithmic scale for better visualization. Source data are provided as a Source Data file.

Moreover, we conducted an extensive comparative analysis of scAGDE with two leading single-cell chromatin data analysis tools, Signac²⁴ and EpiScanpy¹⁶, focusing on dimensionality reduction and visualization aspects. Among them, Signac employs latent semantic indexing (LSI) for dimensionality reduction, while EpiScanpy relies on PCA transformation. To effectively compare and visualize the clustering performance, we used UMAP plots labeled with both the clustering results and the real labels of the cells to provide a clear and intuitive visual representation of the performance of the delineated cellular data. To further demonstrate effectiveness, we conducted an in-depth analysis of scAGDE on three datasets: Leukemia, Forebrain, and InSilico. We calculated a series of visualization-related metrics (ASW, CHI and DBI) and clustering-related metrics (ARI, NMI, F1, and V-Measure) to thoroughly assess scAGDE’s performance. These metrics not only reflect the quality of clustering but also reveal the effectiveness of data visualization post-dimensionality reduction. As illustrated in Fig. 4c, d, scAGDE demonstrated exceptional performance in dimensionality reduction and clustering in scATAC-seq data analysis, particularly in differentiating various cell types.

Specifically, in the Leukemia dataset, scAGDE effectively isolated six distinct cell groups. Notably, the Mono cells and LMPP cells were well-separated, aligning with their vastly different characteristics and roles in the progression of Leukemia, as corroborated by existing studies³². In contrast, the six clusters identified by EpiScanpy¹⁶ were closely placed without clear demarcations, resulting in a mixture of multiple cell types within most clusters. Signac, on the other hand, grouped the six cell types into four clusters, suggesting a different clustering approach. On the Forebrain dataset, scAGDE and Signac showed comparable visualization capabilities. However, scAGDE was particularly proficient in distinguishing and clearly separating the three subpopulations of excitatory neuron cells (EX1, EX2, and EX3). EpiScanpy, meanwhile, struggled to differentiate these excitatory neuron subpopulations and ended up classifying oligodendrocyte cells (OC) into two distinct groups. Further, Supplementary Figs. 21 and 22 show that scAGDE consistently outperformed both Signac and EpiScanpy across the nine datasets. This evidence confirms the superior effectiveness of scAGDE in achieving accurate dimensionality reduction correction and visualization, highlighting its proficiency in ATAC-seq data analysis.

Evaluation of an ablation study and hyperparameter selection

We conducted comprehensive experiments to evaluate the effectiveness of individual components of scAGDE for scATAC-seq data analysis, including the cell graph generation based on the chromatin accessibility-based autoencoder, the employment of a Bernoulli-based decoder and the integration of GNNs. In addition, we evaluated the effect of the different hyperparameters involved in scAGDE on clustering performance, providing insights for future enhancement.

First, we evaluated scAGDE’s cell graph construction against analogous methods in scATAC-seq data analysis. In our study, scAGDE typically employs a chromatin accessibility-based autoencoder to build the cell graph. To demonstrate the effectiveness of scAGDE, we respectively substituted this approach with several alternatives: the PCA transformation followed by the KNN method (KNN), the Shared Nearest-Neighbor graph employed by Seurat and a graph method based on positive pointwise mutual information matrix (PPMI) recently developed in the scMGCA framework³³ for scRNA-seq analysis. Fig. 5a indicates scAGDE’s superior clustering performance with a mean NMI value of 0.83, surpassing the other methods whose performance may be limited by imprecise topological relationships between cells (Supplementary Fig. 16). In particular, the PPMI method, while effective for scRNA-seq, did not yield reliable clustering results on the four sparsest scATAC-seq datasets including the Forebrain, GM12878vsHEK, GM12878vsHL, and leukemia, implying the limited applicability of cell graph strategies from scRNA-seq in scATAC-seq data. For a fair comparison, we explored a range of the number of components in the PCA and found that 50 components had the highest average ARI value of 0.70, which was still lower than 0.81 for scAGDE (Supplementary Note 9 and Supplementary Fig. 23).

Fig. 5: Ablation study of scAGDE framework.

a Table displaying the impact on clustering performance of several components in scAGDE, including the graph construction method, distribution of the decoder and fundamental model architecture. b UMAP visualization of embeddings at three progressive stages, with each cell type label and annotated by ASW value. c The heatmap of Pearson correlation coefficients on the average embeddings of distinct cell types. A value approaching 1 implies a higher correlation between two cell populations. Source data are provided as a Source Data file.

Next, we explored different distribution models for scATAC-seq count data by modifying the decoder to suit various distribution assumptions, including zero-inflated negative binomial (ZINB) distribution, multinomial distribution (Multinomial), negative multinomial distribution (Neg-Multinomial). In brief, the decoder network was utilized to estimate the various parameters of the distributions and get trained with the corresponding modified loss functions and we detailed this part in Supplementary Note 10. The results indicated that within scAGDE, the Bernoulli-based decoder outperformed the other distribution models, followed closely by the multinomial distribution, which exhibited significant differences compared to scAGDE on only four datasets, and the ZINB and negative multinomial distributions, which involve multiple parameters to estimate, demonstrated similar average performance, with no clear leader emerging across all datasets (Fig. 5a and Supplementary Fig. 24). Actually, we developed scAGDE considering the concise parameter settings, which makes it less susceptible to outliers and noise in sparse scATAC-seq data, and the ablation analysis provides further evidence.

Then, we explored the impact of incorporating graph learning in scAGDE. We experimented by substituting the original graph neural network structures (scAGDE-GNN) with linear layers, which are standard in autoencoders, to create scAGDE-AE as detailed in Supplementary Note 11. Besides, we also included an experiment using original GNN structures with identity matrix as cell graph input (scAGDE-Identity), to verify the impact of the cell graph information. As depicted in Fig. 5a and Supplementary Fig. 25, discarding the graph learning mechanism or utilizing incorrect cell graph input significantly reduced the clustering accuracy, which highlights the crucial role of intercellular topological information in scATAC-seq data analysis. To further explore this improvement, we extracted the low-dimensional embeddings obtained at the initial, mid, and near the end stages of the training process, from scAGDE-GNN and scAGDE-AE, respectively. The UMAP visualizations of embeddings revealed a progressively clearer distinction among cell populations, characterized by wider inter-group distances and narrower intra-group distances, with scAGDE-GNN demonstrating a better separation, as reflected in higher ASW scores (Fig. 5b). Notably, scAGDE-GNN was able to differentiate cell types such as IN1, OC, IN2, MG, and excitatory neuron cells earlier in the training process compared to scAGDE-AE, which exhibited a more sequential arrangement with significant overlaps. As the model was trained, the excitatory neurons remained highly similar yet increasingly distinguishable from each other, as demonstrated in the correlation coefficient heatmap in Fig. 5c.

Finally, we explored the impact of various hyperparameters of the scAGDE framework. A primary focus was the number of neighboring nodes, denoted as K, in the cell graph construction. Optimal performance was achieved at K = 15, as demonstrated in Supplementary Fig. 26. Notably, increasing the number of neighboring nodes beyond this threshold led to a decline in overall performance, highlighting the importance of this parameter in the graph construction process. Then, we benchmarked the architecture of scAGDE in terms of the number of consecutive GCN layers and the count of neuron nodes in the hidden layers, and the results evaluated using NMI and ARI are shown in Supplementary Fig. 27. The most effective configuration was determined to be a two-layer GCN encoder with a size of 128-10, providing adequate capacity for compressing information into latent embeddings while avoiding over-complication of the hidden space. Additionally, we explored various weight coefficients for each loss function as outlined in Eq. 29 and identified the optimal settings for scAGDE (Supplementary Fig. 28). By default, we selected the top 10,000 peaks based on the peak importance score as our analysis showed that the performance plateaued at this point (Supplementary Fig. 29).

scAGDE enhances discovery of cell type-specific enhancer regions to infer regulatory mechanisms of key functions

One of the key abilities of scAGDE is its capacity to evaluate and filter peaks based on our proposed peak importance scores, thereby mitigating the curse of dimensionality in downstream analysis. These scores, reflecting the contribution of each peak to the learning of its low-dimensional representation, are derived from the weights of the encoder network, which learns from chromatin accessibility profiles. Thus, they indicate the peak selection preferences of scAGDE. In this study, we detailed the characteristics of the peaks selected by scAGDE and compared these to the peaks identified using EpiScanpy¹⁶, which served as our baseline method.

First, we selected 10,000 peaks from the preprocessed 10×Blood dataset using scAGDE and EpiScanpy, respectively, to determine the accessibility of human immune cells and analyze their accessibility-related statistics. Specifically, after identifying the peaks from scAGDE or EpiScanpy, we generated pseudo-bulk data by merging the accessibility profiles of each peak across different cell types, resulting in a summarized accessibility profile for each peak within nine distinct cell types. Based on this pseudo-bulk data, we calculated the median accessibility and the range (the difference between the maximum and minimum accessibility values across cell types) for each peak, and presented these in the top density plot of Fig. 6a. Additionally, we measured the Shannon entropy of each peak and the percentage of cells where each peak is accessible, which are visualized in the bottom density plot of Fig. 6a. In particular, the peaks identified by scAGDE exhibited lower accessibility and a larger range of accessibility variability, whereas the peaks from EpiScanpy showed higher accessibility and a smaller range of variability. In addition, the variance in accessibility within each cell type in the pseudo-bulk data generated using scAGDE peaks was greater than that of EpiScanpy peaks (Supplementary Fig. 30a). Additionally, we observed an increase in the number of peaks with lower Shannon entropy and a smaller percentage of accessibility, alongside a decrease in peaks with high Shannon entropy, indicating that scAGDE tended to select cell type-specific accessible peaks. To further illustrate this, we performed differential accessibility region analysis and identified 6162 cell type-specific accessible peaks out of the 10,000 peaks, which is significantly more than the 462 identified by EpiScanpy (both with a false discovery rate <0.01). Furthermore, scAGDE demonstrated a higher accessibility difference with larger values of the log-fold-change (Supplementary Fig. 30b, c).

Fig. 6: A further exploration of peaks selected by scAGDE and EpiScanpy.

a Density plots showing various statistics for the peaks selected by scAGDE and EpiScanpy. In the top density plot, the y-axis represents the median accessibility of these peaks across the nine cell types and the x-axis shows the log-transformed accessibility range. In the bottom density plot, the y-axis represents the Shannon entropy based on the pseudo-bulk data for each peak and the x-axis indicates the percentage of cells in which each peak is accessible. b Peak annotation results using ChIPseeker on peaks from scAGDE and EpiScanpy. c Genome track around CD37, CTLA4 and MS4A1. The peak regions discovered by scAGDE and EpiScanpy are separately displayed below the aggregated signals. d The heatmap of peak importance scores during the entire training process. e The numbers of enhancer peaks in discovered regions of each method (n = 40 peaks in EpiScanpy and n = 78 peaks in scAGDE; Fisher’s two-sided test). f Triangle heatmap shows chromatin interaction from the Hi-C experiment data of peripheral blood CD8+ T cells isolated from the peripheral blood of two human donors¹⁰². g Genome track around loci of CD8A with regions of discovered peaks and the super-enhancer, and the results of de novo motif analysis on the highlighted peak regions are displayed. The one-sided hypergeometric test was used and p-values were not corrected for multiple testing. Source data are provided as a Source Data file.

To further dissect the identity of these candidate peaks, we annotated them with peak type using ChIPseeker³⁴ based on their proximity to the nearest transcription start sites (TSS). Fig. 6b shows that peaks from scAGDE were primarily annotated as promoters near TSSs (19.9%), intronic regions (31.0%), and distal intergenic regions (32.1%). In contrast, peaks from EpiScanpy were almost exclusively composed of promoters near TSSs (83.8%). Based on the distance to the TSSs, 77% of the scAGDE peaks can be defined as distal peaks (i.e., greater than 2000 bp), while 83% of the peaks from EpiScanpy are promoters (i.e., less than 200 bp) (Supplementary Fig. 30d). In detail, peaks selected by scAGDE comprised promoters that are useful for cell-type annotation, which were also identified by EpiScanpy, as well as promoters not in EpiScanpy. Additionally, scAGDE included distal and intergenic peaks that were not identified by EpiScanpy (Fig. 6c and Supplementary Fig. 32). For instance, scAGDE uniquely identified the CTLA4 gene-associated promoter and distal peaks, which are predominantly accessible in specific T cell groups. Therefore, scAGDE exhibited a higher preference for peaks in intronic and distal intergenic regions.

To further illustrate these preferences, we uniformly sampled 500 moments from the entire training process of our autoencoder model, and computed the importance scores of all peaks at each moment, as presented in Fig. 6d. To characterize the distinct preference patterns, we organized the 39,565 peak importance profiles into 6 modules by K-means clustering based on the elbow method³⁵. As a result, the identified 6 modules have similar internal distribution and are distinct from each other. Specifically, at the initial stage, these modules exhibited a nearly uniform distribution and as the training progressed, the first two modules (M1 and M2) received increasing attention, whereas other modules, particularly M6, garnered less focus. The variation in peak importance scores is greatest at the 20% stage and gradually stabilizes as the model converges. As expected, M1 and M2 were primarily composed of peaks from intronic and distal intergenic regions, and nearly half of M6 was composed of promoter peaks. At this point, the preference analysis, together with prior annotation results, shows that scAGDE tends to select peaks from intronic and distal intergenic regions. Therefore, we can conclude that scAGDE identified cell type-specific accessible peaks which were predominantly located in intronic and distal intergenic regions, and this discovery of the strong association between specific accessible peaks and their identities is highly consistent with a previous study in which a single-cell ATAC-seq analysis of immune cell populations in the mouse spleen reported that the marker peaks were found primarily in these regions³⁶. Additionally, Chung et al. demonstrated that distal regions are more cell-type specific accessible and informative than promoter-proximal regions³⁷, further supporting the significance of our discovery. Besides, enhancer-associated regions are well known to exhibit cell-type-dependent accessibility³⁸ and have proven to be more effective in distinguishing cell types than promoters or transcriptomic data³⁹. These findings decipher the underlying association between peak attention bias and their unique accessibility patterns in our deep learning models, providing a marked perspective on model interpretability.

To validate the potential functions of the peaks we identified, we compared them to two publicly available enhancer databases: VISTA⁴⁰ and EnDB⁴¹. The VISTA enhancer browser provides catalogs of mammalian enhancers validated in transgenic mice, and EnDB documents over 700 validated enhancers along with related information. We retrieved enhancers corresponding to the human genome from both databases for comparison and discovered that the peaks identified by scAGDE matched more known enhancers, demonstrating a significance of p = 4.28 × 10⁻⁴. Notably, among the scAGDE peaks that did not match acknowledged enhancers, we observed patterns of chromatin accessibility reminiscent of enhancers around key genes. For instance, CTLA4, an immune checkpoint responsible for inhibiting T cell activation and function⁴², is constitutively expressed in regulatory T cells (Treg)⁴³. Within scAGDE peaks proximal to CTLA4, we observed that both conventional T cells (CD4 Helper T cells, Memory CD4/CD8+ T cells, and Naive CD4/CD8+ T cells) and Treg cells demonstrated high chromatin accessibility at the promoter site. Memory CD8+ T cells, in particular, showed even greater accessibility at the promoter site compared to Treg cells (Fig. 6c). In contrast, Treg cells displayed a unique chromatin accessibility peak at approximately 20kb upstream of the CTLA4 promoter, which was uniquely found by scAGDE and we found it overlaps with a super-enhancer (SE_18993) listed in dbSUPER (Supplementary Fig. 31). The overlap with the known super-enhancer suggests that this accessible region may play a key role in regulating the constitutive expression of CTLA4 in Treg cells, potentially highlighting a cell type-specific regulatory mechanism. A similar regulatory pattern may be present near the CD8A gene locus (Fig. 6g). The CD8+ T cells exhibited high levels of chromatin accessibility at both the promoter and 3’UTR regions of the CD8A gene, which encodes the molecular marker of CD8+ T cells⁴⁴. We found that the super-enhancer (SE_12469) from dbSUPER overlapped with two peaks identified by scAGDE near the 3’UTR and confirmed the chromatin interactions between these peaks and the promoter of CD8A (Fig. 6f), supporting our hypothesis that these regions are candidate enhancers. The chromatin interactions can be further validated with additional biologically relevant Hi-C datasets, with consistent results across multiple tissues or cell lines (Supplementary Fig. 33). Interestingly, studies have shown that CD8A is expressed in a subset of NK cells⁴⁵. In our findings, NK cells exhibited similar chromatin accessibility at promoter sites as other immune cell populations, yet displayed increased accessibility in our identified candidate enhancer region. This high accessibility emphasizes the essential role of this enhancer in regulating the transcriptional activity of CD8A, suggesting its substantial impact on gene expression. In summary, scAGDE directs us to regions with potentially functional and unique regulatory programs, offering notable insights into the regulatory mechanisms of key genes.

scAGDE can recover the potential open sites in chromatin open sites and infers cell type-specific cis-regulatory elements

The limitations of the diploid genome structure and the technical constraints inherent in scATAC-seq experiments pose similar challenges as scRNA-seq, leading to the phenomenon of “dropout” where signals from the actual open chromatin sites may be absent⁴⁶. To address this issue, we developed a Bernoulli-based distribution decoder aimed at recovering these dropout events and reducing noise signals present in the original data, thereby enhancing downstream analysis. We first evaluated the effectiveness of scAGDE in peak recovery by comparing it with three state-of-the-art methods specifically tailored to scATAC-seq data analysis, namely SCALE¹³, PeakVI¹⁵, and SAILER¹⁴.

We computed meta-cells for each cell-type group by aggregating individual profiles within the same group, which represented the ground truth representation of chromatin opening landscapes without any dropouts. Subsequently, we computed Pearson correlation coefficients between imputed cells and their corresponding meta-cells for each cell type, with values approaching 1 indicating superior peak recovery performance. Our analysis encompassed nine datasets, and we present the distribution of correlation coefficients in Fig. 7a. The correlation analysis of raw data at the cell level demonstrated significant dispersion, with the lowest mean value recorded at 0.39 (indicated by the dotted line). This dispersion is likely attributed to the presence of accessibility variation, independent of cellular heterogeneity arising from dropout events and noise in the data. After the application of scAGDE’s recovery method, the correlation coefficients exhibited a notably narrower distribution, with a peak approaching 1.0 and a highly concentrated pattern, maintaining an average value of 0.88. This performance was followed by SCALE’s correlation coefficient of 0.85 and PeakVI’s coefficient of 0.78. However, it’s disappointing that SAILER failed to improve the consistency of meta-cell accessibility, and even demonstrated larger variability.

Fig. 7: The effective recovery of open chromatin sites by scAGDE enhanced downstream analysis of mouse forebrain dataset.

a The cell-wise correlations with meta-cells in original data and post-imputation data for each method across all datasets. b The average performance of K-means clustering on all datasets, both before and after imputation by each method. c The UMAP visualizations of imputed data, where the ASW values are displayed at the top-left edge inside the boxes. d The heatmaps of cell-by-peak matrix that contains only the top 15 differential accessibility regions of each cell type from raw, imputed and binary imputed data. Rows represent DARs and columns represent cells, which are both arranged according to eight different cell types. Tiny spacing was plotted between the different cell types therefore the x and y axis were both divided into eight parts. e The recovered cell type-specific peaks by scAGDE and enrichment analysis of known transcription factor motifs and de novo motifs in the peaks. The one-sided hypergeometric test was used for both TF motifs enrichment and de novo motifs enrichment and p-values were not corrected for multiple testing. f The significant results of chromVAR deviation score profile of identified motifs of each cell type. g Enhanced TF deviation scores of NEUROG2, Dlx2 and MEIS2, compared with raw data, displayed on the T-SNE embedding, which was calculated based on TF deviation scores. Source data are provided as a Source Data file.

Subsequently, we applied conventional clustering techniques, specifically PCA transformation followed by the K-means algorithm, to all imputed datasets for evaluation. As expected, scAGDE demonstrated superior performance compared to the other methodologies, achieving the highest average values across all evaluated metrics, as illustrated in Fig. 7b. Conversely, no significant enhancement in clustering outcome was observed following imputation by either PeakVI or SAILER.

Further analysis was conducted on the imputed Forebrain dataset to investigate the observed performance discrepancies. As presented in Fig. 7c, the 2D UMAP visualization demonstrates the ability of scAGDE to effectively delineate different cell populations, closely agreeing with the true labels, especially in the case of excitatory neuronal cells. Additionally, the ASW score for each method, as indicated in Fig. 7c, shows that scAGDE achieved significantly higher scores relative to other methods. Although SCALE and PeakVI preserved the continuous distribution characteristic of the raw data, they tended to blur the boundaries between different cell populations, which elevated the risk of misidentifying cell types. In SAILER, although there was no significant overlap between cell groups, the positioning of these groups was remarkably close, with an almost uniform internal distribution. This arrangement within the UMAP embedding space introduced additional variability at the cellular level, which accounts for the reduced intra-cluster consistency and the suboptimal clustering performance illustrated in Fig. 7a, b.

Beyond this, we found that scAGDE could recover cell type-specific accessibility signals through imputation, thereby enhancing the identification of genomic elements. Specifically, we identified differential accessibility regions (DARs) within the raw, imputed, and binary imputed Forebrain dataset with the Signac toolkit²⁴. As a result, we uncovered 5263 significant DARs by applying the Signac toolkit²⁴ with the default parameters, marking a significant increase from the 626 DARs identified using identical parameter settings prior to imputation. This increase in the number of DARs highlights the impact of imputation on enhancing the detection of differential accessibility regions, particularly in cell types where low coverage may have obscured relevant peaks. For each cell type, we ranked their DARs according to the log2 fold-change values and selected the top 15 DARs, which were used for subsequent analyses. Then, the cell-by-peak matrices from the original, imputed, and binary imputed datasets, containing only these top DARs, were visualized in heatmaps as depicted in Fig. 7d, where the rows represent the DARs and the columns represent the cells, which are both arranged according to cell type. Clear separations are drawn between different cell type groups, reflecting distinct accessibility patterns across the cell populations. The alignment of rows and columns according to cell types ensures that DARs with strong cell type specificity appear along the main diagonal, showing high accessibility in their respective cell types. The heatmaps further revealed that the imputed and binary imputed data exhibited distinct clustering patterns, sharply contrasting with the original data. Specifically, imputation led to a more pronounced clustering of cell types, suggesting that missing or incomplete accessibility information in the original data may have masked these biological patterns. To further understand the biological significance behind this increase in cell type-specific peaks post-imputation, we proceeded with enrichment analysis. The analysis revealed that the enhanced cell type-specific peaks after imputation are significantly associated with cell identity, rather than being arbitrary occurrences (Fig. 7e).

On this basis, we explored the biological significance by characterizing the relevant genes within these recovered regions. Utilizing rGREAT⁴⁷, we meticulously examined the genes associated with the recovered regions, concentrating on a  ±15 kb range from the TSS. This analysis revealed a significant emergence of cell type-specific marker genes. For instance, Aldh1l1 and Slc1a3, markers for astrocyte cells (cluster AC)⁴⁸, along with Neuorod6, Stab2 and other genes significant for excitatory cells (clusters EX1-3)⁴⁹, were identified within the differential accessibility regions (DARs). Similarly, Gad1 and Gad2, associated with inhibitory cells (clusters IN1 and IN2)⁵⁰, and the marker gene Mobp for oligodendrocyte cells (cluster OC)⁵¹, were also discovered. These findings indicate that scAGDE is proficient in enriching the signals of chromatin accessible regions tied to key genes that define cellular identity, which may be the fundamental reason for the significant improvement in clustering results observed with imputed data.

In addition, to further investigate the additional discovered cell type-specific cis-regulatory elements, we performed motif enrichment analysis, seeking to identify known motifs via Signac²⁴ and de novo motifs through HOMER⁵². For the enrichment of known motifs, after filtering out non-significant findings based on ‘p-value’ and ‘fold.enrichment’, we discovered 427 transcription factor binding motifs that displayed cell type-specific enrichment patterns (Fig. 7e). Of note, these included binding motifs of key transcription factors (TFs) crucial for defining cell identity. For instance, motifs for NFIX and NFIB were enriched in regions associated with astrocytes, which have been shown to significantly express these factors⁵³, which are linked to the specification, differentiation, and migration of astrocytes in conjunction with other NFI family members^54,55,56. Excitatory neuron regions displayed enrichment for MEF2C and bHLH family members⁵⁷ motifs, while Dlx2 motifs were enriched in inhibitory neuron regions⁵⁸. Additionally, we identified ETS-factor PU.1 in microglial cell regions⁵⁹ and SOX10 in oligodendrocyte cell regions⁶⁰. In the de novo motif identification, we found that the TFs correlating best with the top enriched motifs were either directly mentioned as key TFs in the specific cell types or belonged to the same motif family, such as the NFI transcription factor NFIA in the AC cluster and the MEF2 transcription factor MEF2A in the EX1 cluster.

The deviation score takes the accessibility matrix as input to calculate accessibility variations for TFs, and scAGDE can restore or erase the accessibility of certain elements harboring TF motifs, thus affecting the chromVAR deviation score. Finally, we evaluated the chromVAR deviation score for each TF motif on the imputed data¹¹. scAGDE can restore or erase the accessibility of certain elements harboring TF motifs, thus affecting the chromVAR deviation score. As expected, we found that the imputation offered more highly significant enriched TF motifs (from 141 to 702) and enhanced the cell type-specific TF activity schema, especially for the EX2 cells within excitatory neurons and the OC cells. (Fig. 7f and Supplementary Fig. 35a). Furthermore, the significantly enhanced specificity improved the t-SNE visualization derived from the TF deviation scores, facilitating clearer distinction between different cell types as shown in Supplementary Fig. 35b. From this figure, we observe that by filling missing values, scAGDE strengthened the patterns of variability for these crucial TF motifs across clusters (Fig. 7g and Supplementary Fig. 35c). Remarkably, the deviation score distribution between IN1 and IN2 cells showed substantial differentiation post-imputation for the MEIS2 motif that plays a critical role in the development of medium spiny neurons⁶¹ (Supplementary Fig. 35d). This observation lends further support to the hypothesis proposed in the literature that cluster IN1 comprises medium spiny neurons⁶². To provide a comprehensive evaluation, we also applied scAGDE to a single-cell multi-omic dataset, demonstrating that scAGDE achieves superior imputation performance (Supplementary Fig. 34). In summary, these findings indicate that scAGDE can effectively uncover potential chromatin accessible regions during the modeling of scATAC-seq data distribution, thereby enhancing the identification of insightful chromatin regulatory elements.

scAGDE reveals subtle functional diversity and regulatory mechanisms in human brain neurons

To investigate the application of scAGDE to single-cell datasets with significant epigenetic variation, we applied the scAGDE algorithm to a brain dataset comprising 14,906 nuclei from single-cell chromatin accessibility assays of adult human frontal cortex tissue⁶³, a region known for its complex and diverse epigenetic landscape. To address data sparsity, the dataset was first preprocessed and subsequently clustered using scAGDE. As depicted in Fig. 8a, we identified eight cell groups, most of which were individually aggregated into clusters, with clusters 2 and 3, and clusters 5 and 6 being adjacent to each other. To further characterize these clusters, we next identified the signature genes of each cluster using the ArchR “gene expression score” matrix²¹. In the eight clusters, we identified 8126 signature genes, many of which displayed unique patterns of expression and aligned with markers found in the PanglaoDB⁶⁴ and CellMarker⁶⁵ databases (Fig. 8b and Supplementary Fig. 36a). On this basis, we assigned specific cell types to each cluster, including oligodendrocyte cells (OG), blood-brain barrier endothelial cells (BBB), microglial cells (MG), GABAergic neurons (GABA+), astrocytes (AC), oligodendrocyte precursor cells (OPC), and two groups of glutamatergic neurons (vGlut+ 1 and vGlut+ 2) (Fig. 8c). Furthermore, our Gene Ontology (GO) analysis of the top 100 marker genes uncovered the biological processes integral to cellular identity, validating the reliability of the clustering and cell type annotation. For instance, oligodendrocytes myelinate and support the axons of the central nervous system (CNS), with their damage leading to demyelination⁶⁶. MG are characterized by their rapid response to even minor pathological changes in the CNS⁶⁷. Additionally, ‘Gliogenesis’ describes the development of OPC into mature oligodendrocytes. Therefore, these cell clusters are distinguished by their unique access to the promoter loci of marker genes (Fig. 8d).

Fig. 8: Single-cell chromatin accessibility data analysis of human brain tissue by scAGDE.

a The UMAP visualization of brain cell clusters. b Heatmap of gene score matrix obtained by ArchR across all clusters. Selected GO terms and correlated genes are shown on the right. c Cluster connectivity changes between initial cluster, refined cluster and scRNA clusters shown as Sankey plots. d Genome track around marker genes from the aggregated scATAC data in all clusters. e Heatmap of 48,578 differentially-accessible CREs in the eight clusters. f Known and de novo motif enrichment results of CREs restricted to all clusters. The top de novo motifs, their best matches and p-values are displayed. The one-sided hypergeometric test was used and p-values were not corrected for multiple testing. g Heatmap of differentially-accessible CREs between two glutamatergic neuron clusters (left) and top-ranked TFs enriched in the two sets of CREs. The top 16 TFs are listed. A one-sided hypergeometric test was performed, and the p-values were adjusted using the Bonferroni correction. Source data are provided as a Source Data file.

Next, we investigated the cis-regulatory elements (CREs) in our eight cell clusters, identifying 48,578 differential chromatin accessibility peaks. As illustrated in Fig. 8e, the largest number of cell type-specific CREs are found in glutamatergic neurons, with comparable amounts of CREs across the four glial cell types. To further dissect sequence features across these cell-type-specific regulatory elements, we measured the enrichment of motifs with HOMER and ArchR in the CREs. This revealed a total of 125 enriched motifs displaying cell type-specific distribution (Fig. 8f); for instance, the SOX TF family are crucial regulators in various developmental processes in oligodendrocytes⁶⁸; The microglial-specific CREs show enrichment for motifs associated with pro-inflammatory TFs, such as STAT2 and IRF1, and the lineage-specific master TF, PU.1⁶⁹. Additionally, ASCL1 is highly enriched in OPC, as evidenced by its active expression in subventricular zone progenitors and OPCs during myelination and remyelination processes⁷⁰. Interestingly, the two groups of glutamatergic neurons exhibited similar motif enrichment patterns, yet they displayed notable differences that warrant further discussion. Notably, the basic Helix-loop-Helix family members such as NEUROG1, NEUROG2, and NEUROG3, known regulators of cell proliferation and neuronal specification⁷¹ were identified in both neuron groups. However, compared to the vGlut+ 2 cells, the vGlut+ 1 cells had an increased number of accessible peaks, with 4957 peaks gained and 1300 peaks lost (Fig. 8g). For the motif enrichment across the gained peaks, we found TFs could be broadly categorized into major families, including FOS and JUN, MEF2, POU and FOX. The FOS and JUN family members can form heterodimers that constitute the transcription factor activator protein-1 (AP-1), regulating the expression of multiple genes in response to external stress stimuli⁷². The MEF2 family members regulate synaptic plasticity, affecting learning and memory formation⁷³. In the lost accessible peaks, motifs from significant TF families such as BHLH and OLIG were identified. Taken together, these findings suggest that there may be even more subtle heterogeneities within the glutamatergic neuron cell groups.

To further explore the subtle heterogeneities, we employed a re-clustering strategy by adjusting the resolution settings of the Leiden algorithm implemented in Seurat⁵, guided by changes in the ASW value. The ASW metric measures both the intra- and inter-class distances, thereby when the number of clusters is larger than the number of major cell types, the clustering resolution increases yet at the cost of a decrease in the ASW values. Here we need to make a trade-off between the cluster numbers and ASW values. As the resolution increased from 0 to 1.0, the ASW decreased more gradually at 0.4, with an ASW value of 0.34 (Supplementary Fig. 36b). This setting refined the cluster assignments as shown in Fig. 8g, where we see GABA+ cells are now divided into two clusters (C1 and C2) and glutamatergic neurons into three clusters (C3–C5). Differential analysis of peak accessibility and gene scores revealed prominent patterns in the three subgroups of glutamatergic neurons (Fig. 9b and Supplementary Fig. 36c).

Fig. 9: Further analysis among three glutamatergic neurons cluster C3–C5.

a The GO (biological process) enrichment results with correlated genes among clusters C3–C5. The biological processes and negative log₁₀p values are displayed in the histogram and are linked by lines to correlated genes, and the corresponding gene score data are described in the heatmap. A two-sided Wilcoxon test was performed and p-values were not corrected for multiple testing. b Heatmap of differentially accessible CREs between cluster C3–C5 (left) and selected top de novo enriched TFs and negative log₁₀ p values. The one-sided hypergeometric test was used and p-values were not corrected for multiple testing. c, d Genome tracks around the RGS4 (c), and the PCDH8 loci (d). Loops shown below these tracks indicate the peak-to-gene linkages and co-accessibility linkages among peaks. The scRNA expression levels are shown in the violin plot for each cluster on the right side. Blue vertical bars spanning each panel highlight CREs linked to gene locus or being co-accessible with gene promoters. Source data are provided as a Source Data file.

To reveal the functional differences between clusters C3–C5, we applied GO enrichment analysis to uncover the biological processes. The results showed that genes associated with cluster-specific CREs of the three clusters were involved in several biological processes (Fig. 9a). Genes from cluster C3 were involved in processes related to synaptic assembly, such as regulation of synapse assembly (GO:0051963), regulation of synapse structure and activity (GO:0051960), and positive regulation of synapse assembly (GO:0051965). Additional processes such as the regulation of synaptic vesicle priming and recycling (GO:1901890 and GO:1901888) are essential for the release and reuse of neurotransmitters^74,75. Cluster C4 was associated with synaptic plasticity (GO:0048167), synaptic transmission (GO:0051966, GO:0050804, and GO:0099177), and sensory perception, notably olfaction (GO:005090, GO:0050911, and GO:0007608). Cluster C5 showed enrichment in synapse regulation-related processes, including synapse organization (GO:0050808) and the regulation of synapse structure or activity (GO:0050803). Expression of genes linked to these processes is specific to the clusters. Furthermore, cell label prediction using single-cell RNA-seq data from Ma et al.⁷⁶ validated our annotation of cell types (Fig. 8c), e.g., clusters C3 and C5 were identified as intratelencephalic neurons from layers 3–5 and 2–3 of the brain, respectively, and cluster C4 was identified predominantly as layer 2–3 intracerebral neurons.

In addition, we applied motif enrichment analysis on the cell type-specific CREs to reveal the correlated TFs for each sub-cluster. For instance, CREs restricted to cluster C3 revealed binding sites of Smad3, which is involved in TGFß-Smad3 signaling pathway that regulates synaptogenesis⁷⁷, and NEUROD1 and Mef2c which are critical for the development and function of the central nervous system^78,79. From CREs restricted to cluster C5, we found MEF2A, which plays a key role in activity-dependent gene expression during synaptic remodeling⁸⁰, and CREB5 of CREB family that is essential for synaptic plasticity⁸¹. CREs restricted to cluster C5 were also enriched in PAX6, a transcription factor regulating the expression of many synaptic proteins⁸².

To reveal transcriptional regulatory differences showing functional heterogeneity, we performed peak co-accessibility analysis and detected peak-to-gene linkages using integrated scRNA data for the three clusters. We identified 3390 peak-to-gene links (Supplementary Fig. 36d). Based on these links, we pinpointed differences in accessibility at non-promoter regions of functionally correlated genes, potentially representing candidate enhancers. For example, within the region upstream of the RGS4 promoter, cluster C3 exhibited higher accessibility and higher expression of gene RGS4 in the scRNA-seq predicted cell population (Fig. 9c). RGS4 is involved in amine transport and regulates G protein-coupled receptor signaling pathways⁸³ and function of glutamate receptors⁸⁴. This suggests that cluster C3 fine controls the transmission of excitatory signals. Additionally, we found higher accessibility in cluster C5 at three distal regions of the PCDH8 gene, which is consistent with observed high PCDH8 scRNA expression levels (Fig. 9d). PCDH8 encodes a cell adhesion molecule that is associated with the regulation of synapse structure and activity, indicating that cluster C5 may help support the maintenance of neural networks.

In summary, our scAGDE algorithm successfully annotated various neuron types with unique regulatory landscapes previously uncharacterized in human frontal cortex tissue. Indeed, scAGDE accurately localized specific markers of each neuron type and thereby provided notable insights into their functional roles and interactions in the brain.

Discussion

Single cell ATAC-seq is an advanced technique for genome-wide profiling of chromatin accessibility at the individual cell level, enabling the study of epigenetic heterogeneity and further characterization of regulatory mechanisms beyond the transcriptional level. However, computational analysis of scATAC-seq data remains challenging due to its limited sensitivity, inherent binarization, and high dimensionality. Here, we propose scAGDE, a multi-step model-based deep graph embedded learning framework to efficiently analyze the extensive and complex scATAC-seq data. We first use a chromatin accessibility-based autoencoder to learn representations from a preprocessed count matrix, then extract peak importance scores from the model parameters of the trained encoder network to filter out less significant peaks. Following this, we utilize a Bernoulli-based graph autoencoder to simultaneously process the data and the cell graph matrix, effectively capturing critical cell-cell topology. To accurately model the distribution of the scATAC-seq data, we introduce a Bernoulli-based decoder network capable of estimating chromatin openness and integrate a combined self-supervised clustering optimization objective into the training guidance, along with other loss functions within scAGDE, to improve overall model performance. Notably, scAGDE incorporates automated cluster estimation to reduce reliance on prior knowledge and allows to bypass clustering components based on task requirements, enhancing flexibility and preserving robust embedding quality (Supplementary Note 16 and Supplementary Figs. 37 and 38).

To investigate the clustering superiority of scAGDE, we performed a comprehensive benchmarking analysis on synthetic and real-world single-cell datasets. In this analysis, scAGDE outperformed eleven other comparison methods in all simulated and real-world datasets, achieving excellent results, even in conditions of poor data quality, confirming the robustness of scAGDE. We found that scAGDE effectively reduced dimensionality and improved data visualization, outperforming five other widely-used methods, evaluated by several metrics, including the ASW value. Notably, scAGDE’s visualization ability could distinguish cell subpopulations and preserve the associations of cell identity in low-dimensional space. Furthermore, our exploration of the direct application of scATAC-seq data for scRNA-seq analysis methods revealed the potential applicability of GNN-based models, although performance was partly hindered by suboptimal cell graph design specific to scRNA-seq data.

Our study revealed that scAGDE exhibited a bias towards chromatin accessible regions with variable regulatory programs in specific cell types during the training process, particularly showing a preference for intronic and distal intergenic regions, in contrast to the promoter-dominated peaks identified by EpiScanpy. Further enhancer enrichment analysis provides evidence of the regulatory significance of scAGDE-identified peaks. Notably, we observed specific chromatin accessibility patterns around key genes such as CTLA4 and CD8A, suggesting the accessible chromatin regions function as candidate enhancers. This offers a deeper understanding of transcriptional regulatory mechanisms in immune cells and highlights the utility of scAGDE in revealing complex regulatory landscapes.

Using a mouse forebrain dataset, we demonstrated that scAGDE not only significantly reduced variation in chromatin accessibility across different cell populations but also enhanced clustering performance and the quality of visualization. Notable increases in the accessibility of gene regions associated with cellular identity were detected. In addition, we found that scAGDE improved the identification of chromatin regulatory elements after recovery of cell type-specific signals, coupled with enhanced enrichment of transcription factor motifs within these chromatin regulatory elements (CREs).

Moreover, scAGDE successfully elucidated epigenetic heterogeneity and regulatory mechanisms in the human brain frontal cortex. When clustering a human brain dataset, scAGDE distinguished several neurons and glial cell types through associations between accessible chromatin regions and marker genes. These annotations were validated by cell-type labels derived from integrated scRNA-seq data. In particular, within glutamatergic neurons, we identified three distinct subgroups, each enriched with subtly varied biological processes, highlighting the further complexity of these cells and the increased diversity of glutamatergic neuronal function.

Additionally, it is worth noting that although scAGDE is not exclusively designed for batch-effect correction, it effectively addresses batch effects while preserving biological heterogeneity using a two-step Harmony-based post-processing strategy (Supplementary Note 17 and Supplementary Fig. 39). This approach enables scAGDE to construct a cross-batch cell graph, facilitating robust integration across different cell batches. Evaluation across multi-batch datasets showed that scAGDE achieves competitive integration performance and effective batch mixing, comparable to dedicated batch-correction tools. Looking forward, scAGDE could be further enhanced by incorporating batch covariates directly into its neural network architecture to directly learn batch-corrected embeddings. This is a promising future direction to strengthen the versatility of scAGDE’s versatility in multi-batch data integration.

As with other deep learning-based approaches, scAGDE also has certain limitations that could affect its performance in specific contexts. One limitation of scAGDE is scalability when dealing with notably large datasets. While scAGDE is designed to efficiently handle typical single-cell ATAC-seq datasets, when the number of cells or peaks increases substantially (e.g., to millions of cells), memory consumption and runtime may rise significantly, leading to potential performance bottlenecks. In such cases, subsampling strategies or distributed computing methods may be necessary to mitigate this issue. Another limitation arises in dealing with notably sparse datasets. Although scAGDE is specifically designed to handle the sparsity commonly found in single-cell ATAC-seq data, leveraging its autoencoder and graph embedding components to extract meaningful patterns from moderately to highly sparse datasets, challenges may still occur when accessibility signals are detected in only a few peaks across most cell types. In these scenarios, the lack of sufficient signal may make it difficult for the model to differentiate between cell types or identify cell type-specific regulatory elements. While scAGDE is robust for sparse data, low signal-to-noise ratios may hinder its ability to extract meaningful patterns, leading to reduced clustering accuracy or embedding performance. This is not a limitation unique to scAGDE; many other algorithms face similar challenges when dealing with notably sparse data. Additionally, as the sparsity increases, managing batch effects becomes more difficult. While scAGDE incorporates the Harmony algorithm for batch effect correction, highly sparse data may require further enhancements to effectively address batch effects. The last limitation is the parameter sensitivity. Like other deep learning-based methods, the clustering results and overall performance of scAGDE can be influenced by the selection of hyperparameters, such as the weighting factors for different loss terms in the objective function. If these parameters are not optimally tuned for the specific dataset, it may result in suboptimal performance.

In summary, scAGDE has demonstrated strong performance across various tasks, including clustering, dimensionality reduction, and imputation, providing notable insights into cellular diversity and epigenetic regulation. In the future, we plan to extend the application of scAGDE to additional omics data types, such as single-cell RNA-seq and proteomics data. Given that scAGDE enables topological information integration, it is particularly suitable for handling the spatial neighborhood relationships inherent in spatial transcriptomics, epigenomics, or even multi-omics data. In addition, inspired by approaches such as scBasset⁸⁵, CellSpace⁸⁶, and SANGO⁸⁷, we have explored a variant of scAGDE, named scAGDE-M, to integrate DNA sequence information with chromatin accessibility to enhance its adaptability in multi-omics integration tasks (Supplementary Note 18 and Supplementary Fig. 40). We are confident that scAGDE has significant potential to enhance genomic research by revealing complex insights across diverse datasets.

Methods

Data preprocessing

In our study, we took the scATAC-seq count matrix ${{{\mathcal{X}}}}\in {R}^{N\times M}$ as input, where N and M denote the number of cells and peaks, respectively. The first step is to filter out peaks that are not expressed in cells, reducing the impact of high dropout rates in scATAC-seq data following the scABC model⁸. Indeed, we retained the peaks present in a minimum of 10 cells with ≥2 reads for the InSilico, GM12878vsHEK and GM12878vsHL datasets; 5 cells with ≥2 reads for the Leukemia dataset; at least 50 cells with ≥2 reads for the Forebrain dataset and at least 50 with ≥1 reads for the 10XBlood, Blood2K and DropBlood dataset. In addition, we kept all the peaks in the Splenocyte dataset. Note that these parameters were determined on the basis of the original sparsity of each dataset.

Then, to facilitate network training and optimization, we excluded rare peaks, occurring in less than δ% of cells with more than two reads, or nearly universal, present in at least (100-δ)% of cells with one or more reads according to reference⁷. Subsequently, we converted the count matrix obtained into binary data by applying a thresholding method where values greater than zero were set to one and all others were set to zero. Finally, we produced the final matrix X_pre ∈ R^N×K to ensure the stability of the analysis, where K denotes the number of filtered peaks. A final binarization was applied to each dataset prior to analysis using our scAGDE method, with no further processing for simulated datasets.

Chromatin accessibility-based autoencoder

We first adopted a chromatin accessibility-based autoencoder to filter out insignificant peaks for potential representation of the scATAC-seq data. The analysis begins with a binarized input matrix M ∈ [0, 1]^N×K defined in the range [0, 1] for N cells and K peaks, representing the cell-by-peak matrix of the scATAC-seq data. The chromatin accessibility profile of a single cell is defined as m and modeled as a Bernoulli distribution. We assign m_i,j a value of 1 if the sequencing fragments are detected at the j-th peak and 0 otherwise. The chromatin accessibility-based autoencoder begins by estimating the latent representation z_i of each observed m_i, using the principles of variational Bayes⁸⁸. The chromatin accessibility-based autoencoder utilized in our study is structured into two main components: the encoder and the decoder. The encoder is responsible for transforming the high-dimensional input data into a lower-dimensional latent space, thereby capturing the essential features of the data. This transformation is mathematically represented by:

$${{{\bf{h}}}}\,=\, \phi ({{{\bf{m}}}}{{{{\bf{W}}}}}_{e})$$

(1)

$${{{\boldsymbol{\mu }}}}\,=\, {{{\bf{h}}}}{{{{\bf{W}}}}}_{\mu }$$

(2)

$${{{\boldsymbol{\sigma }}}}\,=\, {{{\bf{h}}}}{{{{\bf{W}}}}}_{\sigma }$$

(3)

where ϕ denotes a Rectified Linear Unit activation function, which introduces non-linearity into the encoding process, enhancing the model’s ability to learn complex patterns. The variables μ (mean) and σ (standard deviation) represent the parameters of the Gaussian distribution N(μ, σ²). By employing the re-parameterization trick, we can perform efficient backpropagation and stochastic gradient descent during training. Specifically, the sampling process from the Gaussian distribution is reparameterized as follows:

$${{{\bf{z}}}}={{{\boldsymbol{\mu }}}}+{{{\boldsymbol{\sigma }}}}\,*\,{{{\boldsymbol{\epsilon }}}}$$

(4)

where z is the latent representation; ϵ is a random variable sampled from the standard normal distribution N(0, 1). This step ensures that the model remains differentiable and allows the loss gradient to propagate back through the stochastic nodes, thus facilitating effective learning of the representation of latent space.

The decoder, on the other hand, aims to reconstruct the input data from the compressed latent representation, attempting to minimize the loss between the original input and the reconstructed output. The mathematical form of the decoder function is:

$$\hat{{{{\boldsymbol{m}}}}}={{{\rm{sigmoid}}}}({{{\bf{z}}}}{{{{\bf{W}}}}}_{d})$$

(5)

where $\,{\mbox{sigmoid}}\,\left(\cdot \right)=1/1(1+{e}^{-\cdot })$ denotes the sigmoid activation function. The primary objective of model training is to minimize the negative lower bound of the marginal log-likelihood probability of the observed data, which is achieved through a combination of reconstruction loss and Kullback-Leibler (KL) divergence loss between the Gaussian distribution N(μ, σ²) and the standard normal distribution, which are formulated as follows:

$${{{{\mathcal{L}}}}}_{recon}=-{\sum }_{i=1}^{N}\left({{{{\bf{m}}}}}_{i}\log {\hat{{{{\bf{m}}}}}}_{i}+(1-{{{{\bf{m}}}}}_{i})\log (1-{\hat{{{{\bf{m}}}}}}_{i})\right)$$

(6)

$${{{{\mathcal{L}}}}}_{kl}=-\frac{1}{2}{\sum }_{i=1}^{N}\left[1+\log ({{{{\boldsymbol{\sigma }}}}}_{i}^{2})-{{{{\boldsymbol{\mu }}}}}_{i}^{2}-{{{{\boldsymbol{\sigma }}}}}_{i}^{2}\right]$$

(7)

$${{{\mathcal{L}}}}={{{{\mathcal{L}}}}}_{recon}+\beta \,*\,{{{{\mathcal{L}}}}}_{kl}$$

(8)

where ${{{\mathcal{L}}}}$ represents the loss function of the chromatin accessibility-based autoencoder and β is a weighting coefficient for the KL divergence term, often used to balance the influence of the KL term relative to the reconstruction error. This coefficient was set at 0.5 in our study according to the desired emphasis on the quality of the representation over the disentanglement of embeddings, aligning with the central principle of beta-VAE⁸⁹.

Autoencoder-based peak selection in chromatin accessibility data

We implemented a model weight-based peak selection strategy to identify crucial peaks from the original count matrix, leveraging a previously well-trained Chromatin Accessibility-based Autoencoder. The encoder network directly processes the accessibility patterns observed across each peak in all cells, capturing this information within the encoder weight matrix ${{{{\bf{W}}}}}_{e}\in {{{{\bf{R}}}}}^{d\times {d}_{h}}$, as demonstrated in Eq. 1. Consequently, each peak is connected to hidden nodes d_h with connection weights denoted as w_i, which signify the importance of each peak in the model training process. To quantify the importance s_i for j-th peak, we calculate:

$${v}_{i}={{{\rm{variance}}}}({{{{\bf{w}}}}}_{i})$$

(9)

$${s}_{i}=\frac{{v}_{i}-\min (v)}{\max (v)-\min (v)}$$

(10)

Here, s_i measures the variance of the connection weights from the peaks to the hidden nodes, serving as a metric to identify peaks of higher importance. By default, we retain the top 10,000 peaks based on their calculated importance scores. If fewer than 10,000 peaks meet this criterion, all peaks are retained. Then, we can obtain the accessibility profiles X after the autoencoder-based peak selection model.

Cell graph

After refining the feature matrix through our feature selection process, we obtained a data matrix in which each feature (peak) is considered critical for preserving cell heterogeneity. Then, we constructed a cell graph to explore the topological relationships between cells, which was based on the embedded features learned from the previously chromatin accessibility-based autoencoder. By focusing on representation learning, we aimed to improve the accuracy of defining cell-wise relationships.

Indeed, we used the K-nearest neighbors (KNN) algorithm, implemented in the scikit learn package, to build the graph where each node represents an individual cell, and the edges represent relationships between cells. For each cell, its neighbors were identified as the closest k cells, including the cell itself, based on the Euclidean distance between their embedded features. The number k was predefined to control the scale of interaction between the cells. By defining the adjacency of the cell graph, where a connection exists if another cell is within the nearest k neighbors, we effectively mapped out the domain of each cell and its immediate network. The neighborhood ${{{{\mathcal{N}}}}}_{k}(i)$ includes the nearest k cells with the i-th cell as the center. Eventually, we constructed the cell graph A as follows:

$${{{{\bf{A}}}}}_{i,j}=\left\{\begin{array}{ll}1,\quad &j\in {{{{\mathcal{N}}}}}_{k}(i)\\ 0,\quad &j \, \notin \,{{{{\mathcal{N}}}}}_{k}(i)\end{array}\right.$$

(11)

In this manner, we identified the domain of each cell and its neighbors through a local search method in a low-dimensional space, an efficient method proven to model biological networks⁹⁰.

Bernoulli-based graph convolutional autoencoder

To leverage both the accessibility and cell-wise topological connections within the cell graph, we proposed a graph autoencoder that incorporates a GCN and Bernoulli distribution. Specifically, the GCN encoder processes the cell graph and chromatin accessibility profiles simultaneously to learn the highly compressed, low-dimensional embeddings. Subsequently, we employ two decoders: one to reconstruct the cell graph matrix, thus ensuring the accurate preservation of inter-cellular topology; and another to reconstruct the count matrix, thereby capturing the global probabilistic structure.

Indeed, our graph autoencoder operates within the framework of the variational graph autoencoder⁹¹ for its proficiency in managing complex graph structures and enhancing dimensionality reduction. The GCN encoder processes the input X, representing the refined feature data, to estimate the distributional parameters of the latent variables Z, which can be formulated as below:

$${{{\bf{H}}}}=\phi (\bar{{{{\bf{A}}}}}{{{\bf{X}}}}{{{{\bf{W}}}}}_{e})$$

(12)

$${{{{\bf{Z}}}}}_{\mu }=\phi (\bar{{{{\bf{A}}}}}{{{\bf{H}}}}{{{{\bf{W}}}}}_{\mu })$$

(13)

$${{{{\bf{Z}}}}}_{\sigma }=\phi (\bar{{{{\bf{A}}}}}{{{\bf{H}}}}{{{{\bf{W}}}}}_{\sigma })$$

(14)

where $\bar{{{{\bf{A}}}}}={{{{\bf{D}}}}}^{-\frac{1}{2}}{{{\bf{A}}}}{{{{\bf{D}}}}}^{-\frac{1}{2}}$ represents the adjacency matrix of the cell graph and ${{{\bf{D}}}}=diag\left\{({{{\bf{I}}}}+{{{\bf{A}}}}){{{{\bf{1}}}}}_{N}\right\}$ denotes the degree matrix; W_e, W_μ, and W_σ are the model parameters of the corresponding layers. These parameters are critical for defining the Gaussian distribution from which the latent variable Z is sampled, leveraging the re-parameterization trick for efficient gradient descent:

$${{{\bf{z}}}}={{{\boldsymbol{\mu }}}}+{{{\boldsymbol{\sigma }}}}\,*\, {{{\boldsymbol{\epsilon }}}}$$

(15)

Here, ϵ denotes a random variable sampled from a standard normal distribution, ensuring that the model training is differentiable and that the stochastic gradient descent can effectively optimize the variational lower bound; z is one column of the latent variable Z, and μ and σ are elements in certain column of Z_μ and Z_σ, respectively. Following this, we employ an inner product decoder to reconstruct the cell graph matrix from the low-dimensional embeddings. The sigmoid activation function is chosen to map the inner product to probabilities between 0 and 1, which effectively transforms the latent embeddings back into a graph structure. This can be expressed mathematically as:

$$\hat{{{{\bf{A}}}}}={{{\rm{sigmoid}}}}({{{{\bf{Z}}}}}^{T}{{{\bf{Z}}}})$$

(16)

where Z^TZ represents the inner product of the latent representation matrix Z, and the sigmoid function maps these products to the [0, 1] range, indicating the probability of edge presence between nodes.

On this basis, we define our loss function in line with auto-encoding variational Bayes, which includes the reconstruction loss and the Kullback-Leibler (KL) divergence loss between the Gaussian distribution N(μ, σ²). These are formulated as follows:

$${{{{\mathcal{L}}}}}_{recon}=\frac{1}{{n}^{2}}{\sum }_{i=1}^{N}{\sum }_{j=1}^{N}\parallel \!{A}_{ij}-{\hat{A}}_{ij}{\parallel }_{2}^{2}$$

(17)

$${{{{\mathcal{L}}}}}_{kl}=-\frac{1}{2}{\sum }_{i=1}^{N}\left[1+log({{{{\boldsymbol{\sigma }}}}}_{i}^{2})-{{{{\boldsymbol{\mu }}}}}_{i}^{2}-{{{{\boldsymbol{\sigma }}}}}_{i}^{2}\right]$$

(18)

After that, a Bernoulli-based decoder is employed to reconstruct the count matrix, assuming each element follows a multiple Bernoulli distribution. Specifically, for a given cell x_i, the state of accessibility in j-th peak is either open or closed, taking the value 0 or 1. Here, x_ij is modeled as an independent Bernoulli random variable, with the probability of being open (b_ij) at j-th peak. Therefore, the likelihood of the observed accessibility pattern for cell x_i across all peaks is expressed as:

$$f({x}_{i}| b)={\prod }_{j=1}^{K}{b}_{ij}^{{x}_{ij}}{(1-{b}_{ij})}^{1-{x}_{ij}}$$

(19)

To learn the probabilities b_ij, we employ a neural network decoder with a sigmoid activation function at its output, which can be defined as follows:

$${{{\bf{B}}}}={f}_{d}({{{\bf{Z}}}})$$

(20)

where f_d represents a single layer fully connected neural network which can be formulated as: f_d(Z) = sigmoid(ZW_d), and B is a matrix where each element b_i,j represents the probability of accessibility at the j-th peak for the i-th cell. We can then directly use floating probability values as imputed data, reflecting the degree of chromatin accessibility in each peak for every cell, thereby avoiding the complexity and randomness associated with the sampling operation. For training the model and optimizing the parameters B, the objective is to minimize the negative log-likelihood of observation data, serving as our another loss function ${{{{\mathcal{L}}}}}_{m}$:

$${{{{\mathcal{L}}}}}_{m}= -\log {\prod } _{i=1}^{N}{\prod } _{j=1}^{K}{b}_{ij}^{{x}_{ij}}{(1-{b}_{ij})}^{1-{x}_{ij}}\\= -{\sum }_{i=1}^{N}{\sum }_{j=1}^{K}\left[{x}_{ij}\,*\, \log {b}_{ij}+(1-{x}_{ij})\,*\, \log (1-{b}_{ij})\right]$$

(21)

This loss function effectively aggregates the individual log probabilities across all cells and peaks, thereby guiding the neural network to better identify the state of each peak in terms of accessibility.

Self-supervised clustering optimization

We implemented a dual clustering optimization objective that integrates the clustering task directly into the training guidance of our graph autoencoder. Initially, we define the cluster centroids of latent embeddings as c_j, j = 1, ⋯ , N. The soft clustering assignment, which is crucial to our loss function, measures the similarity between each embedded point z_i and centroid c_j using Student’s t-distribution as the kernel. This similarity is defined as:

$${q}_{ij}=\frac{{(1+\parallel {{{{\bf{z}}}}}_{i}-{{{{\bf{c}}}}}_{j}{\parallel }^{2})}^{-\frac{1}{2}}}{{\sum }_{{j}^{{\prime} }}{(1+\parallel {{{{\bf{z}}}}}_{i}-{{{{\bf{c}}}}}_{{j}^{{\prime} }}{\parallel }^{2})}^{-\frac{1}{2}}}$$

(22)

where the embedding z_i ∈ Z corresponds to cell x_i ∈ X and the initial cluster centroids c_j are determined by k-means clustering after pre-training. Additionally, an auxiliary target distribution p_ij can be formulated as follows:

$${p}_{ij}=\frac{{q}_{ij}^{2}/{f}_{j}}{{\sum }_{{j}^{{\prime} }}{q}_{i{j}^{{\prime} }}^{2}/{f}_{{j}^{{\prime} }}},$$

(23)

where f_j = ∑_iq_ij represents the soft cluster frequencies. The objective of soft clustering optimization is to minimize the KL divergence between the soft assignment matrix Q and the target distribution P, defined as follows:

$${{{{\mathcal{L}}}}}_{soft}=KL({{{\bf{P}}}}\parallel {{{\bf{Q}}}})={\sum }_{i}{\sum }_{j}{p}_{ij}\log \frac{{p}_{ij}}{{q}_{ij}}$$

(24)

Furthermore, we introduce a self-supervised learning strategy utilizing high-confidence predictions. Initially, we derive the cluster prediction y_i as a pseudo-label from the clustering assignment q_i using $\arg \max$ function, followed by a one-hot transformation:

$${{{{\bf{y}}}}}_{i}={\phi }_{OH}({\arg \max }_{j}({q}_{ij})),\quad \,{{\mbox{s.t.}}}\,\, j=1,\cdots \,,N$$

(25)

where ϕ_OH denotes the one-hot encoding transformation. The hard self-supervised loss function is then defined as:

$${{{{\mathcal{L}}}}}_{hard}={\sum }_{i=1}^{N}{\sum }_{j=1}^{N}{q}_{ij}^{{\prime} }\,*\, {\phi }_{CE}({q}_{ij},{{{{\bf{y}}}}}_{i})$$

(26)

$${q}_{ij}^{{\prime} }=\left\{\begin{array}{ll}1,\quad &\,{{\mbox{if}}}\,{q}_{ij}\ge r,\hfill \\ 0,\quad &\,{\mbox{otherwise.}}\,\end{array}\right.$$

(27)

where ϕ_CE denotes the cross-entropy loss function and ${q}_{ij}^{{\prime} }$ serves as a weighting factor that filters out low-confidence predictions from q_ij by applying a threshold r, which is adjusted from 0.60 to 0.80 to reflect improvements in cluster assignment accuracy during the training process. The combined clustering loss is defined as:

$${{{{\mathcal{L}}}}}_{c}={{{{\mathcal{L}}}}}_{soft}+{{{{\mathcal{L}}}}}_{hard}$$

(28)

Training of scAGDE

The chromatin accessibility-based autoencoder is first trained using the loss function presented in Eq. (8), which balances the impact of the KL divergence term relative to the reconstruction error through the weighting coefficient β. In our training, we set β to 0.5 to place more emphasis on representation quality than on disentanglement, as suggested in beta-VAE⁸⁹. Next, we initiated training of our graph autoencoder by pretraining it without incorporating the clustering loss function ${{{{\mathcal{L}}}}}_{c}$. During this pretraining phase, we determine the initial cluster centroids using the k-means algorithm on the learned embedding space, setting the stage for further training. In the subsequent phase, scAGDE optimizes a composite loss function defined as follows:

$${{{\mathcal{L}}}}={\gamma }_{1}{{{{\mathcal{L}}}}}_{recon}+{\gamma }_{2}{{{{\mathcal{L}}}}}_{kl}+{\gamma }_{3}{{{{\mathcal{L}}}}}_{m}+{\gamma }_{4}{{{{\mathcal{L}}}}}_{c}$$

(29)

where ${{{{\mathcal{L}}}}}_{recon}$ represents the cell graph reconstruction loss, ensuring the faithful recreation of the original graph structure; ${{{{\mathcal{L}}}}}_{kl}$ regularizes the distribution of the latent variables, encouraging them to follow a predefined standard distribution; ${{{{\mathcal{L}}}}}_{m}$ corresponds to the Bernoulli-based decoder loss, focusing on minimizing discrepancies in the reconstructed count matrix; and ${{{{\mathcal{L}}}}}_{c}$ refers to the self-supervised clustering loss. The coefficients γ₁, γ₂, γ₃, and γ₄ are weight factors assigned to each loss term, controlling the relative importance of the different training objectives. To optimize this composite loss function, we use the Adam optimizer⁹².

Implementation details and comparisons with baseline methods

scAGDE was implemented in Python, taking advantage of the PyTorch framework that is publicly available at https://pypi.org/project/scAGDE/. The framework included both a chromatin accessibility-based autoencoder and a Bernoulli-based graph autoencoder, each featuring a two-layer encoder network (128-10) and a single-layer decoder network. For the coefficients of the loss function, we assigned values of 10, 5, 1, 1 to γ₁, γ₂, γ₃, γ₄, respectively. In this study, scAGDE was benchmarked against several state-of-the-art scATAC-seq data analysis methods including seven non-deep learning approaches and four deep learning-based models, as detailed below:

• Single cell accessibility based clustering (scABC, https://github.com/timydaley/scABC)⁸. scABC employs a weighted K-medoids clustering approach to partition cells and refines the clustering based on landmarks for each cluster, ensuring more accurate clustering assignments.

• An unsupervised Bayesian framework based on topic modeling (cisTopic, http://github.com/aertslab/cistopic)⁹. cisTopic utilizes latent Dirichlet allocation (LDA) with a collapsed Gibbs sampler to iteratively classify regions into regulatory topics and cluster cells based on their regulatory topic contributions.

• Single nucleus analysis pipeline for ATAC-seq (SnapATAC, https://github.com/r3fang/SnapATAC)¹⁰. SnapATAC applies dimensionality reduction on a Jaccard similarity matrix that measures the overlap in accessibility between cells and accounts for sequencing depth variations with regression-based normalization.

• A comprehensive toolkit for single-cell chromatin data (Signac, https://github.com/timoast/signac/)²⁴. The Signac toolkit employs Latent Semantic Index (LSI), leveraging the frequency-inverse document frequency transformation and principal component analysis to reduce the dimensionality of the binarized accessibility matrix.

• An integrative and scalable software package for scATAC-seq data (ArchR, https://github.com/GreenleafLab/ArchR)²¹. The ArchR package implements an iterative LSI for dimensional reduction and call clusters in this reduced dimension sub-space.

• A fast, scalable and versatile tool for scATAC-seq data (SnapATAC2, https://github.com/kaizhang/SnapATAC2. SnapATAC2 is a pairwise-similarity based method and obtains cell embeddings through Laplacian eigenmaps. SnapATAC2 is an enhancement of SnapATAC.

• A graph embedding method that jointly embeds cells and features (SIMBA, https://github.com/sgoldenlab/simba). SIMBA explicitly learns low-dimensional representations of single cells and their defining features, such as chromatin accessible regions, and DNA sequences into a common latent space.

• Single-cell ATAC-seq analysis via latent feature extraction (SCALE, https://github.com/jsxlei/SCALE)¹³. SCALE integrates a probabilistic Gaussian Mixture Model into the VAE to learn latent embeddings.

• Scalable and accurate invariant representation learning (SAILER, https://github.com/uci-cbcl/SAILER)¹⁴. SAILER adopts the VAE model with additional constraints to ensure that the learned representations are independent of confounding factors.

• A deep generative model for scATAC-seq data analysis (PeakVI, https://github.com/YosefLab/scvi-tools/)¹⁵. PeakVI employs a VAE with an additional neural network to estimate technical factors that impact the probability of observing an accessible region.

• A batch-adversarial variational auto-encoder approach (BAVARIA, https://github.com/BIMSBbioinfo/bavaria)¹⁸. BAVARIA uses a VAE and incorporates dimensionality reduction and batch correction through an adversarial learning strategy.

Additionally, we compared five scRNA-seq analysis methods, including both graph-based and nongraph-based, detailed below:

• R toolkit for single-cell genomics (Seurat, https://github.com/satijalab/seurat)⁵. Seurat is an R package specifically designed for analyzing and exploring single-cell RNA-seq data.

• Single-cell variational inference approach (scVI, https://github.com/YosefLab/scVI)⁹³. scVI utilizes stochastic optimization and deep neural networks to analyze single-cell RNA-seq data, effectively aggregating information across similar cells and genes to model the distribution underlying observed gene expression.

• Single-cell model-based deep learning method (scDeepCluster, https://github.com/ttgump/scDeepCluster)⁹⁴. scDeepCluster employs a ZINB model-based autoencoder to simultaneously learn feature representations and perform clustering.

• A single-cell model-based deep graph embedding clustering method (scTAG, https://github.com/Philyzh8/scTAG⁹⁵. scTAG integrates a ZINB-based model with a topology adaptive graph convolutional autoencoder to learn low-dimensional latent representations.

• A graph convolutional autoencoder framework (scMGCA, https://github.com/Philyzh8/scMGCA)³³. scMGCA leverages a graph-embedding autoencoder and introduces a notable method for constructing cell graphs.

The detailed implementation, including function specifications and parameter settings, is provided in Supplementary Note 19.

Benchmarking metrics for clustering and visualization

We employed multiple metrics to evaluate the performance of the clustering algorithms, including NMI, ARI, F1 score, V-Measure and Homogeneity score. The NMI is a metric that normalizes the mutual information (MI) score to a range between 0 and 1 and can be defined as:

$${{{\rm{NMI}}}}(X,Y)=\frac{{{{\rm{I}}}}(X,Y)}{\sqrt{{{{\rm{H}}}}(X){{{\rm{H}}}}(Y)}}$$

(30)

where I() is the mutual information metric and H() is the entropy metric. The ARI score is the corrected for chance of the rand index (RI) score and can be defined as:

$${{{\rm{ARI}}}}=\frac{{\sum }_{ij}\left(\begin{array}{c}{n}_{ij}\\ 2\end{array}\right)-\left[{\sum }_{i}\left(\begin{array}{c}{n}_{i}\\ 2\end{array}\right){\sum }_{j}\left(\begin{array}{c}{n}_{j}\\ 2\end{array}\right)\right]\bigg/\left(\begin{array}{c}n\\ 2\end{array}\right)}{\frac{1}{2}\left[{\sum }_{i}\left(\begin{array}{c}{n}_{i}\\ 2\end{array}\right)+{\sum }_{j}\left(\begin{array}{c}{n}_{j}\\ 2\end{array}\right)\right]-\left[{\sum }_{i}\left(\begin{array}{c}{n}_{i}\\ 2\end{array}\right){\sum }_{j}\left(\begin{array}{c}{n}_{j}\\ 2\end{array}\right)\right]\bigg/\left(\begin{array}{c}n\\ 2\end{array}\right)}$$

(31)

where n_ij represents the entries in the contingency table formed by the true partition and predicted partition. The F1 score combines the precision and recall scores of a prediction and is calculated as the harmonic mean of the precision and recall scores, as shown below:

$${{{\rm{F1}}}}=\frac{2\times {{{\rm{Precision}}}}\times {{{\rm{Recall}}}}}{{{{\rm{Precision}}}}+{{{\rm{Recall}}}}}$$

(32)

V-Measure is an external entropy-based clustering evaluation metric, which can be defined as:

$$v=\frac{2hc}{h+c}=\frac{2\left(1-\frac{{{{\rm{H}}}}\left(C| K\right)}{{{{\rm{H}}}}(C)}\right)\left(1-\frac{{{{\rm{H}}}}\left(K| C\right)}{{{{\rm{H}}}}(K)}\right)}{\left(1-\frac{{{{\rm{H}}}}\left(C| K\right)}{{{{\rm{H}}}}(C)}\right)+\left(1-\frac{{{{\rm{H}}}}\left(K| C\right)}{{{{\rm{H}}}}(K)}\right)}$$

(33)

where $\frac{{{{\rm{H}}}}\left(C| K\right)}{{{{\rm{H}}}}(C)}$ represents the conditional entropy of cell type division given the cluster division condition; h is the homogeneity measure and c is the completeness measure. Eventually, the V-Measure value v is the harmonic mean of h and c measures, and the value is larger when two divisions are closely separated. The Homogeneity score is used to evaluate the clustering algorithm’s ability to assign cells of a single cell type to a single cluster. The score, denoted as h, ranges from 0 to 1, with a higher value indicating better homogeneity. It’s defined as:

$$h=1-\frac{{{{\rm{H}}}}({Y}_{true}| {Y}_{pred})}{{{{\rm{H}}}}\left({Y}_{true}\right)}$$

(34)

where ${{{\rm{H}}}}({Y}_{true}| {Y}_{pred})$ is the probability of assigning true cells to a set of predicted cells, and ${{{\rm{H}}}}({Y}_{true})$ are the labels of the cells. These five metrics all range from 0 to 1, and a higher score denotes a better-quality performance.

Then, to evaluate the quality of dimensionality reduction and visualization, we used the ASW, the CHI, and the DBI. The ASW is calculated using the average intra-cluster distance a and the average nearest-cluster distance b across all samples and the ASW can be defined as:

$${{{\rm{ASW}}}}(x)=\frac{1}{n}{\sum }_{i=1}^{n}\left(\frac{b(i)-a(i)}{\max \left\{a(i),b(i)\right\}}\right)$$

(35)

The CHI measures the compactness and separation of clustering results by calculating the ratio of the between-cluster separation SS_B to the within-cluster dispersion SS_W, normalized by their number of degrees of freedom:

$${{{\rm{CHI}}}}(i)=\frac{{{{{\rm{SS}}}}}_{B}/(k-1)}{{{{{\rm{SS}}}}}_{W}/(n-k)}$$

(36)

where n is the number of samples and k is the corresponding number of clusters. The DBI measures the average maximum similarity of each cluster. Its calculation involves both the inter-cluster distances denoted and the intra-cluster separation degree, denoted as S_i, as shown below:

$${{{\rm{DBI}}}}=\frac{1}{N}{\sum }_{i=1}^{N}{\max }_{j\ne i}\left(\frac{\overline{{S}_{i}}+\overline{{S}_{j}}}{{\parallel {w}_{i}-{w}_{j}\parallel }_{2}}\right)$$

(37)

where ${\parallel {w}_{i}-{w}_{j}\parallel }_{2}$ calculates the Euclidean distance between i-th cluster and j-th cluster, and $\overline{{S}_{i}}$ represents the average distance of all samples of i-th cluster to the cluster centroid. These metrics range from 0 to 1, where a higher score generally indicates a better clustering or visualization quality, except for DBI where a lower value indicates better clustering.

Peaks annotation with ChIPseeker

We conducted an analysis of the peak categories identified by scAGDE in scATAC-seq data, specifically examining their proximity to the nearest transcription start site (TSS) regions. To achieve this, we utilized the ‘annotatePeak’ function from ChIPseeker, an R package specifically developed for ChIP peak annotation³⁴. Considering that our samples were derived from human tissues, we set the parameters ‘TxDb=TxDb.Hsapiens.UCSC.hg19.knownGene’ and ‘annoDb=org.Hs.eg.db’ to ensure alignment with the human genome annotation.

Identifying differentially accessible peaks

To carry out differential peaks analysis, we followed Stuart et al.²⁴ and performed a Wilcoxon rank sum test implemented in Seurat package, using the ‘FindAllMarkers’ function. We set the parameters ‘min.pct=0.15’ and ‘logfc.threshold=0.15’ to keep significantly differentially accessible peaks. When using ArchR package, we applied the ‘getMarkerFeatures’ function with default parameter setting to identify marker peaks.

Identifying peak-associated genes using rGREAT

In our study, to examine the biological relevance between uncovered peaks and cell identity, we used the R package of rGREAT⁴⁷ to identify peak-associated genes. We applied the ‘great’ function with parameters set to ‘basal_upstream = 15,000’ and ‘basal_downstream = 15,000’. The source of TSS was set to ‘txdb:mm10’ and the gene sets collections were obtained from MSigDB⁹⁶ by using the ‘msigdbr’ function with parameters set to ‘species = Mus musculus’.

Known and de novo motifs enrichment analysis

To identify underlying regulators within chromatin accessible regions, we conducted both de novo and known motif enrichment analyses on the datasets. For the de novo motifs discovery, we employed the HOMER tool⁵² with parameters set to ‘-size 200 -mask’, which helps to identify marked regulatory elements that may be associated with the regulation of gene expression. For the enrichment of known motifs, we applied the ‘FindMotifs’ function of Signac²⁴ and the ‘peakAnnoEnrichment’ function from ArchR²¹ across the data. This comprehensive analysis allowed us to contrast the presence and potential functional impact of both newly discovered and established motifs within the chromatin accessible regions of our scATAC-seq data.

Transcription factor deviation scores

We measured transcription factor activity using the chromVAR algorithm, integrated into the Signac package²⁴ in R. Initially, we utilized the ‘addMotifs’ function to incorporate motif information into the Seurat object, sourcing motif position frequency matrices from the JASPAR database via ‘getMatrixSet’. Subsequently, we executed the ‘RunChromVAR’ function on the raw, imputed, and binary imputed count data, setting the genome parameter to ‘BSgenome.Mmusculus.UCSC.mm10’.

GO enrichment analysis

We performed GO enrichment analysis on these datasets using the R package ClusterProfiler⁹⁷. Initially, we selected the top 100 genes ranked by the lowest false discovery rate (FDR) from differentially expressed genes based on each cluster. These genes were identified from the ‘GeneScoreMatrix’ data calculated by ArchR²¹. We employed the ‘enrichGO’ function for this analysis, setting the parameters to ‘OrgDb = org.Hs.eg.db, p-valueCutoff = 0.05, q-valueCutoff = 0.05, with all other parameters at their default values. For enrichment analysis of three groups of glutamatergic neurons, we selected differentially expressed genes with FDR ≤ 0.05 and Log2FC ≥ 0.75. The parameters for this subsequent analysis are consistent with those described previously.

Integrative analysis of human brain scATAC-seq and scRNA datasets

To further evaluate our cell type identifications and detect regulatory differences between three glutamatergic neuron clusters, we integrated the scRNA-seq data of the primate dorsolateral prefrontal cortex⁷⁶. To perform integrative analysis, we first manually filtered scRNA-seq data from human samples. Second, we added the cell embeddings from scAGDE to the ArchR project. Next, we used the function ‘addGeneIntegrationMatrix’ to perform the final integration, with the parameter ‘reducedDims’ to the slot name of scAGDE embeddings. This operation employs the function ‘FindTransferAnchors’ in Seurat to integrate datasets. Following the integration, we identified peak-to-gene linkages by leveraging the integrated scRNA-seq data to detect correlations between chromatin accessibility and the expression of genes of interest. Specifically, we utilized the ‘addPeak2GeneLinks’ function, similarly setting the ‘reducedDims’ parameter to the slot name of scAGDE embeddings to ensure consistency in data handling.

Annotating cell types to cell clusters

In our study, we annotated cell types in the scATAC-seq data by evaluating the cell-type enrichment score that measures the inferred activity of marker genes for each cell type. We first sourced the top 50 marker genes for a range of cell types from two marker gene databases: PanglaoDB⁶⁴ and CellMarker⁶⁵. We then extracted gene activity data from the ‘GeneScoreMatrix’ by ArchR tool, and calculated the average activities across clusters with reads per million mapped reads (RPM) normalization. For each defined cell type, we computed the raw enrichment scores as the mean RPM values across all the marker genes within each cluster. Subsequently, a double z-score transformation was applied on the raw enrichment scores, first across cell types and then across cell clusters, to derive the final enrichment scores. The cell type with the highest enrichment score was selected as the definitive annotation for each cell cluster.

Statistics and reproducibility

The detailed statistical tests were explained in each figure legend. Sample data were obtained from public repositories. Sample size was not predetermined and is the maximum number of samples available for each datasets. No data were excluded from the analyses. No experimental groups were assigned. Our study does not involve group allocation that requires blinding. To reproduce the results, please find the Source Data file we provided.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.