Introduction

Circulating tumor cells (CTCs) are malignant cells that detach from primary or metastatic tumors and enter the bloodstream, where they act as precursors to distant metastasis and serve as real-time indicators of disease status1,2. As early as stage 0, CTCs can be detected in peripheral blood, reflecting the potential for dissemination before radiographic or symptomatic manifestation3. Their abundance and phenotypic characteristics have been linked to tumor aggressiveness, therapeutic resistance, and patient prognosis4,5, making them a valuable biomarker for early cancer detection, risk stratification, and longitudinal monitoring. Unlike tissue biopsies, CTCs can be accessed noninvasively and repeatedly, enabling dynamic insight into tumor evolution and therapeutic response over the course of treatment6.

However, detecting and enumerating CTCs via cytometry, biomarker labeling, or molecular analysis remains challenging due to their extreme rarity and phenotypic variability1. Specifically, a single milliliter of blood contains approximately 10⁶ white blood cells (WBCs) and 10⁹ red blood cells (RBCs)7,8, but may contain fewer than one CTC in early-stage cancers9,10 and rarely more than ten CTCs even in late-stage patients4. CTC surface marker expression is highly heterogeneous. Fluorescent markers such as EpCAM and cytokeratin may identify only a subset of tumor cells, particularly those undergoing epithelial-to-mesenchymal transition (EMT), during which epithelial markers are frequently downregulated10,11. Beyond molecular variability, CTCs also differ widely in physical properties, including size, deformability, and density. These features can substantially overlap with those of WBCs, further complicating accurate discernment12,13,14.

Despite the rarity and phenotypic heterogeneity of CTCs, numerous detection methods have been developed that rely on molecular markers or physical traits, broadly categorized as antigen-based and biophysical separation approaches15,16,17,18. Among antigen-based methods, CellSearch® remains the most clinically validated platform, enriching EpCAM-positive cells via immunomagnetic capture19, with prognostic utility established in metastatic cancers4,5. Subsequent microfluidic platforms like the CTC-Chip20 and Herringbone-Chip21 improved sensitivity through micropost arrays and passive mixing. Recent advancements include aptamer-functionalized nanostructures22, droplet microfluidic single-cell analysis23, and 3D micro-/nanostructured interfaces that enhance capture efficiency via fluid dynamics24. However, antigen-based methods often fail to capture phenotypically heterogeneous CTC subpopulations9,11, while microfluidic platforms with complex architectures face challenges in large-scale implementation24.

Various biophysical methods have been developed to isolate CTCs by exploiting intrinsic properties such as size, density, deformability, and electrical or mechanical characteristics18,25. Specific techniques include size-based filtration such as the early ISET platform26, inertial microfluidics exemplified by the FDA-cleared Parsortix® system27, and deterministic lateral displacement (DLD)28,29, as well as dielectrophoresis13 and acoustic separation30. However, the wide variability in CTC phenotypes and their physical overlap with WBCs often leads to limited specificity, reduced purity, and potential loss of CTCs in clinical implementation18,25,31.

Hybrid strategies have emerged to address the limitations of single-modality CTC detection by combining biophysical and immunoaffinity-based mechanisms32. Examples include the CTC-iChip integrating inertial focusing with magnetic WBC depletion33, size-dictated immunocapture chips enabling geometric enrichment and profiling34, a high-throughput inertial–magnetic sorter for leukapheresis-scale processing35, and integrated inertial–magnetophoretic microdevices36. While these systems improve sensitivity and subtype coverage, they remain limited by device complexity, high cost, and cumulative CTC loss. These challenges highlight the need for streamlined, high-fidelity platforms compatible with clinical workflows.

Digital holographic microscopy (DHM) has emerged as a promising alternative for label-free, high-content imaging of biological samples12. As a coherent imaging technique, DHM captures both the amplitude and phase of transmitted light by recording its interference with a reference wave37. Compared to bright-field and other incoherent imaging methods, DHM provides a significantly larger depth of focus and encodes optical thickness and 3D spatial data, supporting applications such as cell culture monitoring, microbe classification, and particle tracking38,39,40. Despite its advantages, recent explorations of DHM for CTC detection remain constrained by limited throughput, poor specificity at the patient level, and a lack of validation using clinical blood samples41,42.

To address these challenges, we developed a clinically scalable platform that integrates real-time DHM with inertial microfluidic enrichment and orthogonal immunophenotyping. The system is compact, cost-effective, and readily parallelizable, enabling rapid processing of standard 10 mL blood tubes within hours. A customized deep learning model trained on millions of healthy blood cells and a diverse panel of cancer cell lines enables robust classification of patient-derived images in flow. The platform achieves a low patient-level false positive rate of 1 cell/mL and supports downstream analyses such as single-cell multi-omics and 3D tissue culture. In the first clinical pilot cohort application of DHM for CTC detection, our method distinguishes late-stage prostate cancer patients from healthy controls, demonstrating its potential as a clinically viable tool for cancer diagnostics and screening.

Results

Integrated platform for high-precision, high-throughput in-flow CTC detection

Our platform unites three orthogonal technologies—patented saw-tooth inertial microfluidics, DHM, and dual-channel immunofluorescence (IF)—into an optimized in-flow process that overcomes both the low abundance and phenotypic heterogeneity of CTCs (Fig. 1). To begin, whole blood is diluted in a proprietary buffer media aimed at minimizing cell loss through adhesion to plastic surfaces. Next, it flows through Astrin’s patented saw-tooth inertial microfluidic chip (WO 2024/064911 A1), which enriches CTCs by depleting hematologic background. This process removes over 99.999% of RBCs and approximately 99.6% of WBCs (Fig. 2), while retaining 95% of spiked cancer cells (Supplementary Information Section 1, SI §1). Overall, this results in more than a 100-fold enrichment of CTCs in the processed sample (Fig. 2). A detailed description of the microfluidic chip and sample preparation protocols is provided in Methods and SI §1.

Fig. 1: System overview, depicting the detection of CTC candidates based on a combination of microfluidic inertial enrichment, digital holographic microscopy signature, and immunofluorescence (IF) expression.
Fig. 1: System overview, depicting the detection of CTC candidates based on a combination of microfluidic inertial enrichment, digital holographic microscopy signature, and immunofluorescence (IF) expression.
Full size image

Samples begin as whole blood prior to microfluidic enrichment, during which red blood cells are primarily depleted while retaining CTCs and most WBCs. Subsequently, holograms of each cell and IF signal from the field-of-view as a whole are captured during passage through a second microfluidic chip. Detections in the IF signal for emitting cells are visible as broad peaks whose time-scale is inversely proportional to the frame rate (i.e., equivalent to the time taken for a cell to passage the field-of-view). Holograms pass through a neural network to classify each cell and detect CTCs, while the IF data can be used for further filtering operations for cell enumeration.

Fig. 2: Cumulative end-to-end system depletion performance, where each stage builds on the depletion of the prior stage.
Fig. 2: Cumulative end-to-end system depletion performance, where each stage builds on the depletion of the prior stage.
Full size image

Therefore, immunofluorescence depletion numbers are not from immunofluorescence data alone, but rather in conjunction with holography data. Cell counts for whole blood are based on Cleveland Clinic complete blood count data for red blood cells and white blood cells in males8. Microfluidic, holographic, and immunofluorescence depletion data are measured on the imaging system. Fold depletion is calculated in terms of means, rather than medians, as the median value for immunofluorescence stage is zero for RBCs. Each of the four stages shown here mirrors those depicted in Fig. 1.

Subsequently, the enriched sample is hydrodynamically focused into a straight microfluidic channel and analyzed using our dual-modality imaging system (details in Methods). A digital camera captures the holograms generated by a pulsed 405 nm laser, encoding both the optical and morphological information of the cells (Fig. 1). A customized deep learning neural network processes these holograms in real-time (as described in the next section and detailed further in Methods). The network produces a spatial probability heatmap highlighting potential CTC locations, as learned from training on annotated images of cancerous and healthy cells.

Such in-flow CTC classification via holographic imaging provides an avenue for a completely label-free enrichment platform and can improve over time as training datasets grow and deep learning models evolve. Crucially, the label-free DHM system remains compatible with IF-based molecular labeling to enhance the accuracy of the system or provide secondary enumeration metrics (e.g., degree of EpCAM expression). For the system presented herein, two photomultiplier tubes (PMTs) are integrated to capture IF signals emitted by tumor-specific antibody conjugates upon excitation by a co-linear 488 nm laser (Fig. 1). A custom signal processing algorithm (SI §2.1) identifies true cellular fluorescence peaks amid background noise in each PMT channel, performs cross-channel matching, and integrates the resulting IF readouts with predictions from the DHM classifier.

Overall, the integration of biophysical profiling from deep learning based DHM and biochemical readout from IF substantially reduces candidate events and false positives (Fig. 2). The DHM classifier and IF channel each reduce the nucleated cell background by approximately 100 fold, yielding a post fusion false positive rate of about 1 cell per mL, which is sufficient for patient level discrimination in our cohort. At our prespecified operating point, DHM based classification does not meet the per milliliter false positive constraint, so a fused DHM and IF readout is necessary. Conversely, IF by itself lacks adequate sensitivity and stability due to variable antigen expression and occasional nonspecific fluorescence. This dual modality design also mitigates dependence on antigen status, for example during EMT with EpCAM downregulation, because antigen-low CTCs can still be identified by their label-free holographic signatures, while spurious fluorescence is suppressed by the DHM classifier.

Deep Learning Framework

The benefits of using DHM for label-free CTC detection are counterbalanced by several intrinsic challenges. Chief among these is the extreme scarcity of CTCs, which makes it difficult to compile a sufficiently large and diverse set of positive examples for training a robust detection model. Available public datasets are limited and primarily consist of cultured cancer cell lines, and holographic image datasets are even rarer. However, patient-derived CTCs exhibit far broader phenotypic heterogeneity than any single cell line can represent43,44. Therefore, we train the detector to recognize deviation from a large set of white and red blood cells from healthy patient donors, and we regularize the positive class with a diverse set of epithelial cancer cell lines acquired across many independent imaging sessions. This design aims to preserve generalizability to true CTCs, and avoids biasing the label-free model toward antigen-high phenotypes that IF preferentially labels.

Figure 3 summarizes the strategy we adopted to address these challenges while maintaining the real-time throughput required for in-flow assays. We formulated the detection task as an image-to-heatmap transformation in which a streamlined High-Resolution Network (HR-Net)45(SI §3.1) directly yields a probability surface. Local maxima above a chosen confidence threshold correspond to putative cells. Representing cells as Gaussian key points, rather than bounding boxes, obviates the need for precise manual annotation of poorly-defined holographic fringes. The number and depth of model layers were customized to strike a balance between descriptive depth and inference speed—ensuring both high prediction accuracy while supporting live processing of full-field holograms (architectural details in Methods).

Fig. 3: Deep learning model training process.
Fig. 3: Deep learning model training process.
Full size image

Positive and negative samples are generated by imaging cell lines and healthy blood (post-microfluidic enrichment) in separate samples. All images derived from healthy blood are labeled negatively (i.e., lacking any Gaussian keypoint label). Cell lines are spiked into buffer, and all nucleated cells are labeled as positive via a pseudo-labeler neural network, which is itself trained from human-generated labels. The pseudo-labeler is never applied to images with blood cells. Although the pseudo-labeling model is trained from human-generated labels, the CTC detection model can be trained in an automated fashion, enabling large datasets. During each training epoch, the effects of false negatives from the pseudo-labeler (i.e., unlabeled cell line cells) are mitigated using pixel-wise asymmetric loss, wherein CTC model predictions lacking a corresponding pseudo-labeler label (depicted as yellow Gaussian blobs) are penalized at 1/10th the rate of false positives and false negatives. Each training epoch also uses hard sample mining such that images from low-performing runs from healthy controls are preferentially sampled in subsequent epochs.

Training data were deliberately assembled to counter both the rarity of CTCs and their morphological diversity. Two complementary streams were employed (Fig. 3, left side). The negative stream comprised enriched healthy blood, exposing the network to the full spectrum of WBCs, RBCs, debris, and background variation. The positive stream consisted of suspensions of five histologically distinct cancer cell lines imaged in buffer. No blood cells were mixed with these cancer cell line runs, and therefore every nucleated object in these frames is a putative cancer cell. Because such images lack pixel-level labels, a companion HR-Net was first trained on select hand-annotated frames to create provisional “pseudo-labels”. It is important to note that pseudo-labels are only applied to the cell lines. The pseudo-labeler is never given the more difficult task of classifying between healthy and cancer, only to label all nucleated cells in buffer. During subsequent detector training, images that produced misclassifications were resurfaced in successive epochs by hard-sample mining (further details in Methods), ensuring repeated exposure to the most confounding features.

An asymmetric cross-entropy loss embodied the biological class imbalance: weights were skewed towards healthy-blood examples to penalize WBC misclassification while remaining tolerant of occasional pseudo-labeling errors in the positive stream. Further details can be found in Methods. After training, additional cancer lines—25 in total—were sequestered for validation, demonstrating that the network learns invariant features that generalize beyond the limited set of cultures available for optimization. The full list of cell lines, hyper-parameters, and augmentation protocols are provided in Methods and SI §4 Tables 3–4.

Together, the key-point formulation, HR-Net customization, heterogeneity-centered sampling, and asymmetric loss establish a conceptually distinct framework (Fig. 3, center and right panels) that narrows the gap between idealized cell-line imagery and the complex reality of patient-derived CTCs while sustaining the high frame rates required for population-scale liquid biopsy.

System Validation

To evaluate the overall detection performance of the system, we conducted validation experiments using both spiked and unspiked blood samples. Each specimen was prepared using approximately 3 mL of healthy donor whole blood. One group was spiked with LNCaP prostate cancer cells (n = 18), while the control group remained unspiked (n = 18). To enable fluorescence-based confirmation, cancer cells were stained with a cell tracker dye prior to spiking, simulating an ideal condition with uniform marker expression and minimal nonspecific labeling. The unspiked control cohort was used to determine the system’s false positive rate (FPR). In contrast, the spiked samples were used to calculate the recovery rate, defined here as the proportion of introduced cancer cells that were successfully identified by the system. This recovery rate reflects the true positive rate (TPR), also referred to as sensitivity. Full experimental protocols are described in SI §5.1.

To contextualize these results, we also generated a theoretical recovery curve based on a large validation dataset containing 25 cancer cell lines not used during training. As shown in Fig. 4a, the recovery rate increases approximately linearly as the model confidence threshold decreases—from about 30% at a threshold of 0.9 to nearly 80% at 0.2. This trend reflects the expected trade-off between sensitivity and specificity. At higher thresholds, the system is more conservative and yields fewer detections, while lower thresholds allow more potential CTCs to be captured at the cost of admitting more false positives. Across the range from approximately 0.2 to 0.8, the experimental recovery rate was consistently lower than the theoretical prediction, with a maximum deviation of approximately 10%. This discrepancy may be attributed to biological variability between the controlled cell line conditions (i.e., cell lines in pure buffer) versus the more complex background of spiked blood samples (i.e., cell lines in blood). It also reflects necessary differences in how recovery is defined for these two different conditions (further details are available in SI §5). Nonetheless, both curves showed qualitatively similar trends, indicating that the model generalizes well across different sample contexts.

Fig. 4: Model validation using spiked experiments.
Fig. 4: Model validation using spiked experiments.
Full size image

a Measured recovery at different operating thresholds. b False positive rate (FPR), shown as discrete points due to the low number of false positives detected, with only 5 false positive cells found across all unspiked samples (totaling 55 mL of whole blood) at the operating threshold of 0.5.

Given the expected abundance of CTCs in late-stage cancer patients, estimated at around 10 cells/mL, an operating threshold of 0.5 was chosen. At this threshold, the system achieved a TPR of around 60% while maintaining a low FPR. Lowering the threshold would improve sensitivity but introduce more false positives, which can undermine specificity in clinical use. This balance between FPR and TPR is critical for optimizing diagnostic accuracy, especially in the context of rare cell detection. Fig. 4b shows the measured FPR across a range of thresholds, expressed in terms of false positive detections per mL in the control samples. Because false positive events were rare, the plot is composed of discrete points. At the chosen threshold of 0.5, only 5 false positives were detected in total across all healthy blood samples, corresponding to fewer than 0.1 false positives/mL (total unspiked sample volume was approximately 55 mL, i.e., slightly more than 3 mL per sample from the 18 samples). Even at the most permissive threshold of 0.1, the FPR remained below 1 cell/mL. Data concerning the model’s FPR based on nucleated cells in healthy blood without IF data can be found in SI §3.2. This analysis helps clarify how FPR behaves in the absence of orthogonal molecular labeling and supports evaluation of system performance in strictly label-free settings.

To complete the assessment of diagnostic performance, we estimated the positive predictive value (PPV, or precision), which defines the probability that a detected cell is a true positive. While our validation experiments do not permit a direct measurement of PPV, it can be calculated from the experimentally determined TPR and FPR. For this estimation, we assumed a clinically representative CTC abundance of 10 cells/mL. At the chosen operating threshold of 0.5, this model yields a PPV of approximately 0.98. Such a high PPV underscores the system’s strong specificity and indicates that detections are highly likely to be true positives, a critical feature for reliable rare cell analysis.

Patient Sample Performance

To evaluate the efficacy of our platform under clinically relevant conditions, we assessed its ability to detect CTCs in blood samples from prostate cancer patients in comparison to healthy donors. The patient cohort comprised 13 male subjects diagnosed with metastatic castration-resistant prostate cancer (mCRPC), while the healthy control cohort consisted of 8 male donors without any known cancer diagnosis. Blood samples from both cohorts were each collected as part of separate studies. As such, subsequent analysis was not strictly blinded, though significant efforts were made to analyze all samples equally. Samples were collected within the same time window for both cohorts, between November 1, 2023, and February 1, 2024. There were no differences in protocol followed by the laboratory or in subsequent data processing between cohorts, as the same microfluidic enrichment, immunofluorescent staining, data capture and CTC enumeration steps were followed.

Patient samples were subjected to staining for prostate-specific membrane antigen (PSMA) and EpCAM proteins, commonly expressed on the surface of prostate cancer cells and epithelial cells, respectively46. A cell was classified as a CTC only if it exceeded the holography model threshold of 0.5 and exhibited a positive PSMA signal, as determined by PMT scoring (see SI §2.1 for details). This dual requirement was designed to ensure high tumor specificity and reduce the likelihood of misclassifying non-malignant nucleated cells. Although EpCAM expression was also measured, it was intentionally excluded from the criteria for CTC identification. This decision reflects the well-established observation that EpCAM expression can be downregulated in CTCs undergoing epithelial-to-mesenchymal transition (EMT), a process associated with increased metastatic potential and poorer prognosis9,11. Relying on EpCAM alone would risk missing a clinically significant subset of tumor cells. Instead, EpCAM was used as a phenotypic marker to help characterize the epithelial profile of the detected PSMA-positive cells.

This molecular gating strategy revealed a substantial proportion of EpCAM-negative CTCs: across the patient cohort, only 37% of PSMA-positive cells were also EpCAM-positive. This finding has important implications. It suggests that a large fraction of CTCs would likely evade detection in conventional EpCAM-based platforms, particularly in patients with epithelial-to-mesenchymal phenotypic shifts. By leveraging label-free holographic morphology in conjunction with PSMA, the system is capable of identifying a broader spectrum of tumor cells, including EpCAM-low or EpCAM-negative populations.

Figure 5a shows the distribution of detected CTC counts across both cohorts, with values normalized to the initial whole blood volume to account for variability in sample collection. Cancer patients exhibited a markedly higher CTC burden than healthy donors, with median counts of 12.5 cells/ml and 1.5 cells/ml, respectively (counts per patient are provided in SI §6). These findings are consistent with previously reported ranges for late-stage prostate cancer (e.g.47,) and support the system’s ability to reliably distinguish clinical from non-clinical samples. The spread within the patient group reflects the expected biological heterogeneity in disease progression and tumor shedding rates. Additionally, several patients exhibited CTC counts exceeding 20 cells/ml, while healthy individuals exhibited consistently low counts with few outliers, further reinforcing diagnostic discriminability. Representative holograms of CTCs detected in patient samples are shown in Fig. 5b–e, along with corresponding IF signals in Fig. 5f–i. The detected cells display a wide range of morphological features and antigen expression profiles, including EpCAM-negative phenotypes that are often underrepresented in traditional antigen-based CTC detection assays such as CellSearch®.

Fig. 5: Detection rate of the model evaluated on a cohort of healthy donors and donors with late-stage prostate cancer (mCRPC).
Fig. 5: Detection rate of the model evaluated on a cohort of healthy donors and donors with late-stage prostate cancer (mCRPC).
Full size image

a Distribution of detected CTCs in patient samples, normalized to the volume of whole blood in the initial tube to account for sample variability. b-e CTCs identified in prostate cancer patient samples, with corresponding immunofluorescent (IF) signal f–i. For the two IF channels captured, PSMA served as a prostate-specific marker and EpCAM served as a pan-cancer marker (for epithelial cells).

These findings suggest that the system’s applicability extends to the detection of patient-derived CTCs, notwithstanding its initial training using cultured cell lines. Similar to the spiked experiments, the utilization of PSMA staining effectively decreased false positive signals in healthy samples. The results from these patient samples provide critical evidence of the system’s efficacy beyond controlled lab settings and emphasizes the potential of our platform as a powerful tool for cancer diagnostics and screening.

Discussion

Technical Innovation and Comparison with Existing CTC Diagnostics

We present a multimodal CTC detection platform that combines inertial microfluidic enrichment, digital holographic microscopy (DHM), and immunofluorescence (IF) profiling within a deep learning–enabled framework. To our knowledge, this is the first deep learning–based DHM system to achieve both high recovery and a patient-level false positive rate below 1 cell/mL. This level of specificity is particularly critical given that CTC concentrations typically fall below 1 cell/mL in early-stage cancers and average around 10 cells/mL in advanced disease. In this context, even a modest false positive rate from the background of approximately 10⁶ WBCs/mL can produce misleading CTC signals, underscoring the importance of meeting this benchmark for any clinically deployable assay.

At the core of the platform is a customized deep learning model trained and validated on an exceptionally large and morphologically diverse dataset, comprising over 5.9 million healthy cell images and 3.1 million cancer cell images derived from 25 cell lines spanning a broad range of tissue origins. This diversity is essential for enabling robust detection of CTCs, which are not only rare but also phenotypically heterogeneous. The model’s architecture supports flexible adjustment of operating thresholds, allowing tuning of sensitivity and specificity to suit different clinical scenarios. For instance, stringent thresholds can be applied in early-stage screening where false positives must be minimized, whereas more permissive settings may be appropriate for minimal residual disease monitoring or longitudinal surveillance, where detection sensitivity is prioritized.

In contrast to state-of-the-art CTC detection platforms that rely exclusively on biochemical labeling, our system leverages label-free phase-based imaging to characterize cellular morphology and optical thickness independent of surface marker expression. Antigen-based methods, such as CellSearch®, Epic Sciences platform, and other marker-driven systems, depend on the presence of antigens like EpCAM or cytokeratin, and may fail to detect CTCs that have undergone epithelial to mesenchymal transition or exhibit low marker expression9,11,19. This challenge is especially pronounced in real patient samples, where CTCs often diverge morphologically and molecularly from the cancer cell lines used for benchmarking43,44,48. By combining DHM with IF signal profiling, our platform captures a broader range of CTC phenotypes while preserving the specificity required for clinical deployment. The fusion of orthogonal modalities—label-free morphology and targeted molecular markers—enables robust classification even in samples with rare or atypical tumor cells.

Our system also addresses the limitations of physical property–based enrichment techniques, which rely on parameters such as size, density, and deformability to isolate CTCs from whole blood. Technologies including Parsortix®, ClearCell FX, and deterministic lateral displacement chips have shown promise for label-free enrichment, but face a fundamental trade-off in sensitivity and specificity due to overlapping physical characteristics between CTCs and WBCs19,49,50. As with biochemical markers, the morphological and mechanical properties of CTCs in patients can differ significantly from those of cultured cell lines43,44. In our approach, inertial microfluidics is used primarily to reduce sample complexity by removing RBCs and reducing WBC density. However, final classification is not determined by mechanical proxies like size or deformability. Instead, we use image-based deep learning trained on millions of annotated cells. This enables high-fidelity detection of phenotypically diverse CTCs, including those that would be missed by purely physical or biochemical filters.

A key advantage of our platform is its scalability and compatibility with clinical workflows. The entire process—from whole blood collection to quantitative CTC enumeration—is completed within 6 h per 10 mL tube, without requiring manual cell inspection or operator-dependent interpretation. The system remains robust when processing samples up to 24 hours post-collection, even when using EDTA tubes, mitigating challenges posed by pre-analytical variability in blood quality. This is in contrast to widely used systems such as Parsortix®, which typically require same-day processing when using EDTA tubes due to sensitivity to clotting or cell degradation, thereby limiting operational flexibility in real-world settings51,52. In addition, our compact hardware footprint, low reagent consumption, and minimal technician oversight enable high-throughput parallelization across multiple stations. These features collectively position the system for scalable deployment in clinical laboratories, including large-cohort studies and longitudinal monitoring applications.

Toward Broad Clinical Utility in Liquid Biopsy

Our platform is inherently adaptable to accommodate a wide range of clinical workflows and biological sample types. The diversity of cancer cell lines included in the training dataset supports the model’s potential to detect CTCs originating from tumor sites beyond the prostate. Antigens can be selectively incorporated to target cells from specific cancer types or to guide treatment decisions, such as assessing ER/PR or HER2 status in breast cancer. Holographic imaging, being label-free and non-destructive, is compatible with any optically transparent medium. Although this study focused on CTC detection in peripheral blood, the same analytical pipeline can be seamlessly extended to other clinically relevant biofluids such as saliva, urine, pleural effusion, and cerebrospinal fluid. This versatility offers a promising foundation for developing noninvasive cancer diagnostics applicable across diverse tumor types and anatomical compartments53,54,55,56. By integrating label-free DHM with targeted IF, the platform enables robust detection of rare tumor cells, independent of sample origin or matrix composition.

Collectively, these features enable the system to transform a complex, billion-cell blood sample into a concise, high-confidence readout—without requiring expert interpretation by trained technicians or pathologists. A key advantage of the workflow is its real-time detection capability, which operates during continuous sample flow. This live imaging approach not only accelerates CTC enumeration but also preserves cell viability, thereby enabling seamless integration with subsequent isolation and phenotypic or molecular characterization.

This operational flexibility positions the platform for broad utility across the cancer care continuum, including early detection, treatment monitoring, and post-therapy surveillance. The ability to process standard clinical blood volumes with high specificity and rapid turnaround time is particularly advantageous for longitudinal tracking of minimal residual disease and early relapse. Beyond enumeration, the platform enables recovery of viable CTCs for single-cell analyses, including genomic, transcriptomic, and proteomic profiling. These capabilities are well aligned with emerging integrative workflows such as VERSA (Versatile Ex Vivo Rare-cell Sorting and Analysis), which aim to dissect tumor heterogeneity, assess drug sensitivity ex vivo, and support personalized therapy design57,58.

Looking forward, a key direction for future development is the realization of a fully label-free diagnostic workflow powered entirely by deep learning. As the training dataset grows to encompass a broader range of patient-derived CTC morphologies, we anticipate model performance will reach the threshold needed to eliminate dependency on IF labeling. A transition to a purely image-based pipeline would significantly streamline sample processing, reduce reagent use, and shorten assay turnaround, thereby enhancing feasibility in decentralized, resource-limited, or point-of-care settings50,59. Moreover, such a system could enable pan-cancer detection and even subtype classification, relying exclusively on morphological and phase-based features. This shift from antigen-dependent identification to phenotype-aware optical profiling would represent a fundamental advance in the capabilities of liquid biopsy technologies.

This study establishes a platform-level feasibility for label-free DHM detection fused with IF confirmation for prostate cancer. While the present cohort demonstrates patient-level specificity, broader validation across multiple tumor types remains an important next step. Furthermore, activated leukocytes and other atypical cells in non-cancer illness may challenge label-free morphology. Next steps include prospective studies across stages and multiple tumor types with prespecified operating points. We will assess analytical validity through recovery, false-positive rate in healthy donors, repeatability, and operator robustness. We will also extend validation to disease control cohorts that include infectious and inflammatory conditions and other benign states. Clinical validity will be tested using baseline counts and early on-treatment changes, with association to radiographic response, progression-free survival, and overall survival. Clinical utility will be examined in longitudinal monitoring by comparing the timing and magnitude of CTC changes with standard biomarkers and imaging. Thresholds will be tuned for specific use cases such as surveillance, response assessment, or high-risk triage.

Methods

Sample Collection and Preparation

Blood samples were collected into EDTA tubes to prevent clotting. Each donor provided at least two 10 mL tubes of whole blood (actual measured volumes varied) that were then processed with a patented passive RBC depletion and CTC enrichment method, reducing the sample to 3 mL per tube (patent no. WO2024064911A1; further description in SI §1). The enriched samples were stained for protein markers PSMA and EpCAM following an IF staining protocol. All samples are completely processed the day following collection. The active processing time of each sample is approximately 6 hours.

To validate that the microfluidic platform and staining protocols yield enriched samples containing true CTCs, we also assessed a limited set of samples mounted to slides and imaged on a fluorescent microscope. These samples were obtained after microfluidic enrichment and after PSMA/EpCAM staining. Results showed evidence of CTCs in the enriched samples in the context of standard validation methodology, wherein additional stains were applied for DAPI and CD45 that must be positive and negative, respectively, for CTCs. These images are provided and the data are described in SI §6.3.

Artificial CTC-laden blood samples, used in recovery experiments, were created by spiking a targeted number of LNCaP cells into a healthy blood sample following the microfluidic enrichment procedure. The true number of cells observed by the station was validated by collecting the sample from the outlet (i.e., after DHM imaging) into a well plate and counting on a fluorescence microscope. For more details on the spiking procedure and analysis, see SI §5.1 and §5.2.

Experimental protocols were approved by the Institutional Review Board (IRB) of the University of Minnesota (IRB No.: STUDY00013815). The study methods were carried out in accordance with all relevant guidelines and regulations. All patients and donors included in this study provided informed consent prior to having their blood drawn.

Holographic Imaging Station

Digital holographic microscopy was performed using a Gabor holography setup that did not separate the object and reference waves12,60. The light source was a pulsed laser diode (405 nm wavelength), and images were obtained using 10× infinity corrected microscope objective with 0.30 numerical aperture (NA). The focal plane was adjusted to be aligned with the depth-wise center of the flow channel, and a focus quality metric was computed and provided to the user in real-time to help guide this process (further details in SI §2.2). Holograms were captured with a 1.6 mega-pixel machine vision camera with a pixel pitch of 3.45 µm. The camera was synchronized with the laser (450 frames-per-second) with a capture region of interest (ROI) of 1440 × 256 pixels. Images were streamed over USB to the capture and processing computer. There were 18 imaging stations used for the study to avoid over-fitting to station-specific attributes such as the beam profile or dust on the optics. For further details of the system, see SI §7.

Deep Learning Model Development

CTCs are identified in holographic images using a trained deep learning model. Here we outline the architecture and training of the model, including strategies for annotation, sample selection, and postprocessing. We describe the model’s design adaptations for high-throughput inference, the use of pseudo-labeling for large-scale training without extensive manual annotation, and the incorporation of hard sample mining to address class imbalance. Details on the training and validation datasets, including image acquisition and quality control procedures, are also provided.

Model Architecture and Hyper Parameters

A modified HRNet backbone45 was selected for the image to heatmap transformation because it preserves high resolution feature streams throughout the network. This matches the needs of DHM, where small morphological cues, phase texture, and fine diffraction patterns carry signal. We implemented a compact configuration of the HRNet architecture, customized under runtime and memory constraints, to support run-time inference at hundreds of frames per second on the acquisition workstation while preserving classification accuracy. This involved tailoring the number of layers to balance computational efficiency and feature richness. Specifically, the coarsest layers were pruned until real-time inference speeds could be achieved on the target hardware to match the frame rate and flow speed, as inference at run-time greatly increases the future capabilities of the platform for CTC isolation and downstream assays. Pruning had the added advantage of reducing the receptive field, ensuring that the model focused on spatial features most relevant to small target cells. This adjustment allows the model to extract sufficient discriminative features while minimizing the influence of surrounding objects. The throughput was further assisted by running inference using NVIDIA TensorRT, which is optimized for the target GPU with quantization to reduce inference time.

For annotation, we used Gaussian keypoint labels centered on the cytological centroid of each target cell, instead of bounding box supervision. In DHM, salient information extends beyond the in-focus core into surrounding interference fringes, which makes box boundaries intrinsically ambiguous and increases label noise at the periphery. Keypoint heatmaps emphasize the most informative region and allow pixel contributions to decay smoothly with distance, which improves inter-annotator consistency and yields cleaner gradients for training. The approach also reduces annotation effort because raters place a single point rather than define boxes, enabling efficient scaling to large datasets. In practice, the combination of a high resolution HRNet with keypoint supervision produces stable single cell localization in DHM and provides reliable inputs for downstream chain-level counting and fusion with immunofluorescence.

The model weights were initialized using public weights pre-trained on the COCO dataset. The Adam optimizer was used with a learning rate of 0.01 and exponential learning rate decay by a factor of 0.85 each epoch. The pixel-wise loss function is an asymmetric binary cross-entropy loss defined as

$$L=-A\left[y* log\,\left(p\right)\,+\left(1-y\right)* log\,\left(1-p\right)\,\right]$$

where A=0.1 for false positives on positive samples (cell lines) and A = 1 otherwise.

Training Image Annotation

Each image was pre-processed by subtracting a background reference, which was computed using an exponential moving average of preceding frames within the same imaging session. Following background subtraction, the image was normalized using a fixed mean and standard deviation to ensure consistent intensity scaling. Target heatmaps were generated using a pre-trained pseudo-labeling model. For each detected cell location from a cancer cell line, a Gaussian blob with a standard deviation of 2.8 µm was placed to define the spatial confidence region. This representation enabled the binary cross-entropy loss function to emphasize not only the presence of a detection but also allow precise localization accuracy. Note that all negative training data (i.e., images from healthy blood samples) are left with a blank heatmap; the pseudo-labeler is only applied to cell lines in buffer without any blood cells mixed.

Hard Sample Mining

The dataset was inherently imbalanced: many images contained no cells, and healthy blood samples were often dominated by RBCs, which are generally easier to distinguish from CTCs than WBCs. To address this challenge, a hard sample mining strategy was applied to the negatively labeled healthy control dataset. During training, batches were drawn preferentially from healthy control imaging sessions that showed poor model performance in the previous epoch, with sampling weighted by the average loss. This prioritization ensured that the most difficult examples were repeatedly encountered during training. To manage computational efficiency, performance evaluation was conducted at the imaging session level instead of re-scoring individual images after each epoch.

The fundamental goal of this strategy was to improve the detector’s robustness against the heavy-tailed variability of leukocyte backgrounds, where rare white blood cell phenotypes can mimic true positive events. By enriching the training process with these difficult negatives, the model learns to suppress them more consistently, leading to more reliable performance at the low false-positive rates critical for rare-event detection.

Training Dataset

The training dataset was composed of healthy blood samples collected from 32 unique donors, some of whom contributed multiple samples. Positive samples were generated using five established cancer cell lines: AsPC-1 and PanC1 (pancreatic), DU 145 and LNCaP (prostate), and MCF7 (breast) (SI §4.2 Table 3). We intentionally use a diverse set of cancer types, including non-prostate cancers, to avoid overfitting to any given cell line. This is done in lieu of being able to train on a sufficiently large set of true CTCs identified with IF staining, as discussed previously.

In total, the dataset was compiled from over 140 imaging sessions, spanning more than 200 h of acquisition and yielding approximately 300 million raw images. As many of these images were either empty or trivial to classify, an earlier version of the detection model was used to pre-filter the dataset. Images with a model confidence score above 0.3 were retained, resulting in a curated dataset of 7.9 million images, including 5.9 million from healthy samples and 2.0 million from cancer cell lines. These were divided into training and validation sets using an 80:20 split. To prevent data leakage, all images from a given acquisition session were assigned to the same partition, ensuring that repeated captures of the same cell did not appear in both sets. Multiple imaging stations were used during data collection to introduce variability in optical and hardware conditions, thereby improving model generalizability. Importantly, the only manually annotated data used for training came from a small pseudo-labeling dataset consisting of 2500 images. All other annotations in the training dataset were generated automatically.

Validation Dataset

The spiked validation dataset consisted of 36 samples, yielding a total of 185 million images. In addition, 1.1 million images containing isolated cancer cell lines (i.e., spiked into buffer, not blood) were acquired separately to evaluate the model’s theoretical recovery in the absence of background interference or blood-related losses. True positive detections were defined by spatial proximity between the prediction peaks generated by the detection model and those produced by the pseudo-labeling model. This comparison enabled a quantitative estimate of the model’s theoretical recovery, expressed as the true positive rate (TPR), in an idealized context without confounding biological noise (Fig. 4).