Digitalized organoids: integrated pipeline for high-speed 3D analysis of organoid structures using multilevel segmentation and cellular topology

Abstract

Organoids replicate tissue architecture and function and offer a unique opportunity to explore the impact of external perturbations in vitro. However, conducting large-scale screening procedures to investigate the effects of various stresses on cellular morphology and topology in these systems poses important challenges, including limitations in high-resolution three-dimensional (3D) imaging and accessible 3D analysis platforms. In this study, we introduce an AI-based multilevel segmentation and cellular topology pipeline for screening morphology and topology modifications in 3D cell culture at both the nuclear and cytoplasmic levels, as well as at the whole-organoid scale. We demonstrate the versatility of our approach through proof-of-concept experiments, encompassing well-characterized conditions and poorly explored mechanical stressors such as microgravity. By offering a user-friendly interface named 3DCellScope and a comprehensive set of tools for discovery-like assays in screening 3D organoid models, our pipeline demonstrates wide-ranging potential for applications in biomedical research.

Main

The capability to accurately measure 3D deformations of cells within their native 3D structures is of paramount significance for investigations of tissue biology and pathological conditions. While normative mechanical forces and chemical cues are essential for the orchestration of the tissue maturation and optimal functionality, the advent of aberrant stresses can lead to detrimental effects such as accelerated tissue aging, induction of pathophysiological changes and hindered tissue regeneration properties^1,2.

In this context, organoids and organ-on-chip systems have emerged as indispensable tools, offering 3D cellular assemblies that emulate the architecture and function of basic organ units. These tools enable the study of genetic expression, cell signaling pathways and cellular damage in response to external stimuli, providing an ethical and practical alternative to animal models^3,4,5. Compared to two-dimensional (2D) cultures, organoids demonstrate superior physiological relevance^6,7, with recent advances in bioengineered extracellular microniches⁸ enabling precise control of organoid geometry, critical for tissue patterning and cryptogenesis. However, existing studies often focus on overall organoid morphology, gene expression or 2D descriptors and fail to quantify cellular 3D morphology^9,10,11.

Crucially, the cell (and/or intracellular compartments) deformation descriptors rely on precise image-based segmentation techniques. This capacity to gauge cellular deformations must derive from the utilization of ubiquitous cellular markers present across all cell types, thus ensuring maximum independence from cellular differentiation and phenotypic variations. The analysis of complex morphological changes at different scales (single cells, clusters of cells and whole organoids) in combination with approaches based on high-content 3D imaging and screening methods has the potential to reveal new mechanisms of response or adaptation during tissue development, injuries and aging-related diseases.

In the domain of biomedical research and screening campaigns, the demand for accurate 3D cell segmentation from microscopic images has led to the development of various open-source deep learning-based solutions. Open-source segmentation tools, such as Ilastik¹² and OrganoidTracker (nuclei only)¹³, have made progress but face challenges in resolution and image variability. However, existing solutions require high-quality 3D image resolution for robust and accurate segmentation and the training phase for these solutions is often time-consuming and does not readily accommodate changes in resolution, staining, cell types or specific image characteristics. While 3DCellSeg¹⁴ demonstrates robustness for specific models, its training datasets differ from real-world biomedical conditions, which often involve lower resolutions and poorer signal-to-noise ratios (SNRs). Additionally, tools like Cellpose¹⁵ excel in 2D imaging but struggle with 3D datasets in terms of processing time and segmentation accuracy. Other studies, such as those by Beck et al.¹⁶ and Ishihara et al.¹⁷, focus on classifying predefined global morphologies but lack metrics for describing individual cell distributions or morphologies—key features of tissue morphogenesis, cell differentiation and functionality. Moreover, many methods fail to disclose computational time, limiting their practicality for high-throughput applications. All these limitations currently impede the applicability of existing deep learning methods for precise segmentation of compact organoid cells in practical scenarios.

To address these limitations, we introduce a new methodology for assessing 3D cell morphology and topology within organoids. Our approach includes tailored algorithms for multi-scale segmentation and quantification at the nuclear, cytoplasmic and organoid levels, requiring only simple biological markers like nuclei and plasma membranes. Unlike other methods, our system does not demand labor-intensive immunostaining, advanced computing or programming expertise, making it accessible to standard laboratory setups. Our exploration extends to cell positional relationships, allowing for tissue patterning detection and the generation of numerous descriptors. Our method integrates three innovative components: (i) a fast and reliable 3D segmentation process suited for real-world conditions, (ii) analysis of 3D topology descriptors to quantify tissue patterning (descriptors of internal cell-to-cell and cell-to-neighborhood organization) and (iii) the generation of morphological signatures to assess mechanical constraints. Additionally, we provide a user-friendly software interface, 3DCellScope, which allows researchers to import, execute and visualize results from AI segmentation networks with minimal effort. This collaborative approach fosters fairness and facilitates further advancements allowing the incorporation of other 3D segmentation algorithms from diverse research groups, especially considering the rapid evolution of AI networks and the growing needs of 3D biology.

Results

Innovative concept of digitalized organoids and 3DCellScope

We describe the principle of the ‘digitalized’ organoids method, showcasing the capabilities of our analysis techniques (Fig. 1). The 3D digitalization, from subcellular level to whole-organoid scale, involves three levels of segmentation and has been compiled by using open-source software and available AI networks (Fig. 1a). Firstly, nuclei surfaces are extracted using AI-based analysis of DNA-stained images, which is based on a finely trained 3D StarDist convolutional neural network (CNN)¹⁸ model using simulated images that closely mimic real-world conditions. Secondly, cell surfaces are obtained by incorporating both of the nuclei contours as seeds in a grayscale 3D watershed approach based on raw images of actin. Lastly, the complete organoid contour is obtained directly from raw channels using fine-tuned thresholding and morphological mathematics filtering (Extended Data Fig. 1 and Methods). The 3D segmentation layers, the subsequent descriptors generated and the raw images are understandable through data mining, enabling the extraction of biological results (such as cell-to-neighborhood analysis described below).

Fig. 1: ‘Digitalized’ organoids principle.

a, Schematic depicting the pipeline. Top, Organoids were imaged using 3D fluorescence microscopy. Middle, The complete 3D digitalization of organoids consists of three layers of segmentation. Bottom, The resulting data are understandable through data mining. b, Nuclei and cell segmentation applied on a primary PDAC organoid. The white arrows indicate low-intensity nuclei.

By integrating both advanced 3D AI tools and the versatility of user-oriented software under a single software called 3DCellScope (Fig. 1a and Extended Data Fig. 2), we bridge the gap between specialized pipelines and generalist commercial software (Extended Data Table 1), positioning itself as an ideal choice for 3D image analysis of complex biological structure. Furthermore, our interface allows for the direct integration of other AI networks for nuclei segmentation. This capability facilitates collaborative efforts, acknowledges the rapid evolution of the field and uniquely empowers users to assess the performance of different AI solutions with a single click (Supplementary Video 1). To our knowledge, this marks the first instance of single user-friendly software offering such seamless integration (‘Code availability’; https://github.com/quantacell/3DcellScope/tree/main/CompiledVersionForWindows/). Our approach enables researchers to extract valuable biological insights and provide feedback on the raw images, facilitating the interpretation and utilization of the acquired data. Some example images with their corresponding segmentation process and a fully described user guideline are embedded within the demonstration version of 3DCellScope and are freely accessible. Furthermore, our database organization is fully compatible with already developed data mining open-source interfaces such as VTEA (https://imagej.net/plugins/vtea). Alternatively, users can import and generate a similar graphical interface and statistical analysis using KNIME (https://www.knime.com/).

Performance of 3D segmentation on a wide range of datasets

Our segmentation approach is tailored for real-world laboratory conditions, ensuring robust performance under practical constraints. Designed to adapt to diverse image characteristics, it enhances accuracy across various samples while minimizing reliance on user expertise. This adaptability broadens its applicability across experimental setups and biological questions. The method supports a wide range of 3D image qualities—including varying resolutions, anisotropic voxels, distinct microscopy techniques and wavelengths—and accommodates both post-fixation staining (for example, DAPI, NucBlue, actin/membrane binders) and genetically encoded reporters like fluorescent histones (for example, H2B-mNeonGreen, H2B-mCherry). Our approach also prioritizes rapid 3D segmentation for seamless integration into screening workflows, achievable on a standard laptop.

To address the challenges of versatility, time constraints and limited computational resources, we developed DeepStar3D, a pretrained CNN network based on StarDist principles. Utilizing a fine-tuned simulated dataset for the training procedure, it encompasses a wide range of nuclei shapes and image qualities, ensuring robust and efficient segmentation.

A comprehensive description of the training dataset is provided in the Methods. We have chosen StarDist as it represents the state-of-the-art approach for nuclei segmentation well known for its accuracy and speed. Among other 3D instance segmentation frameworks such as U-Net3d¹⁹, Cellpose3d²⁰ and EmbedSeg²¹, StarDist performs extremely well for convex objects such as nuclei, especially in complex biological contexts with a dense group of nuclei and a noisy background, while being comparatively faster than its counterparts^21,22.

We performed an extensive benchmark and comparison analysis between our DeepStar3D CNN and three other networks pretrained based on 3D StarDist: AnyStar²³, Cellos²⁴ and OpSeF²⁵ models (Supplementary Table 1). We manually annotated images from external freely accessible 3D datasets and one from of our own acquisition batch, aiming to cover a large diversity of nuclei shapes, resolutions and staining, resulting in a total of 594 ground-truth nuclei. Data were sourced from independent research institutes, and none of these benchmarked images (including ours) have been used for training of the DeepStar3D model (Methods).

Using the FI intersection over union 50 (F1^IoU50) score (Methods), we show that our model remains particularly robust, consistently ranking first or second and achieving the overall best rank (Extended Data Table 2). Our network consistently maintains a level of quality deemed satisfactory (F1^IoU50 score > 0.5) for the four datasets tested despite large variation in xy and z resolutions, staining procedures, cell types, seeding procedures and imaging modalities (Extended Data Fig. 3a). While Cellos showed superior precision on the data it was trained for, as anticipated, our comprehensive evaluation revealed that DeepStar3D notably outperformed Cellos on all other datasets. AnyStar gave the best results for the colon organoid dataset, as it appears to be more adapted to nuclei with low signal. However, it did not perform well for all other spheroid datasets. Unlike all alternative CNN models, our DeepStar3D did not show significant correlations between the IoU score per nuclei and metrics such as SNR and nuclei density (Extended Data Fig. 3b). This observation underscores the distinctiveness of our model in exhibiting result quality resilience despite modifications of the signal and object density. Another illustration of nuclei segmentation performance using the DeepStar3D CNN on a large organoid of primary pancreatic ductal adenocarcinoma (PDAC), displaying a wide range of nuclear intensity, is presented in Fig. 1b and Supplementary Video 2.

Eventually, our model exhibited the shortest inference time, referring to the duration required to process new data and make predictions. Specifically, for the same unseen image, the inference time was measured at 9.5 s ± 0.7 s for untiled images and 33 s ± 5 s for tiled images. This represents a time saving of between −20% and −70% as compared to other CNNs tested (Extended Data Table 3). Compared to our DeepStar3D CNN, the number of weights in each alternative CNN was between 4.5-fold and 200-fold greater. These characteristics render our approach directly compatible for large-scale batch processing, such as screening campaigns, without necessitating a sophisticated and expensive high-performance computing system.

For cell segmentation, we also aimed to build a method of high efficiency and robustness. Given our accurate nuclei segmentation, we opted to use the contours of the nuclei as seeds to guide a seeded grayscale watershed process²⁶ applied to membrane staining or membrane-enriched proteins such as actin. The seeded watershed transform is a method of choice for image segmentation belonging to the field of classical image processing that does not require any supplemental AI-based training. Figure 1b and Supplementary Video 3 illustrate the precise cell segmentation within the PDAC organoid featuring a heterogeneous architecture with a wide variety of cellular shapes and intensities. On different tumor spheroids of colorectal cancer cells, we also performed a quality-control check of our cell segmentation, showing less than 8% of mis-segmented cells (Methods and Extended Data Fig. 4).

Eventually, the complete segmentation process generates an extensive list of hundreds of morphological and topological descriptors organized into an interpretable database establishing a robust foundation for detailed analysis.

Navigation, manipulations, image feedback and statistics

The three levels of segmentation, graphical filtering and image feedback offer a robust framework for interactive analysis of complex biological data. Researchers can explore data, manipulate visual representations and refine analyses through filters. Examples of basic manipulations, image feedback and result extractions are shown in Fig. 2, using cancer cell spheroids labeled with phalloidin and DAPI. Graphical filtering (‘gating’) allows users to exclude unwanted objects or artifacts (Fig. 2a,b). Real-time feedback on segmented and raw images enhances decision-making, improving result accuracy. Scatterplots reveal multi-scale correlations, such as an inverse relationship between the number of nuclei per spheroid/organoid and mean nuclei volume. For organoids of different sizes (Fig. 2b), we observed that larger spheroids have more compact chromatin (smaller nuclei volume) compared to smaller spheroids. In addition, segmentation quality control can be done to assess whether there is any segmentation bias introduced by factors such as signal quality or intensity (Supplementary Fig. 1).

Fig. 2: Interactivity and data navigation through three levels of 3D segmentation, graphical filtering and image feedback.

a, 2D and 3D visualization of cancer cell spheroids (HCT116), labeled with phalloidin and DAPI. 3D cell contours (middle) and 3D nuclei contours with organoid contours (bottom) are shown. Scale bar, 60 µm. b, Graphical display and gating on a histoplot (top). Manual filtering (red bar) can be adjusted to exclude small objects, with feedback provided on segmented or raw images (right). The bottom plot shows the mean nuclei volume obtained after filtering (gray line for standard error) versus the number of cells per organoid for the four organoids. Scale bar, 15 µm. c, Schematic and histoplot of nuclei centroid distance from the organoid’s border. Plot depicts filtering in two classes: nuclei within 0 µm to 20 µm (green–cyan) and nuclei at a distance greater than 20 µm (red). d, 2D and 3D visualization of nuclei gating based on their distance to the organoid border. Plot shows the mean (±s.e.m.) nuclei volume for two classes: external (<10 µm from the border) and internal (>10 µm; n = 895).****P < 0.0001, using a two-tailed unpaired t-test with Welch’s correction. Scale bar, 30 µm.

Source data

The segmentation of organoid/spheroid contours (Supplementary Video 4) enables the creation of 3D distance maps relative to the spheroid border, offering valuable insights into the distribution and arrangement of cellular nuclei. This aids in understanding cellular behaviors within the spheroid (Fig. 2c). Using the contour, we calculate the 3D distance of nuclei or cell centers to the organoid/spheroid border, allowing the user to apply simple threshold gating based on this distance. For example, in Fig. 2d, our analysis revealed that ‘external’ nuclei, located near the spheroid border (shorter distance), exhibit a significant volume increase (+50 µm³, P < 0.0001) compared to ‘internal’ nuclei, which are farther from the border. This suggests a relative relaxation of mechanical constraints on the external nuclei.

Validating morphological descriptors in predictable conditions

We proceeded to test the accuracy of our 3D segmentation by inducing morphological changes and compaction in cells using an osmotic stress (Fig. 3). Osmotic stress has been used in previous studies as a means to simulate mechanical loading, resulting for example from a stiffer extracellular matrix (ECM) in the local environment²⁷. We validated our morphological parameters and the quality of cellular, nuclear and spheroid segmentations by comparing them to the clear and expected visual effects of osmotic stress, used here as a positive perturbator of cancer spheroid morphology. Our analysis indicates that, under hypertonic conditions, cells within spheroids exhibited a rounder morphology (Fig. 3a and Supplementary Fig. 2). By applying plot gating, we observed, under hypertonic conditions, a higher percentage (+15% as compared to isotonic conditions) of cells within the ‘high roundness’ gate. Owing to the direct image feedback capability of our method, we revealed that cells with a higher roundness are present within the core of spheroids (Fig. 3a), and not only restricted to the external layer of cells, thus indicating that the effect of the osmotic stress seems to propagate through the dense 3D cell culture of a cancer spheroid.

Fig. 3: Proof-of-concept experiment on well-known conditions of morphological modifications to validate multi-scale morphological parameters.

a, Cell roundness was compared under isotonic and hypertonic conditions. Representative actin images are shown. The cell roundness of two representative spheroids (one per condition) are plotted in the histoplot, and filtering was applied using the normalized histoplot with a red bar indicating a high cell roundness threshold. The percentages of cells exhibiting high roundness are displayed (red) on 2D and 3D segmentation. Scale bar, 30 µm. Red arrows highlight cells of high roundness in the core of the spheroid. Violin plots show the cell roundness for isotonic and hypertonic conditions. b, The relative position of nuclei to the organoid border was compared between isotonic and hypertonic conditions. Representative images with DAPI and phalloidin staining are displayed, with cyan contours representing nuclei near the border and red contours marking nuclei farther from the border. Filtering was applied to classify nuclei into two groups based on their relative position to the organoid border. Red arrows point to nuclei in the long-distance class. Scale bar, 10 µm. The histoplot presents the relative position of nuclei for two representative spheroids (one per condition), while the violin plots show the distribution of nuclei positions for five spheroids per condition. c, Chromatin compaction was evaluated by the CV of DAPI staining under isotonic and hypertonic conditions. Clickable nuclei contours, marked by red dots and white hands, link the images to the corresponding histoplot (two spheroids represented, one per condition). Filtering was applied to identify nuclei with high CV values. Violin plots show the distribution of CV values for both conditions. For all violin plots, mean values are represented by blue bars. ****P < 0.0001, using a two-tailed unpaired t-test with Welch’s correction; analysis based on ten organoids (five organoids per condition). Scale bars, 15 µm and 5 µm (zoomed images).

Source data

Next, we looked for modifications of the 3D position of the nuclei relative to the organoid’s border in the same hypertonic condition (Fig. 3b and Supplementary Fig. 2). A notable shift in the distribution of all nuclei positions toward the center of the organoids can be observed and quantified, as shown previously, using a gating approach on the nuclei relative position distribution plot. Multi-gating feedback enhances clarity with two gates shown: cyan for nuclei near the border and red for those farther away (‘high distance’). We observed a higher percentage of nuclei (+21%) within the gate ‘high distance’ under osmotic stress as compared to the isotonic condition. The findings indicate that osmotic shock causes nuclei within spheroids to shift away from the outer border and cluster toward the core. Our pipeline offers users the capability to identify effects through visual inspection of raw images and compile additional descriptors to confirm or refute initial visual impressions. For instance, by visual inspection, we observed first a change in the granularity of the DAPI staining under osmotic pressure. Nuclei from all spheroids exposed to the hypertonic medium exhibited a distinctly more punctiform DAPI staining compared to the control conditions (Fig. 3c). This change is commonly described as a chromatin compaction effect and can be quantified by calculating the coefficient of variation (CV) of the DAPI staining²⁸. Our approach enables users to perform calculations across all features in the database (Extended Data Fig. 5), allowing for the generation of new parameters to quantitatively confirm visual impressions. Users have the flexibility to enrich the database with additional features of their choice. In this case, we validated our visual inspection findings by quantifying the modification of the distribution of DAPI staining with the CV. The control condition displayed a normalized distribution with a narrower gaussian shape, whereas the hypertonic conditions exhibited a dispersed normalized distribution. Furthermore, we observed a significant increase in the mean CV (+30%, P < 0.0001) and a higher percentage of nuclei into the ‘high CV’ class (from 14.5% in the control condition to 49.8% in hypertonic medium; Fig. 3c and Supplementary Fig. 2).

By inducing morphological changes, we validated the accuracy of our 3D segmentation and descriptors. The increase in cell roundness, along with shifts in nuclear positioning and DNA staining granularity, demonstrates the effectiveness of our approach in capturing and quantifying cytoplasmic and nuclear alterations.

Cell-to-neighborhood 3D descriptors for tissue analysis

Within organs and tissues, cells intricately interact to create layers, tubes or complex clusters that possess a distinctive 3D organization (tissue patterning), which is crucial for the proper functioning of the organs. Furthermore, it is widely recognized that local topological features (for example, curvature) of the extracellular microniche plays a pivotal role in facilitating the appropriate differentiation of cells by modifying morphology and cell–cell relative position⁸. Analyzing the tissue patterning and thus cell relative positions is essential for understanding these differentiation and homeostasis processes in 3D. In our previous work, we successfully applied an AI strategy using a dedicated pretrained U-net network (as referenced in Beghin et al.²⁹) for identification and 3D segmentation of neural rosettes, complex internal arrangements specific to neuroectoderm organoids. However, it is important to note that this approach has limitations. A large number of organoids, often exceeding 100, is typically required to build the 3D imaging database for network training. Additionally, generating manual ground-truth contours for these structures is time-intensive and demands input from multiple experts to ensure accuracy and consensus. The specialized U-net network we utilized is specifically tailored to neural rosette structures, relying on particular staining and image quality, which limits its adaptability and requires prior knowledge of the target structure. To overcome these constraints, we propose a versatile solution based solely on the 3D nuclei positions, leveraging cell-to-neighborhood relative positioning without the need for additional markers.

3D topology descriptors were compiled based on the 3D centroid positions of nuclei, as described in Extended Data Fig. 6 and the Methods. In brief, these descriptors not only quantify the intricate cell-to-cell positions (distance and density) in the 3D cell culture but also provide precise measurements of the neighboring nuclei distribution by fitting ellipses around the surrounding nuclei of each nucleus and quantifying this ellipsoidal shape by axis measurements and their ratio (Fig. 4). The ‘prolate’ and ‘oblate’ ratios were computed and then plotted in a 2D scatterplot to unveil the spatial organization in three dimensions and allow multi-gating procedures. Three distinct categories—disk, rod and sphere—were delineated. The shape categories presented here describe the surrounding distribution around each nucleus and are not indicative of the individual nucleus shape or the overall 3D structure. We offer parameters delineating the local distribution around each nucleus and how cells self-organize in three dimensions within a defined radius.

Fig. 4: Topological distribution descriptors.

Schematic representation of ellipsoid fitting around neighboring nuclei, displaying ellipsoid axes and calculation of oblate and prolate ratios. Scatterplot illustrates the relationship between oblate and prolate ratios, with multi-class gating description (each dot represents one nucleus).

Source data

We applied this strategy on two types of experiments. First, we conducted experiments using normal primary epithelial breast cells (HME cells) seeded in microniches composed of ECM. Here, the specific shape of the microniche structure forces cells to position themselves in a typical, nonrandom 3D distribution, forming an adherent monolayer on different types of planes (Fig. 5a). We used this approach to compare two types of microniche configurations: the ECM ‘cup shape’ with a 50-µm diameter (oblique surfaces completely adhesive forming a V-Cup shape, that is, ‘cup’) and a flat-bottom ‘well shape’ (‘well’) with an 80-µm diameter displaying curved vertical adhesive walls and a flat nonadhesive surface at the bottom, along with a spheroid without ECM, meaning the nonadherent surface is available for the cell (Fig. 5a). For clarity, we focused on presenting only the percentage of ‘sphere’ nuclei distribution using this first experiment. The results demonstrated clear differentiation among the microniche types and the spheroid configuration based on the percentage of ‘sphere’ distribution, indicative of isotropic/random cell-to-cell positioning. The 50-µm ‘cup’ microniche exhibited the lowest percentage (<1%) of randomly positioned cells compared to the 80-µm flat-bottom ‘well’, which showed a range of 20% to 40% of nuclei randomly positioned with some anarchic multilayer patterns (Fig. 5a). In contrast, a typical cancer spheroid without an ECM microniche presented the highest percentage of randomly positioned cells (>50%). Through this proof-of-concept experiment comparing microniches, we have validated an analysis based on cell-to-neighborhood descriptors as a versatile tool. This method effectively highlights well-defined 3D cellular patterns and can even facilitate the classification of various biological systems.

Fig. 5: Analysis of 3D topology and comparison of microniche shapes in a complex organoid environment.

a, Comparison of cellular positioning on two different shapes of microniche (blue, ‘cup’; orange, ‘well’) and microniche without ECM (yellow) depicted on a scatterplot of oblate-versus-prolate ratio (one dot denotes one nucleus). Color-coded feedback (red contours denote ‘spherical’ gate, cyan denotes all other nuclei) presented on 3D segmentation views (xy and xz) and a median xy plane (indicated by white double arrows) with the percentage of nuclei exhibiting ‘spherical’ ellipsoids for each microniche (representing an isotropic neighborhood). Red arrows indicate disorganized multilayers of cells. Scale bar, 25 µm. b, Comparison of different regions of interest within a complex primary PDAC organoid. Two complementary strategies were used: a single-step data gating on the scatterplot of oblate-versus-prolate ratio, enabling feedback of three different classes—disk (magenta), rod (green) and sphere (red)—on the 3D view. Red arrows denote enriched areas of ‘sphere’ nuclei distribution (in the core of the organoid), while green arrows highlight ‘rod’ enriched areas (in several buds). ‘Disk’ (magenta) nuclei distribution is not displayed in zoomed images for improved visualization. A two-step strategy is also applied, starting with image gating on segmented images (image gating: core (red), bud (light blue/cyan), borders (orange), with corresponding nuclei counts displayed). Subsequently, the same data gating step (magenta/green/red) is applied to multi-region scatterplots. Additionally, linear regression analysis is performed, and corresponding slope values are presented. Pie charts display the corresponding nuclei percentage in each data gate (disk, rod and sphere) per image gate (regions of interest: core, bud and border). Scale bars, 150 µm.

Source data

We have also tested and validated our cell-to-neighborhood descriptors on a complex 3D structure of a primary PDAC organoid (Fig. 5b). PDAC organoids, derived from dissociated cells, developed into cysts within a week of culture initiation. Following an initial transient multilobular phase, these cysts underwent folding, revealing intricate cell invagination and convolutions internally. Furthermore, the emergence of external buds creates distinctive microenvironmental niches, making them points of interest for investigating the behaviors of cells within these buds, potentially associated with traits such as stemness, invasiveness or proliferative capacity.

Two complementary strategies have been used on this heterogenic structure: a single step using data gating, as previously described, on the scatterplot of oblate-versus-prolate ratio allowed the classification of nuclei into the three distinct classes described—disk, rod and sphere. By image feedback on the 3D view, we were able to see enriched areas of the ‘sphere’ distribution of nuclei predominantly located in the core of the organoid, while ‘rod’ enriched areas were systematically present in the different budding regions. The ‘disk’ distribution nuclei were more uniformly present in all zones of this primary organoid (Fig. 5b and Supplementary Video 5). Additionally, a two-step strategy is applied, beginning with a gating step on the segmented images, and defining three types of regions of interest by the user: ‘core’, ‘bud’ and ‘borders’ (Fig. 5b and Supplementary Video 6). Subsequently, the same data gating step is applied on the corresponding multi-region scatterplot of oblate-versus-prolate ratio. Linear regression analysis is also conducted, and the corresponding slope values are presented. Using this two-step strategy, we measured, for example, a decrease in the slope of linear regression from 1.08 for the ‘core’ region (r² = 0.93) to 0.93 for the ‘bud’ (r² = 0.71) and 0.78 for the ‘border¦ (r² = 0.70). This suggests that the core of this organoid could be associated with a triaxial (‘rugby ball’) and spherical nuclei local distribution, whereas the ‘bud’ and, more prominently the ‘borders’, tend to organize their cells in a disk-like shape. We also present pie charts depicting the corresponding nuclei percentage in each data gate (‘disk’, ‘rod’ and ‘sphere’) and per image gate (regions of interest: ‘core’, ‘bud’, and ‘borders’). The population of the ‘sphere’ nuclei distribution is notably highest within the ‘core’ region (30%) compared to other regions (<8%), while ‘bud’ exhibits the highest percentage of ‘rod’ distribution (9%), and ‘borders’ zones demonstrate predominant ‘disk’ distribution (89%; Fig. 5b). These metrics align precisely with our visual observations, where the PDAC core primarily consists of randomly positioned cells (‘sphere’), borders exhibit a monolayer of cells folded upon themselves (‘disk’) and buds appear as small cysts (monolayer) with internal alignment of cells (a mix between ‘rod’ and ‘disk’). These findings underscore the spatial heterogeneity within PDAC organoids, providing insights into the complex tumor microenvironment and potential implications for therapeutic targeting, to unravel differential sensitivities and drug resistance in particular regions of samples. Other descriptors such as local densities and mean distances (cell-to-cell) between nuclei are also provided (Extended Data Fig. 6b).

We propose a comprehensive analysis with defined metrics to uncover the local 3D arrangement of cells. This innovative approach reveals cell-to-cell and cell-to-neighborhood spatial relationships, highlighting the influence of external microniches or internal heterogeneity in complex multicellular systems. It is particularly relevant for evaluating the organization of 3D structures from primary cells, identifying nonrandom arrangements mirroring native tissues without requiring prior knowledge, extra segmentation steps or AI training. To our knowledge, assessing local topological constraints through 3D morphological signatures without any a priori assumptions represents a novel contribution.

Unsupervised/supervised workflow for extracting signatures

Here we introduce an advanced level of automated analysis, combining unsupervised and supervised methods to extract hallmark features associated with 3D morphological characteristics. The methodology was applied in an exploratory experiment during a parabolic flight (Methods). In brief, hundreds of breast cancer spheroids were subjected to a series of 15 parabolas aboard an aircraft. Samples were fixed either on board after the final parabola (day 0, F for flight) or 24 h later (day +1, F24 for flight fixed after 24 h). Corresponding ground controls (GC and GC24) were cultivated and fixed at matching time points. These extreme conditions simulate cyclic mechanical stresses, with each parabola encompassing phases of 2.5 g of acceleration and a microgravity period (0 g).

It is important to highlight the absence of prior results obtained in such comparable conditions. This void is particularly striking given the contemporary surge of interest in space biology and the concurrent application of organoids in space-related extreme conditions^30,31,32. Earlier deployments of biological experiments in parabolic flight have yielded some first insights into the immediate effects on cellular adhesion but only within a 2D cellular monolayer^33,34,35. The dataset used here exemplifies a discovery-driven experiment, showcasing our unsupervised framework for exploring specific morphological traits of cells within a 3D tissue-like structure, potentially altered by cyclic mechanical stresses. Unlike traditional methods focusing on individual organoids or readouts, we use dimensionality reduction methods like principal component analysis (PCA) to investigate potential morphological signatures of ‘flying’ spheroids. This analysis, conducted without any a priori assumptions, serves as a blind test for examining cellular and nuclear morphology changes in parabolic flight conditions. Figure 6 highlights a holistic analysis, ensuring the robustness and reliability of our results while minimizing potential biases or oversight from classical feature selection.

Fig. 6: Integrated workflow for 3D morphological signature understanding.

a, 3D imaging screening generated 120 organoids under four conditions (day 0: GC and F, day+1: GC24 and F24). Morphological features across 8,215 cells were amalgamated without condition descriptors. b, This ‘blind’ pool is used for unsupervised PCA to reduce data’s dimensionality. The cells, now identified by their respective conditions, were then plotted based on their PC1 values. Subsequent clustering isolated specific cell populations based on distinct signatures. Violin plots represent unsupervised clustering of PC1, illustrating cell volume differences. c, Image feedback showcased a prevalent localization of the ‘high PC1’ population (red) at the organoid border. Scale bar, 50 µm. d, Biologically meaningful gating based on the distance from the organoid’s border was applied. Plots of guided filtering on nuclei distance to the border with significant cell volume differences. For all violin plots, blue bars indicate the mean. ****P < 0.0001, using a two-tailed unpaired t-test with Welch’s correction.

The contingency involved 120 stacks of organoids, which were subjected to the described conditions (Fig. 6a; ground control (GC) and flight (F) on day 0, ground control 24 (GC24) and flight 24 (F24) on day+1). These spheroids were digitized using our streamlined method, resulting in a collection of 3D morphological features from more than 8,000 cells. The cells were numerically pooled ‘blindly’ or anonymously, setting aside the identifiers related to the conditions and to organoids. This ‘blind’ pool was then utilized for unsupervised techniques, such as PCA. This technique was used to effectively reduce the dimensionality of the data to its primary component, PC1. This well-known method enhances key feature visualization, reduces noise, improves generalization and increases computational efficiency for classification³⁶. The 3DCellScope interface allows users to select features to be included for the PCA (Supplementary Fig. 3a).

Subsequently, cells with all their identifiers based on their respective conditions were plotted according to their PC1 values. This visualization strategy highlighted distinct signatures within the data, and specific cell populations can be isolated using gating on the PC1 value. Immediate visualization from this clustering was made available through our interface, allowing for feedback visualization in both 2D and 3D image formats (Fig. 6c). The visualization revealed a concentration of ‘high PC1’ cells at the spheroids periphery, offering 3D localization of cell populations based on multiparametric signatures. This enables users to apply manual gating strategies, such as distance from the border, to isolate and analyze specific populations. Unlike traditional/manual data mining, where prior knowledge guides filtering and hypothesis testing, our approach generates cell signatures blindly and integrates raw image feedback for 3D localization, aiding biological interpretation. This reduces time and increases discovery potential compared to hypothesis-driven methods, which can overlook subtle differences or effects.

Here this strategy revealed that cells have undergone a decrease in their volume due to the exposure to the parabolic flight (Fig. 6b,c), and this significant strong diminution was maximized for cells of the ‘high PC1’ population (localized at the periphery of the spheroids) as compared to the low PC1 population at day 0 (−33% and −14% respectively). Less but significant decreasing of cell volume was also obtained at day+1 (high PC1: −12%, low PC1: −8%; Fig. 6b). The traditional manual gating approach on cell distance to the spheroid border (without PCA reduction; Fig. 6d) was able to confirm this significant decrease in cell volume, and the same difference in response correlated with cell position to the external border. To conclude, the morphological stress generated by parabolic flight on spheroids affects more importantly the morphology of external cells (located at the border of the spheroids) and that this differential effect is still significantly observed after 24 h in normal conditions of culture. Furthermore, this effect impacts not only cell volume but also other features such as cell density and nuclei volume (Supplementary Fig. 3b,c).

Discussion

Our approach combines advanced machine learning with classical image analysis to reveal precise morphological changes in 3D cellular microsystems suitable for common 3D microscopy methods. We introduce 3DCellScope, a tool that enables users to: (i) test and compare AI-based segmentation networks, (ii) receive immediate feedback on images and (iii) generate a comprehensive database of hundreds of parameters.

Our method is based on a high-precision 3D nuclei segmentation, and the subsequent cell segmentation performance heavily relies on it. As a limitation, this approach currently requires nuclei staining procedures, either post-fixation DNA labeling or fluorescent genetically encoded proteins for live imaging. We demonstrate that our nuclei-dedicated AI model, DeepStar3D, consistently outperforms others across diverse independent experiments. Notably, it is independent of specific fluorescence wavelengths and achieves superior performance compared to alternative CNNs across a range of image resolutions and signal qualities. Cell segmentation can then be performed using actin staining or dedicated membrane staining if necessary. For subsequent phenotype analysis, evaluating protein expression levels and measuring fluorescence intensities such as mean, minimum, maximum, median and standard deviation is straightforward. As organoids represent an emerging approach to model the intricate complexity of tumors for example³⁷, our innovation accurately reveals and quantifies this complexity by generating comprehensive 3D patterning signatures—an essential analysis that will also be of importance in drug screening processes based on tumor organoids.

These proof-of-concept experiments demonstrate the reliability of our methodology in characterizing 3D morphological changes in complex multicellular structures like organoids or spheroids. Our method is versatile, accommodating both small organoid sets and large-scale batches in multi-well plates. Not requiring whole-organoid segmentation, it can also analyze other cell culture systems, such as monolayers on flat or curved surfaces. By bypassing differentiation-driven variations, the approach offers a comprehensive understanding of cellular responses to external cues, addressing key questions in tissue engineering.

The quantification of 3D cellular deformations in tissue-mimicking structures advances our understanding of cellular biomechanics, tissue homeostasis and therapeutic interventions. This methodology unlocks the full potential of 3D biology, facilitating innovative therapeutic strategies based on organoids and extending its relevance to other fields, including the emerging domain of space biology. On top of incorporating other imaging modalities such as label-free (non-fluorescence) imaging to address the limitation of the mandatory nuclear staining, future developments will integrate advanced 3D tracking approaches to meet the community’s growing need for dynamic analysis, enabling a powerful 4D (3D + time) system that captures cellular and tissue dynamics over time.

Methods

Ethics

Regarding the participant sample, the study was approved by the Scientific Committee of the tumor bank of Lille and the Department of Pathology in Lille University Hospital. The participant signed an informed consent form.

Maintenance of cell culture and medium

Colorectal cancer cell line HCT116 (91091005, Sigma-Aldrich) and breast cancer cell line MCF7 (86012803, Sigma-Aldrich) were cultured in DMEM high glucose (11965092, Gibco) supplemented with 10% fetal bovine serum (10082147, Invitrogen) and 100 U ml⁻¹ penicillin–streptomycin (15070063, Invitrogen) at 37 °C and 5% CO₂ (that is, complete medium).

Human mammary epithelial (HME) cells, purchased from the American Type Culture Collection (ATCC; PCS-600-010), and primary mammary epithelial cells were cultured in complete Mammary Epithelial Cell Basal Medium (PCS-600-030, ATCC) supplemented with Mammary Epithelial Cell Growth Kit (PCS-600-040, ATCC). HME cells were subjected to no more than ten passages in culture when used in experiments.

Formation of spheroids

After trypsinization, the cell lines (HCT116 cells or MCF7 cells) were suspended in complete medium and adjusted with a concentration of 0.3 × 10⁶ cells per ml. Next, 1 ml of cell suspension was dispensed in p24-well (5826-024, Iwaki) plates containing JeWells that had already undergone long-term passivation with 0.5% Lipidure (CM5206, NOF America), as previously described^29,38. The cell culture plates were placed into the cell incubator (37 °C, 5% CO₂, and 100% humidity) for 5 min to get approximately 20–50 cells per spheroid. The cell suspension was removed, washed three times with DPBS (14190250, Invitrogen) and 1 ml of complete medium was added per well. The medium was changed every 3 days. The cells were fixed after 1 week of culture for staining.

Osmotic stress treatment

After 1 week of HCT116 spheroid formation as described above, osmotic stress was applied by adding NaCl into the culture medium as previously described^39,40. Complete medium supplemented with 100 mM NaCl (7647-14-5, Merck) was used as hypertonic medium. Complete medium of HCT116 spheroids was removed and replaced with the hyperosmotic medium for a 6-h incubation in 5% CO₂ at 37 °C, followed by fixation for staining.

Microniche-shaped ECM

Microniche-shaped ECM arrays were created using elastomeric stamps (polydimethylsiloxane (PDMS)) containing the desired cup-shaped (50-µm diameter) or flat-bottom (80-µm diameter) pillars. Briefly, the PDMS working mold was produced as a replica cast of a primary mold, following standard soft-lithographic procedures. We used Sylgard 184 (Dow Corning), prepared by thoroughly mixing and outgassing the base resin and its reticulation agent in 10:1 weight ratio. Then the mixture was poured on the primary mold and degassing in a vacuum jar was used a second time to ensure complete filling of the cavities and removal of any trapped air. The PDMS was then thermally cured for 1–2 h at 65 °C on a hot plate. Finally, after cooling down, the PDMS was peeled off and diced to final dimensions for usage.

In a 35-mm glass-bottom dish, the PDMS working mold with pillars of defined size and shape was pressed to a hydrogel layer of methacrylated gelatin (GelMA) containing photoinitiator lithium phenyl-2,4,6-trimethylbenzoylphosphinate (LAP) (5272, Advanced Biomatrix) at 50 °C. Next, GelMA-LAP was crosslinked using a UV-LED box (UV-LED KUB2, Kloe France, 365 nm) with a power density of 35 mW per cm² for 1 min, and the mold was carefully peeled off. Microniches were then covered with DPBS and placed in the cell incubator.

Primary cell seeding on ECM microniches

After trypsinization, HME cells were suspended in complete Mammary Epithelial Cell Basal Medium containing 5 µg ml⁻¹ recombinant human insulin, 6 mM l-glutamine, 1 µg ml⁻¹ epinephrine, 5 µg ml⁻¹ apo-transferrin, 5 ng ml⁻¹ recombinant human TGFα, 0.4% ExtractP and 100 ng ml⁻¹ hydrocortisone hemisuccinate and adjusted with a concentration of 0.3 × 10⁶ cells per ml. Then, 2 ml HME cells were seeded at the top of microniches made from GelMA-LAP crosslinked hydrogels. The cell culture dish was placed into the cell incubator (37 °C, 5% CO₂ and 100% humidity) for 5 min to allow cells to enter microniche cavities, allowing approximately 20–50 cells per microniche. The top cell suspension was removed and washed three times with DPBS (14190250, Invitrogen), and 2 ml of complete Mammary Epithelial Cell Basal Medium was added into the 35-mm glass-bottom dish. The medium was changed every 2 days, and the cells were allowed to self-organize into curved layers or epithelial cells adopting the geometry of the hydrogel microwells. The cellular microniches were fixed after 2 weeks of culture for staining.

Primary PDAC organoid culture

Primary PDAC organoids are derived from a resected differentiated PDAC as described in Hadj Bachir et al.⁴¹. Organoids were embedded in domes of Matrigel (Corning, 356231) and cultured with complete pancreatic tumor organoid medium (ADF medium supplemented with Glutamax (1×, Invitrogen, 35050-061), HEPES (1×, Sigma-Aldrich, 83264-100ML-F), B-27 Supplement Minus Vitamin A (Invitrogen, 12587-010), N2 (1×, Invitrogen, 17502-048), N-acetyl-l-cysteine (1 mM, Sigma-Aldrich), Wnt3a/RSPO1/Noggin conditioned medium (50% vol/vol), epidermal growth factor (EGF, 50 ng ml⁻¹, PeproTech AF-100-15), fibroblast growth factor 10 (FGF10, 100 ng ml⁻¹, PeproTech, 100-26), A83-01 (0.5 μM, Tocris, 2939), nicotinamide (10 mM, Sigma-Aldrich, N0636), gastrin (10 nM, Sigma-Aldrich, G9145) and Y-27632 dihydrochloride (10 μM, Tocris, 1254)) as recommended in Boj et al.⁴².

Primary PDAC organoid staining and imaging

Organoid domes were rinsed with PBS and then fixed for 15 min at room temperature in a solution of 4% formaldehyde and 0.1% glutaraldehyde in PBS and washed again with PBS. Staining was performed with NucBlue (Invitrogen) and Alexa Fluor Plus 647 Phalloidin (Invitrogen) for 1 h at room temperature. Ultimately, samples were washed three times for 1 h in PBS and mounted in Fluoromount G mounting medium. Fluorescence images were acquired with a Ti2-E Nikon AX confocal microscope driven by NIS software (Nikon Instruments, Japan). A ×25 Plan Apo Lambda S ×25 Sil objective (1.05 NA) was used to acquire 2,048 × 2,048-pixel images, with a pixel size of 345 nm. Step size was set at 1 µm for z-stack imaging. The sample was excited at 405 nm and 640 nm, and photons filtered with the adequate dichroic filter were collected within the 429–474-nm and 651–721-nm windows, respectively.

Parabolic flight

The parabolic flight experiments were carried out on board a Cessna Citation II (Royal Netherlands Aerospace Centre (NLR)), organized by the Swiss SkyLab Foundation and the UZH Space Hub (Innovation Cluster Space and Aviation of the University of Zurich) as part of the Swiss Parabolic Flight Program⁴³. During a parabolic flight maneuver, 15 consecutive parabolas occurred in three steps. Firstly, in the pull-up phase, the nose of the aircraft is raised to about 60 degrees for 20 s (hypergravity at 2.5 g). Then, in the second phase, the aircraft is in free fall and experiences microgravity for 13 s to 15 s following a parabola trajectory. Finally, in the pull-out phase, the aircraft is tilted down at approximately 60 degrees for another 13 s to 15 s (hypergravity at 2.5 g). During a parabolic maneuver, an aircraft is weightless due to flying on a Keplerian trajectory, an unpropelled body in an ideally frictionless space that is subjected to a centrally symmetric gravitational field⁴⁴.

Cell culture procedure for the parabolic flight

Before the parabolic flight, MCF7 spheroids were cultured, in a tissue culture room close to the airfield, in 24-well plates (n = 4) containing JeWells for 1 week as described above. Two 24-well plates (n = 2) were used as 1 g ground control. The other two 24-well plates were taken on board the aircraft for the parabolic flight using a portable cell incubator allowing normal cell culture conditions during all the transportation and flight: 5% Co₂, 37 °C, Cellbox, following the manufacturer’s procedure. Ground control plates were fixed at the same corresponding time as the plate on board and using exactly the same procedure as described in the fixation procedure below. We referred to ‘GC’ and ‘F’ for the Ground Control and Flying plates that were fixed just after the last 15th parabola (at day 0). ‘GC24’ and ‘F24’ correspond to the same conditions but fixed 24 h after the flight (at day+1). All plates were sealed using CO₂-permeable polymeric films to avoid any medium lost during transportation and flights. After fixation, the plates were washed three times with DPBS and stored at 4 °C for staining and image acquisition.

During the parabolic flight, we maintained continuous monitoring of CO₂ percentage, humidity and temperature using the built-in capabilities of the portable incubator Cellbox. Furthermore, all flight parameters, including cabin pressure and the G vector, were meticulously recorded by the TU Delft Cessna plane crew members present on board. Additionally, to ensure consistency, we used the same lot number, injection methods, volume and containers for fixative, maintaining identical durations between the ground control and the flight experiments.

Fixation and staining for imaging

Spheroids and microniche cell layers were fixed for 15 min in 4% paraformaldehyde (28906, Thermo Fisher Scientific) at room temperature. Then the 3D cultures were permeabilized for 1 h in 0.2% Triton X-100 (T9284, Sigma-Aldrich) solution in sterile DPBS (14190250, Invitrogen) at room temperature on an orbital shaker. Samples were then incubated with 0.5 μg ml⁻¹ DAPI (62248, Thermo Fisher Scientific) and Alexa Fluor 647 phalloidin (A12379, Thermo Fisher Scientific; 1:200 dilution) at room temperature for 30 min on an orbital shaker followed by two fast rinsing steps with washing buffer before image acquisition.

Image acquisition process

The images of spheroids were obtained with a confocal laser scanning microscope (FV3000, Olympus). All images were acquired using a ×30/1.05, WD 0.8 silicone objective, with a z-step size set to 3 μm and an xy resolution of 0.414 μm. Confocal images of DAPI (laser excitation of 405 nm) and phalloidin (laser excitation of 640 nm) were acquired in less than a minute per spheroid (simultaneous excitation mode). Multi-area time-lapse automatic acquisition mode was used to image hundreds of spheroids per well.

Image preprocessing using histogram matching (optional)

The acquired 3D images were subjected to intensity decay along the z direction, necessitating intensity enhancement, particularly in the later z slices. To perform histogram matching using Skimage²⁶, we calculated the 99th percentile intensity for each slice and selected the slice with the highest 99th percentile as the reference slice. Based on the reference slice, a histogram matching procedure was performed for all slices. This preprocessing was carried out for all channels.

Nuclei segmentation by in-house 3D StarDist: DeepStar3D

Nuclei segmentation was performed using a 3D StarDist¹⁸ CNN model. Our pipeline uses a model trained internally on realistic simulated 3D image data with a voxel size of 0.8 μm × 0.8 μm × 1 μm.

The critical training phase was performed internally using a realistic simulated 3D dataset. Generating synthetic data involves several sequential steps, starting with creating a labeled mask, where each voxel gets a unique integer identifier for specific nuclei. To achieve a diverse representation of nuclear morphologies, reflecting the complexity found in real-world scenarios, we integrated a combined strategy involving a parametric sphere representation coupled with a 3D deformation model⁴⁵. The resulting labeled mask was considered as the synthetic ground truth. To produce the corresponding image, voxel values of this mask are transformed using techniques that mimic the acquisition procedures commonly used in microscopic systems. We used Perlin noise to replicate textural variations⁴⁶ and Gaussian blur to emulate point spread function⁴⁷, introducing subtle imaging imperfections. In the end, through the manipulation of a range of simulation parameters, we have produced a dataset consisting of synthetic 3D images paired with labeled masks. This dataset has served as a valuable resource for training and evaluating various segmentation algorithms.

Notably, the generated dataset was constructed with a unique voxel size of 0.8 µm × 0.8 µm × 1 µm. To fit the resolution of the trained network, users can use the auto-resize function for resizing their 3D data to match the voxel size of the training images. Alternatively, data can also be resized by manually inputting (xyz) the resampling factor. To speed up the processing, we cropped our images to 512 × 512 pixels before segmentation. In our implementation, results of our 3D StarDist trained network can be optimized by fine-tuning several thresholds, such as the probability threshold (typically ranges from 0.3 to 0.7) to retain only entities with a probability exceeding this threshold, as well as the non-maximum suppression (NMS) threshold (typically ranges from 0.1 to 0.3), which selects a single entity among overlapping entities. In our approach, size thresholds can also be set to filter out debris.

Cell segmentation by seeded grayscale watershed

Cells were segmented using the seeded watershed method of Skimage²⁶. The nuclei segmentation results were used as the starting seeds. The fundamental principle of this algorithm involves identifying starting points, or ‘seeds’, typically associated with objects of interest in an image. These seeds serve as reference points from which segmentation propagates. The algorithm treats the image as a landscape and assigns pixels to the nearest seed’s ‘basin’. In our context, the initial seeds for segmentation were derived from the nuclei segmentation results, and their propagation was guided based on the intensity channel in which the cells were observed (that is, phalloidin). Notably, the cell expansion was confined within the organoid mask. Additionally, the expansion distance from the nucleus border to the cell border was limited to a maximum distance (which was set at 14 μm by default but can be adjusted if needed). This prevents the over-expansion of cells from the nucleus border. Once the cell label mask was created, it was further refined using a 3D median filter with a sphere kernel.

Organoid segmentation by the Otsu method

Organoid segmentation was performed with the Otsu method⁴⁸ using the mean intensity of all channels. Before segmentation, we enhanced contrast for all z slices by applying contrast limited adaptive histogram equalization⁴⁹ preceded by image normalization. The contrast-enhanced images were then smoothed using a 3D Gaussian filter. The auto-Otsu threshold was calculated using the stack histogram and the final threshold can be adjusted using a scale factor, which was set at 0.8, by default (that is, final_threshold = scale_factor × auto_otsu_threshold). After thresholding, small elements were automatically filtered from the mask, to retain only the largest one in case of out-of-organoid debris or cells. To generate a smooth 3D mask, we conducted a series of mathematical morphological operations on a slice-by-slice basis, that is, dilation with a disk structure element followed by hole filling and erosion with a disk structure element.

Benchmark images for quality assessment of segmentation

We have compiled images of 3D nuclei sourced from independent research groups across various biological and staining models into a unified validation dataset encompassing four types of data. In each instance, manual annotation of nuclei in 3D was conducted using Napari software, resulting in a total of 594 nuclei established as ground truths.

Type 1: The ZeroG breast cancer spheroid dataset originates from parabolic flight experiments, with detailed descriptions of staining and acquisition procedures provided in corresponding sections. Images were captured using a ×30/1.05, WD 0.8 silicone objective, with a z-step size of 3 μm and xy resolution of 0.414 μm. For our benchmark, a single spheroid was selected, comprising 99 annotated nuclei.

Type 2: The monolayer HCT116 cells were transduced with lentiviral particles containing the plasmid K4-EF1a-H2B-Neon-IRES-Puro. Two days after transduction, H2B-mNeonGreen-expressing cells were sorted via bulk fluorescence-activated cell sorting using a Sony SH800 cell sorter. For imaging, cells were seeded onto four-well chambered coverglass (Nunc LabTek II, 155382). Imaging was conducted using a Zeiss LSM 780 laser scanning confocal microscope equipped with a ×63 oil immersion objective (NA 1.4). The xyz resolution was 0.264 µm × 0.264 µm × 1 µm. All images were captured as time-lapse videos. A single frame was extracted for our benchmark, with 38 manual nuclei annotations performed.

Type 3: The human colon organoids were derived from a surgically resected colon specimen, following protocols approved by the Johns Hopkins Medicine Institutional Review Board. Organoids were transduced with pLV-H2B-Neon-T2A-mCherry-CAAX-IRES-Puro, and cells expressing the transgene were selected using 1.0 µg ml⁻¹ puromycin. For imaging, organoids were plated on eight-well chambered coverglass (Nunc LabTek II, 155360). Imaging was conducted 4 days after seeding on a Zeiss LSM 780 laser scanning confocal microscope with a ×40 water immersion objective (NA 1.1). The xyz resolution was 0.2442 µm × 0.2442 µm × 2.0 µm. All images used in our analysis were captured as part of time-lapse videos. A single frame was extracted for our benchmark, generating 358 manual nuclei annotations.

Type 4: The TM00099 cell culture image was obtained from shared data utilized for the Cellos pipeline benchmark²⁴. Clonal cell lines were infected with lentivirus and selected for by sorting for nuclear EGFP or mCherry expression. Cells were seeded at a density of 30,000 cells per well in triplicate in 96-well plates (PerkinElmer) coated with 35 μl Matrigel Growth Factor Reduced (BD Biosciences, 356230). Plates were imaged using the Opera-Phenix High-Content Screening System (PerkinElmer). Twenty-five fields per well were imaged using a ×20 water objective. The xyz resolution was 0.598 µm × 0.598 µm × 5.0 µm. A single field was extracted for our benchmark, with 99 manual nuclei annotations generated.

Benchmark methodology of nuclei segmentation

Segmentation quality of benchmarked StarDist¹⁸ models was evaluated using the F1 score over IoU threshold:

$$\begin{array}{l}{{\rm{F}}}_{1}^{{\rm{IoU}} \ge N \%}=\displaystyle\frac{2 \sum {{\rm{obj}}}_{{\rm{IoU}}\ge N \% }}{2 \sum {{\rm{obj}}}_{{\rm{IoU}}\ge N \% }+\sum {\rm{obj}}-{{\rm{pred}}}_{{\rm{unmatched}}}+\sum {{\rm{obj}}}_{{\rm{IoU}}\ge N \% }}\\\quad\qquad {\rm{where}}\:{{\rm{IoU}}}_{{\rm{obj}}}=\frac{{\rm{Area}}\left({{\rm{Shape}}}_{{\rm{pred}}}\cap {{\rm{Shape}}}_{{\rm{obj}}}\right)}{{\rm{Area}}\left({{\rm{Shape}}}_{{\rm{pred}}}\cup {{\rm{Shape}}}_{{\rm{obj}}}\right)}\end{array}$$

Predictions and ground truths were aligned using a modified Jonker–Volgenant algorithm⁵⁰ designed for bipartite graphs optimizing IoU. Before inference, images were resized to the model training resolution, as performance is strongly influenced by accurate object sizing. Identical inference parameters (NMS = 0.3, probability threshold = 0.5) were applied across all models. We opted for the IoU 50 threshold for the benchmark, given its widespread usage and reliability (Extended Data Table 2 and Extended Data Fig. 3).

To evaluate the degree of independence concerning the precision of nuclei segmentation, indicated by the IoU score per nucleus, across varying qualities of imaging such as SNR and nuclei density (number of neighbors), correlation analyses were conducted using Spearman’s rank correlation coefficient on the image ‘ZeroG Breast Cancer Spheroid’ (N = 99 nuclei; Extended Data Fig. 3b). SNR was obtained using the following equation, with SP representing signal power and NP noise power:

$$\begin{array}{*{20}{l}}{\rm{SNR}}\left({\rm{obj}}\right)=10{\log }_{10}\left(\displaystyle\frac{{\rm{SP}}\left({\rm{obj}}\right)}{{\rm{NP}}\left({\rm{obj}}\right)}\right)\\\quad{\rm{where}}\,{\rm{SP}}\left({\rm{obj}}\right)={\rm{mean}}\left({I}_{i}^{2}\right),\,{\forall}_{i} \in {\rm{obj}}\\\,{\rm{and}} \, {\rm{NP}}\left({\rm{obj}}\right)={\mathrm{var}}\left({I}_{j}\right), {\forall}_{j}{\notin}\, {\rm{obj}} \, {\rm{and}} \,{j} \in B;\\ \quad B = \left[{x}_{\min },{x}_{\max}\right] \times \left[{y}_{\min },{y}_{\max }\right] \times \left[{z}_{\min },{z}_{\max}\right] \end{array}$$

Inference speed was estimated based on ten repeated image predictions (76 × 512 × 512 voxels, image ‘ZeroG Breast Cancer Spheroid’) for both tiled (as described in ref. ¹⁸) and untilled cases, using an Intel(R) Xeon(R) w5-3435 × 2.98 GHz, 128 GB 4,400 MHz RAM. Results are displayed in Extended Data Table 3.

Quality assessment of cell segmentation

We computed the percentage of errors in cell segmentation. For each image, a separate expert manually counted and outlined cellular boundaries in stacks of actin staining images to create a ground truth. The percentage of errors generated by the 3DCellScope were determined by identifying any missing cells or those with major precision issues in their boundaries (where the area differed by more than 30%) as error events, then dividing this number by the total cell count. We used this approach for different z-planes of the same organoid (Extended Data Fig. 4a) and between different organoids at the same z-plane (Extended Data Fig. 4b).

Extraction of morphological and topological descriptors

For each detection (nuclei, cell, organoid), a comprehensive list of morphological features such as volumes, elongation, roundness, equivalent ellipsoid diameters, bounding box sizes, centroid coordinates, distance to organoid border (for nuclei and cell) and intensity statistics for each channel (minimum, maximum, mean, standard deviation) was generated. Other than these morphological features, a density analysis tool was also developed to extract 3D topological descriptors from nuclei position and organoid mask. The 3D topological descriptors are classified into nuclei spatial arrangement descriptors and nuclei density descriptors (Extended Data Fig. 6). Cell-to-neighborhood (nuclei spatial arrangement) descriptors enable quantitative measurements of cell organization by fitting an ellipsoid from all nuclei coordinates in a spheroid of radius r (µm; Extended Data Fig. 6a) that can be specified by users. Ellipsoid axes length, prolate and oblate ratios and various descriptors extracted from the fitted ellipsoid provide insights into internal organization of organoids such as tissue patterning. For cell-to-cell density analysis, nuclei density descriptors computed include distances between nuclei, local density and Ripley’s estimated number of nuclei in a spherical volume of user-defined radius, r (µm; Extended Data Fig. 6b).

Data mining and unsupervised analysis

The complete workflow from segmentation to data mining can be accomplished seamlessly with the developed tools, and Extended Data Figs. 2 and 5 show examples of user-friendly interface and database organization. The 3D segmentation masks in TIFF files and the database in the CSV file, together with the multichannel images (in composite or separate TIFF files), can be used for interactive data exploration, and existing software dedicated for image-based data mining such as VTEA⁵¹, CellProfiler Analyst⁵² or KNIME⁵³ provides important features such as gating, imaging feedback in 2D and 3D and statistical analysis. We incorporated all these well-known functionalities with various other tools such as generating additional features to enable advanced data analysis. In our analysis, we created new feature, named as CV, by dividing the standard deviation with mean intensity (Fig. 3).

For the unsupervised analysis, a PCA, which is commonly used for dimensionality reduction of large databases, was used to generate morphological signatures. Input PCA parameters used to generate data in Fig. 6 were selected so as to be only related to morphological descriptors of nuclei and cells such as volume, roundness, elongation, distance to neighbors and crystal distance. Supplementary Fig. 3a highlights the 3DCellScope selection window used to perform the selection of ten morphological parameters used in this PCA analysis.

Statistics and reproducibility

The precision of image analysis segmentation was carefully assessed as described in Extended Data Figs. 3 and 4 and Supplementary Fig. 1. No statistical methods were used to predetermine sample size. The experiments were not randomized. The investigators were not blinded to allocation during experiments and outcome assessments, except for PCA analysis as stipulated (Fig. 6). No data were excluded from the analysis, unless stipulated (Fig. 1; filter on the size of nuclei). All experiments were reproduced independently with similar results at least three times if not stated in the main text or figure legends.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.