Background and Summary

Cell death is pivotal in multiple physiological and pathological processes1,2. Every day, damaged or dysfunctional cells undergo apoptosis to accommodate new healthy cells, an essential mechanism in tissue development and homeostasis3,4,5. Moreover, cell death is also associated with the defense mechanism entitled to clear dysfunctional or infected cells, thus preventing the onset of pathological conditions like cancer and infectious diseases6. The fine regulation of this mechanism is necessary to limit inflammation7 and to prevent autoimmune disorders2. The death of immune cells is particularly important as it contributes to the initiation of the immune response8, and the selection of antigen-specific cells9, amongst others10.

From a morphological perspective, multiple types of cell death are characterized by the disruption of the cell membrane and the formation of multiple bodies. Amongst these, apoptosis presents hallmarks such as chromatin condensation, cell shrinkage, membrane blebbing, and the formation of apoptotic bodies11. The latter are membrane vesicles that encapsulate the cytoplasm preventing the leakage of alarming signals into the extracellular space12. This process is considered immunologically silent, as it does not further elicit an inflammatory response. Successively, upon the formation of apoptotic bodies, phagocytic cells are recruited locally to clear the resulting apoptotic vesicles13, promoting the end of this process. In this paper, we refer as “apoptotic-like” to all the cell death events that yield to the disruption of cell membrane and formation of apoptotic bodies.

Different imaging techniques can be used to study apoptotic-like cell death. In vitro imaging allows control over multiple experimental conditions, including cell density, migratory cues, or signaling pathways14,15,16. However, this modality cannot capture the complex behavior of the immune system within living tissues. Conversely, techniques such as 2-photon intravital microscopy (2P-IVM) enable the visualization of cells within their physiological environment by delivering in real time17,18,19, four-dimensional time-lapses for different channels (x, y, z, c, t)20,21. Therefore, 2P-IVM represents one of the most informative platforms for the physiological study of cell death in vivo22. However, acquiring high-quality 2P-IVM movies remains non-trivial due to technical and biological challenges such as the surgical procedure, the bleaching of the sample, and the stability of the imaging site, among others23,24,25. As a result, the potential to generate extensive apoptotic datasets in vivo remains limited, whereas the lack of accessibility prevents the reuse of existing 2P-IVM movies25,26. Consequently, the scarcity of 2P-IVM movies negatively impacts studies that require curated datasets, such as comparative analysis or algorithmic solutions. This shortage of data mainly affects the latter aspect, given that mainstream deep learning (DL) algorithms are known for being data-hungry27. Therefore, establishing publicly accessible 2P-IVM repositories, complete with metadata and annotations, becomes pivotal. Such an initiative would broaden the scope of fundamental research on cell death and enable the development of robust algorithms to quantify this biological process in vivo.

In this study, we address this need by providing a comprehensive dataset capturing cell death events with apoptotic-like morphology in vivo28. This dataset encompasses 2P-IVM movies featuring three immune cell type:, neutrophils, eosinophils, and dendritic cells imaged  in various experimental settings associated with the inflammatory response. A schematic workflow is shown in Fig. 1A,B. In addition to the raw data, we provide manual annotations of cell death events, which were curated by multiple operators trained to detect the morphological hallmarks of the process (Fig. 2). Accordingly, each apoptotic-like event is first identified in 2P-IVM fields-of-view (Fig. 2A), and successively annotated for the duration of the process (Fig. 2B). The annotations include centroid-based trajectories of the apoptotic-like events, frame-by-frame semantic annotations describing the cell morphology at any given time step, and 3D volumetric reconstructions of the cellular surface. To determine if a cell presented apoptotic-like morphology, we strictly adhered to the observation of the morphological hallmarks of apoptosis, including shrinkage of the cell, membrane blebbing, and the formation of apoptotic bodies. As a result, the presented catalog is an invaluable resource for comparative studies and the development of analytical solutions for the classification, detection, and segmentation of apoptotic-like cell death in vivo.

Fig. 1
Fig. 1
Full size image

Dataset generation workflow. (A) Representative intravital acquisition set-up. Immune-relevant tissues from an anesthetized mouse were surgically exposed and subjected to 2P-IVM. Resulting in vivo acquisitions (B) are 4D sequential data composed of multiple focal planes per time point and multiple channels. Raw data are annotated by three individual operators tracking cells undergoing apoptotic-like death and generating a 3D mask. Dying cells are associated with a cell state between “blebbing” and “disrupting” that defines the phase of the ongoing process.

Fig. 2
Fig. 2
Full size image

Dataset annotation of apoptotic events. (A) Micrograph depicting a MP-IVM acquisition in the lymph node of a mouse. Neutrophils expressing GFP (green) and a UV-inactivated influenza vaccine are labeled with DiD (red). (B) Dying cells from MP-IVM acquisitions are annotated frame by frame for the duration of the process according to their morphology (top). Frame annotation include semantic labels (pre-apoptotic, blebbing, disrupted), centroid tracking (middle), and 3D volumetric reconstruction of the cell (bottom).

Methods

Mice

Mice were maintained in a pathogen-free animal facility at the Institute for Research in Biomedicine (Bellinzona, Switzerland). Before imaging, mice were anesthetized with a cocktail of xylazine (10 mg/Kg) and ketamine (100 mg/Kg), as previously reported21. All experiments were conducted in compliance with the regulations set by the local authorities and were approved by the Swiss Federal Veterinary Office.

Intravital microscopy

All the videos were acquired through intravital two-photon microscopy (TrimScope, LaVision BioTec) in lymphoid organs (lymph nodes and spleen), under different experimental conditions detailed in Table 1. To acquire videos in the lymph nodes, the organ was exposed through minimally invasive surgery as previously described29. Then, time-lapse images were acquired using a custom upright microscope. To acquire videos in the spleen, mice were anesthetized and immobilized, while the spleen was accessed by making an incision through the skin and musculature as previously described30. After exposure of the organ, the spleen was moisturized by applying pre-warmed PBS. In both surgeries, 4D data encompassing multiple stacks, channels, and time points (Table 1, Fig. 1A) were generated. The imaging system employed two Ti:sapphire lasers (Chameleon Ultra I, Chameleon Ultra II, Coherent) and an optical oscillator that emitted light within the 1,010–1,340 nm range (Chameleon Compact OPO, Coherent). The output wavelength ranged between 690–1,080 nm, ontributing to probe excitation and tissue second-harmonic generation (SHG).

Table 1 Experimental Conditions. This table details the channel specifications and experimental settings for each movie entry in the dataset. Cell types visible in each channel include neutrophils (Neu), dendritic cells (DCs), and eosinophils (Eos). Additionally, when applicable, the channels include second harmonic generation (SHG) and vaccine visualization. Experimental settings cover the used cell markers, fluorophores, target organ, applied stimuli, and microscope magnification.
Full size table

All the videos were acquired under inflammatory conditions. In all, except Neu7, Neu11, Neu17, and Den1, UV-inactivated PR8 virus was injected in the footpad 0 up to 24 hours prior to imaging. Neu7, Neu11, and Neu 17 videos were acquired in an experimental model of Vaccinia virus infection, whereas Den1 was acquired in a model of B16 melanoma metastasis in the lymph node. CFB and GFP-labeled neutrophils were adoptively transferred from a donor to a recipient animal which was CD11c-YFP in the case of Den1. Further details on the staining and experimental stimuli are available through Immunemap31.

Data processing

The microscopy data acquired from the videos were stored in HDF5 files, comprising uint8 or uint16 TIFFs. Prior to image analysis, the raw data underwent no preprocessing. For cell detection, tracking, and volumetric reconstruction of the microscopy videos, we employed Imaris imaging software (Oxford Instruments, v9.7.2). Subsequently, the data obtained from Imaris underwent further analysis using customized Matlab and Python scripts (refer to the code availability section for details).

Manual tracking

The centroids of cells were manually annotated and tracked over time using the “Spots drawing” tool in Imaris (Bitplane). To ensure robustness, three operators independently and redundantly performed tracking without knowledge of reciprocal results. During the tracking process, the operators followed morphological criteria to identify cells showing apoptotic-like morphodynamics. Specifically, they focused on sequences exhibiting the characteristic hallmarks of apoptosis: 1) membrane blebbing, 2) subsequent formation of apoptotic bodies, and 3) eventual cell disruption (Fig. 2B). For each frame within the apoptotic-like sequences, the operators placed a centroid at the center of the cell using the Imaris “Spots” function, resulting in a connected track representing the apoptotic-like event (x, y, z, t, ID). Subsequently, the trajectories generated by each operator were consolidated into a ground truth using a majority consensus scheme that considered the duration and the location of the event. The starting and ending time points were determined by identifying overlaps among at least two of the three tracks of the annotators. For each time point, the two closest spatial coordinates were selected and averaged to create a final ground truth. Then, we performed volumetric 3D reconstruction using the Imaris “Surfaces” function. This reconstruction process generated precise 3D meshes of each cell undergoing apoptotic-like death, enhancing the accuracy and reliability of the dataset.

Semantic annotation

Three independent annotators labeled the state of the cells based on specific morphological criteria. To annotate the interval and dynamics of the cell death process, the annotators assigned to each cell, for each frame within the ground truth sequence, one of two semantic labels: “membrane blebbing” or “cell disruption (Table 2). To ensure robustness and consistency, a majority consensus scheme was employed to establish the final cell state. This scheme considered the overlap between annotations from at least two operators, promoting a unified and reliable consensus on the semantic label for each cell at every time point, hence minimizing potential biases and variations in the annotations.

Table 2  Cell Death Phases Annotation. This table presents the semantic annotations of all cell death events. Each video entry includes a unique identifier along with frame-level semantic annotations indicating the morphological states of dying cells. The latter states are classified as “blebbing” or “disrupting,” each characterized by a start and end frame. Additionally, the table indicates whether cell death events triggers a local immune response (“response”).
Full size table

Computation of quality metrics

To compute the signal-to-noise ratio (SNR) of the entries of the dataset, for each frame of the movies we computed a reference denoised image using a median filter. Successively, we used the SNR function of the Matlab Signal Processing toolbox to compare the reference image to the original one, computing an estimate of the SNR. This procedure was applied for each channel separately.

Computation of density metrics

The density metrics were computed using adaptive thresholding and morphological operations to separate foreground objects from the background in each channel. We then counted the number of cells visible at each time point in each channel. An estimate of the number of cells throughout the movies was generated by averaging these numbers over time. By knowing the spatial coordinates of the cells, we also estimated the average shortest distance between cells and a coefficient of clustering depicting the percentage of cells in close proximity.

Statistical analyses

Statistical analysis and plot visualization were performed using GraphPad Prism 8 (Graphpad, La Jolla, USA). All statistical tests were performed using the non-parametric Kruskal-Wallis test or Mann-Witney test. The significance of the p-value is described as * when p < 0.05, ** when p < 0.01, and *** when p < 0.001.

Data Record

Imaging data, metadata, and experimental details are available through the following channels: https://zenodo.org/records/14551288 or https://app.immunemap.org/cddb. The dataset on Zenodo28 comprises unprocessed raw microscopy videos, each characterized by a unique identifier (ID) and essential metadata. These videos encompass multiple channels and extend across five dimensions: spatial coordinates (x, y, z), time (t), and multiple channels (c). Raw videos are stored in HDF5 format and uploaded individually according to their cell type, for instance “Neu1.h5”. Within each HDF5 file, two types of annotations are included: 1) tracks coordinates of apoptotic-like events (x, y, z, t, ID), and 2) 3D volumetric reconstructions of cell deaths represented by meshes defined by vertices and edges. An additional folder labeled “Annotations” contains two sub-folders:

  1. 1.

    Cell deaths

  2. 2.

    Semantic annotations

The “Cell deaths” folder contains trajectories of apoptotic-like events in comma-separated value format (CSV). Each filename incorporates the identifier of the corresponding microscopy acquisition and the channel where apoptotic-like cells were observed, for instance “Neu1_ch1”. The “Semantic annotations” folder contains semantic labels (CSV) describing the morphological state of cells at various stages of the cell death process. These labels indicate whether dying cells exhibit “blebbing” or “disruptive” morphology and if they elicit an immune response. Start and ending frames are provided to denote the time interval of each state. Lastly, three additional folders contain cropped microscopy movies centered around dying cells:

  1. 1.

    Dendritics_crops

  2. 2.

    Eosinophils_crops

  3. 3.

    Neutrophil_crops

Each cropped movie is named after the original uncropped acquisition and specifies the cell ID of the depicted cell death, such as “Neu1_cell1”.

The dataset on immunemap31 includes all the uncropped HD5F movies, providing complementary experimental details for each acquisition. In Supplementary Information we provide a minimal example in Python to download individual files using the Zenodo API. Moreover, data, metadata, experimental details and annotations (in JSON or CSV format) can be retrieved through the application programming interfaces offered by immunemap and described at https://www.immunemap.org/index.php/doc.

Technical Validation

Imaging acquisitions

Movies in the dataset were imaged with different microscopy settings. These included different voxel sizes, sampling rates, duration, and volume (Supplementary Table 3). As these parameters can influence the overall quality of the movies, we estimated the image quality of our dataset by computing the signal-to-noise (SNR) ratio for each acquisition (Fig. 3A). Higher SNR indicates better image quality, and reportedly, values above five are considered sufficiently good to distinguish foreground objects from the background. Hence, the measured average SNR of 10 can assert the overall quality of the dataset and guarantee that their entries provide reliable and interpretable results.

Fig. 3
Fig. 3
Full size image

Data quality and validation. (A) Distribution of the signal-to-noise ratio of the different movies. (B) Prevalence of the sampling rates used for different microscopy acquisition. (C) Distribution of cell number per field-of-view. (D) Distribution of the shortest distance between each cell. (E) Distribution of cells involved in a local cluster formed by at least three cells in close proximity. (F) Ratios between the annotated timing of cell disruption and sequence duration. A value of 0.5 indicates that the cell disrupts in the middle of the annotation. (G,H) Scatterplots showing the total track distance of annotated apoptotic-like events (G) and their duration in frames (H). (I,J) Inter-operator consensus rate during the annotation of apoptotic-like events (I) and phases (J).

The sampling rate of the movies in our dataset varied between 15, 30, 45, and 60 seconds (Fig. 3B). This aspect is relevant when evaluating dynamic processes in time lapses. Specifically, estimations of general cell motility, such as speed, straightness and cell tracking, can be significantly influenced by the selected sampling rate23. Lower sampling rates may result in the loss of critical information, especially in highly dynamic processes like cell death, where rapid and short-lived changes in the morphology of the cell may occur. To address this concern and ensure an accurate depiction of fast and transient processes, our dataset encompasses movies with high sampling rates.

Aside from the acquisition settings, the interpretability of imaging data can be influenced by the biological content and complexity of the acquisitions. A prominent parameter in this regard is cell density, which significantly impacts the analysis. In time lapses with high cell density, potential side effects, such as cluttering, cell merging, and cell splitting, may arise. These factors challenge cell detection efforts and may hinder crucial information within the data. Consequently, excessive cell density may render specific algorithms incapable of extracting meaningful insights from the movies. Therefore, we estimated pivotal parameters related to cell density and aggregation for each channel in the dataset. These parameters encompass the overall cell number, the shortest distance between cells, and a clustering coefficient that indicates the percentage of cells in close proximity to at least three other cells (Supplementary Table 3, Fig. 3C–E). These measurements underscore the variability in the dataset, with movies ranging from 20 cells to movies containing up to 300 cells and exhibiting a clustering coefficient near 1. By analyzing such diverse time-lapses, researchers can effectively benchmark their detection algorithms on increasingly complex scenarios, ensuring the robustness and applicability of their methods across a wide range of biological conditions and cell densities.

Apoptotic-like events

In the absence of specific apoptotic reporters, the annotated cells might follow a pathway different than apoptosis. For this reason, we adopt the umbrella term “apoptotic-like” to indicate all cell death events sharing morphological similarity with apoptosis. While this does not ensure the annotation of a specific death pathway, the presented catalogue is an invaluable resource for the development of analytical solutions for the classification, detection, and segmentation of cell death presenting apoptotic features in vivo. Cell death events with apoptotic-like hallmarks are inherently diverse, exhibiting variability that can affect classification and detection algorithms. One major challenge arises from the distinct morphologies of cells, leading to a class heterogeneity32,33 that makes it difficult to identify consistent patterns within the same label in supervised classification. Additionally, the variable duration of sequences can pose a challenge to algorithms aimed at classifying the cell death process over time. Finally, the residual motility of cells, caused, for instance, by drifting, can further complicate the accurate identification of cell coordinates during the death process. To address these challenges and aid algorithm development, we estimated the variability between events in our dataset. We quantified the mean timing of cell disruption (Fig. 3F), which provides insights into the typical occurrence of this critical phase. Additionally, we assessed the average displacement of cells (Fig. 3G) and the overall duration of the disruption process (Fig. 3H), offering essential information to guide algorithmic solutions for cell death classification and detection. Hence, by considering these variabilities, researchers and practitioners can better design and fine-tune their algorithms for accurate classification and detection of apoptotic-like events.

Operator annotations

To assess the inter-operator variability during the annotation process, we introduced a consensus ratio, which quantifies the proportion of annotations that achieved agreement between at least two of the three operators involved. The mean consensus ratio for annotating apoptotic-like events was 0.6, indicating a moderate level of agreement among the operators and highlighting the intrinsic challenge of visually detecting apoptotic-like events based on morphological cues (Fig. 3I). However, when annotating the specific phases (Fig. 3J), the mean consensus ratio significantly increased to 0.9, reflecting the agreement between the operators in identifying the distinct phases.

Usage Notes

The proposed dataset represents a valuable resource for researchers seeking to characterize cell death. Cell biologists can utilize the presented dataset as a reference for the death of leukocytes in vivo, enabling the generation of pilot and comparative studies. In addition, they can gain valuable insights into different properties of the process, such as morphology, duration, and the recruiting of neighboring cells. A protocol for opening, downloading, and analyzing the dataset is provided in Supplementary Information and the code repository.

The heterogeneity of the videos included in this dataset (cell type, staining, organ, experimental conditions) should be considered in comparative studies. We recommend the users to always verify the physical parameters of the videos being compared, including volume and temporal sampling rate, as these can introduce a non-linear bias on metrics to quantify cellular morphodynamics such as cell speed. Moreover, the annotation of events relied exclusively on morphological hallmarks, without the usage of reporters for specific cell death modalities. While this is a major limitation of the study, as we cannot pinpoint the specific cell death modality, the dataset provides valuable information to develop computer vision-based methods for cell death quantification.

With access to the raw data and annotations, computer scientists can address the specific tasks of cell death classification, detection, and segmentation. To create a dataset for supervised-learning tasks, users can access the raw microscopy videos and their corresponding annotations provided in Excel format. By utilizing the annotations, it is possible to locate and track the centroids of individual apoptotic-like events over time (x, y, z, t). Using this information, users can define a suitable region around each cell to generate single-cell crops and the respective labels (0 for healthy, 1 for apoptotic). It is important to remark that identifying cell death events in this dataset relied on the analysis of video sequences. Therefore, we recommend that users analyze the entire temporal sequence.

The creation of personalized training datasets offers the possibility to train custom deep learning architectures for the classification of images depicting apoptotic-likw cells based on their morphology (Fig. 4A left). To accomplish this task, a wide array of deep learning architectures is available, such as convolutional networks (CNNs)34, capsule networks35, and vision transformers36,37, among others. Accordingly, recent studies demonstrated the successful classification of a variety of cell death types in static images38,39. Additionally, the presented dataset is best suited for the temporal classification of apoptotic-like sequences (Fig. 4A right), as the latter representation is expected to encapsulate more information than single-frame images. To this end, deep learning architectures tailored for activity recognition40, such as 3DCNNs41, recurrent networks42, or optical flow-based methods43, can be employed. Recurrent architectures have demonstrated effectiveness in classifying sequences acquired in vitro1, showcasing their potential for this task. To achieve whole sequence classification, users can extract consecutive frames as cropped time-lapses, thus preserving the temporal dependencies of the events. With this information, users can train models capable of effectively analyzing the temporal dependencies and patterns exhibited by cells during cell death.

Fig. 4
Fig. 4
Full size image

Dataset usage for deep learning applications. (A) Schematic representation of the classification task applied to single image crops depicting dying cells (left) and to apoptotic-like sequences (right). Single image crops can be classified with the aid of convolutional networks, mapping an image to a corresponding binary value labelling healthy (0) or apoptotic-like (1) cells. Similarly, sequences images can be classified with activity recognition architectures. (B) Schematic depiction of a detection routine applied to full microscopy images containing healthy and dying cells. Input images are processed with a detector that predicts the spatial coordinates of apoptotic-like cells (red bounding boxes). (C) Schematic representation of the image segmentation task applied to full microscopy images depicting dying cells. Input microscopy images are processed with an auto encoder-decoder architecture which learns to map the input into a binary mask that highlights target cells only.

Users can tackle a task of increased complexity with respect to the classification problem by detecting dying cells within uncropped fields-of-view containing multiple cells (Fig. 4B). Accordingly, detecting apoptotic-like events amid other cells poses more significant challenges due to potential cell overlaps and background clutters44. Moreover, the temporal aspect of live cell imaging introduces new challenges, including morphological changes and sudden displacements. As a result, the algorithmic detection of cell death events in live cell imaging remains unexplored. To address this aspect, researchers can use the presented dataset to explore novel approaches for detecting apoptotic-like events in time-lapse microscopy.

Finally, the presented dataset is also suitable for cell segmentation (Fig. 4C) through the utilization of encoder-decoder architectures45. These DL networks have proven effective in several bio-imaging segmentation tasks, thus holding promising applications for segmenting cell death based on their morphology. In this regard, our dataset enables 3D and 2D cell segmentation by providing the 3D binary masks of the events over time. Consequently, researchers can apply 2D or 3D segmentation using each frame of the movies as an independent training sample.