Fig. 6: Bridging the reality gap in RL by pretraining with pySTED.

For real microscopy experiments, the agent was trained for over 12M steps in the simulation. It was then deployed on a real STED microscope to image diverse proteins in dissociated neuronal cultures and cultivated Vero cells. a, Top: simulated images of F-actin in fixed neurons were used during training. Deploying the RL agent to acquire an image of this in-distribution structure in a real experiment allows to resolve the periodic lattice of F-actin tagged with phalloidin-STAR635. Bottom: structural parameters extracted from the images (Methods; the dashed vertical line represents the median of the distribution) were compared with the literature value (solid vertical line), showing that the agent adjusted the imaging parameters to resolve the 190 nm periodicity of F-actin^56,57. b, Top: the trained agent is tested on the protein TOM20, unseen during training (out of distribution). The nano-organization of TOM20 is revealed in all the acquired images. Bottom: the measured average cluster diameter of TOM20 concords with the averaged reported values from ref. ⁶⁰. c, Top: live-cell imaging of SiR-actin shows the model’s adaptability to different experimental conditions (out of distribution). Bottom: the periodicity of the F-actin lattice is measured and compared with the literature. In a–c, STED images are normalized to their respective confocal image (CONF1). The second confocal image (CONF2) uses the same colour scale as CONF1 to reveal the photobleaching effects. d,e, Images acquired by the RL agent on a different microscope of tubulin (d; STAR RED) and actin (e; STAR GREEN) in fixed Vero cells. Image sequences from top left to bottom right show confocal images before (CONF1) and after (CONF2) photobleaching, with CONF2 normalized to the CONF1 image. The STED images are normalized to the 99th percentile of the intensity of the CONF1 image. Images are 5.12 μm × 5.12 μm. The evolution of the parameter selection (left) and imaging optimization objectives (right) showed that optimal parameters and optimized objectives differed for STAR RED (d) and STAR GREEN (e).