Extended Data Fig. 3: Guiding the tip of the robotic sampling arm by a distance-dependent light pattern and a feedback loop between image acquisition, live image analysis, and robotic arm control.

To sample cells from the bacterial swarm, a sampling tip mounted on the robotic sampling arm is moved to the microscope as described in the Methods section and brought into position such that the tip is within the field of view of the microscope, approximately 1 mm above the swarming plate. At this point, a light pattern caused by the refraction of light through the PDMS drop at the end of the sampling tip can be observed through the camera. The shape of this pattern changes depending on the distance between the tip and the swarm surface and can therefore be used to guide the tip towards a careful pickup of cells. Images in panels a-d show different light patterns as viewed from the camera. Insets in each panel depict a binarization of the light patterns, obtained by thresholding, which was used for steering the robotic arm. The distances between the tip and agar surface are: (a), 100 µm, (b), 50 µm, (c), 1 µm, (d), 0 µm. The robotic arm is moved stepwise towards the agar surface, correcting the step size after each movement. Step sizes are chosen to be 50 µm, 30 µm, 20 µm, 10 µm, 2 µm or 1 µm, depending on the correlation of the current binarized image with two reference images representing the distances 100 µm and 1 µm. Step sizes are chosen several times smaller than the estimated distance to the surface to ensure that the tip will under no circumstances penetrate the agar further than 1 µm, which could lead to disruption of the agar surface and a disruption of the further swarm development. During the sampling process, the current image obtained from the microscope camera is displayed in a graphical user interface that allows manual intervention such as stopping the movement of the sampling tip as an additional layer of security. Upon making contact with the surface, the z-position of the point of contact is saved and used as a reference for future z-positioning of the sampling tip. The tip is then moved upwards and out of the humidity chamber before ejecting the sampling tip into an Eppendorf tube.