Fig. 1: An autonomous ultrasound-driven microrobot.
From: Model-based reinforcement learning for ultrasound-driven autonomous microrobots
a, Left, schematic of the experimental set-up, which has an artificial vascular channel with eight PZTs in an octagonal configuration. Right, the schematic illustrates the behaviour of the microbot under ultrasound activation and details methods for its manipulation. b, Guidelines for manipulating microrobots. c, High-level formulation of the RL problem. The environment is our artificial channel. The RL agent is the microrobot with St actuated by the ultrasound frequency and amplitude to achieve a reward Rt after optimal actions At. d, A world model encoder–decoder structure built to simulate and imagine future states in the environment. e, A simulated game environment designed to pretrain the microrobots, thereby reducing the convergence time during experimental training. f, A recurrent ‘dream’ in the latent space. The microrobot envisions several potential paths towards a target in a dreamed environment, allowing it to dream and train on various possible scenarios simultaneously. g, The microrobot agent executes the optimal action, successfully reaches target 1 and proceeds towards target 2 using a newly imagined path. At, action, Rt, reward; RNN, recurrent neural network; St, state; ƒ𝑛, A𝑛, frequency and amplitude, respectively, of the 𝑛th US wave.
