Fig. 4: Two experimental results of long-horizon human-robot interactions.

a The metaAgent recognizes the accidental fall of a user, and initializes communication with the user to help them out of danger based on their language instructions. (upper left) Excerpt of the protocol of communications between the human user and the metaAgent. Herein, the outputs of different domain experts in natural language are seen. (upper right) A sequence of reconstructed 3D human skeletons, based on microwave reconstruction in the sheltered region for privacy preservation, and based on an optical camera otherwise. (bottom) Some photographic snapshots of the robot. More details have been recorded in Video S1. b The metaAgent proactively helps the subject out of danger by checking the subject’s vital signs and contacting a medical expert when it determines that the subject has an emergency and does not communicate with the metaAgent. (left) Excerpt of the protocol of communications between the human and the metaAgent. (right) Some results of the human-robot interactions: (i) Recognition of the abnormal behavior based on a 30 s-long sample of Bob’s breathing rates, (ii) communicate with the subject, but fails to get the response, (iii) proactively monitor the breathing and heartbeat rates of subject, (iv) call the emergence number or human intervene for helping the subject out of the danger. More details have been recorded in Video S2