Fig. 2: Reconstruction of robot geometry and kinematics from a single image.
From: Controlling diverse robots by inferring Jacobian fields with deep networks
a,c,e,g, Visualization of the reconstructed Jacobian and radiance fields (centre) and comparison of reconstructed and measured geometry (sides) from a single input image. Colours indicate the motion sensitivity of the 3D point to different actuator command channels, meaning that our system successfully learns correspondence between robot 3D parts and command channels without human annotations. We show depth predictions (Pred. Dep.) next to measurements from RGB-D cameras (True Dep.), demonstrating the accuracy of the 3D reconstruction across all systems. Pred. Jac., predicted Jacobian. b,d,f,h, The 3D motions predicted using the Jacobian field. We display the motions predicted using the visuomotor Jacobian field (solid circles) for various motor commands next to reference motions reconstructed from video streams using point tracking (dashed circles). Reconstructed motions are qualitatively accurate across all robotic systems. Although we manually colour-coded the command channels, our framework associates command channels with 3D motions without supervision. N, S, E and W are north, south, east and west, respectively.
