Figure 4
From: Deploying and scaling distributed parallel deep neural networks on the Tianhe-3 prototype system

Pytorch distributed Allreduce method.
From: Deploying and scaling distributed parallel deep neural networks on the Tianhe-3 prototype system

Pytorch distributed Allreduce method.