Fig. 2: Illustration of knowledge distillation and knowledge purification.

Knowledge purification requires the selective transfer of the needed knowledge in the process of knowledge distillation to forget the target information instead of simply transferring all information.