Fig. 1: Schematic of analyzing DNN layer weight matrices W. | Nature Communications

Fig. 1: Schematic of analyzing DNN layer weight matrices W.

From: Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data

Fig. 1

Given an individual layer weight matrix W, from either a fully connected layer or a convolutional layer, perform a Singular Value Decomposition (SVD) to obtain W = UΣVT, and examine the histogram of eigenvalues of WTW. Norm-based metrics and PL-based metrics (that depend on fitting the histogram of eigenvalues to a truncated PL) can be used to compare models. For example, one can analyze one layer of a pre-trained model, compare multiple layers of a pre-trained model, make comparisons across model architectures, monitor neural network properties during training, etc.

Back to article page