Table 2 Comparison of pruning algorithms for unstructured pruning in LLMs

Magnitude

No

No

∣W_ij∣

O(1)

SparseGPT²⁹

Yes

Yes

\(\frac{| {W}_{ij}{| }^{2}}{\,{\rm{diag}}\,{(X{X}^{T}+\lambda I)}_{jj}}\)

\(O({d}_{\,{\rm{hidden}}}^{3})\)

LoRAPrune³⁵

Yes

Yes

\({\left\Vert \frac{\partial {\mathcal{L}}}{\partial {{\bf{B}}}_{i,:}}\odot {{\bf{A}}}_{:,j}+{{\bf{B}}}_{i,:}\odot \frac{\partial {\mathcal{L}}}{\partial {{\bf{A}}}_{:,j}}\right\Vert }_{2}^{2}\cdot {\left({W}_{i,j}+{({\bf{B}}{\bf{A}})}_{i,j}\right)}^{2}\)

\(O({d}_{\,{\rm{hidden}}}^{2})\)

Wanda²⁷

No

Yes

∣W_ij∣ ⋅ ∥X_j∥₂

\(O({d}_{\,{\rm{hidden}}}^{2})\)

DsnoT³⁴

No

Yes

\({\mathbb{E}}[{W}_{ij}\cdot {X}_{j}]\cdot \frac{1}{\,{\rm{Var}}\,({X}_{j})}\)

\(O({d}_{\,{\rm{hidden}}}^{2})\)

Flash-LLM²⁸

Yes

Yes

Load-as-sparse, compute-as-dense

\(O({d}_{\,{\rm{hidden}}}^{2})\)

RIA³⁰

No

Yes

\(\left(\frac{| {W}_{ij}| }{{\sum }_{k}| {W}_{kj}| }+\frac{| {W}_{ij}| }{{\sum }_{k}| {W}_{ik}| }\right)\cdot \parallel {X}_{j}{\parallel }_{2}^{0.5}\)

\(O({d}_{\,{\rm{hidden}}}^{2})\)

ADMM³¹

Yes

Yes

∣W_ij∣ ⋅ ∥X_j∥₂

\(O({d}_{\,{\rm{hidden}}}^{3})\)

OWL³²

No

Yes

Outlier Ratio ∝ ∣W_ij∣ ⋅ ∥X_j∥₂

\(O({d}_{\,{\rm{hidden}}}^{2})\)