Table 1 Summary of the 60 error metrics, their corresponding benchmarks, and the fractions of MLIP models that meet the benchmarks
Category | Configuration/dataset | Error metrics | Benchmark | The fraction of MLIP models achieving the benchmark (%) |
|---|---|---|---|---|
Defect formation energy, EfDefect | Vacancy | \(\delta {E}_{{\rm{f}}}^{{\rm{vacancy}}}\) | 0.1 eV (handpicked) | 3.7 |
Split-<110> interstitial | \(\delta {E}_{{\rm{f}}}^{{\rm{split}}- < 110 > }\) | 0.1 eV (handpicked) | 5.0 | |
Tetrahedral interstitial | \(\delta {E}_{{\rm{f}}}^{{\rm{tetrahedral}}}\) | 0.1 eV (handpicked) | 2.8 | |
Hexagonal interstitial | \(\delta {E}_{{\rm{f}}}^{{\rm{hexagonal}}}\) | 0.1 eV (handpicked) | 4.4 | |
Elastic constants | Supercell with 64 atoms | \(\delta {C}_{11}^{{\rm{supercell}}}\) | 1.4 GPa (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}2}\)) | 2.0 |
\(\delta {C}_{12}^{{\rm{supercell}}}\) | 4.0 GPa (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}2}\)) | 8.5 | ||
\(\delta {C}_{44}^{{\rm{supercell}}}\) | 3.4 GPa (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}2}\)) | 8.3 | ||
Bulk modulus, \(\delta {K}^{{\rm{supercell}}}\) | 3.2 GPa (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}2}\)) | 7.3 | ||
Vacancy | \(\delta {C}_{11}^{{\rm{vacancy}}}\) | 492 GPa (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}2}\)) | 77.9 | |
\(\delta {C}_{12}^{{\rm{vacancy}}}\) | 251 GPa (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}2}\)) | 76.0 | ||
\(\delta {C}_{44}^{{\rm{vacancy}}}\) | 84 GPa (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}2}\)) | 75.1 | ||
\(\delta {K}^{{\rm{vacancy}}}\) | 3.8 GPa (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}2}\)) | 9.3 | ||
Split-<110> interstitial | \(\delta {C}_{11}^{{\rm{split}}- < 110 > }\) | 3.2 GPa (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}2}\)) | 7.0 | |
\(\delta {C}_{12}^{{\rm{split}}- < 110 > }\) | 4.9 GPa (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}2}\)) | 12.7 | ||
\(\delta {C}_{44}^{{\rm{split}}- < 110 > }\) | 0.7 GPa (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}2}\)) | 3.2 | ||
\(\delta {K}^{{\rm{split}}- < 110 > }\) | 2.1 GPa (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}2}\)) | 7.0 | ||
Tetrahedral interstitial | \(\delta {C}_{11}^{{\rm{tetrahedral}}}\) | 21 GPa (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}2}\)) | 27.1 | |
\(\delta {C}_{12}^{{\rm{tetrahedral}}}\) | 20 GPa (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}2}\)) | 40.7 | ||
\(\delta {C}_{44}^{{\rm{tetrahedral}}}\) | 26 GPa (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}2}\)) | 21.9 | ||
\(\delta {K}^{{\rm{tetrahedral}}}\) | 5.8 GPa (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}2}\)) | 16.3 | ||
Hexagonal interstitial | \(\delta {C}_{11}^{{\rm{hexagonal}}}\) | 0.5 GPa (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}2}\)) | 1.0 | |
\(\delta {C}_{12}^{{\rm{hexagonal}}}\) | 4.4 GPa (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}2}\)) | 12.0 | ||
\(\delta {C}_{44}^{{\rm{hexagonal}}}\) | 3.0 GPa (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}2}\)) | 9.1 | ||
\(\delta {K}^{{\rm{hexagonal}}}\) | 2.8 GPa (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}2}\)) | 8.2 | ||
Lattice parameter | Unit cell with 8 atoms | \(\delta {a}^{{\rm{unit\; cell}}}\) | 0.16 Å (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 98.2 |
Supercell with 64 atoms | \(\delta {a}^{{\rm{supercell}}}\) | 0.03 Å (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 61.0 | |
Vacancy | \(\delta {a}^{{\rm{vacancy}}}\) | 0.05 Å (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 67.6 | |
Split-<110> interstitial | \(\delta {a}^{{\rm{split}}- < 110 > }\) | 0.04 Å (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 46.7 | |
Tetrahedral interstitial | \(\delta {a}^{{\rm{tetrahedral}}}\) | 0.02 Å (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 32.0 | |
Hexagonal interstitial | \(\delta {a}^{{\rm{hexagonal}}}\) | 0.03 Å (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 38.9 | |
Energy rankings of multiple defects | Ranking dataset \({\mathcal{D}}\)1: 2-vacancies in supercells with 214 atoms | Ranking error rates, \({P}_{{\rm{ranking\; error}}}^{{\mathcal{D}}1}\) | 7.6% (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 0.1 |
Mean \(\Delta E\)DFT of ranking errors, Mean \({\Delta E}_{{\rm{DFT}}}^{{\mathcal{D}}1}\) | 16 meV (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 0.0 | ||
Max \(\Delta E\)DFT of ranking errors, Maximum \({\Delta E}_{{\rm{DFT}}}^{{\mathcal{D}}1}\) | 54 meV (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 0.1 | ||
Ranking dataset \({\mathcal{D}}\)2: 3-vacancies in supercells with 141 atoms | \({P}_{{\rm{ranking\; error}}}^{{\mathcal{D}}2}\) | 7.4% (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 0.2 | |
Mean \({\Delta E}_{{\rm{DFT}}}^{{\mathcal{D}}2}\) | 26 meV (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 0.0 | ||
Maximum \({\Delta E}_{{\rm{DFT}}}^{{\mathcal{D}}2}\) | 67 meV (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 0.0 | ||
Ranking dataset \({\mathcal{D}}\)3: 3-vacancies in supercells with 93 atoms | \({P}_{{\rm{ranking\; error}}}^{{\mathcal{D}}3}\) | 9.9% (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 0.0 | |
Mean \({\Delta E}_{{\rm{DFT}}}^{{\mathcal{D}}3}\) | 78 meV (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 0.0 | ||
Maximum \({\Delta E}_{{\rm{DFT}}}^{{\mathcal{D}}3}\) | 263 meV (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 0.0 | ||
Ranking dataset \({\mathcal{D}}\)4:3-vacancies in supercells with 61 atoms | \({P}_{{\rm{ranking\; error}}}^{{\mathcal{D}}4}\) | 5.5% (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 0.1 | |
Mean \({\Delta E}_{{\rm{DFT}}}^{{\mathcal{D}}4}\) | 69 meV (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 0.0 | ||
Maximum \({\Delta E}_{{\rm{DFT}}}^{{\mathcal{D}}4}\) | 240 meV (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 0.0 | ||
Rare events (REs) | \({{\mathcal{D}}}^{{\rm{RE}}-{\rm{I}}}\) Testing data of 100 interstitial configurations containing REs in ref. 8 | Directional error of forces, \(\Delta {NAC}({{\rm{\delta }}}_{{\rm{\theta }}},{{\mathcal{D}}}^{{\rm{RE}}-{\rm{I}}})\) | 0.22 (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 4.8 |
Magnitude error of forces, \(\Delta {NAC}({|\delta }_{{\rm{F}}}|,{{\mathcal{D}}}^{{\rm{RE}}-{\rm{I}}})\) | 0.20 (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 3.8 | ||
\({{\mathcal{D}}}^{{\rm{RE}}-{\rm{V}}}\) Testing data of 100 vacancy configurations containing REs in ref. 8 | \(\Delta {NAC}({{\rm{\delta }}}_{{\rm{\theta }}},{{\mathcal{D}}}^{{\rm{RE}}-{\rm{V}}})\) | 0.15 (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 15.7 | |
\(\Delta {NAC}({\rm{|}}{\delta }_{{\rm{F}}}|,{{\mathcal{D}}}^{{\rm{RE}}-{\rm{V}}})\) | 0.13 (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 12.7 | ||
\({{\mathcal{D}}}^{{\rm{RE}}-{\rm{I}}}\) | \({\sigma }_{{\rm{E}}}^{{\rm{RE}}-{\rm{I}}}\) | 15 meV atom−1 (handpicked) | 85.8 | |
\({\sigma }_{{\rm{F}}}^{{\rm{RE}}-{\rm{I}}}\) | 0.25 eV Å−1 (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 3.7 | ||
\({{\mathcal{D}}}^{{\rm{RE}}-{\rm{V}}}\) | \({\sigma }_{{\rm{E}}}^{{\rm{RE}}-{\rm{V}}}\) | 15 meV atom−1 (handpicked) | 85.7 | |
\({\sigma }_{{\rm{F}}}^{{\rm{RE}}-{\rm{V}}}\) | 0.17 eV Å−1 (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 11.7 | ||
Energy and force errors, E & F RMSEs | \({{\mathcal{D}}}^{{\rm{enhanced}}-{\rm{I}}}\) Training data from ref. 8 having interstitial REs | \({\sigma }_{{\rm{E}}}^{{\rm{enhanced}}-{\rm{I}}}\) | 15 meV atom−1 (handpicked) | 58.8 |
\({\sigma }_{{\rm{F}}}^{{\rm{enhanced}}-{\rm{I}}}\) | 0.12 eV Å−1 (handpicked) | 15.5 | ||
\({{\mathcal{D}}}^{{\rm{enhanced}}-{\rm{V}}}\) Training data from ref. 8 having vacancy REs | \({\sigma }_{{\rm{E}}}^{{\rm{enhanced}}-{\rm{V}}}\) | 15 meV atom−1 (handpicked) | 60.8 | |
\({\sigma }_{{\rm{F}}}^{{\rm{enhanced}}-{\rm{V}}}\) | 0.12 eV Å−1 (handpicked) | 17.1 | ||
Thermal properties | Bulk: unit cell with 8 atoms | Free energy, \(\delta {E}_{{\rm{free}}}^{{\rm{bulk}}}(T)\) | 0.25 (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 18.3 |
Entropy, \(\delta {S}^{{\rm{bulk}}}(T)\) | 0.33 (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 14.5 | ||
Heat capacity, \(\delta {c}^{{\rm{bulk}}}(T)\) | 0.12 (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 16.6 | ||
Vacancy | \(\delta {E}_{{\rm{free}}}^{{\rm{vacancy}}}(T)\) | 39 (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 60.7 | |
\(\delta {S}^{{\rm{vacancy}}}(T)\) | 92 (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 68.2 | ||
\(\delta {c}^{{\rm{vacancy}}}(T)\) | 41 (\({\Delta }_{{\rm{DFT}}}^{{\rm{K}}1{\rm{to\; K}}4}\)) | 72.2 |
