Table 6 BLEU scores on xx\({\Rightarrow }\)zh.
From: LingualX64: a multilingual benchmark for evaluating symmetry and asymmetry in LLM translation
falcon7b | Mistral-7b | Llama-2-7b-hf | bloom-7b1 | bloomz-7b1-mt | Meta-Llama-3-8B | mpt-7b | vicuna-7b | gemma-7b | |
|---|---|---|---|---|---|---|---|---|---|
ar | 0.65 | 19.09 | 5.33 | 10.15 | 19.33 | 26.55 | 6.38 | 18.66 | 12.63 |
da | 4.96 | 20.77 | 11.34 | 6.95 | 5.96 | 26.73 | 16.32 | 28.9 | 17.9 |
tl | 1.56 | 15.27 | 4.23 | 2.53 | 1.94 | 14.71 | 8.72 | 17.45 | 7.24 |
fr | 12.12 | 23.46 | 17.6 | 24.93 | 22.14 | 29.55 | 17.05 | 30.71 | 20.32 |
de | 9.72 | 22.81 | 14 | 16.47 | 13.68 | 27.43 | 17.4 | 29.38 | 18.9 |
el | 0.47 | 13.86 | 3.67 | 1.36 | 2.82 | 25.58 | 8.86 | 13.86 | 13.09 |
hi | 0.05 | 14.43 | 9.06 | 13.38 | 20.56 | 26.95 | 5.48 | 15.48 | 6.68 |
is | 0.34 | 6.95 | 2 | 0.74 | 0.68 | 20.53 | 4.21 | 3.83 | 6.4 |
it | 13.48 | 22.18 | 14.27 | 22.91 | 19.14 | 27.7 | 15.59 | 29.9 | 18.8 |
ja | 4.63 | 10.72 | 5.27 | 2.37 | 12.05 | 22.34 | 9.39 | 24.08 | 14.96 |
ko | 0.14 | 16.89 | 1.23 | 3.66 | 5.28 | 22.09 | 12.5 | 26.25 | 11.9 |
nl | 4.61 | 24.81 | 13.6 | 11.54 | 7.95 | 28.87 | 19.33 | 30.62 | 20.01 |
no | 3.91 | 20.63 | 12.13 | 7.57 | 5.79 | 26.81 | 16.92 | 29.55 | 16.38 |
pt | 8.76 | 20.49 | 16.68 | 24.47 | 22.7 | 27.01 | 18.88 | 31.58 | 16.11 |
ru | 8.97 | 16.75 | 13.54 | 23.33 | 21.32 | 26.68 | 19.57 | 30.39 | 19.73 |
es | 9.52 | 25.47 | 16.59 | 24.9 | 21.49 | 29.24 | 16.75 | 30.76 | 9.32 |
th | 0.04 | 11.76 | 3.64 | 0.08 | 0.9 | 22.86 | 3.06 | 9.48 | 9.65 |
vi | 0.47 | 17.66 | 2.31 | 15.39 | 21.16 | 26.57 | 10.77 | 28.38 | 15.08 |
en | 26.92 | 34.52 | 21.83 | 29.6 | 21.55 | 34.91 | 18.12 | 37.27 | 26.78 |
af | 0 | 2.23 | 0.96 | 0.9 | 3.77 | 4.48 | 0 | 0.02 | 1.24 |
bn | 0 | 4.83 | 0.25 | 0.76 | 21.24 | 3.84 | 0.01 | 0.01 | 0.15 |
bs | 0 | 5.66 | 0.72 | 0.32 | 3.27 | 8.48 | 0 | 0.02 | 0.19 |
bg | 0 | 14.38 | 0.25 | 0.24 | 7.54 | 7.61 | 0.05 | 0.03 | 0.03 |
ca | 0.2 | 2.29 | 4.36 | 1.68 | 24.56 | 7.93 | 0.24 | 0.14 | 0.62 |
hr | 0 | 7.57 | 1.11 | 0.21 | 2.88 | 9.29 | 0 | 0.13 | 0.14 |
cs | 0.39 | 9.64 | 1.16 | 0.62 | 3.75 | 6.65 | 0.05 | 0.31 | 0.28 |
et | 0.13 | 5.58 | 0.43 | 0.07 | 0.74 | 6.08 | 0.11 | 0 | 0.43 |
fj | 0 | 0.19 | 0.11 | 0.01 | 0.4 | 0.48 | 0 | 0.02 | 0.01 |
fi | 0 | 9 | 2.25 | 0.06 | 0.71 | 6.69 | 0 | 0.01 | 0.2 |
gu | 0.49 | 1.45 | 0.32 | 0.13 | 15.43 | 1.58 | 0.23 | 0.07 | 0.03 |
ht | 0 | 1.64 | 0.11 | 0.13 | 0.55 | 6 | 0 | 0 | 3.56 |
he | 0 | 9.85 | 0.19 | 0.03 | 2.27 | 3.03 | 0 | 0 | 0.16 |
hmn | 0 | 0.24 | 0.03 | 0.02 | 0.25 | 0.5 | 0 | 0.01 | 0.65 |
hu | 0 | 2.29 | 1.11 | 0.01 | 0.77 | 3.53 | 0 | 0 | 0 |
id | 0.62 | 23.66 | 4.98 | 1.03 | 26.34 | 9.38 | 0.32 | 0.39 | 4.06 |
ga | 0 | 0.01 | 0.12 | 0.01 | 0.23 | 3.15 | 0 | 0.02 | 0.06 |
kn | 0 | 1.14 | 0 | 0.04 | 19.13 | 0.96 | 0 | 0 | 0.08 |
kk | 0 | 0.81 | 0.01 | 0.04 | 0.39 | 3 | 0 | 0 | 0.71 |
sw | 0 | 4.59 | 0.33 | 0.32 | 14.18 | 5.99 | 0 | 0.01 | 2.44 |
lv | 0.04 | 4.01 | 0.19 | 0.16 | 1 | 6.95 | 0 | 0.13 | 0 |
lt | 0.1 | 3.44 | 0.44 | 0.27 | 1.83 | 7.34 | 0.16 | 0 | 0.08 |
mg | 0 | 0.07 | 0.02 | 0.05 | 0.22 | 0.83 | 0 | 0.02 | 0.56 |
ms | 0.08 | 12.51 | 3.44 | 0.57 | 21.69 | 6.47 | 0.06 | 0.31 | 2.83 |
ml | 0.06 | 1.33 | 0.25 | 0.15 | 14.44 | 1.57 | 0.04 | 0.06 | 0.05 |
mt | 0 | 0.46 | 0.18 | 0.18 | 0.54 | 9.62 | 0 | 0.01 | 2.05 |
mi | 0.05 | 0.54 | 0.55 | 0.08 | 0.45 | 2.3 | 0.11 | 0.03 | 0.04 |
mr | 0 | 3.97 | 0.09 | 0.01 | 7.88 | 3.51 | 0 | ś0 | 0.01 |
ps | 0 | 0.21 | 0.01 | 0 | 0.5 | 4.86 | 0 | 0 | 0.49 |
fa | 0.07 | 5.5 | 0.85 | 0.07 | 4.03 | 7.96 | 0.51 | 0 | 0.28 |
pl | 0 | 6.66 | 1.58 | 0.37 | 4.21 | 4.04 | 0 | 0.24 | 0.19 |
pa | 0 | 0.48 | 0 | 0.07 | 18.25 | 1.98 | 0 | 0 | 0.7 |
otq | 0 | 0.01 | 0.01 | 0.63 | 1.76 | 0.62 | 0 | 0 | 0 |
ro | 0 | 8.89 | 2.47 | 2.63 | 9.38 | 5.73 | 0 | 0.01 | 0.08 |
sm | 0 | 0.1 | 0.02 | 0 | 0.1 | 0.79 | 0 | 0 | 0.02 |
sk | 0 | 9.5 | 0.51 | 0.27 | 2.13 | 7.26 | 0 | 0.02 | 0.16 |
sl | 0 | 5.21 | 1.8 | 0.41 | 1.37 | 4.2 | 0 | 0.02 | 0.08 |
sv | 0 | 9.54 | 2 | 0.41 | 5.61 | 4.22 | 0 | 0.02 | 0.53 |
ta | 0 | 2.76 | 0.02 | 0.03 | 17.66 | 2.39 | 0 | 0 | 0.18 |
te | 0 | 0.29 | 0.01 | 0 | 17.2 | 1.08 | 0 | 0 | 0.05 |
tr | 0 | 4.78 | 0.58 | 0.03 | 1.39 | 3.83 | 0.01 | 0.01 | 0.07 |
uk | 0 | 4.41 | 0.08 | 0.65 | 7.83 | 4.05 | 0.05 | 0.08 | 0.07 |
ur | 0 | 0.62 | 0.52 | 0.46 | 15.48 | 7.02 | 0.2 | 0 | 0.55 |
cy | 0 | 0.43 | 0.16 | 0.02 | 0.15 | 4.46 | 0 | 0 | 0.15 |