Table 7 COMET scores on xx\({\Rightarrow }\)en.
From: LingualX64: a multilingual benchmark for evaluating symmetry and asymmetry in LLM translation
falcon7b | Mistral-7b | Llama-2-7b-hf | bloom-7b1 | bloomz-7b1-mt | Meta-Llama-3-8B | mpt-7b | vicuna-7b | gemma-7b | |
|---|---|---|---|---|---|---|---|---|---|
ar | 66.12 | 88.32 | 75.13 | 80.33 | 90.77 | 92.33 | 69.88 | 81.08 | 92.9 |
da | 89.7 | 93.77 | 91.19 | 76.11 | 71.95 | 94.14 | 88.75 | 92.59 | 93.98 |
tl | 84.56 | 88.86 | 82.79 | 60.4 | 57.93 | 90.63 | 76.85 | 84.63 | 89.34 |
fr | 93.11 | 93.11 | 91.61 | 92.18 | 91.13 | 93.75 | 90.76 | 93.02 | 89.86 |
de | 92.06 | 93.11 | 90.56 | 85.94 | 86.48 | 93.56 | 90.74 | 92.84 | 93.2 |
el | 70.98 | 87.12 | 79.66 | 59.93 | 62.54 | 93.58 | 79.22 | 78.92 | 92.86 |
hi | 54.96 | 87.7 | 75.96 | 75.4 | 89.33 | 93.9 | 70.63 | 79.31 | 89.24 |
is | 50.97 | 76.17 | 64.36 | 56.09 | 49.43 | 87.28 | 68.15 | 63.07 | 85.17 |
it | 93.32 | 93.59 | 91.93 | 90.45 | 90.51 | 94.02 | 91.58 | 93.52 | 92.1 |
ja | 87.63 | 90.13 | 84.1 | 80.31 | 79.74 | 92.64 | 82.98 | 90.97 | 92.25 |
ko | 61.77 | 88.94 | 79.68 | 76.38 | 65.35 | 93.08 | 81.83 | 89.06 | 91.34 |
nl | 91.88 | 93.38 | 90.77 | 79.36 | 76.63 | 93.89 | 89.66 | 93.62 | 92.77 |
no | 88.96 | 92.97 | 88.96 | 75.67 | 70.34 | 94.07 | 90.17 | 93.52 | 94.22 |
pt | 93.19 | 93.78 | 90.93 | 91.54 | 92.49 | 93.91 | 91.26 | 93.911 | 92.48 |
ru | 92.49 | 93.02 | 89.5 | 89.03 | 91.28 | 93.72 | 90 | 93.28 | 93.17 |
es | 93.54 | 93.86 | 92.24 | 91.76 | 92.08 | 94.27 | 92.02 | 93.95 | 70.62 |
th | 55.68 | 85.18 | 74 | 68.77 | 54.03 | 92.02 | 69.97 | 76.2 | 91.53 |
vi | 73.54 | 90.3 | 84.61 | 84.59 | 89.69 | 92.28 | 78.19 | 91 | 83.76 |
zh | 90.04 | 84.04 | 78.56 | 77.17 | 79.03 | 91.13 | 79.58 | 84.71 | 89.97 |
af | 86.2 | 91.37 | 88.91 | 68.1 | 68.05 | 91.7 | 87.39 | 87.81 | 89.78 |
bn | 48.76 | 85.18 | 70.9 | 82.25 | 89.35 | 90.3 | 60.18 | 56.63 | 89.19 |
bs | 80.92 | 92.11 | 90.44 | 66.88 | 65.35 | 88.94 | 85.75 | 89.69 | 90.07 |
bg | 78.18 | 93.19 | 91.9 | 71.13 | 69.62 | 89.19 | 88.86 | 91.77 | 93.19 |
ca | 86.77 | 92.37 | 91.58 | 92.21 | 92.64 | 91.1 | 89.37 | 90.76 | 91.21 |
hr | 81.39 | 92.75 | 91.35 | 66.47 | 62.96 | 90.3 | 86.22 | 91.49 | 92.8 |
cs | 84.87 | 92.61 | 91.07 | 65.51 | 63.53 | 89.55 | 88.18 | 91.26 | 92.59 |
et | 55.08 | 82.4 | 67.81 | 55.76 | 62.27 | 90.1 | 71.45 | 58.38 | 93.22 |
fj | 37.28 | 42.09 | 40.85 | 41.3 | 37.52 | 44.59 | 41.7 | 41.23 | 45.54 |
fi | 62.74 | 91.3 | 92.2 | 55.6 | 59.46 | 91.89 | 81.15 | 82.61 | 94.22 |
gu | 45.08 | 65.55 | 53.68 | 74.02 | 84.98 | 89.48 | 59.96 | 48.07 | 90.21 |
ht | 54.22 | 67.34 | 54.68 | 52.64 | 43.84 | 73.8 | 52.61 | 51.09 | 72.35 |
he | 49.7 | 83.4 | 73.89 | 49.83 | 53.96 | 89.13 | 59.02 | 63.19 | 92.25 |
hmn | 47.76 | 48.5 | 45.95 | 46.32 | 36.12 | 48.8 | 48.13 | 46.08 | 56.76 |
hu | 56.7 | 92.18 | 90.56 | 54.38 | 57.41 | 89.64 | 80.1 | 90.06 | 92.55 |
id | 89.26 | 92.03 | 91.87 | 91.45 | 91.01 | 90.67 | 90.18 | 86.56 | 91.9 |
ga | 49.3 | 65.29 | 61.83 | 46 | 51.72 | 83.57 | 52.48 | 58.87 | 75.74 |
kn | 41.25 | 72.68 | 50.29 | 78.57 | 87.7 | 89.44 | 52.02 | 50.38 | 89.5 |
kk | 49.03 | 63.69 | 54.52 | 52.15 | 54.57 | 83.63 | 52.2 | 54.06 | 85.82 |
sw | 53.57 | 71.21 | 58.04 | 76.54 | 82.72 | 84.75 | 51.81 | 54.99 | 87.5 |
lv | 57.94 | 77.47 | 65.32 | 56.62 | 60.64 | 88.16 | 69.65 | 61.74 | 91.99 |
lt | 56.83 | 79.43 | 64.31 | 57.95 | 63.26 | 88.03 | 69.11 | 58.86 | 91.48 |
mg | 45.79 | 47.67 | 45.87 | 48.85 | 49.35 | 58.06 | 48.21 | 49.4 | 64.32 |
ms | 87.61 | 91.09 | 89.46 | 88.8 | 89.16 | 88.82 | 88.16 | 83.53 | 91.36 |
ml | 43.14 | 56.89 | 54.19 | 85.34 | 89.44 | 89.69 | 47.99 | 48.31 | 89.57 |
mt | 58.21 | 68.22 | 62.01 | 50.86 | 44.05 | 80.55 | 60.9 | 59.92 | 84.67 |
mi | 46.05 | 58.58 | 47.43 | 40.29 | 34.32 | 60.85 | 41.04 | 45.89 | 58.62 |
mr | 41.2 | 76.17 | 70.23 | 73.55 | 64.65 | 88.4 | 57.67 | 58.32 | 90.02 |
ps | 46.95 | 60.93 | 55.83 | 53.53 | 53.38 | 81.71 | 52.35 | 55.1 | 77.11 |
fa | 56.02 | 86.94 | 82 | 64.5 | 64.2 | 89.61 | 72.11 | 73.39 | 91.99 |
pl | 88.73 | 92.03 | 90.59 | 68.22 | 65.29 | 88.77 | 88.18 | 90.67 | 91.63 |
pa | 42.43 | 59.8 | 51.95 | 82.99 | 90.21 | 89.75 | 52.91 | 46.14 | 83.73 |
otq | 62.75 | 54.61 | 59.73 | 59.46 | 55.25 | 60.31 | 56.6 | 55.84 | 52.84 |
ro | 88.15 | 93.45 | 92.56 | 76.06 | 74.1 | 90.97 | 90.01 | 92.32 | 93.53 |
sm | 39.74 | 45.63 | 40.6 | 41.1 | 33.12 | 53.45 | 39.31 | 41.86 | 52.03 |
sk | 78.74 | 91.24 | 88.1 | 62.55 | 63.73 | 88.21 | 85.94 | 87.16 | 92.49 |
sl | 75.52 | 91.92 | 90.37 | 62 | 60.51 | 89.31 | 81.54 | 89.73 | 92.76 |
sv | 90.31 | 93.49 | 92.81 | 73.31 | 71.58 | 91.73 | 92.18 | 92.85 | 93.53 |
ta | 39.39 | 68.05 | 55.43 | 81.57 | 85.88 | 88.15 | 44.48 | 48.79 | 89.19 |
te | 44.53 | 63.21 | 50.11 | 78.6 | 90.09 | 87.82 | 49.47 | 47.79 | 89.1 |
tr | 68.56 | 90.4 | 84.85 | 58.88 | 57.41 | 91.06 | 84.08 | 71.97 | 93.2 |
uk | 72.3 | 91.33 | 90.09 | 69.87 | 69.22 | 88.46 | 84.49 | 89.99 | 91.1 |
ur | 50.67 | 81.15 | 74.61 | 77.2 | 90.11 | 89.45 | 65.49 | 61.24 | 90.47 |
cy | 50.81 | 70.25 | 62.52 | 55.56 | 61.75 | 87.57 | 50.66 | 53.8 | 85.07 |
