Table 4 Rating concordance and tercile distribution of domains across frameworks

From: By the numbers: ratings and utilization of behavioral health mobile applications

User Experience (UE)

Credibility & Evidence Base (CEB)

Data Use & Security (DUS)

Fleiss’ Exact Kappa (κ)a—0.13

Fleiss’ Exact Kappa (κ)a—0.228

Fleiss’ Exact Kappa (κ)a—0.147

Fractions of expert reviews by tercile

35/54 (0.65)A

Fractions of expert reviews by tercile

12/46 (0.26)A

Fractions of expert reviews by tercile

14/54 (0.26)A

13/54 (0.24)B

32/46 (0.70)B

27/54 (0.50)B

6/54 (0.11)C

2/46 (0.043)C

13/54 (0.24)C

  1. A: top-tercile rating; B: middle-tercile rating; C: bottom-tercile rating
  2. aFleiss’ Kappa (κ) is an index of agreement for more than two raters that is adjusted for chance. Interpretation: <0 (poor agreement), 0.0–0.20 (slight agreement), 0.21–0.40 (fair agreement), 0.41–0.60 (moderate agreement), 0.61–0.80 (substantial agreement), 0.81–1.0 (almost perfect agreement)
Back to article page