Table 1 False positive rates for A/B testing and SMART designs at a sample size of n = 1000
Method | Main effect size (Cohen’s d) | HTE size (Cohen’s d) | False positive rate for age HTE (95% CI) | False positive rate for race/ethnicity HTE (95% CI) | False positive rate for disease HTE (95% CI) | Net benefit range |
|---|---|---|---|---|---|---|
A/B | 0.1 | 0.05 | 0.096 (0.072, 0.130) | 0.278 (0.188, 0.320) | 0.113 (0.065, 0.158) | −0.038–0.009 |
SMART | 0.1 | 0.05 | 0.104 (0.082, 0.151) | 0.26 (0.205, 0.351) | 0.090 (0.062, 0.126) | 0.316–0.522 |
A/B | 0.1 | 0.1 | 0.120 (0.075, 0.186) | 0.229 (0.177, 0.291) | 0.118 (0.052, 0.183) | −0.020–0.039 |
SMART | 0.1 | 0.10 | 0.100 (0.062, 0.136) | 0.235 (0.158, 0.288) | 0.116 (0.079, 0.148) | 0.389–0.598 |
A/B | 0.1 | 0.15 | 0.102 (0.062, 0.167) | 0.271 (0.177, 0.357) | 0.115 (0.065, 0.193) | 0.026–0.087 |
SMART | 0.1 | 0.15 | 0.096 (0.07, 0.128) | 0.255 (0.187, 0.29) | 0.099 (0.039, 0.182) | 0.599–0.654 |
A/B | 0.2 | 0.05 | 0.117 (0.067, 0.156) | 0.249 (0.190, 0.328) | 0.105 (0.055, 0.166) | 0.075–0.151 |
SMART | 0.2 | 0.05 | 0.104 (0.07, 0.138) | 0.235 (0.177, 0.278) | 0.117 (0.100, 0.156) | 0.505–0.568 |
A/B | 0.2 | 0.1 | 0.109 (0.05, 0.189) | 0.254 (0.21, 0.318) | 0.110 (0.072, 0.158) | 0.113–0.151 |
SMART | 0.2 | 0.1 | 0.106 (0.07, 0.153) | 0.273 (0.195, 0.361) | 0.119 (0.082, 0.148) | 0.537–0.754 |
A/B | 0.2 | 0.15 | 0.087 (0.062, 0.118) | 0.259 (0.182, 0.333) | 0.097 (0.062, 0.146) | 0.156–0.206 |
SMART | 0.2 | 0.15 | 0.129 (0.09, 0.193) | 0.243 (0.175, 0.288) | 0.120 (0.082, 0.158) | 0.599–0.654 |
A/B | 0.3 | 0.05 | 0.095 (0.052, 0.128) | 0.257 (0.197, 0.318) | 0.107 (0.065, 0.156) | 0.216–0.271 |
SMART | 0.3 | 0.05 | 0.127 (0.092, 0.16) | 0.243 (0.175, 0.308) | 0.117 (0.080, 0.160) | 0.668–0.724 |
A/B | 0.3 | 0.1 | 0.112 (0.062, 0.171) | 0.260 (0.161, 0.359) | 0.111 (0.075, 0.138) | 0.242–0.280 |
SMART | 0.3 | 0.1 | 0.107 (0.062, 0.138) | 0.278 (0.185, 0.387) | 0.123 (0.082, 0.181) | 0.703–0.769 |
A/B | 0.3 | 0.15 | 0.12 (0.049, 0.178) | 0.252 (0.165, 0.326) | 0.123 (0.080, 0.150) | 0.273–0.327 |
SMART | 0.3 | 0.15 | 0.133 (0.112, 0.158) | 0.253 (0.197, 0.28) | 0.109 (0.070, 0.190) | 0.746–0.787 |
