Table 1 Factors affecting variance estimation and causal variant detection in the simulated phenotypes.

From: Neighbor GWAS: incorporating neighbor genotypic identity into genome-wide association studies of field herbivory

Response

Factors

df

SS

F

p value

PVE accuracy

No. of causal SNPs

1

0.00

0.0

0.95

 

PVEβ

1

0.01

0.6

0.44

 

PVEβ + PVEu

1

0.02

0.61

0.43

 

PVEself:PVEnei:PVEsxn

2

0.25

4.95

0.007

 

α

1

0.96

38.46

6.1e−10

 

s

1

8.49

341.0

<2.2e−16

 

Residuals

4312

107.34

NA

NA

AUCself

No. of causal SNPs

1

13.12

2998.6

<2.2e−16

 

PVEβ

1

0.77

176.6

<2.2e−16

 

PVEβ + PVEu

1

0.02

4.04

0.045

 

PVEself:PVEnei:PVEsxn

2

8.08

923.54

<2.2e−16

 

α

1

0.01

2.19

0.14

 

Residuals

1073

4.69

NA

NA

AUCnei

No. of causal SNPs

1

25.82

2225.1

<2.2e−16

 

PVEβ

1

2.30

198.1

<2.2e−16

 

PVEβ + PVEu

1

0.03

2.24

0.14

 

PVEself:PVEnei:PVEsxn

2

20.97

903.48

<2.2e−16

 

α

1

0.96

83.00

<2.2e−16

 

s

1

0.079

6.83

0.0090

 

Residuals

3232

37.50

NA

NA

Self sensitivity

No. of causal SNPs

1

74.204

1317.15

<2.2e−16

 

PVEβ

1

2.236

39.69

4.0e−10

 

PVEβ + PVEu

1

0.06

1.06

0.30

 

PVEself:PVEnei:PVEsxn

2

11.955

106.10

<2.2e−16

 

α

1

0.089

1.57

0.21

 

Residuals

1073

60.449

NA

NA

Neighbor sensitivity

No. of causal SNPs

1

98.052

1153.56

<2.2e−16

 

PVEβ

1

4.649

54.70

2.0e−13

 

PVEβ + PVEu

1

0.016

0.19

0.67

 

PVEself:PVEnei:PVEsxn

2

23.196

136.45

<2.2e−16

 

α

1

1.852

21.79

3.0e−06

 

s

1

0.096

1.13

0.29

 

Residuals

3232

274.72

NA

NA

MAEself

No. of causal SNPs

1

105.32

323.44

<2.2e−16

 

PVEβ

1

1.80

5.54

0.02

 

PVEβ + PVEu

1

0.14

0.44

0.51

 

PVEself:PVEnei:PVEsxn

2

36.11

55.45

<2.2e−16

 

α

1

0.73

2.23

0.14

 

Residuals

1073

349.41

NA

NA

MAEnei

No. of causal SNPs

1

2.73

15.06

1.0e−04

 

PVEβ

1

16.89

93.17

<2.2e−16

 

PVEβ + PVEu

1

3.51

19.34

1.0e−05

 

PVEself:PVEnei:PVEsxn

2

80.22

221.25

<2.2e−16

 

α

1

0.39

2.15

0.14

 

s

1

45.87

253.01

<2.2e−16

 

Residuals

3232

585.91

NA

NA

  1. The accuracy of the proportion of the phenotypic variation explained (PVE) was defined as the PVE accuracy = (estimated total PVE − true total PVE)/true total PVE. The power was represented by the area under the ROC curve (AUC). The sensitivity to distinguish self or neighbor signals (Self or neighbor sensitivity) was evaluated using the true positive rate of the ROC curve, when the false positive rate = 0.05. The accuracy of the effect size estimates was evaluated using the mean absolute errors (MAE) between the true and estimated fixed effects. ANOVA tables show the degree of freedom (df), sum of squares (SS), F-statistics, and p values. Explanatory factors are the number of causal SNPs, proportion of phenotypic variation explained (PVE) by major-effect genes (PVEβ), total PVE by major-effect genes and variance components (PVEβ + PVEu), relative contribution of self, symmetric, and asymmetric neighbor effects (PVEself:PVEnei:PVEsxn), and distance decay coefficient α. For the neighbor effects, the difference of the reference spatial scales (s = 1–3) was also considered an explanatory variable. NA means not available.