Table 3 Mathematical notations and interpretations

\(n\in {{\mathbb{Z}}}^{+}\)

Number of spatial spots

\({y}_{i}\in {\mathbb{R}}\)

Expression level of a gene at spot i

\({Y}_{i}\in {\mathbb{R}}\)

Random variable representing the expression level of a gene at spot i

\({{\bf{Y}}}={({Y}_{1},\ldots,{Y}_{n})}^{\top }\in {{\mathbb{R}}}^{n}\)

Random vector of a gene’s expression levels at all n spatial spots

\({{{\bf{s}}}}_{i}={({s}_{i1},{s}_{i2})}^{\top }\in {{\mathbb{R}}}^{2}\)

2D spatial location of spot i

\({{\bf{s}}}={\left[{{{\bf{s}}}}_{1},\ldots,{{{\bf{s}}}}_{n}\right]}^{\top }\in {{\mathbb{R}}}^{n\times 2}\)

Matrix of spatial coordinates for all n spots

\({{{\bf{d}}}}_{i}={({d}_{i1},\ldots,{d}_{iL})}^{\top }\in {\{0,1\}}^{L}\)

Spatial-domain indicator vector for spot i

\({{{\bf{c}}}}_{i}={({c}_{i1},\ldots,{c}_{iK})}^{\top }\in {[0,1]}^{K}\)

Cell-type proportion vector for spot i

\({{{\bf{x}}}}_{i}\in {{\mathbb{R}}}^{p}\)

Fixed-effect covariates of spot i

\({{{\bf{z}}}}_{i}\in {{\mathbb{R}}}^{q}\)

Random-effect covariates of spot i

\({\epsilon }_{i}\in {\mathbb{R}}\)

Random measurement error at spot i

\({{\boldsymbol{\epsilon }}}={({\epsilon }_{1},\ldots,{\epsilon }_{n})}^{\top }\in {{\mathbb{R}}}^{n}\)

Vector of random measurement errors with assumed independence

\({\beta }_{0}\in {\mathbb{R}}\)

(Fixed) intercept in the linear mixed model

\({{\boldsymbol{\beta }}}\in {{\mathbb{R}}}^{p}\)

Fixed effects in the linear mixed model

\({{\boldsymbol{\gamma }}}\in {{\mathbb{R}}}^{q}\)

Random effects in the linear mixed model

\({{\mathbf{\Sigma }}}\in {{\mathbb{R}}}^{q\times q}\)

Covariance matrix of random effects γ