Modeling of systematic errors in stereo-digital image correlation due to camera self-heating

Abstract

Errors in strain measurements in stereo-digital image correlation (stereo-DIC) caused by camera self-heating have been experimentally observed in previous research, and have been shown to widely vary from one system configuration to another. Such “thermal errors” are sometimes so large that they strongly compromise the accuracy of the measurements. Despite correcting such errors is crucial when aiming at high-accuracy measurements, the mechanism of the thermal error generation and how it relates to the camera parameters in stereo-DIC are still not clear. In this paper, we first explain in detail how self-heating can introduce large artifacts in the strains measured by stereo-DIC. Using a simplified stereovision model, we provide the main equations that describe the theoretical errors in 3D coordinate reconstruction and 3D deformation measurement. Through several sets of simulations, the effect of camera self-heating on the 3D coordinate, displacement and strain measurements, and the effect of camera parameters on the thermal errors in stereo-DIC were explicitly presented based on the derived theoretical formulas. Finally, several real self-heating tests using a smartphone-based single-camera stereo-DIC system confirm the correctness of theoretical analyses and simulation results.

Introduction

Stereo-digital image correlation (stereo-DIC) is a well-assessed non-contact optical technique capable of performing 3D shape and deformation measurements across a wide range of length and time scales^1,2,3. Compared with two-dimensional digital image correlation (2D-DIC), stereo-DIC can realize 3D shape and deformation measurements of both planar and curved objects without being affected by out-of-plane translations and rotations on the specimen surface, thus offering wider applicability and higher measurement accuracy. With the continuous improvements in reconstruction algorithm and system hardware, stereo-DIC has prone to be a promising tool in scientific researches and engineering applications, such as material properties characterization of various materials^4,5, multi-scale deformation determination^6,7, dynamic deformation tracking^8,9,10 and 3D deformation measurement in harsh environment^11,12.

During the implementation of stereo-DIC, digital image pairs of a calibration target with changing positions and/or orientations and a test object at different loading conditions (times or states) are first recorded by two synchronized cameras or a single camera with auxiliary optical devices¹³. The image pairs of the calibration target are processed to determine the camera parameters (including intrinsic and extrinsic parameters), and the correspondence of the projections in the image pairs of the test object are established using correlation algorithms. Together with the calibrated camera parameters, 3D coordinates of measurement points can be reconstructed based on the determined image coordinates of their projections in the two cameras and the triangulation principle. Usually, the calibrated camera parameters for establishing the world coordinate system are assumed constant during the test, which means all the components inside the imaging system will not change in position or orientation. However, in real situations, the camera parameters do not meet this assumption. For a regular stereo-DIC system, the integrated circuit boards of two cameras will generate heat after being switched on, and the heat then transfers to the other components of the cameras, resulting in slight thermal deformation of the mechanical components of the two cameras. This effect is usually known as camera self-heating or camera warm-up effect. As a result, the structural configurations (e.g., the positions and orientations of the camera lens and sensor) inside the cameras will change accordingly due to slight thermal deformation. In other words, the originally calibrated camera parameters and the established world coordinate system of the stereo-DIC systems are changed by the temperature variations. Unavoidably, these slight changes will cause measuring errors in the measured results. To realize high-accuracy 3D shape and deformation measurements, the errors in stereo-DIC caused by the camera self-heating should be carefully studied and then eliminated when possible.

In literature, the camera self-heating phenomenon was observed in the early 1990s by several researchers in machine vision field^14,15. They found that the image coordinate would drift to some tenth of a pixel during the first hour after camera start-up. To explain the reasons for the observed image coordinate drift, Handel¹⁶ and Podbreznik and Potocnik^17,18 both established mathematical models based on pinhole model for modeling the effect of camera self-heating. Their models assume that temperature will not affect the intrinsic camera parameters and only translation of the camera needs to be considered. In fact, the temperature variation will cause thermal expansions of all the components in the camera, and thus their assumptions may be not accurate in some cases. To address this limitation, Yu et al.¹⁹ proposed a more rigorous model to describe the relationship between variations in the camera parameters and drift in the image coordinates, which consider the changes of the intrinsic parameters. Recently, the camera self-heating phenomenon was found in 2D-DIC measurement with a maximum strain error measured as tens to hundreds of microstrains for different cameras^{20,21,22,23,24}. Later, Ma et al.²² experimentally studied the deformation of the camera components and found the difference in the out-of-plane translations of the camera lens and camera sensor. The theoretical displacement and strain errors in 2D-DIC due to camera self-heating were then derived and fully understood according to these observations. More recently, similar temperature-dependent thermal errors were found in stereo-DIC measurement by Pan et al.²⁵. Our recent research further indicated that the thermal errors would significantly increase to tens of thousands microstrains for a single-camera stereo-DIC system with small baseline distance and short focal length²⁶. However, compared with 2D-DIC using one camera, the mechanism of the thermal error generation in stereo-DIC is more complicated as the structural changes of the two cameras should be considered in the triangulation. To understand the thermal errors in stereo-DIC and explain the difference of the errors in different system configurations, a comprehensive theoretical analysis of the errors in stereo-DIC due to camera self-heating is therefore necessary.

In this work, the mechanism of the thermal error generation in stereo-DIC is first clearly explained through two different models. Then, a comprehensive theoretical analysis is performed to quantitatively investigate the 3D reconstruction and deformation errors in stereo-DIC due to camera self-heating based on a simplified stereovision model. The effect of camera self-heating on the full-field 3D coordinate, displacement and strain measurements and the effect of camera parameters on the thermal errors are presented by conducting a series of simulation tests. Finally, real self-heating tests using a smartphone-based single-camera stereo-DIC system were performed on a flat plate to verify the correctness of the theoretical analysis and simulation results.

Basic Principles of Stereo-DIC Technique

To facilitate the analysis of the thermal errors in stereo-DIC due to camera self-heating, a simplified imaging model based on the pinhole model is adopted here. In this simplified model, the lens distortion and skew of the sensor plane are neglected, and the location of the principal point is assumed to be precisely in the center of the camera sensor. Figure 1 shows the schematic diagram of the imaging model of a stereo-DIC system and its geometrical relationships in the projection plane. As shown in the figure, assuming a physical point P(X, Y, Z) on the object surface has been identified on the left and right sensor planes as P₁(X₁, Y₁) and P₂(X₂, Y₂), then the three-dimensional coordinates of this point in the world coordinate system (O-XYZ, defined at the optical center of left camera) can be derived through geometric relationship as^27,28:

$$\begin{array}{ccc}X & = & \frac{B}{\cot \,{\theta }_{1}+\,\cot \,{\theta }_{2}}\,\cot \,{\theta }_{1}\\ Y & = & \frac{B}{\cot \,{\theta }_{1}+\,\cot \,{\theta }_{2}}\frac{\tan \,{\alpha }_{1}}{\sin \,{\theta }_{1}}\\ Z & = & \frac{B}{\cot \,{\theta }_{1}+\,\cot \,{\theta }_{2}}\end{array}$$

(1)

where θ₁ = ω₁ + φ₁, θ₂ = ω₂ + φ₂, f₁ and f₂ are the focal lengths of the two cameras, φ₁ and φ₂ are the angles between the optical axes of the cameras and the baseline, ω₁ and ω₂ are the angles between the horizontal projection of the light rays PP₁ and PP₂ and the optical axes, and α₁ and α₂ are the corresponding vertical projection angles, and D is the distance between the projection point P’ and the baseline (D is equal to Z in this world coordinate system). Note that ω₁, ω₂, α₁ and α₂ are not independent variables but can be represented by other structural parameters and coordinates in sensor plane as tanω₁ = X₁/f₁, tanω₂ = X₂/f₂, tanα₁ = Y₁·cosω₁/f₁ and tanα₂ = Y₂·cosω₂/f₂, respectively. With these parameters and geometric relationships, θ₁, θ₂ and α₁can be expressed as,

$$\begin{array}{rcl}{\theta }_{1} & = & \arctan \frac{{X}_{1}}{{f}_{1}}+{\phi }_{1}\\ {\theta }_{2} & = & \arctan \frac{{X}_{2}}{{f}_{2}}+{\phi }_{2}\\ {\alpha }_{1} & = & \arctan (\frac{{Y}_{1}}{{f}_{1}}\,\cos (\arctan \frac{{X}_{1}}{{f}_{1}}))\end{array}$$

(2)

Figure 1

(a) Schematic diagram of the imaging model of a stereo-DIC system and (b) the geometrical relationships in XOZ plane (O-XYZ is the world coordinate system of the stereo system defined at the optical center of the left camera).

Once the camera parameters (intrinsic parameters f₁ and f₂, and extrinsic parameters φ₁, φ₂, and B, calibrated using a calibration target) and the coordinates of the two projection points (P₁(X₁, Y₁) and P₂(X₂, Y₂), tracked through stereo-matching process) are determined, the 3D coordinates of point P in the world coordinate system (O-XYZ) can be calculated according to Eq. (1). After repeating this procedure for all the calculation points evenly spaced in the pre-defined region of interest (ROI), the 3D shape of the test object at initial state can be retrieved. Then, the 3D shapes of the test object at deformed states are determined with the same camera parameters and the changed coordinates of the two projection points. By subtracting the reconstructed 3D coordinates of the deformed states from those of the initial state, 3D full-field displacement fields on the specimen surface of different deformed states are retrieved. Finally, the full-field strain maps of the deformed states are calculated by differentiating the displacement fields using a pointwise least square strain estimation approach.

Theoretical Analysis of The Errors in Stereo-Digital Image Correlation Due to Camera Self-Heating

The mechanism of the thermal error generation in stereo-DIC

As introduced above, the 3D coordinates of a point on the test specimen are determined by the calibrated camera parameters and coordinates of the two projection points in the left and right images according to the triangulation principle (Eq. (1)). Usually, the camera parameters are calibrated before or after test and regarded as constants for all the image pairs recorded in a test, while the image coordinates of the two projection points vary from each physical point on the object surface to another. Ideally, the camera parameters and image coordinates of the two projection points should be constant when the cameras and the test sample are all kept stable. However, in the real tests, the structural configurations (e.g., the positions and orientations of the camera lens and sensor) inside the cameras will be changed due to slight thermal deformation. These temperature-dependent changes further will lead to slight alterations in positions (i.e., image coordinates) of the two projection points. With the temperature-dependent image coordinates of the two projection points and constant camera parameters, the reconstructed 3D coordinates will change with the temperature according to the triangulation principle.

To better understand the errors in the stereo-DIC system due to the camera self-heating, here we give two models, including a real model and a calculation model, to explain how the image coordinates and disparities of the two projection points change during the real test and how the reconstruction errors are introduced during the calculation process, respectively. As shown in Fig. 2(a), assuming the two cameras have temperature-introduced translations along the optical axis directions, the two projection points of a real space point P will move outward, which means a decrease in the x coordinate of P₁ and an increase of the x coordinate in P₂. Note that the temperature-introduced out-of-plane translations of the sensor and lens were clearly observed during a real test²². As a result, the disparity (i.e., their difference in x coordinate) of these two projection points in x direction will increase due to the camera translation. During the calculation, the calibrated camera intrinsic and extrinsic parameters were used to build a world coordinate system, as shown in Fig. 2(b). As all the parameters are fixed for all the images recorded in one test, the temperature-dependent disparities undoubtedly will result in temperature-dependent 3D coordinates, and these varying 3D coordinates further lead to temperature-dependent displacements and strains.

Figure 2

(a) A real model to explain how the disparities of the two projection points change during the test, (b) A calculation model to explain how the reconstruction errors are introduced during the calculation process. To facilitate the analysis, the sensor plane was symmetrically placed before the imaging lens.

Image coordinate drifts due to camera self-heating

Image coordinate drift in a single camera due to camera self-heating

To find the connections between the stereo-DIC measurement and the temperature variations, the image coordinate drift (or image distortion) in each camera due to camera self-heating should be modeled. Figure 3(a) shows a simplified pinhole model, which also neglect the lens distortion and skew of the camera sensor. Based on the ideal pinhole model, the relationship between a 3D space point and its projection in sensor plane can be built through three coordinate transformations: (1) from the world coordinate system (R_W) to the camera coordinate system (R_C), (2) from the camera coordinate system (R_C) to sensor plane coordinate system (R_S) and (3) from the sensor plane coordinate system (R_S) to the image plane coordinate system (r_s). By combining these three transformations, the final form of the transformation between the coordinates (X_W, Y_W, Z_W) of a space point P in the world coordinate system (R_W) and its coordinates (x_s, y_s) in image plane coordinate system (r_s) can be written as²⁹,

$$[\begin{array}{c}{x}_{s}\\ {y}_{s}\\ 1\end{array}]=\frac{1}{{Z}_{C}}[\begin{array}{ccc}\frac{L}{{d}_{x}} & 0 & {C}_{x}\\ 0 & \frac{L}{{d}_{y}} & {C}_{y}\\ 0 & 0 & 1\end{array}]([\begin{array}{ccc}{R}_{11} & {R}_{12} & {R}_{13}\\ {R}_{21} & {R}_{22} & {R}_{23}\\ {R}_{31} & {R}_{32} & {R}_{33}\end{array}][\begin{array}{c}{X}_{W}\\ {Y}_{W}\\ {Z}_{W}\end{array}]+[\begin{array}{c}{T}_{x}\\ {T}_{y}\\ {T}_{z}\end{array}])$$

(3)

where Z_C is the Z-directional coordinate of P in the camera coordinate system (R_C), L is the distance between the sensor plane and the optical center (which is usually denoted as f, i.e., focal length, units: mm), d_x and d_y are scale factors (units: mm/pixel) relating pixels to distance, C_x and C_y are coordinates (units: pixel) of principal point in image plane coordinate system (r_s), R = (R₁₁ R₁₂ R₁₃; R₂₁ R₂₂ R₂₃; R₃₁ R₃₂ R₃₃) and T = (T_x T_y T_z)^T are the rotation matrix and translation vector between the world coordinate system (R_W) and the camera coordinate system (R_C). Note that the rotation matrix R can be expressed as¹⁹,

$${\bf{R}}=[\begin{array}{ccc}{R}_{11} & {R}_{12} & {R}_{13}\\ {R}_{21} & {R}_{22} & {R}_{23}\\ {R}_{31} & {R}_{32} & {R}_{33}\end{array}]=[\begin{array}{ccc}\cos \,\beta \,\cos \,\gamma & \sin \,\alpha \,\sin \,\beta \,\cos \,\gamma -\,\cos \,\alpha \,\sin \,\gamma & \cos \,\alpha \,\sin \,\beta \,\cos \,\gamma +\,\sin \,\alpha \,\sin \,\gamma \\ \cos \,\beta \,\sin \,\gamma & \sin \,\alpha \,\sin \,\beta \,\sin \,\gamma +\,\cos \,\alpha \,\cos \,\gamma & \cos \,\alpha \,\sin \,\beta \,\sin \,\gamma -\,\sin \,\alpha \,\cos \,\gamma \\ -\,\sin \,\beta & \sin \,\alpha \,\cos \,\beta & \cos \,\alpha \,\cos \,\beta \end{array}]$$

(4)

where α, β and γ are the Euler angles rotating around X_W, Y_W and Z_W axes. It should be emphasized that the sensor plane coordinate system (X_SO_SY_S) is assumed to be parallel and concentric to the X_CO_CY_C plane of the camera coordinate system in Eq. (3) for an ideal camera model.

Figure 3

(a)A simplified pinhole model and the corresponding coordinate systems, (b) the imaging model of the camera before and after camera heating in Y_CO_CZ_C plane.

Usually, all the parameters (including the object distance Z_C, the image distance L, and the rotation matrix R and translation vector T between the world coordinate system and the camera coordinate system) in Eq. (3) should be constant during an ideal test. However, in a real test, these parameters will be changed by the translations and/or rotations of the camera sensor and imaging lens due to the camera self-heating. As a result, these changes will lead to a drift in the image coordinates (x_s, y_s) of projection point p, and further cause errors in the stereo-DIC measurement according to the analysis in the last section. To quantitatively analyze the displacement and strain errors in stereo-DIC induced by the camera self-heating, the drift of the projection point in the two cameras should be modeled. As evidenced in refs ^22,24, the translations of camera lens and sensor plane along the optical axis direction (i.e., Z_C direction) are the dominant factors caused by the camera self-heating and these observations will help to simplify the analysis of the thermal errors in stereo-DIC. Based on these experimental investigations, here we assume the original image distance and object distance are L(t₀) and Z_C(t₀) at initial time (t = t₀), the thermal-induced out-of-plane translations of the sensor plane and the lens are d_S(t_i) and d_L(t_i) during the test (t = t_i), as shown in Fig. 3(b). Note that the values of d_S(t_i) and d_L(t_i) may differ from one camera to another due to the small differences in camera structures. Then the image distance L(t_i) and object distance Z_C(t_i) of the changed imaging model can be determined through the geometric relationships. Also, the translation vector T between the world coordinate system and the camera coordinate system will be changed by the out-of-plane translation of the imaging lens, while the rotation matrix R is assumed to be constant during the test. With these changed parameters, the locations of the projection point p on the sensor plane can be expressed using the imaging model as,

$$[\begin{array}{c}{x}_{s}({t}_{i})\\ {y}_{s}({t}_{i})\\ 1\end{array}]=\frac{1}{{Z}_{C}({t}_{0})-{d}_{L}({t}_{i})}[\begin{array}{ccc}\frac{L({t}_{0})+{d}_{L}({t}_{i})-{d}_{S}({t}_{i})}{{d}_{x}} & 0 & {C}_{x}\\ 0 & \frac{L({t}_{0})+{d}_{L}({t}_{i})-{d}_{S}({t}_{i})}{{d}_{y}} & {C}_{y}\\ 0 & 0 & 1\end{array}]([\begin{array}{ccc}{R}_{11} & {R}_{12} & {R}_{13}\\ {R}_{21} & {R}_{22} & {R}_{23}\\ {R}_{31} & {R}_{32} & {R}_{33}\end{array}][\begin{array}{c}{X}_{W}\\ {Y}_{W}\\ {Z}_{W}\end{array}]+[\begin{array}{c}{T}_{x}\\ {T}_{y}\\ {T}_{z}-{d}_{L}({t}_{i})\end{array}])$$

(5)

Image coordinate drifts in a stereo-DIC system due to camera self-heating

Left camera: Based on the analysis in last section, we can associate the image coordinate drifts of the two projection points in a stereo-DIC system with a space point on the object surface. Figure 4 shows the schematic diagram of the imaging model of a stereo-DIC system and the changes of the image model in X_WO_WZ_W plane due to the camera self-heating. Same as the previous analysis, the world coordinate system is also moved to the optical center of left camera. As shown in Fig. 4, the Y axis (Y_W) and original point (O_W) of the world coordinate system coincide to those of the left camera coordinate system (O_C1-X_C1Y_C1Z_C1), while the angle between the Z axes of these two coordinate systems can be determined as β_L = −(90° − φ₁). As a result, the rotation matrix R^L between the world coordinate system and the left camera coordinate system can be determined by Eq. (4), and the translation vector T^L is determined as (0 0 0)^T. Then the mathematical relationship between the global coordinates (X_W, Y_W, Z_W) of a space point and its projection point in left camera sensor (x₁, y₁) at initial time (t = t₀) can be expressed as,

$$[\begin{array}{c}{x}_{1}({t}_{0})\\ {y}_{1}({t}_{0})\\ 1\end{array}]=\frac{1}{{Z}_{C1}({t}_{0})}[\begin{array}{ccc}\frac{{L}_{1}({t}_{0})}{{d}_{x1}} & 0 & {C}_{x1}\\ 0 & \frac{{L}_{1}({t}_{0})}{{d}_{y1}} & {C}_{y1}\\ 0 & 0 & 1\end{array}]([\begin{array}{ccc}\sin \,{\phi }_{1} & 0 & -\,\cos \,{\phi }_{1}\\ 0 & 1 & 0\\ \cos \,{\phi }_{1} & 0 & \sin \,{\phi }_{1}\end{array}][\begin{array}{c}{X}_{W}\\ {Y}_{W}\\ {Z}_{W}\end{array}]+[\begin{array}{c}0\\ 0\\ 0\end{array}])$$

(6)

where L₁(t₀) is the image distance of the left camera at initial time (i.e., the focal length), Z_C1(t₀) is the object distance of the left camera and can be estimated as ${Z}_{C1}({t}_{0})=\sqrt{{X}_{W}^{2}+{Z}_{W}^{2}}\,\cos (\arctan \frac{{Z}_{W}}{{X}_{W}}-{\phi }_{1})={X}_{W}$$\cos \,{\phi }_{1}+{Z}_{W}\,\sin \,{\phi }_{1}$ through the geometric relationships in triangle P’O_C1O_C2.

Figure 4

(a) Schematic diagram of the imaging model of a stereo-DIC system and (b) the changes of the image model in X_WO_WZ_W plane due to camera self-heating.

When the camera is working, we assume the out-of-plane translations of the sensor and the lens of the left camera at the time of t_i are denoted as d_S1(t_i) and d_L1(t_i), respectively. The rotation matrix R^L between the world coordinate system and the left camera coordinate system should be kept stable as the relative rotation is neglected, while the translation vector T^L will be changed to ${T}^{L}={(\begin{array}{ccc}0 & 0 & -{d}_{L1}({t}_{i})\end{array})}^{T}$ in the left camera coordinate system. Then the mathematical relationship between the global coordinates (X_W, Y_W, Z_W) of a space point and its projection point in left camera sensor (x₁, y₁) during the test (t = t_i) can be written as,

$$[\begin{array}{c}{x}_{1}({t}_{i})\\ {y}_{1}({t}_{i})\\ 1\end{array}]=\frac{1}{{Z}_{C1}({t}_{0})-{d}_{L1}({t}_{i})}[\begin{array}{ccc}\frac{{L}_{1}({t}_{0})+{d}_{L1}({t}_{i})-{d}_{S1}({t}_{i})}{{d}_{x1}} & 0 & {C}_{x1}\\ 0 & \frac{{L}_{1}({t}_{0})+{d}_{L1}({t}_{i})-{d}_{S1}({t}_{i})}{{d}_{y1}} & {C}_{y1}\\ 0 & 0 & 1\end{array}]([\begin{array}{ccc}\sin \,{\phi }_{1} & 0 & -\,\cos \,{\phi }_{1}\\ 0 & 1 & 0\\ \cos \,{\phi }_{1} & 0 & \sin \,{\phi }_{1}\end{array}][\begin{array}{c}{X}_{W}\\ {Y}_{W}\\ {Z}_{W}\end{array}]+[\begin{array}{c}0\\ 0\\ -{d}_{L1}({t}_{i})\end{array}])$$

(7)

Right camera: For the right camera, the Y-axis (Y_W) of the world coordinate system coincide to that (Y_C2) of the right camera coordinate system (O_C2-X_C2Y_C2Z_C2), while the angle between the Z axes of these two coordinate systems is estimated as β_R = (90° − φ₂). As a result, the rotation matrix R^R and the translation vector T^R between the world coordinate system and the right camera coordinate system can be calculated. Then the mathematical relationship between the global coordinates (X_W, Y_W, Z_W) of the space point and its projection point in right camera sensor (x₂, y₂) at initial time (t = t₀) can be expressed as,

$$[\begin{array}{c}{x}_{2}({t}_{0})\\ {y}_{2}({t}_{0})\\ 1\end{array}]=\frac{1}{{Z}_{C2}({t}_{0})}[\begin{array}{ccc}\frac{{L}_{2}({t}_{0})}{{d}_{x2}} & 0 & {C}_{x2}\\ 0 & \frac{{L}_{2}({t}_{0})}{{d}_{y2}} & {C}_{y2}\\ 0 & 0 & 1\end{array}]([\begin{array}{ccc}\sin \,{\phi }_{2} & 0 & \cos \,{\phi }_{2}\\ 0 & 1 & 0\\ -\,\cos \,{\phi }_{2} & 0 & \sin \,{\phi }_{2}\end{array}][\begin{array}{c}{X}_{W}\\ {Y}_{W}\\ {Z}_{W}\end{array}]+[\begin{array}{c}-B\,\sin \,{\phi }_{2}\\ 0\\ B\,\cos \,{\phi }_{2}\end{array}])$$

(8)

where L₂(t₀) is the image distance of the right camera at the initial time (i.e., the focal length), Z_C2(t₀) is the object distance of the right camera and estimated as ${Z}_{C2}({t}_{0})=\sqrt{{(B-{X}_{W})}^{2}+{Z}_{W}^{2}}\,\cos (\arctan \frac{{Z}_{W}}{B-{X}_{W}}-{\phi }_{2})=(B-{X}_{W})$ $\cos \,{\phi }_{2}+{Z}_{W}\,\sin \,{\phi }_{2}$.

Similarly, we assume the out-of-plane translations of the sensor and the lens of the right camera at the time of t_i are d_S2(t_i) and d_L2(t_i) when the camera is working, respectively. The rotation matrix R^R between the world coordinate system and the right camera coordinate system is also assumed not changed, while the translation vector T^R will be changed to ${T}^{R}={(\begin{array}{ccc}-B\sin {\phi }_{2} & 0 & B\cos {\phi }_{2}-{d}_{L2}({t}_{i})\end{array})}^{T}$ in the right camera coordinate system. Then the mathematical relationship between the global coordinates (X_W, Y_W, Z_W) of a space point and its projection point in right camera sensor (x₂, y₂) during the test (t = t_i) can be written as,

$$[\begin{array}{c}{x}_{2}({t}_{i})\\ {y}_{2}({t}_{i})\\ 1\end{array}]=\frac{1}{{Z}_{C2}({t}_{0})-{d}_{L2}({t}_{i})}[\begin{array}{ccc}\frac{{L}_{2}({t}_{0})+{d}_{L2}({t}_{i})-{d}_{S2}({t}_{i})}{{d}_{x2}} & 0 & {C}_{x2}\\ 0 & \frac{{L}_{2}({t}_{0})+{d}_{L2}({t}_{i})-{d}_{S2}({t}_{i})}{{d}_{y2}} & {C}_{y2}\\ 0 & 0 & 1\end{array}]([\begin{array}{ccc}\sin \,{\phi }_{2} & 0 & \cos \,{\phi }_{2}\\ 0 & 1 & 0\\ -\,\cos \,{\phi }_{2} & 0 & \sin \,{\phi }_{2}\end{array}][\begin{array}{c}{X}_{W}\\ {Y}_{W}\\ {Z}_{W}\end{array}]+[\begin{array}{c}-B\,\sin \,{\phi }_{2}\\ 0\\ B\,\cos \,{\phi }_{2}-{d}_{L2}({t}_{i})\end{array}])$$

(9)

Thermal errors in 3D coordinate, displacement and strain measurements

With above equations, the image coordinates of the two projections (p₁, p₂) on the two camera sensors before and during the camera working process can be estimated provided that the translations of the camera lenses and sensors are measured or pre-estimated. By substituting these temperature-dependent image coordinates into Eq. (1), the reconstructed 3D coordinates of the space point P can be retrieved, and will also present a temperature-dependent trend according to the previous analysis. Then, based on these reconstructed 3D coordinates at different times, the displacement and strain errors caused by the camera self-heating can be evaluated.

Errors in 3D coordinate reconstruction

For a specific space point P (X_W, Y_W, Z_W, in the world coordinate system defined at the optical center of the left camera), the drifted image coordinates of the two projection points can be estimated through Eqs (7) and (9). By substituting these temperature-dependent image coordinates into Eq. (1), the reconstructed 3D coordinates of the space point P at the time of t_i can be written as,

$$\begin{array}{c}{X}_{W}({t}_{i})=\frac{B}{\cot \,{\theta }_{1}({t}_{i})+\,\cot \,{\theta }_{2}({t}_{i})}\,\cot \,{\theta }_{1}({t}_{i})\\ {Y}_{W}({t}_{i})=\frac{B}{\cot \,{\theta }_{1}({t}_{i})+\,\cot \,{\theta }_{2}({t}_{i})}\frac{\tan \,{\alpha }_{1}({t}_{i})}{\sin \,{\theta }_{1}({t}_{i})}\\ {Z}_{W}({t}_{i})=\frac{B}{\cot \,{\theta }_{1}({t}_{i})+\,\cot \,{\theta }_{2}({t}_{i})}\end{array}$$

(10)

where the angles can be represented as,

$$\begin{array}{c}{\theta }_{1}({t}_{i})=\arctan \frac{{x}_{1}({t}_{i})-{C}_{x1}}{{f}_{1}/{d}_{x1}}+{\phi }_{1}\\ {\theta }_{2}({t}_{i})=\arctan \frac{{x}_{2}({t}_{i})-{C}_{x2}}{{f}_{2}/{d}_{x2}}+{\phi }_{2}\\ {\alpha }_{1}({t}_{i})=\arctan (\frac{{y}_{1}({t}_{i})-{C}_{y1}}{{f}_{1}/{d}_{y1}}\,\cos (\arctan \frac{{x}_{1}({t}_{i})-{C}_{x1}}{{f}_{1}/{d}_{x1}}))\end{array}$$

(11)

Note that f₁ and f₂ are the focal lengths of the left and right cameras, and are equal to L₁(t₀) and L₂(t₀), B is the baseline distance, φ₁ and φ₂ are the angles between the optical axes of the left and right cameras and the baseline.

Errors in displacement and strain measurements

By subtracting the reconstructed 3D coordinates at the time of t_i from those at the time of t₀, the 3D displacement errors of the space point P can be derived as,

$$\begin{array}{c}U({t}_{i})={X}_{W}({t}_{i})-{X}_{W}({t}_{0})=\frac{B}{\cot \,{\theta }_{1}({t}_{i})+\,\cot \,{\theta }_{2}({t}_{i})}\,\cot \,{\theta }_{1}({t}_{i})-{X}_{W}({t}_{0})=B[\frac{\cot \,{\theta }_{1}({t}_{i})}{\cot \,{\theta }_{1}({t}_{i})+\,\cot \,{\theta }_{2}({t}_{i})}-\frac{\cot \,{\theta }_{1}({t}_{0})}{\cot \,{\theta }_{1}({t}_{0})+\,\cot \,{\theta }_{2}({t}_{0})}]\\ V({t}_{i})={Y}_{W}({t}_{i})-{Y}_{W}({t}_{0})=\frac{B}{\cot \,{\theta }_{1}({t}_{i})+\,\cot \,{\theta }_{2}({t}_{i})}\frac{\tan \,{\alpha }_{1}({t}_{i})}{\sin \,{\theta }_{1}({t}_{i})}-{Y}_{W}({t}_{0})=B[\frac{\tan \,{\alpha }_{1}({t}_{i})/\sin \,{\theta }_{1}({t}_{i})}{\cot \,{\theta }_{1}({t}_{i})+\,\cot \,{\theta }_{2}({t}_{i})}-\frac{\tan \,{\alpha }_{1}({t}_{0})/\sin \,{\theta }_{1}({t}_{0})}{\cot \,{\theta }_{1}({t}_{0})+\,\cot \,{\theta }_{2}({t}_{0})}]\\ W({t}_{i})={Z}_{W}({t}_{i})-{Z}_{W}({t}_{0})=\frac{B}{\cot \,{\theta }_{1}({t}_{i})+\,\cot \,{\theta }_{2}({t}_{i})}-{Z}_{W}({t}_{0})=B[\frac{1}{\cot \,{\theta }_{1}({t}_{i})+\,\cot \,{\theta }_{2}({t}_{i})}-\frac{1}{\cot \,{\theta }_{1}({t}_{0})+\,\cot \,{\theta }_{2}({t}_{0})}]\end{array}$$

(12)

After repeating this procedure for all the calculation points evenly spaced in the pre-defined region of interest, the 3D displacement error fields at different times can be determined. By differentiating the displacement fields using a pointwise least square strain estimation approach, the full-field strain error fields at each configuration can be calculated.

Simulation of The Errors in Stereo-DIC Due to Camera Self-Heating

Effect of camera self-heating on the 3D coordinate, displacement and strain measurements

To get a first knowledge of the error distributions and trends in stereo-DIC due to camera self-heating, a simulation test is first conducted. According to the aforementioned analysis, the camera parameters to establish the world coordinate system, the out-of-plane translations of the sensor and lens due to camera self-heating and the global coordinates of the test object in the world coordinates are the required inputs for error estimations. To simplify the simulation process, the camera parameters refer to the system configuration of a real stereo-DIC system²⁵, and the out-of-plane translations of the camera sensor and camera lens per Celsius are taken from the experimental observations²². The test object is a planar surface with a physical dimension of 80 mm × 80 mm, and placed in front of the system with a working distance of 730 mm. All these inputs for the simulation are summarized in Table 1. With these inputs, the errors in stereo-DIC caused by the camera self-heating can be estimated through the above equations. Note that, to simulate the matching uncertainties in stereo-matching, zero-mean random noise with a standard deviation of 0.001 pixel was added to the image coordinates of the two projection points in left and right camera sensors.

Table 1 The assumed camera parameters, out-of-plane translations of the sensor and lens and global coordinates of the test object for the error estimation in stereo-DIC.

Figure 5(a) shows a typical temperature variation curve in the imaging camera after switched on and the estimated out-of-plane translations of the camera sensor and imaging lens based on the existed investigation²². With these temperature-dependent out-of-plane translations and the camera parameters, the temperature-dependent image coordinates of the two projections of the physical points on the object surface can be determined through Eqs (7) and (9). Based on the temperature-dependent image coordinates of the two projections and the assumed constant camera parameters, the 3D coordinates of the physical points can be reconstructed. By subtracting the reconstructed 3D coordinates at different time from the 3D coordinates reconstructed at the initial time (i.e., as indicated in Eq. (12)), the full-field 3D displacement errors of the test points were determined. Figure 5(b) shows the 3D displacement errors of a point P(205, 40, 730) in all the test points as a function of time. Clearly, the U, V and W displacement errors all present a similar trend with the temperature and W displacement error is larger than those of U and V. Then, the corresponding strain errors fields can be evaluated by differentiating the displacement fields and the mean in-plane strains are plotted in Fig. 5(c). It is observed that normal strain-time curves show a strong positive correlation with the temperature variations and the values of ε_x and ε_y are almost identical, while the in-plane shear strains are always close to zero. These observations are in good accordance with the experimental results in the previous studies²⁵.

Figure 5

(a) A typical temperature variation curve in the imaging camera after switched on and the estimated out-of-plane translations of the camera sensor and imaging lens, (b) 3D displacement errors of a physical point P(205, 40, 730) on the specimen surface and (c) the measured mean in-plane strain errors as a function of time.

To intuitively present the effect of camera self-heating on stereo-DIC measurement, the full-field U, V, radial $(\sqrt{{U}^{2}+{V}^{2}})$ and W displacement error fields at the times of 1st min, 10th min, 20th min and 30th min were given in Fig. 6. As shown in these figures, the U and V displacement error fields are evenly distributed along X and Y directions, indicating a virtual tensile strain in both directions. The dilatational deformations are more evident from the concentric radial displacement error fields. Ideally, the mean U and V displacements should be close to zeros if the center of the ROI exactly located at the perpendicular bisector of the baseline in XOY plane. Once the ROI deviate from the perpendicular bisector, the mean U and V displacements will not be zero and will change with the camera temperature. Also, it is not difficult to conclude from these displacement fields that the tensile strains in X and Y directions are almost equal, and the in-plane shear strain is negligible. In addition, it is interestingly to find that the W displacement errors at each time vary a little across the whole specimen surface, but will distinctly increase with temperature.

Figure 6

Simulated full-field U, V, radial and W displacement error fields at the time of 1st min, 10th min, 20th min and 30th min based on the assumed camera parameters and camera components translations.

Effect of camera parameters on the thermal errors in stereo-DIC

As the above results only focus on the errors in a specific stereo-DIC system due to the varying temperature, the effects of the camera parameters on these errors are still not clear. To address this problem, we simulated the errors in stereo-DIC systems with varying camera parameters (i.e., focal length f, baseline distance B, working distance D and included angle φ) when the camera sensor and camera lens undergo certain out-of-plane translations at a specific time (e.g., at 60th min). By comparing the measured displacement and strain errors in the stereo-DIC systems with different configurations, the effect of camera parameters on the thermal errors is revealed. It should be noted that two assumptions were made during the simulation. First, the matching accuracy is assumed to be constant and not affected by the other parameters (e.g., shape function, subset size and speckle size, etc.). Then, the sizes of the sensor planes are assumed to be large enough to accommodate the projections of the test object as the projection may go beyond the sensor boundary in some cases. With these two assumptions and the parameters in Table 1, we calculated the thermal displacement and strain errors for the stereo-DIC with different focal length, baseline distance, working distance and included angle, respectively.

Figure 7 shows the simulated 3D displacement error of the point P and in-plane strain errors as a function of the camera parameters, including focal length, baseline distance, working distance and included angle. Note that, in these simulations, the translations of the camera lenses and sensors are constant and taken from the data in Fig. 5(a) at the time of 60th min. From these figures, it can be concluded that the measured W displacement errors are more sensitive to camera parameters than the U and V displacement errors, especially for the system configuration with small baseline distance and included angle. The 3D displacement errors all decrease with the increase of the focal length and baseline distance, and the U and V displacement errors are almost unaffected by the working distance and included angle. For the in-plane strain errors, the ε_x and ε_y strain errors are almost equal and extremely sensitive to focal length and baseline distance. This can explain why a smartphone-based stereo-DIC system (small baseline distance and short focal length)²⁶ is more sensitive to the camera self-heating than the conventional stereo-DIC system²⁵. As shown in Fig. 7, the ε_x and ε_y strain errors will notably decrease with the increase of focal length and baseline distance, while the in-plane shear strain is always close to zero for different system configuration. Also, it is observed that the ε_x and ε_y strain errors are in a positive linear relationship with the working distance, which means a small working distance is preferred for a stereo-DIC system sensitive to the camera self-heating.

Figure 7

Simulated 3D displacement error of the point P and in-plane strain errors as a function of the camera parameters, including focal length, baseline distance, working distance and included angle. The translations of the camera lenses and sensors are constant and taken from the data in Fig. 5(a) at the time of 60th min.

Experimental Verifications

Experimental details

To verify the correctness of the theoretical analysis of the systematic errors in stereo-DIC due to camera self-heating, a series of self-heating experiments were performed. Figure 8(a) shows the experimental arrangement for the verification tests. Here, we adopted a single-camera stereo-DIC system using a smartphone as the imaging system, which is equivalent to a stereo-DIC system using two same virtual cameras. It is much smaller in focal length and baseline distance than conventional stereo-DIC systems, and thus should be more sensitive to the camera self-heating according to the above analysis. Besides, it should be noted that image model of the smartphone camera is similar with that of a conventional camera, which are both based on the pinhole model. The accuracy of the smartphone-based stereo-DIC system in 3D shape, motion and deformation measurement was successfully verified in our previous study²⁶. As shown in Fig. 8(a), the smartphone-based stereo-DIC system is mainly composed of an Android-based smartphone (Mi8, Xiaomi, Inc., China), a homemade optical attachment, and a small tripod (Type 258, Yunteng, Inc., China). The test specimen is a flat plate (150 mm × 150 mm × 5 mm) and placed right in front of the imaging system. Random speckle patterns were decorated onto the specimen surfaces using white spray paints and a black marker pen. The distance between the specimen and the imaging system was first set as 600 mm. During the experiment, the indoor temperature was kept constant, and the planar plate was held static. After that, surface images of the plate were recorded by the smartphone every 30 seconds with the help of an app called Tasker (the smartphone was rooted for emulating touch input). The image recording process lasted for 2 hours and about 240 images were recorded in this test. Note that the imaging system was calibrated before and after self-heating and the temperature variation of the smartphone system during the test was monitored using an infrared thermal camera (SC7900, FLIR). Further, similar experiments were repeated with different working distances, i.e., 300 mm, 450 mm, 750 mm and 900 mm. Finally, to investigate the effect of baseline distance, we conducted an additional self-heating test using the same smartphone and a larger four-mirror device at a working distance of 600 mm.

Figure 8

(a)Experimental set-up for the validation test and (b) a recorded image with defined ROI. The imaging system is a smartphone-based single-camera stereo-DIC system, which has been prone to be effective in 3D deformation measurement and more sensitive to the camera self-heating than a conventional stereo-DIC system.

All the recorded images were then processed to determine the displacement and strain errors due to the camera self-heating using conventional stereo-DIC algorithm. During the calculation, the first image in the image series was adopted as the reference image, and the rest images were regarded as the deformed images. As shown in Fig. 8(b), a rectangular area in the middle of the left image, was selected as the ROI. The displacements were calculated with a subset size of 41 × 41 pixels and a grid step of 10 pixels. Subsequently, a pointwise least squares algorithm using a strain calculation window of 15 × 15 sets of displacement data was employed to estimate the strain fields from the displacement results. As no external loading was applied to the plate, the actual displacements and strains on the plate should be zero and the measured displacements and strains can only be attributed to the errors due to camera self-heating.

Results

Figure 9(a,b) show the 3D displacement errors of a physical point P(65.7, 40.3, 603.8, in world coordinate system) and the measured mean in-plane strain errors on the specimen surface as a function of time, respectively. It should be noted that we compared the camera parameters calibrated before and after self-heating tests, and no noticeable or regular changes were observed. This is because that the variations of the camera parameters due to the nonlinear optimization is not small enough, and may cover up the real changes in the camera parameters. The displacements and strain errors determined from these two different sets of camera parameters are very close. As shown in Fig. 9, the U, V, W displacement and the two normal strain errors all highly correlated with the detected temperature changes on the smartphone back surface. These trends can be separated into two consecutive stages, i.e., the warm-up stage and thermal equilibrium stage. As shown in these two figures, the three displacement components and two normal strains increase rapidly with the temperature in the first stage and then stabilize at certain values in the thermal equilibrium stage. These observations are consistent with the simulated displacement and strain errors and the results in existing researches²⁵. The only difference is that the amplitudes of the displacement and strain errors in this smartphone-based stereo-DIC system are much larger than those in a conventional stereo-DIC system, which can be well explained by the above simulation results. Compared with the conventional stereo-DIC system in the simulation section (f₁ = f₂ = 50 mm, B = 330 mm), the focal length and the baseline distance of the smartphone-based stereo-DIC system (f₁ = f₂ = 5 mm, B = 50 mm) are much smaller. According to the simulation results in Fig. 7, the displacement and strain errors in this smartphone-based stereo-DIC system are unavoidably much larger.

Figure 9

(a) 3D displacement errors of a physical point P on the specimen surface and (b) the measured mean in-plane strain errors as a function of time.

Figure 10 shows the measured full-field U, V, radial and W displacement error fields at the time of 1st min, 10th min, 20th min and 30th min during the self-heating test with a working distance of about 600 mm. Same with the simulated displacement errors shown in Fig. 6, the measured U and V displacement error fields in the real test are also evenly distributed along X and Y directions, respectively. This indicates that the specimen surface undergo a virtual tensile strain in both directions and the virtual strains can be determined from the displacement gradients. The radial displacement. The third row of Fig. 10 shows the resultant displacement fields of the U and V displacement error fields. The superimposed contour lines in this figure denote the magnitudes of the radial displacement vectors, and the homogeneous thermal expansion is evident. Also, the W displacement error fields at 1st min, 10th min, 20th min and 30th min were presented in the last row of this figure. The W displacements on the specimen surface are almost equal at each time step, while the amplitudes are much larger than the simulated W displacement errors (Fig. 6) of a conventional stereo-DIC system. The reason why the W displacement errors in the smartphone-based stereo-DIC system are much larger can be well explained by the simulation analysis.

Figure 10

Measured full-field U, V, radial and W displacement error fields at the time of 1st min, 10th min, 20th min, and 30th min during the self-heating test with a working distance of 600 mm.

The thermal errors in the self-heating tests with different working distances were further compared to indicate the effect of working distance. Figure 11(a) shows the measured in-plane normal strain errors as a function of test time for self-heating tests with five different working distances (i.e., 300 mm, 450 mm, 600 mm, 750 mm and 900 mm). It is clearly observed that the thermal strains all present a rapidly increase stage (0~40 min) and an equilibrium stage (40 min~end), and the thermal errors are more significant for the test with longer working distance. The in-plane strain errors in two directions are almost equal and small differences in some tests (e.g., 450 mm and 750 mm) may be attributed to the small translation and/or rotation of the camera. Taking the mean values of the thermal strains in the equilibrium stage as the ordinate values, Fig. 11(b) shows the mean normal strains in thermal equilibrium stage as a function of the working distance. As seen from this figure, the two strain errors almost linearly increase with working distance with the slopes determined as about 23.51 and 23.95 με/mm. These observations are in good accordance with the simulation results in Fig. 7, and the difference in the slopes can be attributed to the differences in camera intrinsic and extrinsic parameters.

Figure 11

(a) Measured in-plane normal strain errors as a function of test time for self-heating tests with five different working distances and (b) mean normal strains in thermal equilibrium stage as a function of the working distance.

Figure 12 shows the measured in-plane normal strain errors as a function of test time for the self-heating tests with two different optical attachments, i.e., the original optical attachment shown in Fig. 8 for test 1 and a four-mirror device with a larger space between mirrors for test 2. Note that the working distances for the two tests were both set as 600 mm. The camera parameters of the systems in these two tests were then calibrated and list in Fig. 12(c). It is clear that the main difference between the two systems is the baseline distances between the two virtual cameras. As shown in Fig. 12(a,b), the measured in-plane normal strain errors all present a similar trend with the temperature variation in the smartphone. However, strain errors in the thermal equilibrium stage of test 1 are significantly larger than those of test 2, which indicates a larger baseline distance will result in smaller thermal errors. This observation is in good accordance with the simulation test.

Figure 12

Measured in-plane normal strain errors as a function of test time for the self-heating tests with two different optical attachment: (a) original optical attachment shown in Fig. 8, (b) a four-mirror device with a larger space between mirrors. The calibrated camera parameters and working distances for these two tests are present in (c), where c_x1, c_y1, c_x2 and c_y2 are the image coordinates of the principal points for the left and right virtual cameras, f_x1, f_y1, f_x2 and f_y2 are the focal lengths in two directions of the two virtual cameras.

Conclusions and Discussions

In this paper, the systematic errors in stereo-DIC due to camera self-heating is investigated theoretically and experimentally. The mechanism of the thermal error generation in stereo-DIC is first explained, and the theoretical errors in 3D coordinate reconstruction and displacement measurement are derived based on a simplified stereovision model. Simulation results explicitly show the effect of camera self-heating on the 3D coordinate, displacement and strain measurements and the effect of camera parameters on the thermal errors in stereo-DIC. Finally, experimental results from a smartphone-based stereo-DIC system successfully verify the correctness of theoretical analyses.

It is worth noting that the thermal errors caused by camera self-heating and/or temperature variation of the environment are unavoidable in all stereovision systems. These errors are usually small enough to be neglected in 3D measurement but should be paid attention to in some cases, e.g., when a high-accuracy measurement is required or using a stereovision system with a small baseline distance and/or a short focal length. To ensure high-accuracy 3D measurement under these circumstances, the test object should be placed near the system as much as possible to reduce the working distance, and solutions that can mitigate these thermal errors are required if the thermal errors are still not small enough to be neglected. Besides, during camera self-heating process, the rotation matrix R and translation vector T between the world coordinate system and the camera coordinate system may also be changed by the rigid-body translation and rotation of the camera or even environment vibrations. Also, the position of the principle point may be changed by the camera self-heating. These changes will add additional errors to the measured results. However, compared with the clearly observed out-of-plane translations of the sensor plane and camera lens, the rigid-body translation and rotation of the camera are less obvious and very difficult to measure. Once the rigid-body translation and rotation of the camera are determined, the analytical model in this paper can still be applied.