Introduction

Teaching quality is important to improving student attainment, prompting numerous countries to increasingly prioritize teaching quality as a crucial factor. Teaching quality can be improved through three distinct means: enhancing the caliber of teachers employed in educational institutions, providing motivation to encourage greater effort on the part of teachers, and refining teachers’ skill sets through professional development programs (Chetty & Rockoff, 2014). Many studies have shown that teacher training can improve teachers’ ability, teaching quality, and students’ academic achievement (Harris & Sass, 2011; Kraft et al., 2018; Saiti & Saitis, 2006). Teachers’ professional development is significantly influenced by teacher training, which can also increase their expertise (Saiti & Saitis, 2006). Harris and Sass (2011) reported that the productivity of elementary school and middle school teachers increased with informal on-the-job training. A meta-analysis by Kraft et al. (2018) revealed that teacher training had an average impact of 0.15 standard deviations on students’ academic performance. Consequently, teacher training quality is paramount.

Numerous educators and policy-makers agree that evaluation can effectively gauge and improve teacher education (Grammatikopoulos et al., 2013; Liu, 2015; Yang et al., 2021). Therefore, to ensure the efficacy of teacher training, it is necessary to assess its quality. Researchers have developed various assessment methods to evaluate the quality of teacher training. For example, some researchers have done so by assessing teacher performance or testing student achievement, but these methods cannot easily and quickly evaluate the quality of teacher training, especially in large-scale teacher training. Some researchers have also assessed teacher training by measuring its perceived quality, but this method has some shortcomings, such as not taking technology into account and not dimensioning the training content. Given the weaknesses of the above approaches, a new method to evaluate the quality of teacher training is needed.

Numerous researchers have studied the knowledge needed by teachers and have drawn many different conclusions (Harris et al., 2017). At present, the most representative and influential framework is the technological pedagogical and content knowledge (TPACK) framework proposed by Mishra and Koehler (2006), which includes seven kinds of knowledge necessary for teachers. The TPACK framework has received extensive attention in teaching practice and research and has a great impact on research and practice in teacher professional development (Harris et al., 2017). It not only contains the dimension of technology but also divides training content in a logical way. Therefore, evaluating the quality of teacher training on the basis of the TPACK framework avoids the flaws identified in the current research.

Moreover, evaluating the quality of teacher training by measuring the training quality perceived by teachers is fast and convenient. The objective of the present research is to develop a Teacher-Perceived Training Quality Scale from the perspective of the TPACK framework and verify the scale’s reliability and validity by surveying 426 K-12 school teachers in Henan, China.

Literature review

TPACK

Shulman (1986) and Koehler and Mishra (2005) proposed the TPACK framework (see Fig. 1), which includes technological knowledge, pedagogical knowledge, and content knowledge and their interaction. Teachers are required to acquire seven kinds of knowledge in the TPACK framework to carry out classroom teaching effectively (Mishra & Koehler, 2006). The TPACK framework has gained extensive recognition and is frequently applied. Many researchers have developed scales using it. For example, a number have employed it to develop scales of teachers’ self-perceived levels of TPACK (Schmid et al., 2020; Schmidt et al., 2009), and others have used it to develop scales for the impact of Facebook on teacher quality (Lu, 2018).

Fig. 1: TPACK model.
Fig. 1: TPACK model.
Full size image

Reproduced with the permission of the publisher, © 2012 by tpack.org.

Evaluation of teacher training

Cochran-Smith (2001) noted three ways to evaluate the quality of teacher education or teacher training. The first is to assess the professional performance of teacher candidates. The second is through teacher test scores. The third approach is to assess the effect of teacher education on the quality of teaching and student achievement. Previous studies have proposed numerous ways to assess the quality of teacher training. For example, many studies have applied Kirkpatrick’s model, which has four levels. The first level is the reaction, which measures trainees’ satisfaction with the training. The second level is learning, which involves assessing knowledge gains, improved skills, or changes in attitudes. The third level pertains to behavior evaluation, assessing the degree to which trainees apply their acquired knowledge and skills in their respective work areas. The final level is results, which assesses the final changes in trainees at the organizational level (Kirkpatrick, 1979). For example, Zheng et al. (2013) used Kirkpatrick’s model to assess the efficacy of training among 16,264 teachers, using different methods to assess each level, such as understanding the responses of participating teachers, collecting relevant data, and observing teachers’ performance after training. Many studies have assessed the quality of teacher training by interviewing teachers who have participated in the training (Bayrakcı, 2010; Darling-Hammond, 2006; Valcke et al., 2007). For example, Valcke et al. (2007) used the interview method to assess the quality of ICT teacher training in Flanders by conducting in-depth interviews with 185 teachers in almost 100 schools. Moreover, studies have assessed teacher training through changes in students’ academic achievement (Harris & Sass, 2011; Ome et al., 2017).

However, the use of these methods requires extensive manpower and material resources, and measuring large-scale teacher training is difficult. Therefore, for convenience, some studies have evaluated teacher training quality by measuring teachers’ perceptions of it.

Darling-Hammond (2006) formulated a questionnaire to gather perceptual data on the knowledge and skills acquired by participants in a teacher training program. Confirmatory factor analysis was conducted, with the five dimensions of instructional design and teaching, support of various learning styles, evaluation of teaching and learning to provide guidance, creation of a productive classroom, and professional development. Peacock, Matthew. (2009) designed a scale with 22 questions to measure English teachers’ perceptions of the quality of their training. The items on the scale mainly addressed the quality and content of the training course, such as the connections between different courses and whether the program taught classroom management skills. Koh et al. (2013) designed an instrument to evaluate teachers’ perceptions of ICT teacher training course quality with three dimensions: course delivery, content, and environment. It was verified by a survey of 483 preservice teachers in Singapore. To examine what candidates learn in teacher training programs, Akinci and Kose (2019) developed the Teacher Training Curriculum Assessment Scale (TTCES) to measure teacher candidates’ perceptions regarding the quality of teacher training. The scale covers areas such as deficiencies in teacher training courses, teacher training facilities, and teachers’ ability to carry out training.

Through a review of existing teacher-perceived training quality scales, we identified the following deficiencies. First, technical knowledge is not taken into account. Numerous studies have demonstrated the efficacy of instruction facilitated by technology (Herold, 2016). Therefore, it is imperative that teachers today attain proficiency in technical knowledge. Therefore, both teacher training and the evaluation of its quality should consider technical knowledge.

Second, the existing scales have no dimensional division of teacher training content. Through evaluating the quality of different dimensions of training, we can better determine deficiencies and make teacher training more targeted. Therefore, the evaluation of teacher training quality should be divided into dimensions.

The TPACK framework has gained widespread acceptance in both educational practice and academic research. It has arisen as a theoretical framework with the purpose of defining the knowledge needed for teaching in the digital technology era (Willermark, 2018). Therefore, teacher training must help teachers master TPACK knowledge; that is, it should include the framework’s seven dimensions. It is feasible and necessary to establish a teacher training quality scale from the perspective of the TPACK model.

Research questions

Given the gaps in extant research, this study aims to develop a Perceived Training Quality Scale for teachers from the perspective of the TPACK framework and test its reliability and validity. It is hoped that the establishment of such a scale can help evaluate the quality of a teacher training program or a certain period of teacher training programs to enable teacher educators to improve teacher training.

Research methods

Instrument development

Creating a literature-based item pool

Many studies have developed TPACK scales, and some have developed Chinese TPACK scales for Chinese teachers (Archambault & Crippen, 2009; Koh et al., 2010; Schmid et al., 2020; Schmidt et al., 2009). By reading and referring to the items of these scales and exploring the actual situation of Chinese teachers and teaching, this study developed the items of the Teachers’ Perceived Training Quality Scale. It is a five-point Likert scale with 37 items and 7 factors.

Gathering expert viewpoints on the content validity and linguistic structure of the scale

First, eight experts in teacher education—four professors, two associate professors, and two doctoral students—were asked to evaluate the scale and make recommendations. Second, five primary and secondary school teachers were asked to determine whether the scale items would be easily understood by other primary and secondary school teachers and to verify the scale’s content validity. Finally, two linguists were invited to check the grammar of the scale.

On the basis of the eight experts’ suggestions, we revised the scale many times, and the final scale contained 33 items and seven dimensions: the training quality of technological knowledge (TK-TQ), content knowledge (CK-TQ), pedagogical knowledge (PK-TQ), pedagogical content knowledge (PCK-TQ), technological content knowledge (TCK-TQ), technological pedagogical knowledge (TPK-TQ) and technological pedagogical content knowledge (TPACK-TQ).

The following are several questions on the scale: “Overall, what do you think of the quality of the training you received in improving teachers’ technical knowledge and level?” “Overall, what do you think of the quality of the training you received in improving teachers’ ability to adjust teaching according to students’ foundation and ability?” “Overall, what do you think of the quality of the training you received in improving teachers’ ability to choose appropriate technologies to improve the effectiveness of teaching methods and students’ learning ability?” Each question had five response options: very high, relatively high, average, relatively low, and very low.

Data collection

An internet-based questionnaire was created utilizing the Wenjuanxing platform. The questionnaire had three distinct sections. The first addressed the objective of the survey and underscored its voluntary nature. The second asked questions about the participants’ demographic attributes, such as their gender, age, and teaching experience. The third contained the items of the Teachers’ Perceived Training Quality Scale. The data were gathered by disseminating the questionnaire in September 2022, using a random accessible sampling approach. The questionnaire link was sent to K-12 teachers in Henan, China, and 426 teachers responded.

Participants

A total of 426 teachers from K-12 schools in Henan Province, China, completed the survey. According to the demographic data collected, 39.9% of the participants were male, and 60.1% were female. Additionally, 4.2% were from rural schools, 70.9% were from county schools, and 24.9% were from urban schools. The participants were aged between 23 and 60 years (M = 40.64, SD = 9.06), and their teaching experience varied from 1 to 39 years (M = 17.14, SD = 10.52). China’s Ministry of Education stipulates that teachers must receive at least 240 h of training cumulatively every five years, and there are many national, provincial, and municipal teacher trainings every year in China. All of the participants had participated in several types of teacher training in the past year.

Data analysis

First, SPSS software was used to conduct item discrimination. Second, to verify the structural validity of the scale, exploratory factor analysis (EFA) was performed on the collected data via SPSS software. Third, to validate the reliability of the scale, SPSS was used to calculate the Cronbach’s alpha (α) of the scale. Finally, confirmatory factor analysis (CFA) was conducted via Amos software.

Results

Item discrimination

To determine the effectiveness of the Teachers’ Perceived Training Quality Scale in distinguishing between individuals who possess high and low levels of competence, item discrimination statistics were computed. First, the total score of each individual was computed. The samples were then arranged in ascending order on the basis of their cumulative score. Among them, 27% of the samples with the highest scores were classified into the upper group, whereas 27% of all samples with the lowest scores were classified into the lower group. Finally, an independent sample t-test was performed to determine whether the difference between the upper and lower groups in terms of the score of each item and the total score of the scale was significant. The results are presented in Table 1.

Table 1 T-test results for the item means of the upper and lower 27-percent groups.
Full size table

As shown in Table 1, significant differences were found between the lower 27% and upper 27% groups in terms of the score of each item and the total score of the scale. The results reached a statistically significant level (p < 0.001) and indicate that each scale item effectively differentiated between individuals affiliated with the lower and upper groups.

Results of EFA

Exploratory factor analysis (EFA) was carried out to determine whether the seven theoretical factors were present in the data collected. Principal component analysis with the varimax rotation method was employed to extract factors during the EFA, and the values were computed for Kaiser‒Meyer‒Olkin (KMO) test and Bartlett’s test of sphericity (BTS). The KMO value was 0.975, and the BTS value was [chi-square (v2) = 14032.81; df = 528; p = 0.000]. These results suggest that the gathered data are appropriate for examination through the utilization of exploratory factor analysis (EFA) (Barbara, Tabachnick, 2007).

On the basis of the findings of the exploratory factor analysis, a total of 33 items were retained, which represents 68.39% of the total variance. Table 2 displays the factor loadings of these items, which were categorized into six distinct factors. The original constructs of five factors—namely, TK-TQ, CK-TQ, PK-TQ, TK-TQ, and PCK-TQ—were retained without any alterations. However, TPK-TQ was merged with TPACK-TQ. As a result, the combined factors of TPK-TQ and TPACK-TQ are referred to below as TPACK-TQ. Additionally, Factors 1 through 6 accounted for 23.97%, 13.10%, 10.93%, 10.85%, 9.72%, and 9.55% of the variance, respectively.

Table 2 Factor loadings, Cronbach’s alpha coefficients, and mean values for the survey.
Full size table

The reliability of the scale was assessed by computing Cronbach’s alpha, which was suggested to be 0.70 (Nunnally, 1978). The final results showed that the Cronbach’s alpha of the total scale was 0.942, and the Cronbach’s alphas of the TK-TQ, CK-TQ, PK-TQ, PCK-TQ, TCK-TQ, and TPACK-TQ subscales were 0.920, 0.918, 0.919, 0.916, 0.929, and 0.962, respectively. These results suggested that the entire scale and its subscales had adequate internal consistency. Therefore, the reliability of the survey for assessing teacher-perceived training quality was confirmed.

Results of CFA

Confirmatory factor analysis (CFA) is utilized to assess the unobservable structure of a measuring instrument for the purpose of validating constructs (Brown, 2015). To establish the validity of the scale’s seven-factor structure, this study conducted a confirmatory factor analysis on the gathered data and employed maximum likelihood. The CFA results verified the seven factors and the structures are presented in Fig. 2.

Fig. 2
Fig. 2
Full size image

Standardized CFA results.

The CFA findings indicated that the items’ factor loadings ranged between 0.794 and 0.893. Furthermore, the chi-square/degrees of freedom ratio (χ2/df), standardized root mean square residual (SRMR), normed fit index (NFI), comparative fit index (CFI), and root mean square error of approximation (RMSEA) were used to test the model data fit. RMSEA was determined to be 0.058, SRMR was 0.0356, NFI was 0.920, CFI was 0.951, and χ2/df was 2.448. The goodness-of-fit criteria accepted for the model after the confirmatory factor analysis were as follows: 0.05 < RMSEA < 0.10, SRMR < 0.05 (Schermelleh-Engel et al., 2003), NFI ≥ 0.90 (Hair, 1998), CFI ≥ 0.90 (Bentler (1990)), and χ2/df < 3 (Kline, 2005). According to the findings, the stated model exhibited satisfactory fit index values.

Discussion

Teacher training plays a significant role in determining the quality of teachers and the academic achievement of students. Therefore, evaluating the quality of teacher training is important. However, existing evaluation methods, such as observing teachers’ performance and measuring students’ performance after training, cannot quickly and conveniently evaluate teacher training quality. Moreover, the existing Teachers’ Perceived Training Quality Scale does not consider information technology and does not evaluate all the dimensions of teacher training; thus, it cannot evaluate all aspects of teacher training quality. The TPACK model has been widely accepted. Therefore, this study developed a scale for assessing teachers’ perceived training quality from the perspective of TPACK.

Six factors were acquired in the exploratory factor analysis, of which five were as expected, whereas TPACK-TQ and TPK-TQ were merged into one factor, which also occurred in previous studies. For example, Qiu et al. (2022) studied the factor structure of TPACK among 286 prospective teachers of Chinese as a second language. The researchers utilized exploratory factor analysis, leading to the identification of a six-factor structure. The findings suggested that teachers were unable to discern the demarcation lines between TPK and technological content knowledge (TCK) and between TPK and TPACK. Additionally, Chai et al. (2015) examined the construct validity of the TPACK scale by surveying preservice primary school teachers participating in ICT programs in Singapore and reported that one of the TCK items merged with TPACK. They theorized that this may have occurred because ICT tutors do not pay enough attention to TCK.

This may be due to the following reasons: first, the teachers surveyed were unable to distinguish between TPACK training and TPK training. Second, there is no good distinction between these two kinds of knowledge in teacher training. Third, as TPACK and TPK may often be combined in teacher training, teachers’ evaluations of these training methods are more consistent, resulting in a combination of factors.

Owing to the theoretical basis of the TPACK model, we still set the model as having seven factors in the confirmatory factor analysis. The results showed that the model fit well. Therefore, the seven-factor structure was still valid. Moreover, the total scale and subscales had high reliability.

Finally, the scale contained 33 items and seven dimensions: the training quality of technological knowledge (TK-TQ), content knowledge (CK-TQ), pedagogical knowledge (PK-TQ), pedagogical content knowledge (PCK-TQ), technological content knowledge (TCK-TQ), technological pedagogical knowledge (TPK-TQ) and technological pedagogical content knowledge (TPACK-TQ).

The survey of teachers in Henan Province revealed that they could distinguish among different trainings and evaluate their quality, which indicated that the scale could effectively measure the quality of different trainings perceived by teachers. In addition, there was a significant positive correlation between teachers’ perceptions of the quality of different types of training. This result was in line with our expectations because the city and school where a teacher was located would lead to positive correlations among the quality of the different trainings they received.

In conclusion, the findings indicate that the Teachers’ Perceived Training Quality Scale is a reliable and valid instrument to evaluate the quality of teacher training and may provide a useful resource for future research. It can measure teachers’ perceptions of the quality of the training they receive so that they can change its content and methods and improve its quality.

Conclusion and limitations

Drawing from the theoretical framework of technological pedagogical content knowledge (TPACK), examining the previous self-perception scale developed for teachers’ TPACK, and considering the specific circumstances prevalent in China, this study designed the Teachers’ Perceived Training Quality Scale, which has 33 items and seven dimensions. The scale was subsequently used to survey 426 K-12 teachers in Henan, China. Its validity was evaluated through both exploratory factor analysis and confirmatory factor analysis, and its reliability was also assessed. The results show that the scale has good reliability and validity. Therefore, the Teachers’ Perceived Training Quality Scale developed in this study is a reliable and valid tool for assessing the quality of teacher training.

This study has the following shortcomings. First, the TPACK and TPK factors were merged into one factor in the EFA, and this step may need to be identified more clearly in future studies. Second, this study used the convenience sampling method and had fewer samples. At the same time, there were fewer teachers in rural schools in the sample. These factors may have led to deviations in the results of the study. It is recommended that the scale be tested on large samples to confirm its validity and reliability. Additionally, it should be tested in different cultures to confirm its psychometric properties. An English version of the scale was prepared and is presented in Supplementary Appendix 1. Third, the Teachers’ Perceived Training Quality Scale is a self-reported scale that reflects teachers’ own perceptions, which may lead to certain errors. Future research can triangulate self-declarations with other measures, such as lesson observations or performance assessments, to overcome potential biases.