Introduction

The effective implementation of health projects holds paramount importance in enhancing overall health outcomes and the quality of services1. It is crucial to study and assess the performance of projects to identify areas for improvement and enhance the efficiency and effectiveness of health systems2. Performance evaluation of projects can be categorized into ex-ante performance evaluation, mid-term performance evaluation, and ex-post performance evaluation, depending on the stage of implementation. Common performance evaluation frameworks include the Balanced Scorecard and the Donabedian Mode3,4. In recent years, significant progress has been made in researches on the performance of health projects, with researchers developing various frameworks and corresponding indicators to assess the effectiveness of health projects5,6,7,8. These frameworks assess a variety of factors, including patient outcomes, service quality, efficiency, and cost-effectiveness9. The findings of these studies have made significant contributions to understanding the factors influencing project performance, providing valuable insights for improving the performance of health projects.

Currently, most studies, including aforementioned frameworks, primarily evaluate the ex-post performance of health projects. Ex-post performance evaluation focuses on measuring the final performance of projects. In contrast, ex-ante performance evaluation can arguably be more critical, as it can effectively regulate project budgets by comparing project expenditures with market prices, thus reducing budget size and preventing fund wastage. Pre-evaluation has broad applications in fields such as environmental science, public funding projects, agricultural research and operations management10,11,12,13. Roper et al. established a pre-evaluation framework for publicly supported R&D projects based on the private and social benefits of the projects and their shares14. Fonseca and Gibson redefined the components of existing frameworks into a more applicable pre-assessment framework, focusing on Environmental Impact Assessment (EIA)15. In the field of health projects, a study in Myanmar conducted a cost-effectiveness analysis using a decision tree model, evaluating the ex-ante performance of a maternal and child health project. It compared the cost per disability-adjusted life year averted by the project with the current situation16. Overall, there is limited application of ex-ante performance evaluation in the current health project performance assessment, and further research is needed on the development of evaluation frameworks.

In April 2021, China National Health Commission issued the implementation plan for budget performance management in health projects, emphasizing the necessity for ex-ante performance evaluation of health projects. Health project implementing organizations must submit project self-assessment reports, evaluate the performance of each project using the ex-ante performance evaluation framework, and obtain assessment scores. Projects are then ranked based on the scores of ex-ante performance evaluation. In general, budgets are allocated to projects based on their rankings, with higher-ranked projects receiving priority. Consequently, in cases of limited financial resources, non-essential projects may be rejected.

However, under the current budget management system, ex-ante performance evaluation has not been fully implemented, because of the difficulty of accurately assessing the effectiveness or efficiency of projects, the complexity of obtaining performance information, inadequate use of performance information for decision-making, and weak performance incentives or constraints. Continuous research in this area to strengthen the health system and ensure the provision of quality services to patients is highly worthwhile.

The study attempts to develop an ex-ante performance evaluation framework for health projects and establish a project ex-ante performance self-assessment report template. We are evaluating 698 health projects in Tianjin between 2021 and 2022 using our ex-ante performance evaluation framework. This enables us to gain insights into the ex-ante performance of health projects in China, identify issues and explore the factors that influence the ex-ante performance of health projects, and make suggestions about improvement strategies.

Methods

Ex-ante performance evaluation framework

The ex-ante performance evaluation framework development followed the three stages outlined by McMeekin et al.: (i) identifying relevant data, (ii) developing the framework, (iii) validating, testing and refining the framework17.

Relevant data includes literature review, previous research result18,19, and interviews with key figures in health administrative departments, project implementation organizations and project beneficiaries. To develop the framework, the findings from existing research were pooled and organized by the authors to create indicators and key points. The preliminary ex-ante performance evaluation framework was refined, validated and tested by the experts from Tianjin Medical University and Tianjin Basic Public Health Service Quality Control Center. We invited experts from the aforementioned organizations to conduct weight analysis on 5 indicators using the Analytic Hierarchy Process and adjusted the weights of each indicator appropriately to make their scores an integer that is easy to score.

The ex-ante performance evaluation framework primarily examines five aspects, namely, necessity of project, economy of input, rationality of performance indicator, feasibility of implementation plan, and compliance of financing. By comprehensively considering the preparation of health projects, these five aspects can offer a sufficient reference for project process management and project performance prediction and possess a authority and scientific validity. The details are presented in Table 1.

Table 1 The ex-ante performance evaluation framework of the health project in China.
Full size table

This indicator system has been transformed into a project ex-ante performance self-assessment report template to obtain the necessary information for ex-ante performance evaluation. Experts from Tianjin Medical University and Tianjin Basic Public Health Service Quality Control Center have reviewed the ex-ante performance self-assessment template and provided suggestions for modification. Multiple health project leaders have filled out ex-ante performance evaluation reports to test the usability, comprehensiveness, and logical clarity of the ex-ante performance evaluation report template. During this process, further modifications were made to the ex-ante performance evaluation report template to form the final ex-ante performance self-assessment report template (“Supplementary Material”).

Data collection

Sample projects cover a total of 698 health projects in Tianjin between 2021 and 2022. Out of these, there are 245 health projects scheduled for 2021 and an additional 453 health projects planned for 2022. These health projects encompass all projects implemented by the municipal health administrative department and its affiliated units between 2021 and 2022. The leaders of these health projects have prepared ex-ante performance self-assessment reports according to the template. The health projects mainly involve the development of traditional Chinese medicine, the training of health professionals, the provision of basic public health services, health management, the construction of public health systems, the prevention and control of infectious diseases, the improvement of women’s and children’s health, and the development of health information technology.

We gathered fundamental data on health projects, including the project name, project content, project budget, project frequency, whether it pertains to epidemic prevention and control, project implementation organization, project objective, project funding category, and project year. We have manually classified the project content, project objective and other information. Manual classification information is classified separately by two researchers. If there is any inconsistency, the two researchers will discuss and reach an agreement to classify the project into the closest existing category. Attributes of various projects are presented in Table 2.

Table 2 Attributes of various projects.
Full size table

The study protocol was reviewed and approved by The Ethical Committee of Tianjin Fourth Central Hospital (SZXLL-202304-OKY02) in accordance with “International ethical guidelines for biomedical research involving human subjects(2002)” developed by Council For International Organizations of Medical Sciences in collaboration with World Health Organization. All participants provided informed consent prior to taking part in the study.

Measurement

Experts from 6 different fields, namely health services, personnel subsidies, discipline construction, equipment purchase, physician training, and engineering, were recruited to evaluate the health projects within their respective categories. Three experts with intermediate or higher professional titles are employed in each field. Choose the most senior expert from each field to serve as the team leader, who will then guide the other two members to successfully complete the scoring task. All experts were trained on the ex-ante performance evaluation framework of the health project and qualified.

Experts score 5 indicators for each health project based on self-evaluation reports, but not scoring for each key point. The degree of conformity between project characteristics and key points is the important basis for scoring this indicator. The higher the degree of conformity, the higher the score. If the total score of two experts for the same project is less than or equal to 10, then calculate the average score. If the difference between the total scores of two experts for a project is greater than 10, the team leader will re-score the project and calculate the average score of the team leader’s score and the closest score to the team leader’s.

Statistical methods

The statistical analysis was carried out by SPSS 25.0. Descriptive statistics were utilized to characterize the sample health project and ex-ante performance evaluation scores, as well as the scores for each dimension. The measurement data were identified using “mean ± standard deviation” while the count data were identified based on frequency and composition ratio. Comparisons between groups were conducted using either the independent-samples T Test or one-way ANOVA. To analyze the factors influencing the ex-ante performance evaluation scores of the health projects, multiple regression analysis was employed with a test level of α = 0.05.

Results

Ex-ante performance of health projects

The overall ex-ante performance evaluation score for the sample projects was (91.19 ± 8.02). The sub-scores for necessity of project was (14.74 ± 1.85), economy of input was (34.45 ± 2.43), reasonableness of performance indicator was (8.27 ± 2.28), feasibility of implementation plan was (23.99 ± 3.84), and compliance of financing was (9.75 ± 1.47).

The sample health projects primarily mainly were health service projects, once projects, projects budget less than 5 million yuan, non-epidemic prevention and control projects, projects implemented by medical institutions, projects aiming to protect other specific targets, non-central direct fund projects, and projects scheduled for 2022. The details are presented in Table 3.

Table 3 Basic information of health projects in China.
Full size table

Mono-factor analysis of ex-ante performance of health projects

The results of the independent samples t test indicated that the project’s necessity, economy of inputs, and overall ex-ante performance score for the 2022 project were significantly higher than those of 2021 projects (P < 0.05). Furthermore, the necessity of the project, feasibility of the implementation plan, and compliance of financing scores for projects with < 5 million yuan were significantly lower than those of projects with ≥ 5 million yuan (P < 0.05). The compliance of financing scores of the epidemic prevention and control program were significantly higher than that of the non-epidemic prevention and control program (P < 0.001). The reasonableness of the performance indicators score of the epidemic prevention and control program was significantly lower than that of the non-epidemic prevention and control program (P < 0.05). The necessity of project and total ex-ante performance scores of central direct funds projects were significantly lower than those of non-central direct funds projects (P < 0.01).

The one-way ANOVA results indicated that the differences were statistically significant (P < 0.001) when comparing the reasonableness of performance indicators among projects of various categories. Compared to engineering projects, the reasonableness of the performance indicators for health service projects are 0.78 scores higher, the reasonableness of the performance indicators for personnel subsidy projects are 0.41 scores higher, the reasonableness of the performance indicators for discipline construction projects are 1.32 scores higher, the reasonableness of the performance indicators for equipment acquisition projects are 0.10 scores lower, the reasonableness of the performance indicators for physician training projects are 1.46 scores higher (P < 0.05).

The one-way ANOVA results indicated that the differences in the reasonableness of performance indicator scores were statistically significant when projects implemented by different organizations were compared(P < 0.001). Compared to projects implemented by other organizations, the reasonableness of the performance indicators for projects implemented by government are 1.93 scores lower (P < 0.05).

The one-way ANOVA results indicated that the differences were statistically significant (P < 0.01) when comparing the reasonableness of performance indicators with varying project objectives. Compared to other target projects, the reasonableness of the performance indicators for bond repayment project are 0.74 scores lower (P < 0.05).

Table 4 provides detailed information on this comparison.

Table 4 Ex-ante performance of health projects in China.
Full size table

Multi-factor analysis of ex-ante performance of health projects

The regression results indicated that factors such as whether it was an epidemic prevention and control project, project budget, project content, project objective, and project implementation organization were influencing the total score of ex-ante performance of health projects (P < 0.05). The regression results indicated that project budget, year, project content, project objective, and project implementation organization were the factors influencing the necessity of project (P < 0.05). The regression results indicated that factors such as whether it was an epidemic prevention and control project, project budget, project content, project objective, and project implementation organization were significant factors influencing the economy of input (P < 0.05). The regression results indicated that the factors influencing the reasonableness of the performance indicators include whether it was an epidemic prevention and control project, project budget, project content, project objective, and project implementation organization (P < 0.05). The regression results showed that factors such as whether it was an epidemic prevention and control project, project budget, project content, project objective, and project implementation organization influenced the feasibility of implementation plan (P < 0.05). The regression results showed that factors such as project budget, project content, project objective, and project implementation organization significantly influenced compliance of financing (P < 0.05). Table 5 provides detailed information on these factors.

Table 5 Factors influencing the ex-ante performance of health projects in China.
Full size table

Analysis of deduction reasons

The overall project performance is commendable, but there are still some deficiency. Following expert evaluations, discussions were held with the experts to gain insights into the deductions in scores. Regarding the necessity of the projects, discrepancies were identified in the alignment of certain projects’ content with national policy, and some projects overlap with existing ones. Additionally, the beneficiaries of some projects are not the service recipients within this jurisdiction. Regarding the economy of input, the budget of certain projects is not sufficiently reasonable, with prices appearing to be either excessively high or low in comparison to the market price. Furthermore, there is a mismatch between the total cost of some projects and their individual items. Regarding the reasonableness of performance indicators, some projects’ indicators are not refined or quantified. The relevance of some indicators to the project content is inadequate. The expected values of some indicators were deemed unreasonable. Moreover, some projects have duplicate or missing performance indicators. Lastly, the primary performance indicators of some projects are not effectively linked to secondary indicators. Regarding feasibility of implementation plans, some project implementation plans are simplistic, while others utilize unreasonable project implementation methods leading to increased costs. Additionally, certain project implementation organizations lack essential resources, including organizational and management capacity, as well as adequate staffing for the projects. Regarding compliance of financing, discrepancies were identified in certain projects where the main expenditure contents did not align with the support scope of special funds, and certain projects include the procurement of equipment or materials that are prohibited for purchase in their expenditure contents.

Discussion

China National Health Commission issued an implementation plan for budget performance management in health projects in April 2021, highlighting the requirement for all health projects to undergo ex-ante performance evaluation prior to implementation. The study attempts to develop an ex-ante performance evaluation framework for health projects and establish a project ex-ante performance self-assessment report template to gather relevant information. The framework development process followed an established methodology17. The ex-ante performance evaluation framework primarily examines five aspects: necessity of project, economy of input, the reasonableness of performance indicator, feasibility of implementation plan, and the compliance of financing. Weight analysis was conducted using the Analytic Hierarchy Process. Subsequently, the indicator system was transformed into a project ex-ante performance self-assessment report template.

The study also attempts to analyze the ex-ante performance of health projects in China, identify issues and explore the influencing factors and propose improvement strategies. Sample projects from Tianjin between 2021 and 2022 were evaluated using a self-assessment report and fundamental data obtained from the Tianjin municipal government information disclosure platform. Experts utilized the ex-ante performance evaluation framework to score the projects.

The study found that the reasonableness of performance indicators for physician training projects are 1.46 scores higher than engineering projects. This improvement can be attributed to the China Medical Association’s comprehensive indicator system for resident physicians, which provides a framework for developing performance indicators. In addition, previous studies focusing on the evaluation system for resident physicians undergoing standardized training may relate to the higher scores for the reasonableness of performance indicators in physician training projects20,21,22,23. On the other hand, the reasonableness of the performance indicators for equipment acquisition projects is 0.10 scores lower than engineering projects. Equipment acquisition projects tend to prioritize the quantity of purchased equipment while overlooking factors such as category, quality, efficiency and utilization rate of the equipment. The lack of attention to essential aspects of equipment acquisition may contribute to the lower scores in reasonableness of performance indicator for these projects. Studies have also highlighted common challenges in the equipment purchasing process, including a lack of comprehensive expertise and differing opinions among decision-makers24, which could further impact the reasonableness of performance indicators for equipment acquisition projects.

Compared to other targets projects, the reasonableness of performance indicators for bond repayment project is 0.74 scores lower. This discrepancy may be attributed to certain bond repayment projects lacking focus on key aspects such as the assets that can be generated by the project, the provision of health services after project completion, and the project’s alignment with the overall health strategy. The oversight of these critical performance indicators in bond repayment projects could contribute to the lower scores of the reasonableness of performance indicators for bond repayment projects.

The necessity of the project, feasibility of implementation plan, and compliance of financing scores for projects with budgets less than 5 million yuan were significantly lower than those of projects budgets equal to or more than 5 million yuan. Projects with larger budgets tend to receive more elaborate planning and review due to the increased attention from relevant leaders. The input of substantial funds triggers a scale effect, leading to more fruitful outcomes. Moreover, research suggests that adequate project funding positively impacts project performance by enabling the allocation of essential resources, equipment, and personnel, thereby enhancing project outcomes25. This correlation may explain the higher scores observed for projects with budgets equal to or greater than 5 million yuan.

In contrast, the reasonableness of performance indicators for projects implemented by government is 1.93 scores lower compared to projects implemented by other organizations. Government projects often face challenges such as aligning policy targets with project outcomes, encountering bureaucratic hurdles, limited flexibility, and slower decision-making processes26,27,28. Moreover, the government’s heavy workload may hinder its ability to thoroughly address every detail of each project, potentially leading to lower scores in the reasonableness of performance indicators for projects implemented by government.

The reasonableness of the performance indicators score of the epidemic prevention and control project was significantly lower than that of the non-epidemic prevention and control program. In 2022, epidemic prevention and control projects garnered more attention due to their specific objectives, resulting in increased budget. Given the urgent character of most epidemic prevention and control projects, there is a dearth of past experience to draw from, resulting in a suboptimal level of preparedness for epidemic prevention and control projects. This may be related to the lower scores of the reasonableness of performance indicators for epidemic prevention and control projects. Now, China has optimized the epidemic prevention and control policies, easing restrictions and potentially reducing the need for epidemic prevention and control projects in the future.

The ex-ante performance of sampled health projects is generally good, yet certain issues persist, particularly concerning the reasonableness of performance indicators and the feasibility of implementation plans. These problems are linked to the time constraints in the annual project declaration process, which often result in inadequate time for the project’s responsible personnel to conduct a thorough and scientific design of the project implementation plan. Moreover, these issues are associated with the project manager’s work attitude and technical proficiency.

This study also has some limitations. Due to the lack of relevant data, the study failed to validate the correlation between ex-ante performance and ex-post performance. The two-year data spanning is insufficient to establish a performance benchmark that would serve as a comprehensive reference for future evaluations. In the future, further research could be conducted on the basic of more comprehensive and detailed data.

Conclusions

We developed an ex-ante performance evaluation framework for health projects and establish a project ex-ante performance self-assessment report template for collecting relevant information. We could evaluate the ex-ante performance of health projects using the ex-ante performance evaluation framework. This enabled us to gain insight into the ex-ante performance of health projects in China, identify issues and explore the factors that influence the ex-ante performance of health projects and make suggestions about improvement strategies.

Project implementation organization should meticulously oversee the projects that are regularly carried out in our daily work, refine the documents and workflow to improve health project management. Furthermore, it is crucial to enhance business training and draw on external experience to elevate the awareness and proficiency of project managers regarding health project budget performance management.

Government should track the performance changes over the years of projects carried out frequently, analyze the cost calculation standards of similar expenditures in different projects, and form benchmarks for project evaluation, improve the ability of ex-ante performance evaluation, and eliminate ineffective projects through ex-ante performance evaluation.