Satisfactory completion of dental vocational training in Scotland: A system of assessment

Abstract

Formal assessment of dental vocational training (DVT) has been an issue since 1993 when DVT became mandatory for graduates wishing to practice within the NHS. There have been a number of other drivers for change ,including concerns about the capabilities of new graduates, a lack of standardisation of the training experience¹ and accountability for standards of training. All of these factors contributed to the decision, by the Scottish Council for Postgraduate Medical and Dental Education (now part of NHS Education for Scotland - NES), to support a programme of research into the development of a competency-based system of assessment for DVT². During the life of this project the argument for introducing assessment of DVT and General Professional Training (GPT) has become ever stronger.

Main

'It appears the age of accountability in education and training is upon us and the dental profession should welcome this opportunity for change.'

A commitment to improvements in quality within the NHS was set out by the government in the NHS Plan³, following which specific recommendations for change within dentistry emerged 'to meet the needs and wishes of patients', with particular emphasis being placed upon the provision of high quality dental care⁴. In a pledge to improve quality, the report Modernising NHS Dentistry states that the government will 'root out poor performance more effectively' and 'strengthen the self regulation of the dental profession so that it is easier to deal with poor performance before serious harm is done'⁵. Also, the intention to introduce significant change (including assessment) in the delivery of medical education and training was clearly laid out in recent consultation documents Unfinished Business and the proposals to set up the Postgraduate Medical Education and Training Board.⁶,⁷,⁸ Reform in this field is now imminent and a key part of the changes is better (evidence–based) assessment based upon defined competencies.

It appears that the age of accountability in education and training is upon us and the dental profession should welcome this opportunity for change. Ensuring the competence of those completing DVT and GPT will maintain high standards of patient care and improve the quality of the training. Some of the difficulties in managing poor performance of Vocational Dental Practitioners (VDPs) are a result of the wording of the NHS regulations,⁹,¹⁰ in that VDPs have only to complete the period of training, rather than having to meet any predetermined standard of competence during the DVT year. To ensure accountability for the quality of training and patient safety the profession and the Scottish Executive Health Department have agreed that the DVT regulations will to be changed to include the need for satisfactory completion of DVT, and that a robust system of assessment should be introduced to support this decision.

If the assessment system to support satisfactory completion is to be considered robust and fit for purpose, it should be based on evidence and meet the published criteria for good assessment, in being valid, reliable and feasible. Recent research into the assessment of health professionals has focussed on competency-based or outcomes-based assessment. Dentistry is no exception to this general trend and a competency-based approach to assessment has been recommended by a number of authors.²,¹¹,¹²

'The identification of competencies for DVT was the first and crucial step in the development of an assessment system for this training.'

A recent study investigating the strengths and weaknesses of existing assessment systems in dentistry identified a tendency to record numbers rather than quality¹¹. DVT schemes across the UK have traditionally used a logbook, the Professional Development Portfolio (PDP) to monitor training. Although the PDP by definition is not 'assessment', it is regularly completed by the VDP during training and it aims to promote learning through reflection on clinical encounters. However, there is growing evidence which suggests that whilst logs are useful to the coordinators of training programmes in monitoring the extent and types of experiences encountered during the training, they may be of little educational value to the trainee.¹³,¹⁴ If a log of experience is used to gauge the quality of training in terms of content, there should be an evidence-based list of training objectives which allows comparison of progress against targets. A comprehensive 'competency document' containing such information has been produced as a result of three years research by NES. The extensive validation of the competencies required in DVT make this document unique.¹⁵ A second version of this document has also been produced which considers the continuum of education and training, and links the competencies expected in DVT/GPT with those prescribed for undergraduates by the GDC in the First Five Years – 2,¹⁶ although the evidence-base or validation of the undergraduate competencies has not been published.

The identification of competencies for DVT was the first and crucial step in the development of an assessment system for this training. The assessment system must ensure that all areas of competence are sampled appropriately and assessed to a level which can provide evidence for judgement against all-round competence. There are many challenges involved in the development of competency-based assessment for clinical practice which involves a complex combination of skills including technical ability, decision making, clinical judgement, communication, professionalism and practice manage-ment.¹⁷ Different methods of assessment may be required for these different areas if the necessary validity and reliability is to be ensured.¹⁸ However, a priority for educationalists is often the less exciting prospect of feasibility. This paper describes a system of assessment which has been developed with the aim of meeting this challenge.

Assessment System Overview

The DVT assessment system has been developed over the last three years taking account of international educational research evidence and extensive consultation with experts and stakeholders about methodology and patient care issues. Where possible, small scale pilots involving different assessment methods were carried out, within individual DVT schemes, as preliminary feasibility studies.

To promote validity and reliability of the system for assuring satisfactory completion four different methods of assessment have been integrated to ensure triangulation of evidence. Fig 1 shows how the system is organised. The centre of the diagram illustrates the overall aim of obtaining evidence of 'all-round competence' with the four domains from the DVT competency document – clinical, communication, professionalism and managerial - emerging as the major areas requiring evaluation. The blue sides to these blocks indicate which methods of assessment within the system will cover each of the competency domains. The four methods employed are the Longitudinal Evaluation of Performance (LEP), a Patient Assessment Questionnaire (PAQ), a Test of Knowledge (ToK) and an assessed programme of the Management of Medical Emergencies (MME).

Figure ig1

The LEP is a method of continuous assessment and is used throughout the training. It ensures the assessment of competencies across all domains by direct observation of the trainee delivering treatment for patients.¹⁷ Supporting this method in the clinical domain is MME training with assessment specifically targeting the competencies required to ensure safe basic life support. The PAQ is used to get a judgement of VDP competence in the communication and professionalism domains.¹⁹ It is unique in that it uses direct patient involvement in the assessment of graduates. The ToK focuses on assessing the areas of knowledge included in the managerial domain which are generally not covered in the undergraduate curriculum and this area is considered 'new ground' by many VDPs.

The consideration of practitioner time and resources has been paramount in this study, and care has been taken not to 'over-assess' or create other risks to successful implementation of satisfactory completion. Assessment methods have been chosen for their educational value and their feasibility so that they may be incorporated into the existing DVT programme.

Indicators of quality assessment (reliability, validity, feasibility) are important for each of the methods adopted. However, the overall success of the system rests on the quality indices being demonstrated for the entire system. Important outcome measures will link with the key themes underlying the project; 'Does the assessment system identify poor performance?', 'Does the assessment system help all VDPs progress through their training?', 'Does the assessment system increase regulation within DVT for trainers?'.

Evidence supporting the use of each of the assessment methods and justification of the design of the system is outlined below.

Longitudinal Evaluation of Performance (LEP)

It is important that assessment of DVT detects poor performers yet is of educational benefit to all VDPs. Individuals will inevitably have different strengths and weaknesses when they enter their DVT year, and will progress at different rates depending on a variety of factors such as the range and volume of experience available and the support given by trainers. It would be difficult to defend the validity of a single, (summative) assessment used at the end of training and this approach would have limited educational value.²⁰ Continuous assessment is used to monitor the progress of each trainee from the onset of training and provide regular and valuable feedback (from trainers). A key requirement is that the standards expected upon completion of training, are made clear to VDPs and evidence of when that requirement has been met is collected and recorded.

The LEP involves direct observation of VDP performance in general practice, on a regular basis, by trainers. A detailed rationale for the design of the LEP has been published elsewhere.¹⁷ Ratings of performance are given against a nine point scale covering the criteria 'Need Improvement'/'Satisfactory'/'Superior', across the skill areas that embrace the competency domains. Trained evaluators give ratings using their judgement against the standard expected upon completion of this training as a reference point. This is important for feedback as the VDP is then able to judge exactly how much improvement is required for satisfactory completion. Concerns were initially expressed that this approach could be demotivating in the early stages of training as on some occasions lower ratings would be inevitable. For example with more complex procedures or those clinical situations seldom experienced as an undergraduate. However, preliminary results have suggested that when VDPs were fully informed of the process from the onset, demotivation is not a major issue. Concerns have been further reduced as a result of trainer and VDP feedback from preliminary pilot studies, which lead to the descriptor for the lower ratings on the 9-point scale being changed from 'Unsatisfactory' to 'Need Improvement'.

The number of LEPs carried out during the training and the number of different evaluators contribute to the reliability of this method of assessment. The difficulty of obtaining multiple evaluators in a general practice setting is acknowledged¹²,¹⁷ although it is easier for those in GPT who spend time in the hospital or community dental services where a wider range of evaluators is available. This issue has been addressed by using Advisers to carry out LEPs (in addition to the trainer) to allow a minimum of two evaluators. In addition, the regular use of this assessment i.e. 1 LEP per week (total 46 LEPs throughout the DVT year) should increase the reliability of this assessment tool.

High content validity can be ensured as the clinical focus of each encounter assessed is noted for each case, in addition to a more detailed description of the procedure. A range of competencies encompassing all areas within the clinical domain, different patient age groups and different case complexities, must be completed throughout the year.

The principle aim of the LEP is to identify the VDP's strengths and weaknesses as early as possible, thus allowing training to target areas requiring support and to provide evidence of improvement with time. No penalty is given for LEP ratings given in the 'Need Improvement' range, as long as evidence of improvement to the necessary standard is subsequently provided before the end of DVT. The proposed criteria for satisfactory completion of DVT for each of the methods within the system of assessment are given in Table 1

Table 1 Criteria for Satisfactory Completion of DVT.

Patient Assessment Questionnaire (PAQ)

A number of policy documents have highlighted the NHS commitment to greater patient involvement in a whole range of areas, including patients' views on the service.³,²¹,²² Extensive research, carried out by the American Board for Internal Medicine, on the use of patient questionnaires for the assessment of resident doctors has demonstrated that if used under certain conditions this method can be a useful tool in the identification of 'humanistic skills'.²³ The validity of a PAQ should be high considering that the judgements are being made by the people who are being treated by the dentist. However, the biggest justification for this method lies with its feasibility in the general dental practice setting. The PAQ allows specific evidence to be collected on the performance of the dentist with very little or no time commitment from either the VDP or trainer.¹⁹ At designated time periods within the training, PAQs are handed out (by reception staff) to consecutive patients following their consultation. Questionnaires may be completed in the waiting room and the sealed reply paid envelope handed to reception staff for posting, or alternatively patients may wish to complete the questionnaires at home and post them to the collation centre. Two rounds of data collection (in November and February) allow feedback to be considered and poor performance addressed if necessary. For reliability purposes a minimum of 20 PAQs are required for analysis each time. VDPs in practices with a low return rate can be sent more PAQs if required.

The PAQs have been designed using Teleform software²⁴ and can be optically read, allowing automated data entry. Results can be analysed quickly and feedback reports quickly sent to VDPs and trainers. Average scores are given for each of the questions within the PAQ (based on judgements received along an EVGFP (Excellent, Very good, Good, Fair, Poor) scale. As with other elements of this assessment system, no penalties are given for lower ratings as long as improvements are demonstrated, through collected evidence, before the end of training. Data collected from preliminary pilot studies is extremely positive.¹⁹

Test of Knowledge

Whilst the level and application of knowledge during practice will be assessed throughout the training using the LEP it was considered appropriate that 'new' knowledge specific to DVT (including NHS rules and regulations, financial issues, legislation relevant to practice etc) was assessed in order to increase triangulation of evidence in this domain. It was considered important that this test was not seen as another 'final exam' but rather it should be a learning tool as well as an assessment method. Questions on the test are drawn from a large bank and randomised in order that VDPs may attempt the test on numerous occasions throughout training, with feedback being given to promote learning. Whilst many study day programmes have been successful in delivering courses in relevant areas there are significant variations between different schemes and regions. Realistically it could not have been otherwise given the lack of a validated curriculum for DVT.

VDPs can access the Test of Knowledge throughout the year, with the criteria for satisfactory completion of DVT being that a pass certificate is present as evidence in their portfolio at the end of training (Table 1).

Management of Medical Emergencies

Assessment of how VDPs manage medical emergencies is included in the system of assessment to address specific obligations with regard to patient safety. Serious medical emergency situations are rare in general dental practice but additional training in this area was considered important in terms of accountability and for the confidence and competence of those completing DVT 'satisfactorily'.

Several medical emergencies potentially relevant to dental practice were identified and incorporated into six scenarios (Table 2). Training is carried out in small groups using a highly sophisticated manniquin at the Scottish Clinical Simulation Centre (SCSC) at Stirling Royal Infirmary. The course is built into the study day programme and facilitated by a faculty of dental trainers in addition to SCSC staff. The VDPs are assessed to Basic Life Support standard. The training incorporates videos of performance and debriefing sessions on all scenarios performed in teams.

Table 2 Medical Emergencies incorporated into the DVT simulator training

Record of Progress and Achievement

All evidence in support of satisfactory completion of DVT is collected in a portfolio or Record of Progress and Achievement (RPA) which is given to VDPs during their induction when they receive training on the assessment system. Detailed information on the criteria for satisfactory completion (Table 1) is included in this document and is thus available at all times to the VDP. In addition to evidence collected using the four assessment methods described above, the RPA holds the contract of service, evidence of attendance at study days and trainer / adviser statements on the performance of the VDP. All criteria must be fulfilled before satisfactory completion can be awarded.

Satisfactory Completion of DVT - A National Pilot Study in Scotland

'Satisfactory completion of DVT should be regarded as an award for achieving the high standard of competence that is required by DVT authorities.'

All VDPs and GPTs who started their training in August 2002 in Scotland used this system of assessment in a national pilot study of satisfactory completion. The pilot has been supported by Scottish Executive which provided additional funding for the trainers involved in the process. Implementation has included training courses on assessment for all trainers and every VDP involved. Data on the use of this system with over 100 VDPs will be available for analysis from September 2003.

Evaluation of individual methods will provide important information which will be used to refine the system if necessary. However, the 'holy grail' of evaluation is that the 'whole' system works together to fulfill the original aims of assessment of this training. Satisfactory completion of DVT should not be regarded as another 'hurdle' to clear before becoming a principal within the NHS, but rather an award for achieving the high standard of competence that is required by DVT authorities.