PROTECTING VALIDITY IN PROFESSIONAL DISCUSSION ASSESSMENTS

How to get the most out of Professional Discussion methods in student assessment…

Professional Discussion (PD) is playing an increasingly prominent role in the assessment of vocational and technical qualifications.

Not only is it a useful way to address any gaps in knowledge, skills and behaviours that are harder to assess through other methods (such as practical demonstrations or a portfolio), but interviews like these are also an opportunity to support students in developing new communication skills that will benefit them in their future careers.

Essentially a two-way discussion between an apprentice and an independent assessor, PDs work best as a means of measuring the depth of understanding an apprentice has about their role.

But how can an assessor ensure these discussions are as accurate and reliable as possible?

Protecting validity

The very nature of PD leaves it open to questions of validity. Unlike the formal structure of a written exam, for example, the way a conversation flows can vary greatly between different people and various factors can influence a student’s answers that wouldn’t be at play when answering questions on paper.

One common factor that can have a bigger influence on an interview-style assessment is for apprentices to be distracted by nerves. The stressful feelings they might have about taking part in an interview scenario could prevent them from fully engaging in a two-way discussion, which might mean they’re unable to show how much they truly understand.

There are several measures that can be taken to safeguard the validity of a PD assessment, both by the apprentice themselves and the assessor.

For instance, apprentices should take part in mock PDs as much as possible and have the chance to listen back to review their performance. Plenty of opportunity to practice will go a long way in calming nerves and helping apprentices to feel prepared.

For their part, End-Point Assessment Organisations and independent assessors must be self-aware and mitigate potential for bias, unconscious or otherwise, while taking steps to help put nervous apprentices at ease.

A note on diversity

While it’s important to protect validity by taking a consistent approach to PD assessment, it’s also necessary to remember that the varying needs of individual students means that some will require a different kind of interaction during the two-way sessions.

For example, there are a range of factors that could impact on the performance of an apprentice outside the assessable areas of their knowledge and ability, such as attention, memory and use of language. Such factors can influence performance positively or negatively – not having to translate thoughts into the written word might be beneficial for some, while others may miss the opportunity to revisit their answers as they would in a written assessment.

With this in mind, it’s important for assessors to be clear and transparent on assessment timings and to listen carefully to apprentices and show an interest in the answers given, especially for those who are feeling nervous.

Guidance on grades

When it comes to grading a professional discussion, there are frameworks in place that can guide an assessor on levels of achievement. For example, apprentices at lower levels may offer examples of their knowledge and skills in a matter-of-fact manner, while those at a higher level would demonstrate a broad range of theoretical and technical knowledge through their skills in practice.

Such frameworks offer valuable tools in helping an assessor to award grades that are in line with an apprentice’s understanding of their role.

As PDs come to play a bigger part in apprenticeship assessment, supporting everyone taking part is key. Clear guidance helps protect the validity of the process by giving direction to assessors, while fostering a setting that gives apprentices the best chance to succeed.

Paul Kelly, Qualifications Director at Professional Assessment Ltd (PAL)

PAL’s new white paper, ‘Maximising the Benefits of Professional Discussion and Minimising Threats to Validity’ explores PD as a method of assessment, including practical advice on protecting validity and assessment criteria.