The art of the possible in end-point assessment observation during Covid-19

Over the last 2-3 months there has been a huge amount of effort by employers, #EpAOs, External Quality Assurance (#EQA) providers and the Institute for Apprenticeships and Technical Education (@IFAteched) to identify flexibilities in the delivery of end-point assessment (#EPA) to enable EPA to continue in the midst of Covid-19.

This has brought organisations together to identify solutions that maintain rigor, robustness, integrity and fairness, in EPA whilst ensuring the safety of apprentices and their assessors.

I, like many, believe that is it important to watch and learn lessons from EPA delivery during Covid-19, harnessing what we have learnt to continuously improve. The IfATE have been capturing apprentice and employer feedback via surveys, and also have published some apprentice feedback on their website, so I thought it would be good to also begin capturing feedback from assessors, EpAOs and providers.

This article, an interview with assessors that have begun delivering remote EPA observations as a result of flexibilities approved by the IfATE, is the first step in this process.

I will be talking with EpAOs over the next few weeks to begin to capture their feedback.

The interview was with assessors from the International Assessment Centre (IAC), an organisation that provides industry expert assessors to EpAOs to carry out EPA support services via their Independent Technical Expert Network.

The IAC support a wide range of industry sectors including: engineering, construction, manufacturing, oil and gas, offshore, nuclear, utilities and food production. At the end of May 2020, IAC Technical Experts began delivering their first remote and video-based observations for EPA, so I spent some time talking with their assessors to find out how it went.

As assessors, were you involved in the design of the remote delivery options, or was it determined by the EpAO?

As soon the guidance was published by the IfATE and ESFA, we held a remote standardisation workshop, with our technical expert network at the IAC, to debate ways in which to facilitate EPA remotely, whilst ensuring EPA processes and practices were not compromised.

The outcomes of this workshop were then shared with our EpAO clients and they then approached their EQA provider to discuss and seek approval from the IfATE. In our case, and based on the plans and procedures they were presented with, the IfATE granted approval.

What remote delivery mechanisms have you used for EPA Observations?

To date we have facilitated 2 types of remote observations. The first observation option we explored with the employer and EpAO was a live stream of the apprentice undertaking the EPA activity. This required that the employer check the assessment location, prior to assessment, for conductivity and bandwidth by running some mock footage.

Where conductivity and bandwidth were found to be unreliable, we moved to the second observation option where the apprentice was filmed undertaking their EPA activity and the film submitted to our technical expert assessors in a given format and using a secure file transfer platform. Both of these methods had been approved by the IfATE for the specific EPA standards we are involved in end-point assessing.

Who holds the camera?

The video camera/mobile phone was held by a member of staff identified by the employer. In our case it was agreed that the filming for all apprentices would be conducted by the same person. It was important that this person was not just randomly selected, so prior to appointment it was agreed that the cameraperson would need to hold health and safety awareness, recent occupational experience, an understanding of assessment practices and not have known or been involved with the apprentices training.

This would ensure that, should the apprentice inadvertently place themselves or others at risk, someone could intervene and stop the apprentice. We were also governed by health and safety procedures and Covid guidelines.

How did you prepare the apprentice and make sure they demonstrated all the required KSBs during their remote observation?

We acknowledged that even without a camera being pointed in your direction, any form of assessment is a daunting prospect to an apprentice. We therefore arranged, through collaboration with the EpAO and Employer, and in advance of the date of assessment, that the Technical Expert Assessor would brief the apprentice on what they were expecting to observe. To be honest, we got the feeling that this was not as big a deal with the apprentices than we initially thought, as in these modern times most are far more advanced with the use of technology than us “oldies”. This short but vital communication briefing appeared to settle the apprentices allowing them to focus on what they needed to prepare for assessment and ensure that they could showcase their “masterpiece” on the actual day.

On the day of assessment, the Technical Expert Assessor contacted the apprentice to provide a more detailed briefing. This included the normal briefing on expectations as well as the importance for the apprentice to verbally articulate each stage of the task they were about to facilitate (what they are doing and why). The verbal articulation at each stage was not as critical for the live streaming, as the Technical Expert Assessor could ask questions via the two-way communication facility on the live streaming application, but was important for the recorded assessments when the Technical Expert Assessors were assessing the film footage that had been submitted.

How are the assessments that are not live-streamed assessed?

Every effort was made to live stream all assessments, but this was not always possible, so when the apprentice’s performance was to be recorded the cameraperson was provided with a set of guidance notes that detailed which aspects of performance MUST be captured.

These guidelines were developed by the Technical Expert Assessor in collaboration with the EpAO in line with the marking criteria of the Assessment Plan. Whilst this did not limit the amount of footage captured, it did ensure that key aspects were recorded in more detail than others.

The footage was also broken down into stages for example, Pre work preparations and communications, formulating a plan, preparing the work area etc. This not only helped with file transfer but played a big part in standardisation and subsequently equality. Once the Technical Expert Assessor received and assessed the recorded footage, we held a technical discussion with the Apprentice via an established communication application, in this case MS Teams.

The Technical Expert Assessor would then share screens and ask pre-set and specific questions, in relation to the performance and the assessment plan KSBs, to the apprentice. Where considered appropriate, and following agreement with the apprentice, these questions were sometimes recorded to support the assessment decision.

Does that mean it takes longer to carry out a recorded observation than if delivering face to face?

Once all the planning and assessment methodologies are set and communicated the actual assessment time is no different.

However, like with anything that new some of the first assessments did take a little bit longer whilst the Technical Experts Assessors became accustomed to the process.

Does it require more preparation than if delivering face to face?

We would say that it takes a great deal more preparation up front due to the site conductivity checks and sourcing a suitable person to film the apprentice undertaking the work. However, the planning and preparations at this stage appear to be well spent, as feedback from all parties were very positive. We should also not forget that we are saving on travel time for the Technical Expert Assessor, and in some cases, the need to complete site induction.

Does it cost more to deliver in this way?

There are initial upfront investment costs to consider, as video recording equipment is required, but let’s not forget that modern mobile phones are able to capture high quality films, so the purchasing of recording equipment may not be required.

However, if it is required, it can be reused for future assessments. There is also the cost consideration of the time of the person that is holding the camera to record of live-stream the apprentice undertaking their assessment. But there are also costs saving in terms of assessor travel and overnight accommodation costs.

So long as these costs are discussed and agreed between the EpAO and employer prior to the delivery, then it should not be a barrier to delivery.

How have the technical expert assessors found it?

Initial feedback suggests that it was a good experience, and that the collaborative planning and preparations ensured a robust assessment judgement could be made.  

In a few weeks’ time we will be conducting a standardisation meeting to capture conclusive feedback from assessors, employers and apprentices.

Is there anything that hasn’t worked so well?

It would have been an improvement to have clearer and more direct vocal communications. For future assessments we would explore the possibility of having microphone and ear pieces for the apprentice. This would help when working in noisy locations.

We cannot get away from the fact that remote assessment takes away the personal factor and that some apprentices may become more nervous being filmed than if they were working alongside the Technical Expert Assessor. However, apart from that we believe we could be sowing the seeds of a new dawn in end-point assessment.

When Covid-19 is over, do you think that remote delivery methods should be retained as an option for EPA observations?

Let’s not forget that this was not a simulated environment, it was simply that the eyes of the Technical Expert Assessor were replaced by the lens of a camera. Technology will always carry some element of risk when delivering remote observations especially when live-streaming, but careful planning and testing will reduce the level of risk.

If we can assure health and safety, use reliable communication applications, and comply with GDPR, then I hope that our regulators see that assessment outcomes are not compromised and that remote EPA observations can continue to be a viable delivery option going forward.

THANK YOU to the International Assessment Centre for their time sharing their experiences.

Jacqui Molkenthin, JEML Consulting

If you would like to know more about end-point assessment, I have authored a range of articles that you may find helpful:

• EPA during Covid-19
• EPA from the perspective of assessors and EpAOs , the full survey report can be downloaded here.
• Ofqual for micro EpAOsOfqual for micro EpAOs 
• Conflicts of Interest in end-point AssessmentConflicts of Interest in end-point Assessment 
• Assessing behaviours in EPA
• Overcoming the challenges of EPA
• Translating a standard into a curriculum
• How to become an EpAO
• Gateway to EPA, more than just a tick box exercise
• Lifting the lid on EQA
• Employer led EQA