The Rise and Fall of End-point Assessment

At the EPAO conference on 24^th Feb in Birmingham, I shared some insights in to the trends from the past 3 years of apprenticeships and end-point assessment, alongside some of my predictions.

I received lots of positive feedback and follow up from my presentation, so I thought I would expand on my conference presentation to add some more insights and predictions for the sector. 

But, before I begin, I must add some caveats to the data I have provided:

1. The ESFA and IfATE spreadsheets of standards never exactly match (I think it is due to time lags)
2. The ESFA Find and EpAO service does not exactly match the ESFA spreadsheet list of EpAOs (you will see an example in my article)
3. The ESFA do not publish details of who has left the register of EpAOs, only those who have joined, so I have relied on my own monthly analysis of the register over the past 3 years to identify who has left the register and when
4. The ESFA, Ofqual and IfATE do not produce a list of the EpAOs that fall within each phase of the Ofqual transition and how they are progressing, so I have had to use multiple spreadsheets to try and deduce which EpAO fits into which phase and at what stage they are in the process.
5. Ofqual does not publish data on the average length of time it takes from first application to approval, nor does it publish data on the percentage of applicants that are approved on their first application, so I am only able to provide estimates from my experience.

The number of standards and EpAOs has grown dramatically at higher levels

Between November 2018 and February 2022, the number of apprenticeship standards increased from 324 to 688, a 112% increase.  Over the same period the number of EpAOs has grown from 181 to 313, a 73% increase, however this statistic masks a worrying and rapid decline in recent months in the number of EpAOs, I will explain these statistics later. 

In terms of level, the number of level 2 standards has increased from 77 to 148 a 92% increase; the number of level 3 standards has increased from 124 to 224, an 81% increase; the number of l level 6 standards has increased from 44 to 104, a 136% increase; the number of l level 7 standards has increased from 15 to 59, a 293% increase. The proportion of standards across levels is now 22% level 2, 33% level 3, 15% level 6 and 9% level 7.

The massive growth in the number of level 6 and 7 standards has also led to a massive rise in the number of HEI/university end-point assessment organisations. In November 2018 there were 31 (17% of all EpAOs), but by Feb 2022 is was 86 (27% of EpAOs), a 177% increase. 

There are a large number of standards without apprentice starts

Looking at the ESFA list of standards and the ESFA data for apprentice starts from 2018/19 academic year to date, there are 102 standards without any apprentice starts. Of these standards 7 were approved for delivery pre 2018, 8 were approved for delivery in 2018, 22 were approved for delivery in 2019, 28 were approved for delivery in 2020, and 37 were approved for delivery in 2021.  I can understand why standards approved in 2021 will not show starts as they will not have been approved in time for programmes to be developed and apprentice starts to be showing on the system. I can also understand why starts may not show from the beginning of 2021 for standards that have no EpAO because a new rule was introduced stating that “Funding for apprenticeship starts on new standards will only be permitted where we have received and confirmed an EPAO in principle commitment” (equates to 30 standards), but this still leaves a lot of standards that have never had an apprentice start.

This analysis leaves me with a number of questions floating around my head:

1. Why were the standards with no apprentice starts developed? Standards are supposed to be developed following employer demand. If the employer demand was there to design a new standard, why is it not translating into apprentice starts? Perhaps the standards are not fit for purpose, for example the KSBs aren’t realistic, or the funding band is insufficient?
2. Is there a minimum number of employers, with potential apprentices, required before it becomes viable for a training provider to invest in the development work required to set up and offer an apprenticeship training programme?

A number of standards remain without an EpAO

Back in November 2018, 89 of the 324 standards (27%) did not have an EpAO. Roll forward to Feb 2022 and 47 of the 688 standards (7%) do not have an EpAO. However, there are a further 57 who do not have an EpAO, but who do have an EpAO in principle, which technically brings the number of standards without an EpAO to 104 (15% of standards).

Using the ESFA spreadsheets of standards, EpAOs, and ESFA data sets on apprentice starts, you discover that 30 of the 47 standards without an EpAO have never had an apprentice on programme despite all being approved for delivery long before the introduction of the ESFA rule that funding for apprenticeship starts on new standards will only be permitted where there is an EPAO in principle.  Of the remaining 17 standards without an EpAO; 8 have had less than 10 apprentices, 3 between 11 and 20 apprentices, 2 between 21 and 50 apprentices, and 3 with more than 100 apprentices. The standards with more than 100 apprentices but no EpAO are Data Scientist [integrated degree], Heritage engineering technician and Cyber Security technologist (2021), although the find and apprenticeship service is showing “BCS, The Chartered Institute for IT” as an EpAO for the Cyber Security technologist (2021). What also seems odd is that these standards have apprentice registrations in 2021/22 despite the ESFA rule that funding for apprenticeship starts on new standards will only be permitted where there is an EPAO in principle.

This analysis leaves me asking lots of questions:

1. What has happened, or is going to happen, to the 994 apprentices who have been or are on programmes where there is no EpAO? Is there, or has there been, an EpAO on the ESFA “Emergency End Point Assessment Framework Agreement” that has, or will, pick them up? If yes, how is this communicated, and how is the quality of assessment assured?
2. Do the standards without EpAOs have fit for purpose assessment plans, for example, is the EPA possible to deliver or cost effective?
3. Is there a minimum number of apprentices required on an apprenticeship before it becomes viable for an EpAO to invest in the development work required to set up an end-point assessment?

The number of EpAOs has now gone into reverse, and those remaining are growing in size

I mentioned earlier that we are starting to see a decline in the number of EpAOs. When I first began monitoring the register in Nov 2018 there were 181 EpAOs, there are now 313 (EFSA Feb 2022 data). This is a 73% increase, but that increase ceased in March 2021 when it reached a peak of 325 EpAOs. Since then, the number of EpAOs has begun to steadily, and more recently rapidly, decline.  Since I began monitoring the register, I have seen 70 EpAOs disappear from the register, these 70 EpAOs had been approved to EPA 154 standards.  81% of these EpAOs have disappeared from the register since the announcement of the transition to Ofqual in August 2020 (they provided EPA for 123 standards).  These numbers are likely to increase significantly over the coming months as I am aware of a number of EpAOs that have failed their second Ofqual submission who have now received letters from the ESFA about their removal from the ESFA register from May 2022. Unfortunately, I am unable to report on whether these EpAOs had or have any apprentices registered with them that have, or will need to transfer to other EpAOs. I am also unable to report on whether the loss of EpAOs leaves any particular standards without EpAOs as that data is not released by the ESFA.

Alongside the decline in the number of EpAOs we have seen an increase in the size of the remaining EpAOs. When I first began monitoring the register in Nov 2018 69% of EpAOs assessed 1 or 2 standards, this figure now stands at 47%. Over the same period, the percentage assessing 5 or more standards rose from 22% to 37%. If you were to look at the average numbers of standards per EpAO, the average number of standards per EpAO was 4 in November 2018, fast forward to Feb 2022 and that figure is 9, but please bear in mind that these averages are distorted by some very large EpAOs.

Covid has bought challenges and opportunities

Covid has no doubt thrown a curve ball at the sector, but the sector resilience and adaptability has been clear. Those EpAOs able to adapt and innovate have been the best at survival, and many of the Covid flexibilities are now here to stay. However, the Covid impact is not over. Starts fell during Covid, the impact of which will not be felt by some EpAOs until 2024. For other EpAOs they are currently overwhelmed with the backlog of apprentices requiring EPA. It is going to take a few years for the sector to get back to an equilibrium post covid.

Employers are not in the driving seat

The Richard review called for the “Handing purchasing power to the employer” (page 11/12), and the Funding Rules refer to employer selection, but this is sadly not the reality. All too often I hear, and see evidence of, the training provider driving the choice of EpAO.  However, there is an underlying problem which exacerbates this issue. How can the employer make an informed choice when: (a) there may only be one EpAO; (b) the rules of employer selection are not monitored or enforced; (c) there are no published metrics about the quality of an EpAO to inform employer selection? Unless changes are made to enable and facilitate employer selection, then the employer selection of an EpAO will be lost.

Ofqual recognition is taking longer that the IfATE and Ofqual planned

I have heard much speculation that the move to Ofqual was a ploy by the policy makers and Ofqual to reduce the number of EpAOs in the market place and to create larger, more generic, EpAOs. The IfATE have always said that this has never been their intention, but there is no disputing the early evidence that the transition to Ofqual is having a negative impact on the number of EpAOs in the market place.  However, on a more positive note, as of the 22^nd February 2022 list from Ofqual, there are 95 EpAOs recognised by Ofqual, of which 37 are sole EpAOs. The increasing number of sole EpAOs is proof that organisations of all shapes and sizes can gain Ofqual recognition. Based on my analysis across the ESFA, IfATE and Ofqual spreadsheets I estimate that there are approximately 100 EpAOs still to be recognised by Ofqual. Some of those will have received letters of removal from the register by the ESFA, some will still be in the Ofqual review process, and others are yet to apply to Ofqual.

Personally, I am an advocate of the move to a consistent approach to EQA, but I do believe that that the lack of transparency about how long it really takes to understand Ofqual and gain recognition, and the lack of user-friendly sector specific guidance in a timely manner is what has led to the decline in the number of EpAOs.

Ofqual do not publish how long, on average, it takes from first submission to approval, nor do they publish how many organisations, on average, get recognised on their first application, which has meant that it has never been clear how long this journey of transition will take an EpAO.  I know from experience that it usually takes between 2 and 3 submissions before an organisation is recognised by Ofqual, a timeframe of approximately 15 months, if achieved at second submission (3-4 months to prepare the first submission, 3 months Ofqual review time, 2-3 months to meet with Ofqual for feedback, take stock and resubmit, 3 months for the Ofqual review of the second submission, followed by 1 month between application approval, panel and recognition letter). First time application approval and recognition is possible, and I have worked with a number of EpAOs that have achieved this, but from experience I would say that the likelihood of approval/recognition on a first application is as low as 10%, and even if you do achieve it first time, it still takes around 7 months.  

In my EpAO survey report last year, I summarised the guidance materials that have been produced by Ofqual, and the sector feedback on the guidance. Much of the most practical, and well received, guidance was not released until March-June 2021, 7 months after the transition was announced and 2 months before the phase 1 application deadline. 

There is a worrying lack of information on the quality assurance of integrated degree end-point assessment

The focus on Ofqual appears to have led to eyes being taken off the ball when it comes to the quality assurance of integrated degree end-point assessment by the OfS.  Apprentices are on-programme and yet there is absolutely nothing, in the public domain, from the OfS about the quality assurance of the 86 University/HEI EpAOs.  Why is so much rigour applied to non-degree EPA, and yet there appears to be nothing in place to assure the quality of integrated degree EPA?

On a final note, what are my predictions for the next few months?

Bearing in mind that I have heard words such as “it’s just a storm in a tea cup” from regulators and policy makers, you may want to take my predictions with a pinch of salt but here they are:

1. The number of EpAOs, for standards with Ofqual as the EQA provider, will continue to fall.
2. We will start to see the rise of the large generic EpAO.
3. There will be a (temporary) increase in the number of standards with apprentices on programme without an EpAO.
4. A number of standards will either be retired due to low or no apprentice take up, or will be amended, via the trailblazer group, to enable them to be delivered. This may even trigger a funding band review.
5. The ESFA will decide to allow EpAO subcontracting, under the guise of collaboration (Subcontracting is not permitted under the ESFA Conditions for EpAOs, section 3.5), to try and plug the gaps from the loss of EpAOs.
6. The number of HEI EpAOs will continue to rise, until such times as there is clear and proper quality assurance of integrated degree EPA.
7. Unless action is taken to enable employers to be in the driving seat when it comes to EpAO selection, the principle of employer purchasing power will be lost. However, with such drastic loses of EpAOs in the market place, they may no longer have a choice of EpAO anyway!