External Quality Assurance (EQA) starts to get to grips with the building momentum

This is a critical and exciting time for the apprenticeship reforms.

Now a few years in we are beginning to see momentum build as volumes of apprentices on standards begin to ramp up. There are over 400 Standards approved and 200 more in the system. The number of end-point assessment organisations is also building with over 200 on the Register of End-point Assessment Organisations.

EQA can now begin to assess how well these reforms are working, identify where changes may be required and really drive continuous improvement of the apprenticeship system, whilst ensuring apprenticeship assessment is working as employers intended.

Apprenticeships are important in contributing to the efforts to improve levels of productivity and are key in capturing the imagination and attention of employers, helping redress the decline in investment in training that has occurred over recent times.

Engaging employers in the design and development of EQA helps them own the solution

On this latter point, engaging employers in the design and development of apprenticeships and now in the external quality assurance phases of the apprenticeship programme is a powerful way to stimulate and engage the demand side, inviting employers to help own the solution.

NSAR is an example of the employer-led model for external quality assurance of standards-based apprenticeships and we provide external quality assurance for 14 digital apprenticeship standards, four freight and logistics standards, and five rail standards.

Trailblazer groups developing the apprenticeship standards can choose an employer-led approach to EQA, a professional institution approach (where appropriate), OfQUAL or default to the Institute, as last resort.

But isn’t NSAR rail? The National Skills Academy for Rail (NSAR) is evolving and we are developing our external quality assurance footprint and our skills support services. We are building on our strong and long track record in external quality assurance in the non-public training and competence assessment space, allowing us to move into other areas where this expertise can be developed and applied.

The rail sector wanted employer-led, external quality assurance of their apprenticeships and our EQA methodology and the approach has resonated with employers beyond just rail. Our methodology is derived from the needs of each assessment plan, how risks to the validity of assessments are mitigated and how occupational competence ensured.

Why should employers take on EQA in apprenticeships?

End-point assessment in apprenticeships is a paid for service accessed via a marketplace for end-point assessment services. The ability to offer assessment services in these marketplaces is controlled with only those end-point assessment organisations on the Register of End-Point Assessment Organisations for that apprenticeship standard able to offer assessment.

A significant proportion of the public funding made available for apprenticeships is used to buy assessment services in these marketplaces. There are even greater sums of public money and employer co-investment committed to training provision for apprentices and much of the return on this investment is determined in end-point assessment. So ensuring end-point assessment is works properly is important and this is where EQA organisations come in.

For the assessment marketplace to work, employers (as customers) need to feel confident about the choices they make. Employers need to be sure that regardless of whichever end-point assessment organisation they choose to assess their apprentices, the assessment outcome will be the same. Just as good end-point assessment organisations work hard to ensure the assessment outcome is the same regardless of whichever assessor assesses an apprentice, we work hard to make the same claim about the EPAOs operating within each apprenticeship standards marketplace. This way, employers can have confidence in the assessment marketplace and in the assessment of their apprentices. Knowing that whichever EPAO is chosen assessment will be consistent and outcomes comparable are also important reassurances for training providers and apprentices.

Given the massive investment employers have made in creating standards and assessment plans, and their desire to maximise the quality and impact of apprenticeships in their industry it is understandable that they want to be have a key role in assuring the quality of what results. Employer involvement in EQA also sends a strong signal to other employers, giving them confidence in these assessment market places, signalling that it’s a ‘safe’ space to ‘shop around’ when procuring assessment services. Our employer-led Boards are made up of employers who are committed stakeholders, many are Levy payers and all are apprenticeship users. They have a real vested interest in making sure there is a strong focus on occupational validity, continuous improvement and that apprenticeships in their areas really do meet the needs of industry.

What do EQA organisations do?

We have a responsibility for ensuring fair, consistent and comparable assessment practice and outcomes in each apprenticeship standard. We work to ensure end-point assessment organisations interpret and implement apprenticeship standard assessment plans in ways that comply with the employer intentions and that they are implemented consistently. We see our role as creating, maintaining and curating an assessment community with a shared understanding of the how to interpret and implement the assessment plans. This means coming to an agreement across the EPA community where there is any uncertainty or confusion and helping to reinforce this consensus. Having helped establish this community of shared understanding we also work to de-risk the EPA assessment market place by ensuring new entrants are brought up to speed and share this understanding.

Where necessary, we become the facilitator and curator of the ‘case law’ that emerges ensuring any wrinkles in interpretation of assessment plans are ironed out. This puts everyone on the same page and means that there is a ‘memory’ and understanding about meaning in the assessment plan and how to implement it that can be shared with any new entrants to the market. Working through compliance and implementation challenges is a practical way to support greater consistency in assessment.

We also have a role in supporting end-point assessment organisations with their internal assessment consistency and fair assessment. Here the focus is on the internal quality assurance processes and procedures and we look at how they standardise their assessors, how assessment outcomes are moderated and how risks in the assessment process are mitigated. Our role is to support and to offer appropriate professional challenge to help with continuous improvement.

We are just starting to look at comparability of assessment outcomes across end-point assessment organisations. Our goal here is to facilitate a process that does two things. The first, is to allow end-point assessment organisations see how their judgements align with others assessing the same apprenticeship. The second is to allow us to look across the assessment outcomes and to take a view on their comparability.

So, we run what we call a 3C model: compliance, consistency and comparability.

What’s different about the employer-led model?

The most immediate feature/attribute that jumps out is accountability. The Digital, Rail and Logistics Apprentice Quality Boards (made up of employers) will guide, direct and provide challenge for our EQA delivery and we report our work into them.

Employers also take a strong interest in the occupational validity of what apprentices do in end-point assessment and we support this by using occupational specialists to help us scrutinise what EPAOs do and what the apprentices produce in response to assessment plans.

Our employers ask for a strong focus on continuous improvement. This means, working to establish a culture where there is the right balance of professional challenge and support, developing a trusted partner/critical friend style relationship, providing the feedback needed to help EPAOs on their own continuous improvement journeys. It means respecting commercial sensitivities and intellectual property but at the same time working to build openly collaborative relationships that help drive quality improvement.

What about charging for EQA services?

Paying for EQA is an exercise in cost-recovery for all external quality assurance organisations, regardless of which type. The difference is really about how you recover that cost. Our employer-led model has to recover our costs from within the apprenticeship funding allocation. Other approaches and EQA variants are able to access their cost-recovery from outside the apprenticeship funding model.

We do not arrive at our EQA charge using a cross-subsidy model which allows for a flat rate charge, i.e. a single price applied for all apprenticeships regardless of volume, need or standard. This cross-subsidy approach has the benefit of the simplicity of a single rate but brings with it other challenges we think our model overcomes.

Our model is based on combining a number of cost considerations and this is applied standard by standard. The needs of the assessment plan, the nature of the assessment marketplace (e.g. how many end-point assessment organisations offering the end-point assessment), the cost of accessing the right occupational competence to help comment on the occupational validity and the number of apprentices on the standard all play a factor in determining the charge.

We also take a proportionate approach to EQA, acknowledging the amount and nature of EQA contact we have with an end-point assessment organisation on other standards. This allows us to find efficiencies, strip out duplication within our practices and to not to charge for EQA work that we don’t do (e.g. there is no need for comparability work where there is only one EPAO in the marketplace). This approach helps keep us running at or very, very near cost-recovery levels.    

When it comes to looking at EQA cost, we do need to reframe the discourse a little. We need to consider how much public funding (and employer resource) is invested in apprenticeship standards. This brings a sense of proportion to the debate.

For example, the Infrastructure Technician apprenticeship has a maximum funding band of £15 000; Rail Engineering Technician has a maximum funding band of £24 000. This is just the public funded element. Multiply in the volumes of apprentices on these standards and you end up with substantial levels of public investment.

The external quality assurance charge for the effort to safeguard and externally quality-assure the outputs of these market places is close to one-third of one percent of this funding. So, there is a need for some sense of proportion when weighing up the cost-benefit and in debate about charging.

Where next for EQA?

We understand the need to make external quality assurance efficient, stripping out unnecessary burden or any duplication for end-point assessment organisations. The Institute is working on a framework to bring greater consistency to EQA approaches. This will help.

As an EQA though, there is much that we can do. For example, we are talking with other employer-led EQAs about how to streamline our processes, looking at how we get to a place where we can acknowledge what each has done to ease some of the demands on end-point assessment organisations.

We are very aware that one EPAO may have to work with a range of EQAs and so the onus is on EQAs to find ways to make this work efficiently. There is a range of other ways we can lighten the load in our compliance work and we are busy implementing these, now we are up and running.

We are seeing some very interesting and innovative approaches from a range of different end-point assessment providers. Each type of provider (awarding organisation, training provider, private company, professional institution, and employer) brings something different to the table and offers different types of choice and value proposition.

So, we are working hard to ensure apprentices can be confident that whichever end-point assessment organisation is chosen they will be assessed fairly and get the result their performance merits.

Barry Smith, Head of Assessment, The National Skills Academy for Rail (NSAR)

Any groups interested to find out more about an employer-led approach can contact us about the feasibility of setting something up with our support.