Hints and tips for EpAOs on designing multiple choice questions and answers

Across my work, with a wide range of EpAOs, I have seen a variety of approaches to the design of questions for knowledge tests. Knowledge tests requirements across the assessment plans vary, for example, they may be multiple choice test (MCQs), short answer, scenario based, and some contain a mixture. But there is one thing in common, the assessment plan does not write the questions or answers, that is the role of the EpAO. This means that EpAOs must design question and answer banks that are fit for purpose, which essentially means that they must be valid, reliable, comparable, manageable and minimise Bias (Ofqual Condition D1.2). In the Ofqual technical evaluation of end-point assessments in June 2020, they identified “more than one correct answer in multiple choice questions” in over a quarter of EPAs they reviewed (page 7). The report then went onto highlight effective practice (page 8).

The Institute for Apprenticeships and Technical Education (IfATE) provides guidance on the use of knowledge tests in end-point assessment, but does not publish guidance on writing knowledge test questions. Ofqual provide guidance within their criteria (Criteria D) detailing how they expect organisations to explain how they develop their multiple choice tests, but they do not provide guidance on how to write them.

Given how critical it is for EpAOs to make sure they get the question, and answer, design right, I thought it may help if I share a few hints and tips on writing multiple choice questions. Readers may find it helpful to read this article in conjunction with another article of mine about ensuring the correct level in assessment design. But before I get started, readers may want to get used to some terms that are often used for multiple choice question design, and are seen across internet searches and academic papers/reports. The multiple-choice question or scenario may be referred to as the “stem”, the correct answer may be referred to as the “key”, and the remaining incorrect answers may be referred to as the “distractors”.

As with all my articles, I am not a representative of Ofqual, IfATE or the ESFA; what I am sharing are hints and tips based on practical experience, which I hope are useful to EpAOs. Within this article, I have chosen to use the, well known, highway code to help provide examples so that readers can understand the principles I am trying to explain:

1. Make sure the question is clear and that all answers are plausible:

Do not use words that could be misleading and do not include unnecessary information. Make sure the apprentice can understand the question without having to read the answer options, and make sure that all answers are plausible but that only one is correct. For example:

QUESTION – When should a lorry driver stop at the stop line at traffic lights?

ANSWER – When the traffic lights are: a) red; b) amber; c) red and amber; d) green

This question is misleading as 3 of the answers could be correct: a) Red: Stop on red; b) Red and amber: Stop and wait; c) Amber: Stop unless you have already crossed the line.

This question contains some unnecessary / irrelevant / distracting information by adding in the word lorry driver. It would perhaps have been better to say ‘when should a driver’.

This question has also automatically enabled the apprentice to rule out option d, an obvious incorrect answer, meaning that they only have to choose from 3 possible answers.

Perhaps a more suitable question would be “at traffic lights, what does it mean when the red and amber light is showing?”.

2. Avoid open or multiple interpretation questions:

Avoid questions that lead the reader to think that the correct answer is based on their opinion rather than fact, for example, words such as could, should, might, or think (just like the example question in bullet point 1). Instead use words such as must, compulsory or required. Outside of MCQs, if you look at Ofqual conditions, you start to understand the importance of such wording. The video series “Ofqual Explains” details when and how they use specific words, such as ‘must’, ‘where’ and ‘any’, and their importance, demonstrating just how critical the use of a words it to its meaning and interpretation.

3. Avoid Negatives:

Make sure the question enables the apprentice to demonstrate that they know the correct answer and have the required knowledge. To do this, you need to avoid negative questions, and avoid answer options such as ‘none of the above’. For example:

QUESTION – When do you not need to stop at the stop line at traffic lights?

ANSWER – When the traffic lights are: a) red; b) amber; c) red and amber; d) none of the above

Not only does this question have a high risk of being misread, it fails to enable to apprentice to demonstrate that they know the right answer. It also fails to take into account a whole range of other factors that could influence why, or when, they do or do not need to stop at traffic lights.

4. Avoid using pictures and charts where possible:

For example, if the traffic light question had the answers displayed as pictures of traffic lights, rather than text answers of red, amber, red and amber, green; this could have a negative impact on apprentices if, for example, they are colour blind, or of the assessment was paper based and the papers had been printed in black and white. I am not saying, do not use pictures and charts, but think carefully about their accessibility to all users and all types of assessment delivery options before choosing to use them.

5. Avoid lengthy questions, but be aware of the level:

Lengthy questions take up valuable test time, and can have an impact on equality. For example, a long question may penalise people with English as second language or those with reading difficulties. Longer questions may also cause apprentices to panic because of it taking up valuable test time to read. However, remember to take into account the level of the assessment, a level 2 multiple choice question, focussed around recall, will look different to a level 6 multiple choice question, focussed around critical thinking. It is less common to see multiple choice questions used at higher levels, but it is still worthwhile taking this into consideration.

6. Keep answer lengths similar:

One answer longer than the others tends to give away the answer, as longer answers tend to be the correct answer. You may not believe me on this, but I have found that when I have written questions and answers, I have unintentionally made the correct answer longer because I was making sure that the answer was correct beyond doubt (this tends to be more of a challenge at higher levels).

7. Use stand-alone questions:

Make sure the answer to a question can’t be found in a question elsewhere in the test, and make sure that an apprentice doesn’t have to get an answer correct in one question in order to be able to understand and answer another question. When writing questions, you are often writing them based on a theme, or a particular Knowledge, skill, or behaviour (KSB) of the standard, and when writing on this basis it is easy to unintentionally come up with questions that link together.

8. Use neutral language where possible:

Ensure questions and answers are free of gender, ethnic, age, political, cultural stereotyping or discrimination. For example: avoid things such as TV show names as they may not be recognised by some apprentices because of age, culture or personal preferences; and avoid gender stereotypes such as fireman, air stewardess, and barmaid, and replace with gender neutral language such as firefighter, flight attendant and bar tender. In fact, you may recall recent press coverage about the change of cricket terminology from ‘batsman’ to ‘batter’.

9. Avoid Homonyms:

Avoid using words that sound the same but are written differently and have different meaning and words that are written the same but have different meaning. For example: ‘pain’ and ‘pane’; ‘loan’ and ‘lone’; ‘break’ and ‘brake’; lead (dog lead, or metal, or to be in front); plane (woodworking tool or aeroplane). If they must be used, ensure that the meaning is clear to the apprentice.

10. Mapping:

Map each question to the KSBs of the standard, in accordance with the assessment plan. In other words, when writing a multiple-choice question, do not write questions that do not map to the KSBs, and do not write questions for KSBs that are not assessed via the multiple-choice knowledge test.

Jacqui Molkenthin, JEML Consulting

I hope my ideas and suggestions are helpful. You can find a whole range of other articles of mine supporting the EpAO sector on FE News and my LinkedIn updates.