FE and the law of small numbers

It will not be a surprise to you if I say the number one is bad news in FE. One student on a course, one supplier, dependence for apprenticeships on large employer; these are all bad news.

Beyond the number one, small numbers can still be problematic; and often we don’t even realise it.  The Law of small numbers means we can try to fool anyone with “research evidence”; but worse still we manage to delude ourselves that our research is valid.

I’m a great believer in testing and measuring wherever possible.  The benefits are enormous when carried out correctly.  When carried out badly, testing and measuring is a dangerous exercise and can severely damage a college.  For example, you might decide to carry out a test into your latest prospectus cover design.  You go and ask a selection of ten students which of two designs they prefer.  You get a very clear result with 90% of them preferring one of the designs.  Clearly the result is conclusive and the winning design is used.

But is it that clear?

The chance of this result being replicated across your target audience is very low.  Let’s look at a very simple explanation why this is the case.  Let’s ask three prospective students which cover they prefer, and as they can only answer for or against a cover, let’s assume they toss a coin to determine their answer.  There is a 50:50 chance of getting an affirmative answer when first one to tosses their coin.  The same chance occurs with the second and third throw of the coin irrespective of the previous result.

When all three have their results, we have several possible answers.  All can have chosen cover 1, (111) or all could have chosen number two (222) or we could have 112, 121, 211, 212, or even 121 again (think about it).  What we can’t have is a 50:50 result.

The actual result is dependent on chance.   Of course real life isn’t subject to tossing a coin but this does demonstrate the risk of focusing on small numbers when an answer isn’t obvious and chance plays a part.

More than three people participating will provide a result that approaches what we will get if you interview hundreds of people.  To get a true picture I suggest we need to include a large number of people in a survey as only then do we obviate chance playing too big a part.

Marketing companies use the Law of Small Numbers to their advantage.  Think about the adverts that say 89% of 187 women found that product X decreased wrinkles!  What these companies have done is carry out many small interviews and have obtained a wide range of results, perhaps from 89% for to 89% against the argument.  They have then chosen the result that suits them and have honestly explained that it was the result of a small group.  What they didn’t say was that the aggregate answer wasn’t impressive. They use the Law of small Numbers to their advantage.

The danger within FE is that we sometimes conduct surveys and jump to conclusions, having forgotten that small numbers (perhaps influenced by the gender, dress, questioning technique or manner of the interviewer) are unlikely to represent the reality that larger numbers provide.

Stefan Drew is a marketing consultant, and was previously director of marketing at two FHE colleges. He now works with providers throughout Europe and the US – visit www.EmployerEngagementStrategies.co.uk or www.StefanDrew.com for more details