or How I learnt to love statistics and be an Honest Consultant
Chris Dillow at Stumbling and Mumbling often notes how remarkable it is that so few people know or understand Bayes’ Theorem. C.P. Snow wrote his famous “Two Cultures” essay drawing on his observation of how few people know about something as fundamentally important as the Second Law of Thermodynamics, and that this is the equivalent of not having heard about the works of Shakespeare.
I’ve often thought the same about the Normal Distribution, often called the Bell Curve after its distinctive shape.
The Normal Distribution is a result of probability theory that we all learnt in secondary school maths, and shows how we would expect a random range of related variables to be distributed.
In the Normal Distribution illustrated above, the average of all of the values of a given set of numbers (for example, people’s height) is in the middle. In any distribution, over 68% of all of these values will fall within one standard deviation – the amount of variation from the average. Over 95% of all values will be within two standard deviations of the average.
(Applying this example to women’s height, we find that the average height for women is 5ft4in with a standard deviation of 3in. Thus, 68% of women’s height will fall between 5ft1in and 5ft7in, and 95% of all women’s height will be between 4ft10in and 5ft10in.)
The Normal Distribution is a really useful way of thinking about where single instances fit into an overall picture.
How could we apply the Normal Distribution to the idea of people’s experiences of public services? Let’s think of the horizontal (x) axis as a quality continuum, with the very worst experience on the left, through the average in the middle, to the very best on the right; and let’s think of the vertical (y) axis as a frequency continuum, from the very rare at the bottom to the very often at the top.
Thinking in this way, what we see is that people’s experiences of public services follow a Normal Distribution. The vast majority of people’s experiences are average or thereabouts; only very rarely (i.e., more than two standard deviations away from the average, or around 0.1% of the time) do people have either the very best or the very worst experience of public services.
I think this observation is important for two main reasons.
The first is about how politics, policy and campaigning is conducted. Although most people’s experiences are average or thereabouts, the experiences and examples we hear most about are, almost by definition, unusual. They exist at either end of the Normal Distribution. So politicians often talk about the very best case scenario in the new idea they’re introducing or in the White Paper case studies that are cited. Similarly, organisations present the best examples of the work they’ve done, promoting these through various communications channels or capturing them in funding bids or contract tenders.
In the above diagram, politicians and organisations operate mainly at point P on the Normal Distribution.
At the other end, we hear of nightmare stories of people’s experiences of services in the headlines of newspapers, or of scare stories from campaigners which highlight the very worst impact of this or that policy change. Newspapers and campaigners operate at point N on the Normal Distribution.
(Of course, each respective group can swap which end of the Normal Distribution they operate at depending on what purpose they are seeking to serve; think of politicians talking about “Broken Britain”, for example.)
The second reason it’s important to think about the Normal Distribution of people’s experiences of public services is to note that, most often, the very rare is what drives most activity. Trying to prevent or minimise the very worst in public services is the realm of regulators, legal teams and complaints procedures; trying to promote the very best is the business of funding bids, think tank proposals, job applications etc.
And it’s this difference between the ends and the middle of the Normal Distribution that creates the problem in the space of people’s expectations of public services. The gap between what the Normal Distribution says our experience is most likely to be (95% of people will be within two standard deviations of average) and what we think our experience will be – the space of N and P represented by newspaper headlines and political rhetoric – leads to expectations that, in reality, can very rarely be met.
A politician gives a speech in the space of P in which they say how things are going to be much better for us all. But, across a whole population, only 0.13% are likely to feel that full impact; the rest will have average experiences whilst some will have terrible experiences, so that both groups feel the promise of the politician hasn’t been delivered.
A newspaper reports in the space of N of an appalling case that occurred because of this or that change. Many will think that this is more than typical than it is, despite only 0.13% of the relevant population having that experience and the vast majority having an average or thereabout experience.
A commissioner commissions a new service from a provider based on the promises in the space of P the provider gave in its funding proposal. The reality is that the service provided is average, with some great outcomes and some very poor ones. This is exactly what the Normal Distribution could have told the commissioner, but they remain disappointed because of their original expectations.
Where does this leave us? I think that understanding and using the Normal Distribution could help us have a more honest approach to what people can expect from public services.
In an area that is “good” at what is does, what we’re really saying is that people’s experiences are generally slightly better than average; the Normal Distribution of such a place would like this (blue line) compared to the normal Normal Distribution (black line).
In this area, slightly more people have a better experience, slightly fewer people have a poor experience, and a very small proportion of people still have extremely good or bad experiences. The net effect is that the average experience of all people in the area is slightly better than normal; effectively, the Normal Distribution has been slightly shifted to the right.
In an area that is “poor” at what it does, the Normal Distribution would look like this (blue line) compared to the normal Normal Distribution (black line):
In such a “poor” area, slightly more people have a poor experience, slightly fewer people have a good experience, and a very small proportion of people still have extremely good or bad experiences. The average experience of all people in the area is slightly worse than normal; effectively, the Normal Distribution has been slightly shifted to the left.
The subtitle of this post comes from my personal feeling that an Honest Consultant is one whose pitch to a potential client would entail a discussion about the Normal Distribution and how the results of their work will make things a little bit better than average for most people in an area.
Such a consultant is unlikely to be successful in their work. But, in understanding and using the Normal Distribution and what it tells us about people’s experiences of public services, the Honest Consultant is at least managing the expectations of the potential client and the public they represent. If in doing so this reduces the gap between what people expect and what they experience, and so increases people’s trust and understanding of what public services can or can’t do, then the Honest Consultant’s use of the Normal Distribution will have been worthwhile.
arbitrary constant 
Superb! I came to this while thinking about and researching the bell curve of political opinions and how, increasingly, politicians are appealing to minorities on either end of the spectrum.