Mark Twain’s ‘returns of conjecture’ from ‘investment of fact’

One gets such wholesale returns of conjecture out of such a trifling investment of fact.

This was Mark Twain on science in “Life on the Mississippi”, but he could well have been talking about any aspect of the modern world.

It’s a human folly to interpret new information in a way that confirms everything you already think. In danger of repeating this folly, though, the Twain quotation hits home with me for two current reasons.

First, I recently reflected on my relationship with news and decided to make a conscious and proactive choice to opt out of the news, social and other media. This was because:

I’d grown tired of most sources of media. Their focus seemed only to be on trivial, untrue or highly creative interpretations of things to do with politics and policy, or fanning the flames of these things with news stories and opinion pieces. I’d also grown increasingly tired with social media, the majority of which was people sharing either trivial, untrue or highly creative interpretations of things relating to politics and policy, or sharing a news story or opinion column that had fanned the flames of their outrage. On top of this, I found myself frustrated with the never-ending wealth of blogs, reports, videos and so on which offered organisation x’s perspective on the latest thing y or z.

This clearly reflects Twain’s observation. In this case, ‘trifling fact’ was the latest policy idea or something political that had happened, and the torrent of interpretations, opinions and perspectives were the ‘wholesale returns of conjecture’.

The second is on the nature of evidence.

Instead of considering, questioning and reflecting on new evidence – what it might actually tell us, what limitations it has, what our interpretations of it might reveal of our own assumptions – our current policy and political approaches tend to use any new evidence as the jumping off point for any number of other directions. As I intimated before, this is because we don’t live in a rational, evidence-led vacuum that is protected from the whims of politicians and public opinion. This in itself is fine (well, it’s reality); Twain’s observation reinforces to me we should invest more in what is currently ‘trifling fact’ than where most effort is made – on ‘wholesale returns on conjecture’.

Advertisement

How many countries are there? On evidence and PHBs

Image via http://damirbecirovic.com/

How many countries are there?

This seems like a straightforward question to answer, doesn’t it? Most primary school children could give you an answer, and even if they couldn’t they could quickly look it up in an Atlas.

But perhaps it’s not as simple a question to answer as we think. Scotland and Wales are countries, aren’t they(?), and yet they don’t appear on the list of countries recognised by the United Nations: the UN reckons there are 193 countries, including “the United Kingdom”. My Times World Atlas from 1986 says there were 173 countries. And football’s governing body, FIFA, currently has a list of 209 countries with football rankings.

So, in order to know how many countries there are we need to ask ourselves at least two prior questions: (1) What do we mean by a “country”?; and (2) Who are we asking?

Maybe the question is a bit complicated, so let’s ask ourselves an easier question by going up a level: how many continents on the world are there?

Erm, well. National Geographic reports: “By convention there are seven continents… [but] some geographers list only six [and] in some parts of the world students learn there are just five continents.” Which means the answer again depends on asking other questions, including: (1) What do we mean by a “continent”?; and (2) Who are we asking?

This “facts” business is tricky, isn’t it?

I share this by way of thinking about what we mean by “evidence” in the context of “evidence-based policy” and the recent example of Personal Health Budgets.

A significant announcement by Simon Stevens, the Chief Executive of NHS England, about Personal Health Budgets gave rise to some teeth-gnashing earlier this month.

The gnashing focused on the evidence base that underpins the effectiveness of Personal Health Budgets. Some folks, especially the well-known Ben Goldacre of Bad Science fame, are not convinced by the current status of the PHBs evidence. They think there should be at least a Randomised Control Trial (RCT) to test whether Personal Health Budgets work. Others, including advocates of personalisation in public services more generally, noted both the results of the existing evaluation of the Personal Health Budgets pilot and the value of all types of evidence, especially including the views of patients/users themselves.

Both groups therefore lay claim to “evidence-based policy”, which leads me to two reflections:

1. It’s hardly an original thought (indeed, there are entire disciplines dedicated to such questions) but we must remember there is value associated with all different types of evidence and research methods. The value derived, and of the associated evidence arrived at, depends on what types of answers you’re hoping to uncover, how questions are framed and what pre-questions and/or assumptions underpin the framing of those questions. Different people have different thresholds for evidence and research methods, quite aside from the fact that one type of evidence or research method that’s a gold standard in one discipline could be next to useless in another.

For me, this is the equivalent of the first pre-question we came to in considering countries and continents: What do we mean by “evidence”?

2. Let’s not even get into the “policy” bit of “evidence-based policy”. For example, when has policy ever been based on evidence anyway? Does policy making happen in a rational, evidence-led vacuum that is protected from the whims of politicians and public opinion which, heaven forfend, may not be evidence based? Notwithstanding questions of what we mean by evidence, it’s safe to say that not all policy is based on what evidence there is. This is therefore the equivalent of the second pre-question we came to in considering countries and continents: Who are we asking what we mean by “evidence”?

The up-shot of this in the context of the evidence base for Personal Health Budgets is that Ben Goldacre and advocates of personalisation are both right, and they’re both wrong. There cannot be a definitive answer to the question of whether Personal Health Budgets are effective until some other, perhaps unanswerable questions, are considered.