In defence of anecdotal evidence

Anecdotal evidence is worthless, right? It comes about through uncontrolled conditions, and the people reporting it may report selectively (whether or not they intend to be biased). Thanks in part to the works of writers such as Richard Dawkins, we have learnt to dismiss anecdotes and personal testimonials, bringing us closer towards a world governed by Reason and statistics. And we can consign anything supported merely by anecdotes to fire. Hurrah!

For many things in the natural world, it is relatively straightforward (if expensive) to isolate the thing to be tested, conduct experiments or controlled trials, and then quantify the effect of that thing, with well-defined error bars. There are well-established principles and procedures for designing clinical trials, which is why we can resolutely label things like homeopathy, claims about the MMR vaccine, and everything Deepak Chopra says as bullshit, even if there are occasional success stories.

But – and perhaps Dawkins and co. haven’t realised this yet – humans are complicated, and social phenomena, which involve multiple humans, are very complicated. It is impossible to control the environment in which they arise. Also, individual experiences are unique, and it is difficult to give meaningful definitions or boundaries (see also this post), and to ensure that everybody uses the same definition. Related to this, people do not always accurately report their experiences: something that is perceived to be ‘shameful’ will be underreported even if it is actually quite common.

For these reasons, many social phenomena have not been studied quantitatively. But

Absence of evidence is not evidence of absence.

If anything, it is evidence that you haven’t yet done a good enough job collecting evidence on the subject.

When nothing else is available, and when it is not possible to conduct a systematic, controlled and quantitative study, then anecdotal evidence is the best you can do, and it needs to be taken into account, provided it comes from a credible reporter, who has no vested interests. And you must hear that evidence, even if you do not give it much weight.

In more technical language, I am arguing that probabilities are subjective measures of a degree of belief, not objective, and that any evidence should update your posterior probability, even if not by very much.

What I’ve said so far has been relatively abstract, but a failure to understand this has truly harmful effects when we dismiss anecdotal evidence. When hundreds of people report that they have been victims of something, then we need to start taking their testimonials seriously.

The Everyday Sexism Project has collected reports from tens of thousands of women about the sexist abuses that they have suffered. These are idiosyncratic and can’t be categorised; they might have happened repeatedly over a long time, or be one-off events. These acts are often not visible: even the person doing or saying the sexist things might not realise that they are being hostile. An individual claim of sexism might be dismissed by suggesting a variety of mitigating circumstances, or even by assuming bad faith on the part of the reporter! But what is more likely: that misogyny exists in our society, or that thousands of women have conspired together to make up that myth? (You may find Occam’s razor useful.)

Everyday sexism is just one example of microaggression, which also happens in other contexts such as race and religion. Moreover the fear of being subject to a racist attack is just as relevant as the number of actual attacks. Fear has a chilling effect on society, and has a measurable effect on the economy, but by its very nature it is difficult to measure.

Other examples include people’s testimonials of an NHS (or other public service) that is unable to provide a good experience. When thousands of people across the country complain about this, then it is no longer an egotistic individual or a problematic local service; there is something nationwide happening.

In summary,

When thousands of anecdotes are given, then it is no longer "merely" anecdotal evidence.

As with the etymological fallacy, the failure to give anecdotal evidence the weight that it sometimes deserves is a dangerous fallacy, because it is easy to commit it, thinking that you are rational and your opponent is not. This arrogant attitude poisons a discussion.

One thought on “In defence of anecdotal evidence”

  1. In the language of probability, the difference between the statements ‘Sexism exists in our society’ and ‘Astrology works’ is that the latter has a much lower prior probability because it has absolutely no physical basis. The former is at least possible.

    Moreover, ‘evidence’ for astrology tends to be nebulous and doesn’t exclude other possibilities, while ‘Lots of people do sexist things’ (almost tautologously) supports the statement that ‘Sexism exists in our society’. Therefore, according to Bayes’ law, the posterior probability for astrology isn’t updated much by that evidence, while the posterior probability for sexism is. It’s therefore not inconsistent to label astrological success stories as being merely ‘confirmation bias’, while not doing the same for the testimonials on Everyday Sexism.

Leave a Reply

Your email address will not be published. Required fields are marked *