Anecdotes (alone) are not evidence

Today I'm expanding on a guest post I did for @scientist_with_a_board on Instagram. You can find it here: https://www.instagram.com/p/CAIxtUqniSh/

Anecdotes (alone) are worthless as evidence, because you cannot use them to establish causal relationships. False positives (believing there is a connection between A and B when there is none) are more often harmless than false negatives (believing there’s not a connection between A and B when there is one). So from an evolutionary perspective, we had to trust our experiences and observations and use correlations to survive. If we tended to run into bears at the watering hole in the morning, we likely visited in the afternoons. If we felt sick after eating a new type of plant, we probably would've avoided eating it again.

But now that we are surrounded by hundreds of options, ie: products, supplements and diets. And now that we have the scientific method and (better) reasoning skills, so we can (and should!) do so much better. Anecdotes are unreliable, often cherry picked, not representative, not controlled, and a sample size of just one.

Two events can happen by coincidence. And when it does, it’s hard to know if there’s an causal effect without controlled studies or mechanism. We also suffer from cognitive bias, we're more likely to remember and comment on notable or unusual experiences. But for the 4000 people who didn’t perceive a benefit, they’re less likely to talk about it. And of course the placebo effect is very real. Just the act of taking an inert substance can affect mood and belief and have significant effects on subjective measurements like pain and effort, like Michael Jordon’s secret stuff (yep going with a Space Jam reference). If I am sick, eat pizza, and feel much better the next day, was it the pizza or was it the fact that colds only last a few days and we usually get better without treatment? (Now replace “pizza” with “vitamin C”). If I call my mom after suffering from arthritis over the last week and trying all of my usual go-tos, was it the conversation with my mom that made it go away or is arthritis well known to come and go? (Now replace “conversation with mom” with “celery juice”) If I start wearing blue shorts and start gaining substantial muscle, was it the blue shorts or the lifting, fueling, and resting that I also started doing? (Now replace “wearing blue shorts” with “weird workout supplement”) And now replace the variables in those scenarios for diet fads, superfoods, supplements, teas, and wraps (and even vaccines to observe the non existent relationship with autism). Wait let me say that again. Despite the anecdotes floating out there, no studies have ever found any sort of connection between vaccines and autism. So while you have every right to try things just to experiment a little on your own body (at least when it doesn’t harm anyone else), it’s super unethical to sell and market placebos and anecdotes, especially when it’s harmful itself or if it hinders people from getting evidence- based care. Anecdotes help inform new hypotheses, and they can support research proposals and rationale for new research, but anecdotes require additional data to count as evidence. So if you’re a consumer, ask for the evidence! And if you’re a company or a practitioner, get someone in research to help you conduct a real controlled study.

For more info, here are a few good articles:

https://thelogicofscience.com/2016/02/10/5-reasons-why-anecdotes-are-totally-worthless/

https://www.scientificamerican.com/article/how-anecdotal-evidence-can-undermine-scientific-results/