Understanding the different types of scientific studies

Let’s talk about evidence and scientific studies..

Evidence- or data- is only as good as the study that the data is from. Some studies inherently produce better data than others, so always pay attention to the study design when you’re trying to interpret results and limitations.

Anecdotal stories and expert opinions are just that- stories and opinions. While these can be super interesting, always always always look for- or ask for- the evidence that supports it. And then do a quick search yourself (browse ~10 abstracts) to see if that evidence is cherry picked. In the case of HCQ, an example would be a physician who claims to have treated 20 COVID-19 patients with HCQ or just believes that it works. Super interesting, but these alone are not data, evidence, or facts. You cannot determine cause and effect here, and you have no way to know if these cases represent what would happen to a larger group of people.

Pros: sometimes the earliest indication that something is worth studying, it's super cheap and easy to make observations

Cons: these aren't evidence, they aren't controlled, representative, or causal

Example: which Youtube video should I not link here?

Case reports or case studies are usually interesting cases: a patient or two with an interesting disease or interesting symptom. They are just observational. They cannot tell you if this is normal or abnormal or what the cause is, but they can spark interesting questions that can later be studied. These reports at least present as much of the data as possible: the demographics of the patient, their medical history, all of the medications they’re on, and what physicians noted about them in the clinic. This is somewhat stronger than an anecdotal story, as you have more context and documented information, however, you still cannot determine any type of cause and effect here, and you still don’t know if these cases represent what would happen to a larger group of people.

Pros: sometimes the earliest indication that something is worth studying, it's super cheap and easy to record particularly interesting observations

Cons: these aren't evidence, they aren't controlled, representative, or causal

Example: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7165082/

Studies in cells or animals are great at helping us to understand potential mechanisms and potential cause and effect since they are actually experimental studies. We can manipulate genes and proteins to really understand what causes a disease, we can look at dose responses and time courses, and we can identify potential targets for therapeutic benefits. However, in a dish, you lose the effects of surrounding cell types, hormones and nutrient fluxes, your circadian rhythm, and more! Moreover, while mice and other model organisms are actually fairly similar to humans, they aren’t humans. We have evolved to have different proteins and feedback loops that have a dramatic effect on the diseases we develop and therefore, the drugs that work. For example, mice can’t naturally get atherosclerosis, but humans sure can, and we don’t have any good mouse models of sepsis because their response is just different enough. There’s also much more variability in human genetics, our environments, and psychological factors that can impact our response to disease development and drug treatments. Basically, we can learn SO much from studies in cells and animals, but we need further human testing to really understand what happens in humans.

Pros: they can tell us a great deal of if and how something works on the target cell type

Cons: they don't ensure the same results will be observed in humans, the studies often take a few years to finish, and they're not usually very cheap- although the time and money depend on how in depth the study goes and the exact question being asked

Example: https://pubmed.ncbi.nlm.nih.gov/32150618/

Cross sectional and case-controlled studies are other types of observational studies. Cross sectional studies take a large snapshot of often thousands of people to look for associations between behaviors, demographics, symptoms, prevalence of disease, or mortality rate, but remember, correlations ≠ causation. For example, a cross sectional study could include the demographics of every COVID patient to look at what age groups it primarily affects, however, as you all know by now, that data was very dependent on other factors: environment, lockdowns, school closings, etc, and would’ve changed depending on when and where you collected it.

Pros: you can determine if you see the same associations with a large number of people fairly cheap, quick, and easy. Often these are based on hospital records or surveys

Cons: correlation ≠ causation

Example: https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2768834

Retrospective (planned after the fact) case controlled studies typically look back through medical records of patients with a disease or outcome of interest and compare it to another similar group of patients. This is more controlled than just publishing information about your group of interest, however, you have no control over the two groups you’re comparing and you’re limited in the information you can collect on them, which limits what you can infer from the data. Especially with a new disease that we’re learning about rapidly, hospital protocols can substantially differed over a few weeks, so if we’re talking about case controlled HCQ studies, the patients on HCQ could’ve been treated a few weeks before or after the controls, and there could’ve been many other variables that influenced their outcomes including when they were diagnosed, how quickly therapies were offered, what type of other supportive care the patients received. We actually saw this in the Henry Ford Health System study: the patients in HCQ groups were significantly more likely to also be on steroids (~80%) vs. the controls (~30%). So while cross sectional and case controlled studies begin to look at more representative populations and begin to use controls, and they’re great studies to guide further research, they don’t tell us causation.

Pros: you can start to compare a treatment to a control, and find doses/ patients/ timing that are more likely to hold up in future trials

Cons: while they're partly controlled, they still cannot suggest causation

Example: https://www.ijidonline.com/article/S1201-9712(20)30534-8/fulltext

Prospective (planned ahead of time) cohort studies will be the last type of observational studies we discuss. These enroll and then track patients over a long period of time to often determine risk factors associated with disease. Oftentimes, these are done when controlled trials are unethical. For example, you can track smokers and non-smokers and look at how often each group gets lung disease (we cannot give patients cigarettes if our hypothesis is that they may cause lung cancer). We can also keep track of diet and exercise over a prolonged period of time to look at the types of behaviors associated with patients who get heart disease, although controlled studies would be better to determine causality for this. The main studies I’ve seen published using prospective cohort design for COVID look at the likelihood frontline workers vs. the general public acquire COVID-19 infections. While this can provide important information about who we should protect or further study, they don’t give us complete cause, effect, and mechanistic data.

Pros: these are best used when it's unethical to provide a treatment to subjects

Cons: they still do not account for enough variables to ensure causality

Example: https://www.thelancet.com/journals/lanpub/article/PIIS2468-2667(20)30164-X/fulltext

Finally getting to the good stuff, or prospective controlled trials. These are often randomized (even better), so patients are randomly designated to either a control group (placebo or standard of care) or an experimental group to reduce bias. These are often double blinded as well (even even better), so not only do patients no know what group they’re in, researchers may also be blinded to who’s receiving the placebo vs. treatment. These studies range from smaller pilot studies to larger clinical trials which determine safety and efficacy of a drug, and because patients are all enrolled in the study throughout the duration of their care, most variables are removed apart from the experimental variable, or drug, which provides a much better basis to determine if the treatment or drug truly causes an outcome.

Pros: you can better determine causation AND if something can work in human beings!

Cons: these are more expensive, more time and labor intensive, and require subject recruitment. Moreover, the data is often specific to the subject demographics and the specific treatment protocol.

Example: https://academic.oup.com/cid/advance-article/doi/10.1093/cid/ciaa1009/5872589

https://www.nejm.org/doi/full/10.1056/NEJMoa2016638?page=1&sort=oldest

The last type of study I’ll mention is a systematic review, or meta-analysis. These studies combine data from many smaller studies to get a better picture of a broader population. The main thing I’ll say here is that understanding how a drug affects more people is great, however, as with all calculations, the quality of the data you put in has a dramatic impact on the data you get out, and when there aren’t many great studies, doing a meta-analysis on them doesn’t give you great data. So never assume results from a meta-analysis are the end all be all without doing a lot more work to understand all of the available studies and data, and the current consensuses in the field.

Pros: they can provide information on efficacy on a wider scale and reduce the variability (or how important uncontrolled variables that occur within each study are). They're also cheap!

Cons: the results are only as good as the data put in, and so you need a good number of well controlled studies for the best effect

Example: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7390912/

So as you’re asking for evidence or looking at data for claims around COVID, or vaccines, or collagen, or whatever the hot new supplement or diet is this month, pay attention to the type of study done. And most importantly, pay attention to what the study can and can’t tell you.

----

And if you're interested in reading an article about how a national healthcare system could help promote faster more coordinated controlled studies in future pandemics: https://www.wired.com/story/covid-19-drug-research-is-a-big-huge-mess/