Date: January 4th, 2021
This is an SGEM Xtra episode. I had the honour of presenting at the McGill University Emergency Medicine Academic Grand rounds. They titled the talk "Evidence-Based Medicine Master Class". The presentation is available to watch on YouTube, listen to on iTunes and all the slides can be downloaded (McGill 2020 Part 1 and McGill 2020 Part 2).
Five Objectives:
Look at the burden of proof and talk about what is science
Discuss EBM and give a five step process of critical appraisal
Talk about biases and logical fallacies
Do a check list for randomized control trials
Record a live episode of the SGEM
1) Who has the Burden of Proof and What is Science?
Those making the claim have the burden of proof. It is called a burden because it hard - not because it is easy. We start with the null hypothesis (no superiority). Evidence is presented to convince us to reject the null and accept there is superiority to their claim. If the evidence is convincing we should reject the null. If the evidence is not convincing we need to accept the null hypothesis.
It is a logical fallacy to shift the burden of proof onto those who say they do not accept the claim. They do not have to prove something wrong but rather not be convinced that the claim is valid/“true” and this is an important distinction in epistemology.
What is science? It is the most reliable method for exploring the natural world. There are a number of qualities of science: Iterative, falsifiable, self-correcting and proportional.
What science isn’t is “certain”. We can have confidence around a point estimate of an observed effect size and our confidence should be in part proportional to the strength of the evidence. Science also does not make “truth” claims. Scientists do make mistakes, are flawed and susceptible to cognitive biases.
Physicians took on the image of a scientist by co-opting the white coat. Traditionally, scientists wore beige and physicians wore black to signify the somber nature of their work (like the clergy). Then came along the germ theory of disease and other scientific knowledge.
It was the Flexner Report in 1910 that fundamentally changed medical education and improved standards. You could get a medical degree in only one year before the Flexner Report. The white coat was now a symbol of scientific rigour separating physicians from “snake oil salesman”.
Many medical schools still have white coat ceremonies. However, only 1 in 8 physicians still report wearing a white lab coat today (Globe and Mail).
Science is usually iterative. Sometimes science takes giants leaps forward, but usually it takes baby steps. You probably have heard the phrase "standing on the shoulders of giants"? In Greek mythology, the blind giant Orion carried his servant Cedalion on his shoulders to act as the giant's eyes.
The more familiar expression is attributed to Sir Isaac Newton, "If I have seen further it is by standing on the shoulders of Giants.” It has been suggested that Newton may have been throwing shade at Robert Hooke.
Hooke was the first head of the Royal Society in England. Hooke was described as being a small man and not very attractive. The rivalry between Newton and Hooke is well documented. The comments about seeing farther because of being on the shoulders of giants was thought to be a dig at Hooke's short stature. However, this seems to be gossip and has not been proven.
Science is also falsifiable. If it is not falsifiable it is outside the realm/dominion of science. This philosophy of science was put forth by Karl Popper in 1934. A great example of falsifiability was the claim that all swans are white. All it takes is one black swan to falsify the claim. There are some philosophers that refute Popper's claim about falsifiability.
Science is self-correcting. Because science is iterative and falsifiable it is also self correcting. Science gets updated. We hopefully learn and get closer to the “truth” over time. Medical reversal is a thing and there is a great book and by Drs. Prasad and Cifu on this issue called Ending Medical Reversal: Improving Outcomes, Saving Lives.
The evidence required to accept a claim should be in part proportional to the claim itself. The classic example was given by the famous scientist Carl Sagan (astronomer, astrophysicist and science communicator). Did the TV series Cosmos and wrote a number of popular science books (The Dragons of Eden). Sagan made the claim that there was a “fire-breathing dragon that lives in his garage”.
The quality of evidence to convince you of something should be in part proportional to the claim being asserted. The summary is the famous quote by Carl Sagan that "extraordinary claims require extraordinary evidence".
Science does not make claims about the truth. It gives an approximation of the the best point estimate of the observed effect. It’s the best known method for exploring the natural world. Science has no agency but rather it is a process. However, scientists are flawed individuals who make mistakes. As Blaise Pascal said: "There is not such thing as the truth, we can only deliver the best available evidence and calculate a probability".
Real World Example:
Marik et al made the claim in 2016 that vitamin C cocktail (hydrocortisone, thiamine and vitamin C) could cure sepsis. He published a before and after observational study with 94 patients. The result was a 32% absolute decrease in mortality (NNT 3). We covered this study on SGEM#174: Don’t Believe the Hype – Vitamin C Cocktail for Sepsis. Dr. Jeremy Faust (FOAMCast) and I had eleven other skeptics comment on Dr. Marik's study. Our bottom line was that vitamin C, hydrocortisone and thiamine was associated with lower mortality in severe septic and septic shock patients in this one small, single centre retrospective before-after study but causation has yet to be demonstrated.
Higher-quality studies have since been published looking at they issue. Putzu et al had a SRMA of RCTs including critically ill patients (not just sepsis). They found no statistical difference in mortality. This was covered in SGEM#268: Vitamin C Not Ready for Graduation to Routine Use.
There has been a RCT published by Fujii et al in JAMA 2020. It specifically looked at 216 patients with septic shock and found no statistical mortality benefit to vitamin C.
Has the burden of proof been met that vitamin C is a cure for sepsis? I am not convinced by the available evidence. Note that this is different than claiming vitamin C does not work. That would shift the burden of proof. I am simple accepting the null hypothesis of no mortality superiority of vitamin C compared to placebo in septic patients.
It is ok to say "I don't know" if vitamin C works. It reminds me of a quote from Dr. Richard Feynman. I have degrees of confidence or certainty about various positions. These positions are tentative and subject to change. I am not absolutely certain about anything. To be absolutely certain could be considered a logical fallacy (nirvana fallacy). Logical fallacies will be discussed later.
2) Evidence-Based Medicine and a Five Step Process to Critical Appraisal
This was defined by Dr. David Sackett over 20 years ago (Sackett et al BMJ 1996). He defined EBM as “The conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients.” I really like this definition, and the only tweak I would have added would be to include the word "shared".
The definition of EBM can be visually displayed as a Venn diagram. There are three components: The literature, our clinical judgement, and the patients values/preferences.
Many people make the mistake of thinking that EBM is just about the scientific literature. This is not true. You need to know about the relevant scientific information. The literature should inform our care but not dictate our care.
Clinical judgement is very important. Sometimes you will have lots of experience and other times you may have very limited experience.
The third component of EBM is the patient. We need to ask them what they value and prefer. The easiest way to do this is to ask the patient. It should start with patients care and it ends with patient care. We all want patients to get the best care, based on the best evidence.
Levels of Evidence:
There is a hierarchy to the evidence and we want to use the best evidence to inform our patient care. The levels of evidence is usually described using a pyramid. The lowest level is expert opinion. the middle of the hierarchy is a randomized control trial and the top is considered a systematic review.
The systematic review +/- a meta-analysis is put on the top of the EBM level of evidence pyramid. However, we need to watch out for garbage in, garbage out (GIGO). This means if you take a number of crappy little studies (CLS), mash them all up into a meat grinder and spit out a point estimate down to the 5th decimal place that results is some impressive p-value is an illusion of certainty when certainty does not exist.
EBM Limitations:
Harm and the parachutes argument - Smith and Pell BMJ 2003, Hayes et al CMAJ 2018, and Yeh et al BMJ 2018
Most published research findings are false - Ioannidis PLoS 2005
Guidelines are just cookbook medicine
Good evidence is ignored
Too busy for EBM
Five Alternatives to EBM:
This was adapted from a paper by Isaacs and Fitzgerald BMJ 1999. To paraphrase Sir Winston Churchill, EBM is the worst form of medicine except for all the others that have been tried.
Eminence Based Medicine - The more senior the colleague, the less importance he or she placed on the need for anything as mundane as evidence. Experience, it seems, is worth any amount of evidence.
