Evidence-Based Medicine is a Lie
Wheels In — Intro + Opening Thoughts
Here at “The Surgeon’s Record,” we aim to be evocative and a bit contrarian without engaging in hyperbole or baiting clicks. That said, I’ll admit the title of this week’s post is just a tiny bit over-the-top. To be clear, I’m a believer in practicing evidence-based medicine (EBM), following the data, and avoiding low value, unindicated care.
Like other popular healthcare topics—value-based care, practicing at the top of one’s license, quality over quantity—EBM is oft-referenced but rarely discussed in detail. The truth is that, in medicine, evidence is rarely as ironclad, definitive, or black and white as we pretend it is. This week, I’ll discuss why EBM often falls short, how we can evolve our approach to evidence-based treatment, and the need to balance science with experience.
Dem Dry Bones
From the EIC’s Desk
A few years ago, I developed what I call “Schwartz’s Law of the Orthopedic Literature.” For every study, there is an equal and opposite study. Find a study about a particular topic, condition, or treatment, and you’re likely to find at least one other study whose findings conclude the polar opposite.
Some examples in ortho:
Tibial stems to reduce total knee failure rate in obese patients: Pro / Con
Betadine irrigation to reduce postoperative infections: Pro / Con
Safety and effectiveness of aspirin to prevent blood clots after joint replacement: Pro / Con
Keep in mind, this is just a small sampling. In most cases, there are multiple articles to support each side. Unbelievably, conflicting studies sometimes appear in the same edition of a journal. Based on Schwartz’s Law, one might logically assume that if you look hard enough, you can find evidence to support almost any practice. It’s both sublime and absurd.
What’s the problem here? Is it bad science—poor study design/methodology, statistical manipulation, unsupported conclusions? Is it the complexities of medicine and the unpredictability of the human condition? Is it incomplete, unreliable data capture and reporting? Is it the expense and difficulty of high-quality research? Is it failure to separate statistical significance from clinical significance? Is it biases, both inherent and hidden? Yes … yes it is.
Statistical fragility further muddies the waters. The Fragility Index (FI) is a tool used to “measure the robustness (or fragility) of the results from a clinical trial that uses dichotomous outcomes.” FI is calculated by switching patients in the control/intervention arm to an event or non-event until the P-value is > 0.05. In other words, FI measures just how easy it is to flip the results of a study from statistical significance to statistical insignificance. The results of highly fragile studies become insignificant if only a small number of patients switch from the event group to the non-event group (or vice versa). When’s the last time you considered a study’s fragility?
Low FI is especially problematic for studies in which patients are lost to follow up (which is nearly all of them). Accounting for the outcome of even one such patient could render research findings insignificant. The take home message? We shouldn’t rely solely on p-value (i.e., statistical significance) to determine a study’s clinical relevance. In other words, don’t take the evidence at face value.
Despite our reputation as purveyors of low-value treatment, orthopedics has undertaken multiple initiatives to guide clinical decision making. The AAOS regularly publishes Clinical Practice Guidelines (CPGs) on topics such as carpal tunnel syndrome, osteoarthritis of the hip, osteoarthritis of the knee, and surgical site infection.
CPGs are established by workgroups of clinicians who review the medical literature and develop recommendations based on the strength of the available evidence. While CPGs are a necessary and welcome tool, their guidance is often limited or vague. There simply aren’t enough high-quality studies to justify stronger recommendations.
What’s an EBM believer to do? Hang their head in defeat? Shake their fists at the sky? Succumb to EBM nihilism? The goals of EBM are still noble: deliver high-value care to patients rooted in science. Just because doing so is hard and imperfect doesn’t mean it’s without merit. As care evolves, EBM must evolve with it.
I’d propose three mechanisms to improve our ability to leverage EBM:
Create a new information dissemination paradigm
Emphasize real-world, practical outcomes
Leverage technology to improve data capture/analysis
Paradigm Shift
Clinical robustness of health tech solutions is … lacking but evolving. Startups should not be given a free pass on providing evidence. White papers and commissioned (purchased) third-party data evaluations are a minimal step toward EBM. But it’s only fair that we create a new paradigm to support early-stage research, transparency of results/outcomes, and crowdsourced peer review. We can’t criticize healthcare innovators for being non-scientific if we don’t level the playing field.
Open-source sharing of experience and innovation is already underway. During the pandemic, orthopaedic surgeon Dr. Ira Kirschenbaum launched the Journal of Orthopaedic Experience & Innovation (affectionately known as JOEI). JOEI was founded on “the belief that information in orthopaedics should be on a more open platform, have increased accessibility, respect surgeon experience with procedures and the practice of Orthopaedics and give a voice to early innovation.” JOEI is a shining example of the needed paradigm shift in medical publishing.
Emphasize Practicality
The best research produces actionable information for doctors and patients—high-quality papers that offer practice-changing insights. There will always be the need for high-level academic research, but there’s also need for EBM rooted in real-world practicality. It can be maddeningly difficult to extract meaningful conclusions from traditional medical literature.
Case studies, cohort studies, and anecdotes are considered weak evidence. But higher-level studies (including RCTs) aren’t as strong as we think they are. Many are fragile, contradicted by other studies, and/or plagued by clinical irrelevance (hand waved by statistical significance). Negative studies are less likely to get published, even when their conclusions are entirely valid.
The dirty secret is that many physicians learn most from the experiences of others. As doctors, we love sharing practical tips, firsthand experiences, triumphs, and failures. This may sound unscientific, but it doesn’t have to be. Practical research can reflect real world outcomes without conforming to the rigidity of traditional study methodologies. Findings should be presented in a transparent way, trusting the reader to parse conclusions presented. Open-source publishing encourages presenting all results “as is” without the need for p-hacking and other tricks.
Leverage Technology
This is the part where I just casually mention artificial intelligence and move on. Kidding aside, technology holds the key to evolving EBM. In an ideal world, unfiltered data (structured and unstructured) is captured in totality and available for unbiased analysis. AI can be trained to determine what’s important and what isn’t and to make connections without the biases of traditional research. We are a long way from this ideal world, but the first steps are being taken now.
Garbage in, garbage out is still a problem, but we can start by improving data capture. As discussed during our recent webinar, ambient scribes are a significant move in this direction. I’m excited about the potential for ground up approaches to solve the GIGO problem. Novel practice models that free us from cut-and-paste EMR billing platforms will help too.
Medical knowledge advances should be continuous, constantly updating and evolving as more information becomes available. It should be personalized and precise. Technology will help us get there—valid concerns over privacy, black boxes, and inherent biases notwithstanding. Next generation EBM will stem from insights, correlations, and causations even the most robust traditional research can only imagine.
I want to be crystal clear that I am not anti-science or against the idea of EBM. Quite the opposite. Research is the cornerstone of improving medical knowledge and advancing treatment. There are plenty of cases where the science is strong and irrefutable. The point here is that we need to think critically about evidence as it’s presented, allow room for experience and intuition, and design better mechanisms to capture data and develop insights.
We also need to stop manipulating data, misrepresenting or misinterpreting results, and cherry-picking spurious studies to support narratives and ulterior motives. There’s as much of that going around as there is low value care and non-evidence-based treatment.
- Ben
Small Incisions
Quick Takes on Timely Topics
Steward hospitals staying open after deals reached, Healey says (Boston Globe)
The Steward Health saga is gradually winding down in Massachusetts. Stewardship Health, Steward’s physician group, is being sold to PE-backed (and curiously named) Rural Healthcare group. Six of the eight Steward-owned hospitals will remain open, split between Boston Medical Center, Lawrence General Hospital, and RI-based Lifespan Health. This is a win for patients and access. Ralph de la Torre still has his date with Congress. What lessons have we learned? The evil that men do lives after them; the good is oft interred with their bones. So let it be with Steward.
What Will the Future of Retail Health Look Like? - MedCity News
On the heels of our post last week, MedCity News weighs in with their take on the current state of Retail Healthcare. “People don’t want to go to random providers in a super transactional way,” says Centivo CEO Ashok Subramanian. Preach, Ashok.
“People under siege try to extend the moat, not engage the enemy, and that describes the ‘strategy’ of most health economy stakeholders,” says Hal Andrews, President & CEO, Trilliant Health. Yep. Incumbents have created a nearly impenetrable morass, and they continue to add insular layers of protection. The solution: don’t play the game (or re-invent it). We also need to hold those who talk a big game but fail to produce accountable for empty words. Andrews throws his hat into the “consumer-driven” healthcare ring and correctly calls out legislators for picking on bit players while giving bullies a free pass.
Why Will Healthcare be the Industry that Benefits the Most from AI? | Andreessen Horowitz (a16z.com)
A trope I hope we retire in 2025: Fax machines in healthcare. We get it. Meanwhile, I’m also bullish on the idea that healthcare could be the industry that benefits most from AI (despite the hype cycle). I also agree that clinical judgement is the scarcest asset. It’s a feature, not a bug. The push for AI and technology to replace … er … enhance clinicians continues.
Many patients don't want Value Based Care — Will Kurtz, MD
The idea that patients don’t want VBC may incite pearl clutching for some. It shouldn’t. This excellent piece by orthopedic surgeon Will Kurtz explains why. I added my own thoughts here.
The Light Box
Healthcare Visuals
From the Gallery
Amplifying Community Voices
“If you are a healthcare outsider trying to disrupt the system with VBC, you need to understand how patients inadvertently and sometimes willfully increase inappropriate care if you want to successfully lower MSK spend.”
— Will Kurtz, MD, “Many Patients Don’t Want Value-Based Care”
Wheels Out — Signing Off and Looking Ahead
We shouldn’t use the imperfections of EBM as a justification for a Wild West approach where anything goes or as an excuse to be ignorant of science. A well-reasoned rationale for going against accepted norms is one thing. Substandard treatment stemming from an unwillingness to accept novel information goes against the principles of medicine. Worse yet, pseudo-science, false claims, and the willful practice of unnecessary or harmful treatment for personal gain have no place in healthcare.
Until next time,
Ben Schwartz, MD, MBA
Editor-in-Chief/Senior Clinical Fellow
Enlightening article. Love the idea of the FI. Read this interview this morning and I think is a good complement to this discussion. https://journals.lww.com/clinorthop/fulltext/9900/editor_s_spotlight_take_5__arthroscopic_partial.1706.aspx
An interview with Teppo Järvinen MD, PhD who directs FICEBO—The Finnish Centre for Evidence-Based Orthopaedics (www.FICEBO.com) specifically about arthroscopic meniscectomy for degenerative meniscal tears but really about the broader topic of EBM and changing practice patterns based on data and not anecdote and experience.