Thank you for this thoughtful approach to making the evidence hierarchy more relevant and understandable. I hope to use it ( and reference you) with my grad students.
As for its utility for the general public, I must add a major caveat. There is over 30 years of evidence showing that more than half of the adult US population reads at an 8th grade level and lower and struggles to understand basic health and science information – “low health literacy”. A person’s fundamental literacy and health literacy play an important role in the information they seek out, understand, and use, as well as their health outcomes (Nielsen-Bohlman et al., 2004; Schwartzberg, Van Geest, & Wang, 2005; Zarcadoolas, Pleasant, & Greer, 2006). Other important factors influencing consumers’ understanding are the technological skills they have (technological literacy).
Understanding the basic laws of physics, or distinguishing an opinion piece from a scientific report, is getting harder and harder. Just read many of the articles on the vaccine skeptical Children’s Health Defense Fund site https://childrenshealthdefense.org/ I recently listened to portions of the new ACIP meetings discussing vaccine safety and admit I was tripped up a number of times on whether something presented was valid evidence or cherry-picked.
The enormous and growing gaps between “elites” who understand these elements and the large number in the public sphere who do not is that millions now question the fundamental validity of science and medicine, feel betrayed by experts, and find truth in compelling alternative narratives?
I welcome you and other readers who would like to join me in thinking through, perhaps another period to augment the work you’ve done, targeted at the population I have mentioned here.
I completely agree… I envision this tool to be used in conjunction with an explanation by an expert, a simple scale they can point to when dissecting studies to say “here’s where this ranks”. Asking the public to determine where different studies rank entirely by themselves is certainly challenging and beyond the scope of what this can accomplish.
Again, I really like what you’ve done and I’ll need to think about it a bit more to see how it might Ridge to something. I’m working on particularly around the issue of evidence and measles vaccine positions.
What I have always appreciated about the Cochrane Library Reviews site, for example, is their practice of making a very clear differentiation between a conclusion for professional users, and a "Plain Language Summary" of the identical conclusions - presumably - for a lay audience. I believe, for the most part, they largely succeed.
Nevertheless, as someone who regularly follows a number of high profile sites that routinely can provoke five hundred comments after every post - and I am thinking of Paul Offit's site specifically - highly "contentious" and routinely graphic exchanges void of accurate, foundational information seem the point. As you say, "alternative narratives" dominate one side of the argument, while attempts at overwhelming with data dominates the other, frequently in the crudest and most insulting terms possible. It is a rare occasion that anyone ever cedes a point to another.
Likewise, I have noted that it rarely takes but moving a post or two past the opening post on SubStack these days to run headlong into a post from someone with an outrageous claim, that should you click, you generally will find a link to a legitimate study that is grossly misinterpreted. And I can only believe they must have the presumption that no one will actually read the study in question. For example, on a single day last week, I found such posts claiming that a study from India reported mass excessive deaths among elderly vaccinated men, without ever mentioning the authors claim of an association with the monsoon season & other weather events in India alone; claims of reduced maternal fertility after the Covid-19 vaccine, without mentioning it was a study conducted among rats alone; increased miscarriages following vaccination, without mentioning the article was devoted solely to an algorithm specifically developed to mitigate such problems; and lastly, a panel discussion of physicians who had all been stripped of either their medical licenses, specialty certifications, or hospital admission privileges, raging about how mRNA had "invaded" DNA, when the Karlinska Institute demanded they stop claiming their highly experimental paper be used in such a manner.
These are not typical times by any stretch of the imagination, and they demand specific, focused techniques to address them, if only because they are being sponsored and supported by the Sec. of HHS and his followers. This will, indeed, take a lot of hard thinking to properly address.
I received a thoughtful question on this post -- what do we do when we don't have the higher quality studies, either because the research isn't being prioritized or the disease is too rare to study in large trials? In that case, I think it's totally reasonable to go with the highest quality data that is available, understanding the limitations. If all that is available is case reports because the disease is so rare -- that's still valuable information, but should also be interpreted with the limitations in mind. Many times clinicians must make decisions with limited information, and a case report can be a valuable tool to help guide them. But if we have better studies, then those take precedence. That's the overall intent of the pyramid: to help people rank the quality of information, not to dismiss every study that doesn't fall into the top of the pyramid.
Kristen, this is great work. I also appreciate Cristina‘s comments about the challenges of reaching a wider audience. The misuse of evidence is so fundamental to the wide and growing critique of the current administration that finding a way to reach people is critical. I wonder if a good application of your tool might be some sort of comparative analysis on the use of evidence around a particular issue or event, for example, the recent ACIP meeting or statements related to measles vaccine. You might break out specific claims made, and rate them according to the quality of evidence. unfortunately, all too often the evidence cited by HHS officials is exactly zero, but that in itself is telling. Keep up the great work!
Thank you Kristen. This easy-to-grasp scale sets the bar for the level of information required to provide excellent medical care and is an easily used tool to refute the nonsense gushing from Kennedy/MAHA
Thank you for this thoughtful approach to making the evidence hierarchy more relevant and understandable. I hope to use it ( and reference you) with my grad students.
As for its utility for the general public, I must add a major caveat. There is over 30 years of evidence showing that more than half of the adult US population reads at an 8th grade level and lower and struggles to understand basic health and science information – “low health literacy”. A person’s fundamental literacy and health literacy play an important role in the information they seek out, understand, and use, as well as their health outcomes (Nielsen-Bohlman et al., 2004; Schwartzberg, Van Geest, & Wang, 2005; Zarcadoolas, Pleasant, & Greer, 2006). Other important factors influencing consumers’ understanding are the technological skills they have (technological literacy).
Understanding the basic laws of physics, or distinguishing an opinion piece from a scientific report, is getting harder and harder. Just read many of the articles on the vaccine skeptical Children’s Health Defense Fund site https://childrenshealthdefense.org/ I recently listened to portions of the new ACIP meetings discussing vaccine safety and admit I was tripped up a number of times on whether something presented was valid evidence or cherry-picked.
The enormous and growing gaps between “elites” who understand these elements and the large number in the public sphere who do not is that millions now question the fundamental validity of science and medicine, feel betrayed by experts, and find truth in compelling alternative narratives?
I welcome you and other readers who would like to join me in thinking through, perhaps another period to augment the work you’ve done, targeted at the population I have mentioned here.
I completely agree… I envision this tool to be used in conjunction with an explanation by an expert, a simple scale they can point to when dissecting studies to say “here’s where this ranks”. Asking the public to determine where different studies rank entirely by themselves is certainly challenging and beyond the scope of what this can accomplish.
Again, I really like what you’ve done and I’ll need to think about it a bit more to see how it might Ridge to something. I’m working on particularly around the issue of evidence and measles vaccine positions.
What I have always appreciated about the Cochrane Library Reviews site, for example, is their practice of making a very clear differentiation between a conclusion for professional users, and a "Plain Language Summary" of the identical conclusions - presumably - for a lay audience. I believe, for the most part, they largely succeed.
Nevertheless, as someone who regularly follows a number of high profile sites that routinely can provoke five hundred comments after every post - and I am thinking of Paul Offit's site specifically - highly "contentious" and routinely graphic exchanges void of accurate, foundational information seem the point. As you say, "alternative narratives" dominate one side of the argument, while attempts at overwhelming with data dominates the other, frequently in the crudest and most insulting terms possible. It is a rare occasion that anyone ever cedes a point to another.
Likewise, I have noted that it rarely takes but moving a post or two past the opening post on SubStack these days to run headlong into a post from someone with an outrageous claim, that should you click, you generally will find a link to a legitimate study that is grossly misinterpreted. And I can only believe they must have the presumption that no one will actually read the study in question. For example, on a single day last week, I found such posts claiming that a study from India reported mass excessive deaths among elderly vaccinated men, without ever mentioning the authors claim of an association with the monsoon season & other weather events in India alone; claims of reduced maternal fertility after the Covid-19 vaccine, without mentioning it was a study conducted among rats alone; increased miscarriages following vaccination, without mentioning the article was devoted solely to an algorithm specifically developed to mitigate such problems; and lastly, a panel discussion of physicians who had all been stripped of either their medical licenses, specialty certifications, or hospital admission privileges, raging about how mRNA had "invaded" DNA, when the Karlinska Institute demanded they stop claiming their highly experimental paper be used in such a manner.
These are not typical times by any stretch of the imagination, and they demand specific, focused techniques to address them, if only because they are being sponsored and supported by the Sec. of HHS and his followers. This will, indeed, take a lot of hard thinking to properly address.
I received a thoughtful question on this post -- what do we do when we don't have the higher quality studies, either because the research isn't being prioritized or the disease is too rare to study in large trials? In that case, I think it's totally reasonable to go with the highest quality data that is available, understanding the limitations. If all that is available is case reports because the disease is so rare -- that's still valuable information, but should also be interpreted with the limitations in mind. Many times clinicians must make decisions with limited information, and a case report can be a valuable tool to help guide them. But if we have better studies, then those take precedence. That's the overall intent of the pyramid: to help people rank the quality of information, not to dismiss every study that doesn't fall into the top of the pyramid.
Kristen, this is great work. I also appreciate Cristina‘s comments about the challenges of reaching a wider audience. The misuse of evidence is so fundamental to the wide and growing critique of the current administration that finding a way to reach people is critical. I wonder if a good application of your tool might be some sort of comparative analysis on the use of evidence around a particular issue or event, for example, the recent ACIP meeting or statements related to measles vaccine. You might break out specific claims made, and rate them according to the quality of evidence. unfortunately, all too often the evidence cited by HHS officials is exactly zero, but that in itself is telling. Keep up the great work!
I love this idea!
Thank you Kristen. This easy-to-grasp scale sets the bar for the level of information required to provide excellent medical care and is an easily used tool to refute the nonsense gushing from Kennedy/MAHA
Yes!!! This is extremely helpful! Thank You!!