34 Critical appraisal
One of the first great methods for assessing the quality for evidence based practice came from the concept of critical appraisal, which was being taught in medical schools as early as the 1980s. Critical appraisal is the process of carefully assessing the scientific outcomes of research studies to judge their trustworthiness, value, and relevance to a particular context. It generally considers the type of article for strength of evidence and relevance to the topic at hand.
One commonly used tool in critical appraisal is the pyramid of evidence, which has many incarnations, like the one below:
As the pyramid goes up the quality of evidence tends to be stronger, depending on its relevance. This is intended to be a guide for helping clinicians remember which types of articles tend to better demonstrate evidence than others. However, it is important to remember that the methodology of a given article determines its efficacy and quality of evidence more than its type. A randomised controlled trial may have poor methodology and not demonstrate a high level of evidence despite it being a “high level” experimental study.
At the very bottom of the pyramid is background information, expert opinions, and non-evidence based guidelines, which do not demonstrate any level of evidence. Individual case reports are single instances of an occurring phenomena and are observational in nature, often relying on medical records and patient recall and not high in evidence but can be useful. Case Series and Case studies demonstrate slightly more evidence than a single case report because they are observational but will pool a series of case reports to prove more than a single instance of a phenomena. A cohort study takes an existing body of effected individuals and studies them, thus rounding out observational studies with the highest quality of an observational study.
In the experimental study realm are both non-randomised and randomised controlled trials. Randomised reduce more bias in its ability to study effects of an intervention, but this is not always possible, so sometimes non-randomised controlled trials must occur instead of experimental study
In the realm of critically appraised literature are evidence-based guidelines, which summarise evidence into recommended practices. Then comes systematic review—though the types of articles reviewed in the systematic review will also indicate how strong the evidence presented in the systematic review is as well. And lastly a Meta-analysis sits atop the pyramid for its ability to use statistical modelling to combine results of research studies.
How can I tell?
- The title of the article indicates what kind of research it is.
- Sometimes you can even include the type of article you would like to find (such as “case report” or “systematic review” or “randomised controlled trial” as a search term)
- The methods described by the article help you deduce what kind of article it is.
- When an article mentions using a retrospective approach to look back at medical records of people with a certain type of condition to look for certain factors or outcomes that is a cohort study.
| Study Type | Definition | Title words | Methods |
|---|---|---|---|
| Experimental | Experimental studies are ones where researchers introduce an intervention and study the effects. Experimental studies are often randomised, meaning the subjects are grouped by chance. | clinical trial, controlled trial, randomised controlled trial | Researchers will outline methodology thoroughly for experimental studies. Groups are screened for eligibility and assigned treatment and control groups using scientific method. |
| Observational | Observational studies are ones where researchers study the effect of a risk factor, diagnostic test, treatment, or other intervention without trying to change who is or isn’t exposed to it. | Cohort, case control, case study, case report, case series, retrospective | Researchers look at groups that are “linked” by common factors, they might use medical records, for example, or rely on patient recollection. They also might find an already effected population for data collection. |
Other factors to consider
There are many parts of a paper to review when critically analysing it for strength of evidence. Firstly, is the reduction of bias in the process of its creation. This can mean something as simple as conflicts of interest on behalf of the authors, which may be financial or bias in other ways (such as overlapping projects that may intersect with the interests of the article). Additionally, a look at the methodology of an article takes on importance as looking at whether or not randomisation is appropriate as a method in an experiment or if the statistical models used are appropriate for the approaches taken.
This chapter is adapted from “Critical appraisal” in Evidence Based Practice by Carrie Wade, used under a Creative Commons Attribution-NonCommercial 4.0 International Licence, except where otherwise noted.
