“You have to evaluate each hypothesis in the light of the evidence of what you already know about it.” – R.A Fisher
For the past two years, I have worked with Emory DPT students to develop and write systematic reviews (SR). It has been a steep learning curve and a challenge; however, it has taught me a great deal about the assessment of literature. One of the tools used when conducting an SR is an assessment of rigor. In essence, these charts provide a framework for evaluating the quality of evidence through the assessment of potential sources of bias and the soundness of the methodology.
For example, The Medlicott & Harris Evaluation of Methodological Rigor assesses the following: randomization, presence of inclusion/exclusion criteria, if the subject pools were similar at baseline, repeatability of the treatment protocol, whether outcome measure reliability and validating are included, was the assessment blinded (could be broken down to single vs. double-blind), presence of long-term follow-up, and mention of adherence to home program (obviously, specific to rehab). When you scan this list, how many of these qualities do you look for when reading a study? This list is not exhaustive either (e.g. selectivity biases and conflicts of interest).
There are additional subtleties to consider depending on the type of study being conducted – randomized control trial, cohort, quasi-experimental – but it should highlight the importance of analyzing how the methods are developed and carried out.
Another important factor to assess is the controlling of variables. Whether it is the pain pill experiment or assessing a real vs. sham manipulation, we, unfortunately, cannot control for all the variables that may impact the outcome of a treatment. In the case of the DOMS reducing drug, we certainly have more control though. The intervention itself is pretty straightforward…swallow the pill. For the manipulation, however, we have more to consider.
How skilled is the practitioner? Aside from the difference in the delivery of the techniques, what is the patient’s belief about manipulation and cavitations? If they hate the sound of cracking knuckles or do not like being up close and personal with someone, they may tense or simply respond poorly as the experience itself was negative. How about the state of mind and body of the patient? This can affect any type of experiment and intervention. If they came into the experiment and had a rough night of sleep, had a fight with a significant other, were recently laid off at work, or they are a Browns fan and the NFL season just started, chances are their foul mood could result in a dampened experience; now is not the best time to conduct many types of trials.
Conversely, the subject who recently received a raise went on a great first date or slept gloriously the previous night will be in a far better state of mind and more likely to respond well to the treatment. Again, this list is not exhaustive – their diet, past experiences, connection with the clinician, temperature and comfort of the treatment room, the number of times they have watched the current episode of Fixer Upper playing on the TVs in the clinic – but it underscores the inability to control all variables. We can draw incomplete and inaccurate conclusions if we simply look at the outcome following an intervention and ascribe all the credit to said intervention.
In my next post, I will discuss the vital section to read in a paper, the results section. In the meantime, I encourage you to dive deeper into the methods section of the papers you are reading. There will legitimately be papers that warrant no further consideration after the methods section as you determine the paper is essentially useless. It is frustrating, but unfortunately, often the case. Your ability to translate evidence into clinical practice will improve as you develop your critical thinking skills. You will better understand the reasoning for treatment choices you may not currently utilize in the clinic. It will take practice – I am still learning more about this every day – and you will have to research about research to have a greater grasp on the concepts (yes, it is as fun as it sounds). You will also start to note that abstracts are heavily biased and only tell a portion of the story. This will become even more apparent after the next post. I promise it is worth the added effort and will help you become a more skilled and evidence-informed clinician