Watch Out For These Issues in Research

Zachary Walston

Reading research can be daunting. I cannot possibly cover all the ins and outs of understanding research in a single article - or a series of articles. I would like to point out a few key elements.

Controls and Placebos

On the first day of the study, you complete a protocol of seated knee extension exercises at 75% of your 1 rep max to failure for 6 sets. You are asked to return in 48 hours. When you waddle back into the clinic due to the massive amount of delayed onset muscle soreness (DOMS), you are provided a pill that that has been approved for human testing. The medicine shows great promise in reducing DOMS severity following physical exertion which will allow for more rapid return to activity. You have been informed of the risks and are a willing participant. You take the medicine and wait. The researchers check in with you after 20 minutes, 1 hour, and 4 hours to assess your pain levels. Remarkably, you feel much better at the 4-hour mark and quickly abandon the waddle. You feel so good in fact, that you are ready to take on another set of leg extensions to prepare for beach season and justify your recent purchase of debatably too short swim trunks. Alas, the researchers ask you to hold off and report pain scores at 24- and 72-hours post administration of the medicine.

You, as the reviewer, later find out the participant took a sugar pill, thus displaying the elevated power of the placebo effect. This warrants a closer look, however.

  • What if both the experimental group and the control group improved but the magnitude for the experimental group was greater?
  • How much greater would the difference have to be to matter?
  • What if the designed study did not include a control at all but the outcomes were outrageously positive?
  • Is the study still valuable and can you conclude the pill created the effect?
  • Would the placebo effect have been present if no information about the drug’s efficacy in previous trials was disclosed?

These questions are vital to thoroughly understand.

Selection Bias

Even if you decide research is for nerds – proud nerd here – and you determine you will never need to know how to set up a study, understanding the methodology of study design will allow you to critically appraise both literature and individuals who claim they’ve read the literature and are attempting to synthesize relevant information to regurgitate to the masses. Some do this far better than others (i.e. professors/researchers > journalists…most of the time) but how do you know when the reported information is accurate?

When looking at a study, the methods and results sections are by far the most important. In the methods, you immediately want to know about the setup and potential sources for bias, as these can muddy the waters of your results. For example, if the study above included patients ranging from age 12-92 years old and did not control for training experience, then there is a high likelihood of volatility in the results due to the variability in the participants. Conversely, if the population consists of healthy, trained college athletes ranging from age 18-22, the predictability is greater and you have a greater ability to translate to the general population, but only within those specific parameters (i.e. other healthy, trained college athletes ranging from age 18-22). The DOMs would likely be significantly dampened for someone with a history of training, which stimulates the protective phenomenon of the repeated bout effect, compared to a novice individual.

This highlights how the recruitment of individuals and the inclusion and exclusion criteria of studies can dramatically impact the results. We need to determine if the researchers obtained a representative population. Did they recruit participants similar to the target group the pain-relieving medicine is intended for upon release, or did they perhaps lure a bunch of people seeking a quick ‘fix’ for their addiction and the opportunity to try a new pain drug? Selection bias is a primary concern when evaluating the quality of a trial and determining the applicability of the results. Selection bias essentially means the researchers did not obtain a randomized group of participants and thus the sample is not representative of the population. This is important as the end goal is to be able to apply the results to future individuals with a reasonable amount of predictability.

Let’s say the recruitment was correctly carried out and you have a representative population who are appropriate for the trial. Next, we must determine the set-up for administering the treatment.

  • What was the education provided to the patient? Were they told to expect pain relief or not?
  • Were the experimenters blinded to who received the treatment and who received the placebo?

You may be surprised, but the latter can have a substantial impact on the outcomes. How is that you ask? Great question.

I Know Something You Don't Know

Studies have demonstrated that experimenters or providers act differently when they believe the intervention is a sham rather than the real deal.[1] This is a critical point for the world of rehabilitation. While we may be able to blind a patient to the sham intervention, such as setting up a thrust joint manipulation while only providing a light shear force, a clinician will know whether they provided a legitimate joint manipulation or not. Subtle changes in body language, effort, communication, and facial expressions can cue the patient that they are not receiving the experimental treatment and thus the potential improvement will be dampened. Let’s take it one step further.

The type of sham intervention matters as well. In many cases, a sham intervention may be a treatment. While a thrust did not occur during the sham manual technique, what was the effect of placing your hands on a patient? Was the hand placement on an area that was painful? Perhaps the patient experienced relief from the light touch acknowledgement of their pain. The sham can potentially highlight what components of the intervention are actually creating an effect. This leads to a lot of debate and controversy among treatments provided. In physical therapy, dry needling comes to the forefront regarding methodological quality and sham treatments.

It's a Sham

The intent of this post is not to dive into the efficacy of dry needling[2] but instead to demonstrate the need to evaluate sham procedures. At Combined Sections Meeting this year, I gave platform presentations on my two recent publications. In a platform presentation, 8 presenters are each given a 15-minute block to present their research and answer a couple of audience questions. In my second session, one of the presentations, “Short-Term Outcomes of Dry Needling for Patients with Mechanical Neck Pain: Randomized Clinical Trial” by Gattie et al, used sham dry needling.

The study utilized a sham needle device that blocked the needle from penetrating the skin (thin rubber material). The clinician was still able to replicate the tapping of the needle (when you insert it) and its manipulation (such as twisting), but the point would not penetrate the skin. The authors compared the pain and outcome scores of two statistically similar groups presenting with shoulder pain following physical therapy treatment. Each group received the same exercise protocol, but one received sham periscapular dry needling and the other received actual dry needling. The results demonstrated statistically similar improvements for both sham and experimental groups. Unfortunately, they did not have a control group only receiving exercise. But, the most fascinating outcome pertains to the assessment of the sham procedure.

At the end of the interventions, the participants were provided a questionnaire asking whether they believed they received the sham, the actual needle, or were unsure. No participant in either group (total n of 77) believed they received the sham, with only a couple more in the sham group stating they were unsure compared to the experimental group. Similar findings have been reported with other types of sham needles. Again, this research does not determine the efficacy of dry needling, but it is something to consider when attempting to determine what is contributing to an observed effect.

Comments / 0

Published by

I am a physical therapist, researcher, and educator whose mission is to challenge health misinformation. You will find articles about health, fitness, medical care, psychology, and professional development on my site. As the husband of a real estate agent, you will also find real estate and housing tips.

Atlanta, GA

More from Zachary Walston

Comments / 0