When the index trials are removed, the effect sizes drop between 40% to 70% in each systematic review or clinical guidelines. ((Image by O’Connell et al., licensed under CC BY 4.0)
A group of British, Australian, and Finnish researchers identified eight clinical trials that skew the results of 32 systematic reviews and 10 clinical practical guidelines for spinal pain, according to their study published in The Journal of Pain in 2023. These trials examine cognitive behavioral therapy (CBT) and/or physical therapy for chronic back pain or neck pain.
When they removed these trials—which they called “index trials”—from the data, the effect size was reduced by an average of 58% with an interquartile range (IQR) of 40% to 74%. The IQR is the spread or range of the middle half of a set of numbers. It gives a sense of where most of the typical values lie without being thrown off by extreme outliers—like a huge or tiny value.
In this case, this means the middle 50% of all the comparisons had reductions between 40% and 74%.
“We saw the same pattern of reduced effect sizes when we grouped analysis by outcome (pain or disability) or clinical population (chronic pain or postsurgical pain),” the researchers wrote.
All eight index trials were headed by Dr. Marco Monticone from the University of Cagliari in Sardinia, Italy. According to Retraction Watch in 2024, Monticone had seven retractions total with three papers pulled in 2022.
Dr. Neil O’Connell of Brunel University in the U.K., who was the lead author of 2023 review, told Retraction Watch that they found “serious concerns” in the evidence that warrants consideration for retraction.
“The [two] trials that look like outsiders were both small pilot studies (2014, 2018) with some concerns but less clear. However, after our investigation we flagged all 10 trials to the respective editors and shared our work. There is a wider question of trust when concerns are raised across multiple papers from a single author team,” O’Connell said.
Back to the 2023 review, O’Connell et al. identified key concerns in the eight trials, including:
- Lack of research governance (e.g. no pre-registration, no documentary confirmation of ethical approvals, lack of sharing data upon request)
- Data anomalies (e.g. having duplicates or data with highly similar results)
- Low to no attrition of participants in some studies
- Implausible results (e.g. “extremely large” effect sizes that diverge from scientific literature)
“On that basis, we recommended that they should not be included in evidence syntheses and clinical guidelines in this area,” they wrote.
Skewing data and clinical guidelines
O’Connell et al. reported that 10 of the systematic reviews did not conduct meta-analyses, rather, they used a narrative review. Nine of these reviews “drew broadly positive conclusions regarding the effectiveness of the interventions [CBT and exercise]…underpinned in part by evidence from the index trials.”
Only two reviews reported the “heterogeneity”—or mixed research methods. One of these reviews wrote that the Monticone trial is “an example of the potential benefits of group-based therapy, and another commented that studies with a higher number of treatment sessions found larger effects on pain intensity than those with fewer sessions,” O’Connell et al. wrote.
“No review raised specific concerns regarding the veracity of the data from index trials,” they added.
The researchers also found:
- 20 of the systematic reviews reported “broadly positive conclusions regarding the interventions”
- Six reported “cautiously positive conclusions which were qualified by issues of the quality of the evidence or the size of treatments”
- Six reported negative or questionable conclusions
Impact on clinical practice guidelines
O’Connell et al. identified nine CPGs used index trials in their evaluations of treatments, such as CBT, exercise, and multimodal or biopsychosocial interventions for spinal pain. All but one guideline made positive treatment recommendations based on these trials, and none expressed concerns about the quality or trustworthiness of the index trials.
They highlighted the following:
- The American Physical Therapy Association included a 2016 trial by Monticone et al. that was the only one showing benefit for general exercise, which influenced a recommendation for using it.
- The Canadian Agency for Drugs and Technologies in Health cited a 2017 Monticone trial as the only new trial showing benefit and support of multidisciplinary pain treatment.
- The Canadian Chiropractic Guideline included a 2012 Monticone trial that showed no long-term benefit but still made positive recommendations based on the other trials.
- The Finnish Medical Association included the same 2012 Monticone trial and said the evidence for CBT was weak and lacked detail.
- The Japanese Orthopedic Association included a 2013 Monticone trial. Removing this trial in re-analysis led to smaller treatment effects, but the guideline still recommended physiotherapy.
- The Dutch NHG Guideline included a 2013 Monticone trial and stated the results were unclear but still questioned the relevance of CBT for chronic low back pain.
- NICE (U.K.) and KCE (Belgium) included a 2013 and 2016 Monticone trial and found mixed evidence but relied heavily on these trials to recommend multidisciplinary rehab as cost-effective.
- The North American Spine Society (NASS) included a ENTRUST-PE is a start marker and recommended fear-avoidance interventions but gave no clear guidance on adding CBT to exercise.
- The Russian Pain Society included 2013 and 2016 Monticone trials (same as NICE and KCE) and recommended multidisciplinary programs but provided little detail on how evidence supported this.
Including the index trials made treatments look more effective than they really are, caused more inconsistency in the data, and made the results seem more precise and positive than they actually are, according to O’Connell et al.
“In some cases, excluding index trials shifted effects from statistically significant to nonsignificant,” they wrote.
“We have not accused Dr. Monticone of misconduct. We have only identified what we think are important concerns about the trustworthiness of those papers,” O’Connell said in an interview with Massage & Fitness.
He also said that the clinical guidelines would need to be updated based on the new evidence and see whether or not the recommendations are still robust without the index trials.
Reflections
O’Connell and another group of colleagues wrote a letter to The Journal of Clinical Epidemiology in 2023 to build trust in research.
“While error and fraud are acknowledged as widespread, some editors appear disinclined to believe that their journals are affected; and that whistleblowers may still need to pursue concerns that are editors’ responsibilities,” they wrote. “Left unaddressed by those tasked with tackling them, such data go on to influence evidence reviews and guidelines and increase the risk of harm to patients. We no longer worry about being too skeptical–we worry that we are not skeptical enough.”
In 2024, O’Connell and his colleagues developed the ENTRUST-PE framework (ENhancing TRUSTworthiness in Pain Evidence) to address the systemic problems in pain research. The framework revolves around seven core values:
- Equity, inclusion, and diversity
- Governance and integrity
- Patient and public involvement
- Methodological rigor
- Transparency and openness
- Balanced communication
- Data authenticity
O’Connell said in an interview that ENTRUST-PE is a start marker to allow conversations within the pain research community about improving trustworthiness in pain research.
Back to the 2023 review, O’Connell et al. wrote that if untrustworthy trials are not identified and removed during the development process of reviews and clinical guidelines, then “the conclusions and recommendations of those reviews and guidelines are at risk of being incorrect, with a potentially major impact on patient care.”
“This issue is compounded by an academic and publishing system that is generally slow, inefficient and inconsistent in dealing with scientific error, issues of misconduct, and research integrity, and where mistakes are often uncorrected, raising the likelihood of negative impact,” they wrote.
Nick Ng is the editor of Massage & Fitness Magazine and the managing editor for My Neighborhood News Network.
An alumni from San Diego State University with a bachelor’s in graphic communications, Nick also completed his massage therapy training at International Professional School of Bodywork in San Diego in 2014. In 2021, he earned an associate degree in journalism at Palomar College.
When he gets a chance, he enjoys weightlifting at the gym, salsa dancing, and exploring new areas in the Puget Sound area in Washington state.
