Copyright © 2006 by The Journal of Bone and Joint Surgery, Inc.
Commentary & Perspective
Commentary & Perspective by
Seth S. Leopold, MD*,
University of Washington Medical Center, Seattle, Washington
A "pilot study." Those words, in the title of an article, tend to evoke a conflicted response from me. As a clinical investigator, I recognize the value of preliminary data as a guide for future work on a topic. On the other hand, as a clinician, it is sometimes hard to avoid the temptation to take pilot data and use them to guide therapy.
But yielding to the temptation to use the data presented in this study to guide therapy would be ill-advised. Even within the context of a pilot study, this report had substantial methodological problems and a surprisingly overt bias in its presentation in favor of the product in question.
From a methodological standpoint, some readers may perceive that the authors "dredged" the data to arrive at the results presented in the paper. In a study of only seventeen patients, the authors defined, a priori, one primary and eight secondary study end points. Counting all the subdomains of the outcomes instruments used, there were, in fact, more end points analyzed than patients who completed the study. Yet despite this, most of the results section is spent analyzing end points that were either not among the nine pre-defined major goals of the paper, or were arbitrary and nonvalidated modifications of those goals. For example, after they learned that Hyalgan was not more effective than placebo at improving pain or ankle arthritis scores, the authors analyzed, post hoc, four "levels of discrimination" (>30-mm difference, as well as ≥20%, 50%, and 70% changes) to try to discern a level of "clinical responsiveness" that was not a component of any of the nine predefined study end points. Fortunately or unfortunately, even with this sort of statistical sleight of hand, nothing hit the magic p < 0.05 level. And although the overall SF-12 scores were not significantly different between Hyalgan and placebo, after combing through the various SF-12 subdomains, the authors concluded that Hyalgan increased patient "vitality" at the p = 0.03 level. Whether the use of a joint injection for ankle arthritis can somehow affect a patient's "vitality" (without alleviating that patient's pain or improving that patient's function more than could be effected with use of placebo) or whether this is spurious significance resulting from conducting dozens of analyses at the p < 0.05 level is something readers will have to conclude for themselves.
Take-home message? Hyalgan and saline placebo injections both resulted in modest improvements in ankle arthritis scores, although Hyalgan was no better than placebo (p = 0.32). And as it turned out, the positive deflections in ankle arthritis scores in both study groups were in the range of what one would expect from a true pharmacological placebo1-4: in this case, an improvement of about 34%. So, in essence, this was a negative study. There's certainly nothing wrong with reporting a negative result; in fact, analyses of publication bias (including ones on viscosupplementation5) suggest that small negative studies systematically fail to get published, and this represents a critical problem with our literature. Yet despite data showing, fairly clearly, no difference between Hyalgan and placebo, the authors (who were funded by the manufacturer of the product in question) spend much of the paper emphasizing "differences" that didn't achieve statistical significance, and questioning the literature, which of late has drifted away from its earlier support of viscosupplementation5. They look for reasons why their placebo saline injections somehow might not have behaved as a placebo, but rather served as some kind of an unexpected treatment whose effects lasted an amazing six months. And despite a "no difference" results section, the authors emphasize the safety and/or efficacy of Hyalgan at least five times in the context of a brief discussion section, leaving a strong impression that the investigators are conducting advocacy journalism. In the context of commercially-funded research, this sort of writing is likely to cause readers to wonder whether modifications ought to be made to peer review that would require disclosures of financial relationships earlier in the process.
Although the conclusions drawn by the authors went well beyond what can be substantiated on the basis of the study design and the results, and although ultimately the trial was not convincing as clinical research, this paper certainly has several strengths. The study confronts a reasonably common, highly morbid condition (ankle arthritis), for which there are few good therapeutic options. The authors used randomization to allocate treatments, a good approach that is only used in about 3% of orthopaedic publications6. If subsequent investigators are able to read past the conversion of a no-difference data set to a product endorsement and focus instead on the usable results presented in the paper, there probably are sufficient raw data here to help render sample-size calculations in future studies, although a more explicit presentation of pain scores (means or medians, as well as measures of the spread of the data) would have been more helpful. And finally, the authors candidly state that their work requires confirmation in larger trials of appropriate design, presumably before recommending the use of the product in question in a clinical setting for ankle arthritis. I agree with this last caveat wholeheartedly.
*The author did not receive grants or outside funding in support of his research for or preparation of this manuscript. He did not receive payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity. No commercial entity paid or directed, or agreed to pay or direct, any benefits to any research fund, foundation, educational institution, or other charitable or nonprofit organization with which the author is affiliated or associated.
References
1. Mulrow CD, Williams JW Jr, Trivedi M, Chiquette E, Aguilar C, Cornell JE, Badgett R, Noel PH, Lawrence V, Lee S, Luther M, Ramirez G, Richardson WS, Stamm K. Treatment of depression--newer pharmacotherapies. Evid Rep Technol Assess (Summ). 1999;7:1-4.
2. Depression Guideline Panel. Depression in primary care: volume 2, Treatment of major depression. In: Clinical practice guideline, Number 5. Rockville, MD. U.S. Department of Health and Human Services, Public Health Service, Agency for Health Care Policy and Research. AHCPR Publication No. 93-0551. March 1993.
3. Hughes R, Carr A. A randomized, double-blind, placebo-controlled trial of glucosamine sulphate as an analgesic in osteoarthritis of the knee. Rheumatology (Oxford). 2002;41:279-84.
4. Karlsson J, Sjogren LS, Lohmander LS. Comparison of two hyaluronan drugs and placebo in patients with knee osteoarthritis. A controlled, randomized, double-blind, parallel-design multicentre study. Rheumatology (Oxford). 2002;41:1240-8.
5. Lo GH, LaValley M, McAlindon T, Felson DT. Intra-articular hyaluronic acid in treatment of knee osteoarthritis: a meta-analysis. JAMA. 2003;290:3115-21.
6. Leopold SS, Warme WJ, Fritz Braunlich E, Shott S. Association between funding source and study outcome in orthopaedic research. Clin Orthop Relat Res. 2003;415:293-301.
|
