This Article
Right arrow Full Text
Right arrow Full Text (PDF)
Right arrow Letters to the Editor: Submit a response
Right arrow Alert me when this article is cited
Right arrow Alert me when Letters to the Editor are posted
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My File Cabinet
Right arrow Download to citation manager
Right arrowReprints and Permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Bhandari, M.
Right arrow Articles by Wright, J. G.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Bhandari, M.
Right arrow Articles by Wright, J. G.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us   Add to Technorati  
What's this?
The Journal of Bone and Joint Surgery (American) 86:1717-1720 (2004)
© 2004 The Journal of Bone and Joint Surgery, Inc.

Interobserver Agreement in the Application of Levels of Evidence to Scientific Papers in the American Volume of The Journal of Bone and Joint Surgery

Mohit Bhandari, MD, MSc1, Marc F. Swiontkowski, MD2, Thomas A. Einhorn, MD3, Paul Tornetta, III, MD4, Emil H. Schemitsch, MD5, Pamela Leece, BSc1, Sheila Sprague, BSc1 and James G. Wright, MD6

1 Department of Clinical Epidemiology and Biostatistics, McMaster University Medical Centre, 1200 Main Street West, Room 2C9, Hamilton, ON L8N 3Z5, Canada. E-mail address for M. Bhandari: bhandari{at}sympatico.ca
2 Department of Orthopaedic Surgery, University of Minnesota Medical School, 420 Delaware Street S.E., Box 492, Minneapolis, MN 55455
3 Department of Orthopaedic Surgery, Boston University Medical Center, 720 Harrison Avenue, Suite 808, Boston, MA 02118
4 Department of Orthopaedic Surgery, Boston University Medical Center, 818 Harrison Avenue, Boston, MA 02118
5 Division of Orthopaedic Surgery, Department of Surgery, St. Michael's Hospital, University of Toronto, 55 Queen Street East, Suite 800, Toronto, ON M5C 1R6, Canada
6 Division of Orthopaedic Surgery, The Hospital for Sick Children, 555 University Avenue, Suite 107, Toronto, ON M5G 1X8, Canada

Investigation performed at the Department of Clinical Epidemiology and Biostatistics, McMaster University Medical Centre, Hamilton, Ontario, the Departments of Orthopaedic Surgery, University of Minnesota Medical School, Minneapolis, Minnesota, Boston University Medical Center, Boston, Massachusetts, and the Division of Orthopaedic Surgery, St. Michael's Hospital, University of Toronto, Toronto, Ontario, Canada

The authors did not receive grants or outside funding in support of their research or preparation of this manuscript. They did not receive payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity. No commercial entity paid or directed, or agreed to pay or direct, any benefits to any research fund, foundation, educational institution, or other charitable or nonprofit organization with which the authors are affiliated or associated.

Background: Since January 2003, all clinical scientific articles published in the American volume of The Journal of Bone and Joint Surgery (JBJS-A) have included a level-of-evidence rating. The aim of the current study was to evaluate the interobserver agreement among reviewers, with varying levels of epidemiology training, in categorizing the levels of evidence of these clinical studies.

Methods: Fifty-one consecutive clinical papers published in the American volume of JBJS were identified by a computerized search of the table of contents from January 2003 through June 2003. Each paper was blinded so that only the title, abstract (without the level of evidence designated), and methods section were provided to the reviewers. The papers were coded and were randomly organized in a binder. Six surgeons graded each blinded paper for (1) the type of study (therapeutic, prognostic, diagnostic test, or economic or decision analysis), (2) the level of evidence (on a scale of I through V), and (3) the subcategory within the particular level of evidence. Three surgeons were members of JBJS American Editorial Board, two surgeons were reviewers for JBJS-A, and one surgeon was an active researcher not formally associated with JBJS-A. The reviewers did not receive any formal training in the application of the classification system, but each was provided with a detailed description of the classification system used by JBJS-A. Intraclass correlation coefficients with 95% confidence intervals were determined for the reviewers' agreement regarding the type of study, level of evidence, and subcategory within the level of evidence.

Results: The majority (69%) of the fifty-one included articles were studies of therapy, and 57% of the studies constituted Level-IV evidence. The intraclass correlation coefficients for the agreement among all reviewers with regard to the study type, level of evidence, and subcategory within the level of evidence ranged from 0.61 to 0.75. Reviewers trained in epidemiology demonstrated greater agreement (range in intraclass correlation coefficients, 0.99 to 1.0), across all aspects of the classification system, than did reviewers who were not trained in epidemiology (range in intraclass correlation coefficients, 0.60 to 0.75).

Conclusions: These findings suggest that epidemiology and non-epidemiology-trained reviewers can apply the levels-of-evidence guide to published studies with acceptable interobserver agreement. The validity of this system remains a question for future research.


Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Technorati Technorati    What's this?


This article has been cited by other articles:


Home page
 JBJSHome page
K. Okike, M. S. Kocher, C. T. Mehlman, J. D. Heckman, and M. Bhandari
Publication Bias in Orthopaedic Research: An Analysis of Scientific Factors Associated with Publication in The Journal of Bone and Joint Surgery (American Volume)
J. Bone Joint Surg. Am., March 1, 2008; 90(3): 595 - 601.
[Abstract] [Full Text] [PDF]

Home page
 JBJSHome page
N. Fletcher, D. Sofianos, M. B. Berkes, and W. T. Obremskey
Prevention of Perioperative Infection
J. Bone Joint Surg. Am., July 1, 2007; 89(7): 1605 - 1618.
[Full Text] [PDF]

Home page
 JBJSHome page
J. G. Wright
A Practical Guide to Assigning Levels of Evidence
J. Bone Joint Surg. Am., May 1, 2007; 89(5): 1128 - 1130.
[Full Text] [PDF]

Home page
 JBJSHome page
T. R. Lenters, A. K. Franta, F. M. Wolf, S. S. Leopold, and F. A. Matsen III
Arthroscopic Compared with Open Repairs for Recurrent Anterior Shoulder Instability. A Systematic Review and Meta-Analysis of the Literature
J. Bone Joint Surg. Am., February 1, 2007; 89(2): 244 - 254.
[Abstract] [Full Text] [PDF]

Home page
 Anesth. Analg.Home page
I. T. Cohen and K. Patel
Peer Review Interrater Concordance of Scientific Abstracts: A Study of Anesthesiology Subspecialty and Component Societies.
Anesth. Analg., May 1, 2006; 102(5): 1501 - 1503.
[Abstract] [Full Text] [PDF]

Home page
 JBJSHome page
W. T. Obremskey, N. Pappas, E. Attallah-Wasif, P. Tornetta III, and M. Bhandari
Level of Evidence in Orthopaedic Journals
J. Bone Joint Surg. Am., December 1, 2005; 87(12): 2632 - 2638.
[Abstract] [Full Text] [PDF]

Home page
 JBJSHome page
P. A. Cole and M. Bhandari
What's New in Orthopaedic Trauma
J. Bone Joint Surg. Am., December 1, 2004; 86(12): 2782 - 2795.
[Full Text] [PDF]