 |
 |
 |
|
| home | CME | help | feedback | subscriptions | archive |
|
© 1996 The Journal of Bone and Joint Surgery, Inc. Interobserver Reliability and Intraobserver Reproducibility of the Modified Ficat Classification System of Osteonecrosis of the Femoral Head*   , CHARLOTTE, NORTH CAROLINAInvestigation performed at the Department of Orthopaedic Surgery, Carolinas Medical Center, Charlotte
Anteroposterior and lateral plain radiographs of 116 osteonecrotic femoral heads were reviewed to assess the interobserver reliability and intraobserver reproducibility of the modified Ficat classification system. The radiographs were reviewed initially and then again six months later by three adult reconstructive surgeons, two general orthopaedic surgeons, two orthopaedic residents, and one musculoskeletal radiologist. All eight observers agreed on the classification of twenty hips (17 per cent) at both the first and the second review of the radiographs. Paired comparisons revealed a mean interobserver kappa reliability coefficient of 0.46 (range, 0.30 to 0.67) for the first review and 0.45 (range, 0.30 to 0.66) for the second. For all observers, the mean rate of perfect agreement between the first and the second review was 68 per cent (range, 56 to 80 per cent). The mean kappa value for intraobserver reproducibility was 0.59 (range, 0.44 [one of the residents] to 0.73 [one of the general orthopaedic surgeons]). No observer or pair of observers had excellent reproducibility or reliability (  > 0.75).
The poor interobserver reliability and fair intraobserver reproducibility diminishes any meaningful comparison of studies in which the modified Ficat classification system has been used and illuminates the need for a more reliable and reproducible classification system.
Osteonecrosis of the femoral head has been most commonly classified according to the system of Ficat4,7,10,11,26 and, more recently, the modified system of Ficat15,23,33,34 (Table I). Many authors have recommended treatment on the basis of the symptoms and the Ficat classification as determined with the use of plain radiographs1,19,23,26,33. Treatment with pulsed electromagnetic fields or core decompression, or both; bone-grafting and decompression; and rotational transtrochanteric osteotomy have been suggested for Ficat stage-I lesions1,3,15,23,25,28,29,32,33. These procedures as well as vascularized fibular grafting, rotational transtrochanteric osteotomy, and intertrochanteric osteotomy have been recommended for stages IIA and IIB1,3,12,18,25,26,28,29,32-34. All of these procedures as well as use of surface replacement and total hip replacement have been recommended to treat stage-III lesions1,3,6,12,16,18,19,25,26,29,34. Pulsed electromagnetic fields, surface replacement, and total hip replacement have been recommended for the treatment of stage-IV lesions3,6,13,16.
The choice of treatment and the judgment of its efficacy often are based directly on the Ficat stage. Determination of the Ficat stage, therefore, has important consequences, as it has a direct effect on the patient's clinical course. A classification system is a means of description that should be biologically meaningful and be reproducible from one observer to the next as well as by one observer on separate occasions. The absence of reproducibility clouds the comprehension and comparison of studies and treatment recommendations that are based on such a classification system. The purpose of the present study was to assess the degree of interobserver reliability and intraobserver reproducibility of the modified Ficat classification system for osteonecrosis of the femoral head.
One hundred and twenty-one symptomatic, untreated, osteonecrotic femoral heads were assessed consecutively with standard anteroposterior and lateral plain radiographs at our clinics between 1988 and 1993. Hips that did not have any radiographic changes had changes evident on magnetic resonance imaging that were consistent with osteonecrosis20. The acceptability of each radiograph was determined by an orthopaedic surgeon who was not an observer in this study. The radiographs of five hips were determined to be unacceptable; therefore, 116 hips were included in the study. All identifying data on the radiographs were obscured, and the radiographs were numbered randomly. Each radiograph was reviewed and classified by eight observers: three adult orthopaedic reconstructive surgeons, two general orthopaedic surgeons, two fifth-year orthopaedic residents, and one musculoskeletal radiologist. All of the observers were familiar with the Ficat classification system7 and had used it in clinical situations previously. One week before the radiographic review, each observer was provided with and asked to review a copy of Ficat's description of his classification system7 and each was given information regarding the modified Ficat system. The observers were also provided with a copy of the classification system during testing and were allowed to refer to this as often as necessary. No questions or discussion were allowed during or after testing, and the observers were given as much time as they needed to review each radiograph. After a decision had been made, the radiographs of the next hip were presented until all 116 hips had been classified. The radiographs were classified by each observer on two separate occasions, six months apart. In the interim, the radiographs were not available to any of the observers and no feedback was provided. The second review was performed in a similar manner, except that the order of the radiographs was reversed.
Statistical Analysis The guidelines of Svanholm et al. were used for interpretation of the kappa values30. Values of less than 0.50 indicated poor agreement and those of more than 0.75 indicated excellent agreement. Accuracy, or how close an experimental observation lies to a true value, was impossible to measure because the correct classification for each osteonecrotic hip was not known. We therefore assessed the level of agreement between paired observers (interobserver reliability) and between the reviews of the same observer (intraobserver reproducibility) over time.
Interobserver Reliability Paired comparisons among the classifications of the eight observers for each radiograph produced twenty-eight paired analyses for each of the two reviews of the radiographs. The mean interobserver reliability coefficient was 0.46 (range, 0.30 to 0.67) for the first review and 0.45 (range, 0.30 to 0.66) for the second. The mean reliability coefficient with respect to the level of expertise was 0.41 for the orthopaedic residents, 0.51 for the adult reconstructive surgeons (who had a mean of 7.7 years of experience), and 0.60 for the general orthopaedic surgeons (who had twenty and fifteen years of experience).
Intraobserver Reproducibility The mean kappa intraobserver reproducibility coefficient was 0.59 (range, 0.44 to 0.73) among the eight observers (Table II). The mean reproducibility was 0.60 (0.66, 0.58, and 0.56) for the adult reconstructive surgeons, 0.49 (0.54 and 0.44) for the orthopaedic residents, and 0.62 (0.73 and 0.55) for the general orthopaedic surgeons. The mean perfect agreement between the first and the second review was 68 per cent (range, 56 to 80 per cent).
Paired comparison of the two observers with the highest reproducibility revealed kappa reliability coefficients of 0.49 and 0.55 during the first and second reviews, respectively.
Orthopaedic classification systems provide subdivisions in the spectrum of presentation of certain disease processes. These subdivisions, in turn, may be studied individually with respect to diagnosis, treatment, prognosis, and outcome. For such a classification system to be useful, it must be reproducible among different observers as well as by the same observer on separate occasions. Without these qualities, a precise language on which scientific study and comparison can be based is impossible. Reported evaluations of orthopaedic classification systems2,5,9,17,21,22,31 have yielded disappointing results, with kappa values ranging from 0.40 to 0.57 for interobserver reliability and from 0.58 to 0.69 for intraobserver reproducibility. Recently, Kay et al. reported a kappa statistic of 0.82 for intraobserver variability and 0.56 for interobserver variability for twenty-five hips that were assessed with use of the Ficat classification system on three occasions by six observers14 (Table III).
Svanholm et al. stated that "since we are dealing with the minimal requirement that the criteria under study are reproducible, and since the lower limit of the reproducibility defines the upper limit of the significance of the parameter in biological terms, we advocate that  0.50 should be taken as poor, and  0.75 as excellent reproducibility."30 We modified this scheme so that kappa values that were between 0.50 and 0.75 were interpreted as fair. According to these modified guidelines, the present study demonstrated poor interobserver reliability and fair intraobserver reproducibility, with no instances of excellent intraobserver reproducibility or interobserver reliability. In simpler terms, during approximately one of every three reviews, an educated observer classified an osteonecrotic hip differently than a colleague did and, in fact, differently than he himself had classified the hip six months previously. The observers were least likely to change their classification of hips that they had initially classified as stage I or IV, presumably because these stages represent clear visual images of either a normal hip without changes or a severely osteonecrotic hip with dramatic changes. Hips that were classified initially as stage IIA, stage IIB (transition), or stage III were twice as likely to be classified differently by the same observer than those classified as stage I or IV. While we did not address the reasons for this, the discrepancy may result from various interpretations of the phrases used in Ficat's original description of the middle stages7, such as diffuse sclerosis in the description of stage II; crescentic line, segmental flattening (or out-of-round appearance) in the description of transition; and sequestrum and collapse in the description of stage III. The spectrum of the osteonecrotic disease process, stratified into rigid classifications, creates inevitable gray areas at the boundaries of each grade. The use of language to describe a radiograph is fraught with imprecision and is subject to the observer's interpretation of both the language and the radiograph. An out-of-round appearance to one observer may be collapse to another. These gray areas and the various interpretations of Ficat's language cause variability in the application of this classification system. These results, especially with regard to the middle stages of classification, are important and disturbing for several reasons. In the absence of excellent interobserver reliability, it is not sound to compare studies of similarly classified hips from different centers30. Also, in the absence of excellent intraobserver reproducibility, it is not sound to rely on outcome studies that are based on plain radiographs that were assessed over a period of time, before and after treatment, within the same center, or by the same observer or group of observers. Many methods of treatment for the various stages of osteonecrosis of the femoral head have been described, with variable results1,3,15,16,18,19,23-29,32-34. Widely divergent recommendations for treatment as well as greatly differing prognoses often are based on use of the Ficat classification as it reflects the presence or absence of subchondral, segmental, or advanced collapse of the femoral head. It is reasonable to assume that one cause of the present conflict in treatment recommendations is the different interpretation and use of the Ficat classification by different investigators. The modified Ficat classification for osteonecrosis of the femoral head does not have acceptable interobserver reliability and intraobserver reproducibility on which to base treatment protocols and determinations of outcome. Magnetic resonance imaging and computed tomography offer a more detailed view of the involvement of the femoral head, subchondral collapse, narrowing of the joint space, and acetabular changes found with the progression of this disease. Future treatment recommendations and outcome analysis may be more reliably and reproducibly based on these imaging techniques.
NOTE: The authors give special thanks to Walter B. Beaver, M.D.; James Coumas, M.D.; Thomas K. Fehring, M.D.; Forney Hutchinson, III, M.D.; and Jeffrey G. Mokris, M.D., for participating in the study. The authors also thank Kim Gravitte for her technical assistance in the statistical analysis of the data.
*No benefits in any form have been received or will be received from a commercial party related directly or indirectly to the subject of this article. No funds were received in support of this study.
¶Tennessee Orthopaedic Associates, 301 21st Avenue North, Nashville, Tennessee 37203.
  CiteULike  Connotea  Del.icio.us  Facebook  Technorati  Twitter What's this?
This article has been cited by other articles:
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Top
Abstract
Introduction
| home | CME | help | feedback | subscriptions | archive | site index |
|
Stanford University Libraries' HighWire Press (TM) assists in the publication of JBJS Online. |