 |
 |
 |
|
| home | CME | help | feedback | subscriptions | archive |
|
© 1997 The Journal of Bone and Joint Surgery, Inc. Quality of Data Regarding Diagnoses of Spinal Disorders in Administrative Databases. A Multicenter Study*  , MARSHFIELD, WISCONSIN , KNOXVILLE, TENNESSEEInvestigation performed at Marshfield Clinic, Marshfield, Wisconsin; Knoxville Orthopedic Clinic, Knoxville, Tennessee; Rothman Institute, Philadelphia, Pennsylvania; Orthopedic Associated of Dallas, Dallas, Texas; Texas Back Institute, Plano, Texas; Washington Hospital Center, Arlington, Virginia; Minnesota Spine Center, Minneapolis, Minnesota; and University of Iowa, Iowa City, Iowa
The purpose of the present study was to evaluate the accuracy of data regarding diagnoses of spinal disorders in administrative databases at eight different institutions. The records of 189 patients who had been managed for a disorder of the lumbar spine were independently reviewed by a physician who assigned the appropriate diagnostic codes according to the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM). The age range of the 189 patients was seventeen to eighty-four years. The six major diagnostic categories studied were herniation of a lumbar disc, a previous operation on the lumbar spine, spinal stenosis, cauda equina syndrome, acquired spondylolisthesis, and congenital spondylolisthesis. The diagnostic codes assigned by the physician were compared with the codes that had been assigned during the ordinary course of events by personnel in the medical records department of each of the eight hospitals. The accuracy of coding was also compared among the eight hospitals, and it was found to vary depending on the diagnosis. Although there were both false-negative and false-positive codes at each institution, most errors were related to the low sensitivity of coding for previous spinal operations: only seventeen (28 per cent) of sixty-one such diagnoses were coded correctly. Other errors in coding were less frequent, but their implications for conclusions drawn from the information in administrative databases depend on the frequency of a diagnosis and its importance in an analysis. This study demonstrated that the accuracy of a diagnosis of a spinal disorder recorded in an administrative database varies according to the specific condition being evaluated. It is necessary to document the relative accuracy of specific ICD-9-CM diagnostic codes in order to improve the ability to validate the conclusions derived from investigations based on administrative databases.
Administrative databases have been used to evaluate many aspects of the health-care-delivery system, and a variety of conclusions have been drawn from data collected in this manner. Because such data influence the current debate on health-care policy, these conclusions have important implications for patients, providers, and society as a whole5,6,13,15. The validity of conclusions derived from research involving administrative databases depends on the quality of the data in the databases. To our knowledge, the quality of the data regarding the diagnosis of spinal disorders in administrative databases has not been studied previously. The processing of information in medical records to be entered into databases for later analysis follows a typical sequence in most hospitals. First, the clinical information in the medical record and the discharge summary is abstracted by personnel in the medical records department. The data identified include demographic characteristics, the diagnoses, the procedures that were performed, and any complications. Numerical codes for diagnosis, procedure, and complications are assigned according to the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM). The ICD-9-CM codes for each hospitalization then are collated into what is called a discharge abstract, which is meant to reflect the actual inpatient clinical experience. Discharge abstracts from different hospitals may be pooled at a central data-processing center where a particular database is managed, such as the Commission of Professional and Hospital Activities or the Health Care Financing Administration. In addition, more than forty of the fifty states in the United States have statewide databases for discharge abstracts; such databases are frequently used by health-service investigators addressing issues in clinical and health-policy research7. There is an increasing reliance on these electronic administrative databases in many fields of medicine. For example, administrative databases have been used to compare regional and local variation in the rates of complications at hospitals and for the development of severity indices9,16. Not surprisingly, debate about and an awareness of the problems with administrative databases are developing. Investigators as well as professionals involved in quality-assurance and quality-improvement programs in hospitals, who frequently rely on the information in administrative databases, have raised concerns about the quality of data. The problems that have been identified include institutional differences in coding practices and in definitions of data as well as incomplete and inaccurate data1-3,10,13,14,18,20. To better understand the limitations of conclusions derived from data obtained from administrative databases, we sought to evaluate the accuracy of commonly used ICD-9-CM codes for diagnoses related to the spine.
Eight surgeons from eight hospitals each provided the records from consecutive patients in whom a procedure had been performed on the lumbar spine. Seven surgeons each provided twenty-five records, and one provided twenty-two. The hospitals were a convenience sample of spine centers at which the surgeons were active members of either the Scoliosis Research Society or the North American Spine Society. None of the hospitals or the practices of the participating physicians were connected in any fashion with regard to their coding practices. The personnel in the medical records department at all of the hospitals used the St. Anthony's ICD-9-CM Code Book as its reference for coding. The records of patients who had a tumor, deformity, or fracture were excluded. A research coordinator received all of the clinical data that had been collected during each patient's hospitalization; these data included the medical history and physical findings, the progress notes, the discharge summary, the operative report, and the radiological reports. The personnel at each of the eight hospitals also were asked to include the regularly coded discharge abstract, which contains the ICD-9-CM codes for the diagnosis and procedure, for each record. Information identifying the patient was obliterated from the records. One hundred and ninety-seven records were reviewed for completeness by the research coordinator. The records of eight patients did not meet the entry criteria: six patients had involvement of the cervical spine, one had an infection, and one had a tumor. Therefore, 189 records, none of which were missing data, were available for analysis. A spine surgeon (T. F.) reviewed all of the clinical data in each record and assigned all ICD-9-CM spine diagnosis codes that applied to that hospitalization. When a patient had more than one such diagnosis, all of the diagnoses were entered without regard for which were primary or secondary. The physician was blinded to the ICD-9-CM data in the discharge abstract. The codes assigned by the physician were assumed to be the so-called gold standard in the present study. A second physician reviewer (D. F.) blindly assigned codes for a random sample consisting of twenty of the 189 records. There were no disagreements between the two physicians regarding the principal diagnosis and only minor disagreements regarding secondary diagnoses. The statisticians involved in the present study created data files of the ICD-9-CM codes that had been assigned independently by the hospital personnel and the physician reviewer. Before the physician reviewer evaluated the data, he was asked to list the specific ICD-9-CM codes that corresponded to each diagnostic category. These specific codes then were collapsed into six major diagnostic categories for comparison: herniation of a lumbar disc, a previous operation on the lumbar spine, spinal stenosis, cauda equina syndrome, acquired spondylolisthesis, and congenital spondylolisthesis. These were the six most commonly coded diagnoses and, historically, the ones that have been studied most often with use of information from administrative databases9,6,16. The codes assigned by the physician reviewer were compared with those assigned by the hospital personnel without adjustment for the number of operations performed at each institution. The accuracy of coding also was compared among the hospitals, but this analysis was limited because of the small numbers of records from each institution. The sensitivity, specificity, positive and negative predictive values, per cent agreement, and kappa coefficient were calculated for the hospital codes in each of the six diagnostic categories. The kappa coefficient is an indicator of the amount of agreement beyond that expected by chance alone; a coefficient of one indicates perfect agreement, and a coefficient of zero indicates no agreement beyond that expected by chance. The diagnostic codes that had been listed in each record by the hospital personnel were compared with those assigned by the physician reviewer.
Six of the eight institutions were teaching hospitals, and none was a staff-model health maintenance organization (Table I). The personnel at each hospital reported that the entire medical record had been evaluated when the codes were assigned. At seven of the eight hospitals, coding was done after the patient had been discharged (range, immediately after to seven days after discharge). At the remaining hospital, coding was done during the patient's hospitalization. An average of nine (range, two to sixteen) individuals performed the coding at each hospital. The training for such personnel varied both among and within the hospitals. Half of the institutions employed Certified Coding Specialists, but only ten of the sixty-eight individuals who performed the coding had that certification. The greatest proportion of the individuals (thirty [44 per cent] of sixty-eight) had participated in a two-year program and were Accredited Record Technicians. There were a variety of other credentials, but eighteen individuals (26 per cent), including at least one from each hospital, had received only on-the-job training. At six of the eight hospitals, the diagnosis and procedure codes from inpatient records were reported to an external database, agency, or registry.
The age range of the patients was seventeen to eighty-four years. Specific ICD-9-CM codes were used to describe each of the six diagnoses listed in the Methods section (Table II). A previous spinal operation was defined as Deyo et al. described it in previous studies of the spine involving administrative databases5,6.
The number of times that each diagnostic code was assigned or not assigned by the physician reviewer and the number and percentage of correct diagnostic codes that had been assigned by the hospital personnel (the sensitivity and specificity) were tabulated (Table III). The code for spinal stenosis was the one assigned by the physician reviewer most frequently (eighty-seven [46 per cent] of the 189 records). The code for congenital spondylolisthesis was never assigned by the physician reviewer, but it was included in the study because it had been used by hospital personnel.
Sensitivity is the percentage of patients who had a particular diagnosis (as determined by the physician reviewer) that was coded correctly by the hospital personnel. When hospital coding is not sensitive, the correct diagnoses are underrepresented in a database. The sensitivity of hospital coding ranged from 28 per cent (seventeen of sixty-one records), for a previous spinal operation, to 94 per cent (seventy-nine of eighty-four records), for herniation of a disc, and 100 per cent (two of two records), for the relatively uncommon diagnosis of cauda equina syndrome (Table III). The sensitivities of hospital coding for spinal stenosis and acquired spondylolisthesis were estimated to be intermediate (75 per cent [sixty-five] of eighty-seven records and 71 per cent [thirty-two] of forty-five, respectively). Specificity is the percentage of patients who did not have a particular diagnosis (as determined by the physician reviewer) and were not assigned the code for that diagnosis by the hospital personnel. When hospital coding is not specific, the database indicates that certain patients had a diagnosis when, in fact, it was incorrectly coded. The hospital coding of all six diagnoses was highly specific: the lowest specificity was 94 per cent (ninety-nine of 105 records), for herniation of a disc (Table III). Sensitivity and specificity indicate how well the truth is represented in the database. Two related measures, the positive and negative predictive values, describe the accuracy of hospital coding. The positive predictive value of hospital coding is the percentage of patients assigned a diagnostic code (by the hospital personnel) who actually had that diagnosis (as determined by the physician reviewer). When the positive predictive value is low, a database indicates that a substantial percentage of patients had a particular diagnosis when, in fact, they did not. In the present study, the positive predictive value was lowest for congenital spondylolisthesis (zero of nine records) (Table IV). The positive predictive value was higher for the more frequently assigned diagnostic codes: 89 per cent (seventeen of nineteen records) for a previous spinal operation, 89 per cent (thirty-two of thirty-six) for acquired spondylolisthesis, 93 per cent (seventy-nine of eighty-five records) for herniation of a disc, and 97 per cent (sixty-five of sixty-seven records) for spinal stenosis (Table IV).
The negative predictive value of hospital coding is the percentage of patients not assigned a particular diagnostic code (by the hospital personnel) who did not have that diagnosis (as determined by the physician reviewer). When the negative predictive value is low, a database indicates that a substantial percentage of patients did not have a particular diagnosis when they actually did have the diagnosis. The negative predictive value was the lowest (74 per cent [126] of 170 records) for a previous spinal operation and was 100 per cent for cauda equina syndrome (184 of 184 records) and congenital spondylolisthesis (180 of 180 records) (Table IV). Sensitivity, specificity, and positive and negative predictive values are functions of the number of either false-positive or false-negative results of hospital coding. Over-all, the hospital personnel correctly classified 94 per cent (178) of the 189 records with regard to the diagnosis of herniation of a disc; the kappa coefficient of 0.88 indicated a high rate of agreement after adjustment for the agreement expected by chance. The hospital personnel correctly classified 98 per cent (186) of the 189 records with regard to the diagnosis of cauda equina syndrome; however, since neither the physician reviewer nor the hospital personnel assigned this code very frequently, there could be few false-positive or false-negative results; thus, much of the agreement may be attributable to chance, with a kappa coefficient of 0.56 (moderate agreement). The observed agreement was lowest (76 per cent [143] of 189 records) for the diagnostic category of a previous spinal operation, with a kappa coefficient of 0.32 (fair agreement). The kappa coefficients for acquired spondylolisthesis (0.73) and spinal stenosis (0.74) indicated substantial agreement after accounting for chance. Of the total of 279 occurrences of the six diagnoses, as coded by the physician reviewer, eighty-four (30 per cent) were missed by the hospital personnel; forty-four (52 per cent) of the missed diagnoses were previous spinal operations. In twenty-six instances, a diagnostic code that was not assigned by the physician reviewer had been assigned by the hospital personnel. The hospitals appeared to differ somewhat with regard to the sensitivity of coding in diagnostic categories in which the number of false-negative codes was sufficient for study. At each hospital, there was at least one instance of false-negative coding in the category of a previous spinal operation. The hospital personnel from four institutions had missed all twenty-six of the previous spinal operations that were so coded by the physician reviewer. The personnel at two hospitals had correctly coded three of the four and nine of the thirteen previous spinal operations that were coded by the physician reviewer. At two hospitals zero of five and three of nine diagnoses of spinal stenosis (as determined by the physician reviewer) were correctly coded, whereas at two other hospitals ten of ten and thirteen of thirteen of these diagnoses were correctly coded. When all six diagnostic categories were considered together, the sensitivity of hospital coding ranged from 62 per cent (twenty-four of thirty-nine) to 75 per cent (twenty-seven of thirty-six) among the hospitals; there was no significant over-all difference among the hospitals (p = 0.93). There were too few false-positive codes for any of the diagnoses to allow comparison of the specificities among the hospitals. However, there were at least two instances of false-positive coding at each hospital, and there were never more than five instances of false-negative coding. The nine instances of false-positive coding of congenital spondylolisthesis were attributed to four of the eight hospitals.
Databases have been used recently in an attempt to describe many aspects of the delivery of health care, and a variety of conclusions have been drawn from research relying on such databases. These conclusions can have important implications for patients, providers, and society as a whole, as such information is used in the current debate on health-care policy. The ICD-9-CM is currently the most widely used system in North America, and its availability at this critical time in health care has made its acceptance sufficiently universal to guarantee perpetuation of its use. The ICD was not the first and is not the only system for the classification of diagnoses. In fact, it is the descendent of a long line of systems of diagnostic classification. Its predecessors were used for the classification of causes of death, and early editions were used to track infectious diseases such as tuberculosis. The seventh revision (ICD-7), the first to be called the International Classification of Diseases, was sponsored by the World Health Organization and appeared in 1955. The version in current use, the ICD-9 is expected to be superseded by the ICD-10, which is targeted to be released in the year 2000. ICD databases were not created to be used in clinical research or research related to health policy. They are used because they are a readily available source of data. As such, they are not a substitute for data collected primarily, with a hypothesis established before the procedures for collecting data have been designed. In most cases, the codes themselves frequently were developed before the nomenclature of the specific diagnoses and clinical syndromes were defined. As is true for research involving the collection of primary data, assessing the quality of the data is important for an understanding of the results of research involving administrative databases. Investigators should be able to assess the quality of specific data in the database that they use, and they should report on that quality. With the exception of reports on data from Medicare hospital claims, there have been few studies on the quality of data from administrative databases11,12. According to studies on the accuracy of coding that were conducted before the introduction (in 1983) of Medicare's prospective payment system for hospitals, the age and gender of the patient and the dates of admission and discharge generally were accurate, but the diagnosis and procedure codes were not4,11,12,16,18. In a frequently cited article, Ramirez and Thisted reported the prevalence of complications of lumbar discectomy, as determined from a large national database. To validate the data from the discharge abstracts provided by the hospitals to the database, the investigators asked hospital personnel to identify the principal diagnoses and procedures for a sample of patients who did and did not have complications recorded in the discharge abstract. They were also asked to confirm any complications that had already been coded in the database and to reconsider the possibility of complications in patients for whom no complication was listed in the database. The investigators used the results from this sample to provide corrected estimates of the prevalence of complications. They found 96.5 per cent agreement on the combination of a primary diagnosis of displacement or degeneration of a lumbar disc and a primary procedure consistent with excision of a disc. They also found that nearly half of the coded complications were pre-existing or trivial conditions. Although that study was based on a small sample of ostensibly average cases, in only one instance was a serious complication that was identified during the study not recorded in the discharge abstract. When the study population was analyzed, the sensitivity and specificity of the hospital coding in the discharge abstract were 62 and 99 per cent, respectively16. There are limitations to the validity testing performed by Ramirez and Thisted, however. The rate of response of the hospital personnel to the authors' queries was relatively poor, and the authors relied on hospital personnel to verify the coding by other hospital personnel. In addition, the individuals performing the coding were not blinded to the discharge abstracts in the medical records, and the rate of false-negative coding was not measured precisely. Finally, the results regarding the coding of certain diagnoses and procedures may not be generalizable to the coding of others, although some authors have attempted such generalization5,6. Addressing the limitations noted by Ramirez and Thisted, one of us (T. F.) and colleagues7 performed an independent detailed review of medical charts in order to examine the accuracy of the coding of complications of anterior spinal procedures. The accuracy was found to vary depending on the particular complication being evaluated: the prevalence of cardiac and pulmonary complications tended to be overestimated, and that of wound infections, other wound-related complications, genito-urinary complications, and gastrointestinal complications tended to be underestimated. When a large database is used in an analysis, a small degree of random misclassification would be expected to weaken associations; thus, relationships among procedures, diagnoses, and outcomes would be understated. A potential source of random error is the incorrect entering of codes into the database: a code other than the code of interest could be erroneously entered or, conversely, the code of interest could be erroneously entered when another code was intended. Our analysis was not based on data that had been entered into an administrative database, so this type of error and other processing errors that may occur at the central data-processing center were not a factor. Many mechanisms for introducing error into such databases are not expected to be random, however. If the individuals performing the coding are not properly trained to recognize and code certain diagnoses, there is a bias toward the undercoding of those diagnoses. If such a diagnosis (for example, a previous spinal operation) provides a partial explanation for poor outcomes or for an increased number of complications, undercoding would result in an overestimation of the risk of poor outcomes for patients who do not have that particular diagnosis. Besides improper training or supervision of hospital personnel performing coding, other non-random sources of error are possible. Reimbursement regulations may provide incentives for institutions to favor one code compared with another or to code a disorder as being more severe than it is. Some hospitals limit the number of codes that can be assigned, increasing the likelihood that codes for less important disorders will be used less frequently, particularly in complex cases for which more than one code is needed to describe the case. The mere presence of administrative databases and the threat that institutional comparisons will be made public may bias the wording in medical records and possibly even the coding. Improving the accuracy of the information in these databases poses a major challenge because of the lack of incentives, the added cost, and the number of institutions involved. A possible limitation of our study was the use of one physician reviewer as the so-called gold standard. A certain amount of subjectivity is required when assigning some codes, and it is possible that other observers would have coded some of these records differently. The coding of a sample of twenty records by a second physician reviewer suggested that the use of one physician reviewer was not an important source of error, although this conclusion is limited by the small number of records in the sample. Our data show a tendency for hospital personnel to undercode—that is, to not recognize certain diagnoses. This was particularly true for previous lumbar operations: forty-four of sixty-one were not identified by the hospital personnel. The undercoding was not always this dramatic, and some diagnoses were overcoded as well. Undercoding may have been the result of similarities between diagnoses. For example, the hospital personnel missed thirteen cases of acquired spondylolisthesis but incorrectly assigned the code for congenital spondylolisthesis nine times. The net effect of errors in coding on the analysis of information obtained from an administrative database is unpredictable. The effect depends on the type of analysis (descriptive or comparative), the degree and direction of the error, the importance of the diagnostic code in the analysis, and the relationship between the diagnostic code in question and the outcome. Knowing the accuracy of the coding for particular diagnoses seems to be essential when interpreting studies that include information from administrative databases. Deyo et al. used information from an administrative database to evaluate the relative risks of various complications and other outcomes of an operation on the lumbar spine, with and without arthrodesis. They found that the patients who had arthrodesis more frequently had had a previous spinal operation, less frequently had herniated discs, and more frequently had possible instability of the spine than did the patients who did not have arthrodesis6. Therefore, those authors made statistical adjustments for the preoperative differences in their analysis of the relative risks of adverse short-term events associated with arthrodesis. They stated: "Inaccurate diagnosis or procedure coding would tend to minimize observed differences among procedures or patient groups." We contend that the effects of inaccurate coding in that study are probably unpredictable and depend on the degree of miscoding and on whether miscoding was somehow associated with whether arthrodesis was performed or with the risk of an adverse event. In any case, the findings of the present study suggest that the degree of miscoding may be substantial, especially regarding the diagnosis of a previous spinal operation. Unless the effects of coding errors in a particular study are explored, the possibility of incorrect or overstated conclusions must be considered. Logistically, it may be difficult to obtain data from a random sample of medical records in a large multicenter database, especially if personal identifiers are not provided or if patient confidentiality is a concern. If such an effort could be made with a large enough sample, the effects of errors in coding could be evaluated by comparing analyses of that subset of the data with original codes with analyses of the subset with corrected codes. The size of such a validation sample might be based on the over-all results and the expected reduction in precision associated with sampling. Although adjustments for errors in the complete data set are useful for the interpretation of over-all results, it may not always be possible to make such adjustments (for example, in complex comparative studies). We realize that many outcome analyses require adjustment for comorbidities, but we did not study the accuracy of hospital coding of these diagnoses in the present study. Future research should specifically address this important issue, as it also relates to the validity of the conclusions that can be drawn from analyses of information in administrative databases. In conclusion, we believe that the errors in the assigning of diagnostic codes by hospital personnel range from minor to substantial, with the degree depending on the particular diagnosis and, to a lesser extent, on the hospital. Whether the errors are great enough to affect the conclusions of research depends on the rate of error and on the importance of a particular diagnosis in the analysis. We recommend that, whenever possible, the information in an administrative database be validated before publication of results and that the findings of that effort at validation be reported in the article. It would be particularly helpful to explore or at least to mention the likely effect that the identified errors may have had on the results of the study. There are substantial limitations in the usefulness of administrative databases in clinical and health-policy research involving spinal operations, and these limitations should be discussed in reports of studies based on these sources of data.
NOTE: The authors thank the surgeons who contributed cases for this study: Todd Albert, M.D., Rothman Institute, Philadelphia, Pennsylvania; Craig Callewart, M.D., Orthopedic Associated of Dallas, Dallas, Texas; Richard Guye, M.D., Texas Back Institute, Plano, Texas; Neil Kahanovitz, M.D., Washington Hospital Center, Arlington, Virginia; Mike Smith, M.D., Minnesota Spine Center, Minneapolis, Minnesota; and James Weinstein, D.O., University of Iowa, Iowa City, Iowa. The authors also thank Jayne Frahmann, R.N., Study Coordinator, Marshfield Medical Research and Education Foundation, Marshfield, Wisconsin.
*No benefits in any form have been received or will be received from a commercial party related directly or indirectly to the subject of this article. Funds were received in total or partial support of the research or clinical study presented in this article. The funding source was the Physician Research Fund at Marshfield Clinic, Marshfield, Wisconsin.
  CiteULike  Connotea  Del.icio.us  Facebook  Technorati  Twitter What's this?
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Top
Abstract
Introduction| home | CME | help | feedback | subscriptions | archive | site index |
|
Stanford University Libraries' HighWire Press (TM) assists in the publication of JBJS Online. |