The Inter-rater Reliability of the Clinical Signs and Symptoms Checklist in Diabetic Foot Ulcers

/Home/Complications of Diabetes/Diabetes and Foot and Leg Problems/Diabetes and Foot Ulcers/Diabetic Foot Ulcer Symptoms

The Inter-rater Reliability of the Clinical Signs and Symptoms Checklist in Diabetic Foot Ulcers
PUBLICATION DATE: Jan 01 2007
Sue E. Gardner, PhD, RN; Rita A. Frantz, PhD, RN; Heeok Park, MSN, RN; and Melody Scherubel, BSN
Ostomy Wound Management

The Clinical Signs and Symptoms Checklist (CSSC)1 is a tool designed to objectively measure 12 clinical signs and symptoms of localized chronic wound infection. Five of these signs and symptoms are commonly known as the classic signs of infection (pain, erythema, heat, edema, and purulent exudate); the other seven are signs specific to wounds healing by secondary intention (serous exudate, sanguineous drainage, delayed healing, discoloration of granulation tissue, friable granulation tissue, wound base pocketing, foul odor, wound breakdown).2 Each sign and symptom is represented with a specific descriptor; the conceptual definitions and the development of their descriptors have been reported elsewhere.1 During first testing, the reliabilities of the items on the tool were found to range between 0.53 and 1.00.
Since its initial development, the CSSC has been revised to include sanguineous drainage as a sign or symptom because wound drainage with a bloody composition did not fit the descriptors for serous exudate or purulent exudate (see Figure 1). Adding sanguineous drainage enhanced discrimination regarding wound drainage. The reliability of the revised CSSC has not been examined. Furthermore, the reliability of the CSSC has not been examined in a homogenous sample of chronic wounds. The purpose of this study was to examine the reliability of the revised CSSC in a sample of diabetic foot ulcers. The findings reported here were obtained in conjunction with a larger study designed to examine the validity of each sign and symptom for identifying infection in diabetic foot ulcers.

Methods and Procedures

Participants. In this observational, cross-sectional design, two members of the research team independently assessed patients with diabetic foot ulcers for the presence of signs and symptoms of infection using the CSSC. In addition to the principal investigator (PI) — a Registered Nurse, three other Registered Nurses were trained in CSSC use in order to examine its reliability. All participating nurses had training and experience in chronic wound management — two of the nurses were certified wound care nurses and the other two had more than 3 years of wound care experience. Each nurse reviewed the items on the CSSC and practiced using this tool to assess wounds in conjunction with the PI. Competency was typically achieved with fewer than five practice wounds.
Study subjects included patients with diabetic foot ulcers. Subjects were screened and enrolled based on the following criteria: 1) 18 years of age or older, 2) presence of a full-thickness diabetic foot ulcer, 3) white blood count >1,500 cells/mm3, 4) platelet count >125,000/mm, and 5) no coagulapethies. If a subject had more than one diabetic foot ulcer, one was randomly selected for inclusion in the study. Human subject approval was obtained from the Institutional Review Board at each site before data collection commenced. Informed consent was obtained from all patient participants. Subjects were enrolled in the study from August 2001 through August 2004.
Setting. A Department of Veteran’s Affairs Medical Center and a university-associated tertiary hospital served as settings for the study.
Wound assessment. The CSSC was used to assess the presence of the clinical signs and symptoms of infection. Decisions regarding presence of signs and symptoms on the CSSC required observing, palpating, and manipulating the wound as well as observing the dressing for character of wound exudate. After the dressing was removed, the study ulcer was cleansed by gently rubbing the surface with saline-soaked coarse-mesh gauze.1 The dressing, which typically had been in place overnight, was retained for assessment of the presence and type of wound exudate. Study ulcers were not treated with dressings and or topical treatments (eg, hydrocolloids or silver sulfadiazine) that would interfere with the ability to characterize wound exudate; hydrocolloids and silver sulfadiazine interact with wound fluids in a manner that makes the wound fluid appear falsely purulent. Two members of the research team trained in the use of the CSSC independently assessed and recorded the presence of each clinical sign and symptom of chronic wound infection to determine inter-rater reliability.
Statistical analysis. Inter-rater reliability of each sign and symptom was analyzed using percent agreement (ie, total number of concordant observations divided by total number of paired observations) and Kappa coefficient (ie, observed agreement minus expected agreement divided by one minus expected agreement). Three measures of percent agreement were used because estimates are influenced by the manner in which agreement is defined and total agreement can be inflated by a high percentage of non-occurrence agreements when few occurrence agreements occur. Because percent agreement does not correct for chance agreements, Kappa statistics, which correct for chance agreement, also were calculated for each Checklist item.

Results

Participants and wounds. Patient participants (N = 64, 40 from the tertiary hospital, 24 from the VA facility; mean age 55 years, SD ± 11.4 ) were predominantly Caucasian (n = 63; 98%) and male (n = 49; 77%). The subjects had non-arterial, neuropathic diabetic foot ulcers located on the plantar surface of the foot. The most commonly used dressing (n = 44; 69%) was moistened saline gauze covered with a Telfa® (Tyco Healthcare/Kendall, Mansfield, Mass), which prevented evaporation and drying of the gauze. The second most frequently used dressing was dry gauze (n = 18; 28%). From a sample of 64 diabetic foot ulcers, 64 paired, independent observations were made using the CSSC.
Checklist findings. Table 1 presents the reliability findings for each item on the CSSC, including total percent agreement (TPA), occurrence percent agreement (OPA), non-occurrence percent agreement (NPA), and Kappas. All of the items had TPA values >70%. In addition, all items had Kappa values >0.345. Sanguineous drainage (0.385), discoloration of granulation tissue (0.381), and foul odor (0.345) had the lowest Kappa values. An overall Kappa value for the CSSC was not calculated.

Discussion

The measurement literature generally suggests an agreement of 70% is necessary, 80% is adequate, and 90% is good.3,4 A Kappa >0.80 generally is almost perfect agreement, between 0.60 and 0.80 substantial agreement, between 0.40 and 0.60 moderate agreement, and between 0.20 and 0.40 fair agreement.5 Based on TPA and Kappa values, all of the items on the CSSC, including the newer sanguineous drainage item, had “fair” to almost perfect reliability estimates. These findings are consistent with initial findings on the reliability of the CSSC,1 although the sample used in this study was twice as large and more homogenous in terms of chronic wound type.
In addition, the reliability of the items on the CSSC compare favorably with other studies that have examined inter-clinician agreement with respect to wound infection status. Wirthlin et al6 found that agreement between surgeons with respect to wound cellulitis/infection and the presence of erythema were -0.04 and 0.22, respectively. Cutting7 found that agreement between nurses with respect to the infection status of granulating wounds was only 47.5%. Neither of these studies employed a structured tool from which to identify the specific signs and symptoms of infection.
Consistent with the findings from the initial study,1 the signs of heat, discoloration of granulation tissue, and foul odor had only moderate or fair agreement. These signs require more subjective judgment than other signs and symptoms because they include the characteristics of touch, color, and smell. The newly included sign (sanguineous drainage) also had only fair agreement, which may indicate the descriptor of this item needs to be revised.
It is important to note that the majority of ulcers in this study were dressed with gauze dressings, which are more conducive to assessing type of wound exudate/drainage than other dressing types such as hydrocolloids.1 Hydrocolloids interact with the wound fluid, producing a creamy substance that is difficult to distinguish from purulent exudate. Gauze dressings allow more direct observations of exudate/drainage color and consistency. In clinical settings that use non-gauze dressings for wound care, wound exudate/drainage can best be assessed by cleansing the ulcer, dressing the ulcer with a dry gauze dressing for an hour, and then completing the assessment of the wound for type of wound exudate.1
In addition, the findings of this study are based on the assessments of nurses trained in chronic wound assessment and the use of the CSSC. Other clinicians, such as physical therapists, physicians, or podiatrists trained in wound assessment and the use of the CSSC, should be able to achieve similar results.
Finally, the fact that most ulcers were plantar neuropathic diabetic foot ulcers may have resulted in all having similar signs and symptoms. This similarity may have inflated the inter-rater reliability findings reported here. Nevertheless, the findings on inter-rater reliability reported here are similar to the findings reported on the CSSC in a mixed sample of chronic wounds. Also, the fact that most of the subjects were Caucasian may have made erythema easier to identify than it may be in persons of color. This may have inflated the inter-rater reliability for erythema. The findings from our first study1 also were based on a primarily Caucasian sample.

Implications for Practice

The early identification of infection in diabetic foot ulcers is essential to preventing infection-related complications, such as amputation. The Infectious Disease Society of America (IDSA) recently published guidelines8 specific to identifying and managing diabetic foot infections. The IDSA states that “wound infections must be diagnosed clinically on the basis of local (and occasionally systemic) signs and symptoms of inflammation.”8 Specifically, the guidelines regarding a diabetic foot ulcer note the presence of purulent exudate or the presence of two or more signs of inflammation, such as redness, warmth, swelling or induration, and pain or tenderness. The items on the CSSC relevant to these signs include purulent exudate, increasing pain, erythema, heat, and edema. The identification of purulent exudate had substantial inter-rater reliability using the CSSC, as did increasing pain, erythema, and edema. However, the routine assessment of diabetic foot ulcers often does not include a systematic approach to identifying these specific signs and symptoms of infection. Incorporating the CSSC into wound assessment may improve clinician skill and accuracy in identifying the specific signs and symptoms (eg, purulent exudate and signs of inflammation) delineated as most important by the IDSA.
Although the findings from this study suggest that the CSSC may improve the ability to consistently identify signs and symptoms of diabetic foot ulcer infection, the findings reported here do not address the validity of these signs and symptoms as accurate indicators of infection in diabetic foot ulcers. A study9 designed to examine the validity of clinical signs as indicators of diabetic foot ulcer infection revealed that the IDSA definition (purulent exudate or two or more signs of inflammation) is a better indicator of infection status among diabetic foot ulcers than any one sign or symptom alone. Moreover, the signs specific to secondary wounds were better indicators of infection in diabetic foot ulcers than the classical signs of infection. More study is needed to identify which signs and symptoms, or combinations of signs and symptoms, are most indicative of diabetic foot ulcer infection.

Conclusion

A study to examine the inter-rater reliability of the revised CSSC in diabetic foot ulcers found that inter-rater reliability was lower but still substantial regarding the more subjective signs and symptoms of infection (eg, heat, discoloration of granulation tissue, and foul odor) than the inter-rater reliability regarding secondary signs and symptoms such as wound breakdown, pocketing, and increasing pain. A systematic approach to wound infection assessment potentially can improve diagnosis and subsequent treatment, making instruments such as the CSSC important tools in the management of diabetic foot wounds specifically and chronic wounds in general.



This study was funded in part by the Nursing Research Initiative, Health Services Research and Development, Department of Veteran’s Affairs (NRI 01-005-1) and with support from the Gerontological Nursing Interventions Research Center NIH #P30 NR03979 (PI: Toni Tripp-Reimer, The University of Iowa College of Nursing) and the Hartford Center for Geriatric Nursing Excellence, The John A. Hartford Foundation (PI: Kathleen Buckwalter, The University of Iowa College of Nursing). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the Department of Veteran’s Affairs or the National Institute of Nursing Research.
References:

1. Gardner SE, Frantz RA, Troia C, et al. A tool to assess the clinical signs and symptoms of localized infection in chronic wounds: development and reliability. Ostomy Wound Manage. 2001;47(1):40–47. 2. Cutting KF, Harding KG. Criteria for identifying wound infection. J Wound Care. 1994;3(4):198–201. 3. Hartmann D. Considerations in the choice of interobserver reliability estimates. J Applied Behav Anal. 1977;10(1):103–116. 4. House A, House B, Campbell M. Measures of interobserver agreement. Calculation formulas and distribution effects. J Behav Assess. 1981;3(1):37–57. 5. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–174. 6. Wirthlin DJ, Buradagunta S, Edwards RA, et al. Telemedicine in vascular surgery: feasibility of digital imaging for remote management of wounds. J Vasc Surg. 1998;27(6):1089–1100. 7. Cutting KF. Identification of infection in granulating wounds by registered nurses. J Clin Nurs. 1998;7(6):539–546. 8. Lipsky BA, Berendt AR, Deery HG, et al. IDSA guidelines: diagnosis and treatment of diabetic foot infections. Clin Infect Dis. 2004;39(7):885–910. 9. Gardner SE, Frantz RA, Park H, Scherubel M. Diagnostic validity of clinical signs in identifying localized diabetic foot ulcer infection. Manuscript in preparation.

© 2009 HMP Communications, LLC