British Journal of Radiology (2006) 79, 468-472
© 2006 British Institute of Radiology
doi: 10.1259/bjr/53513558
The impact of a short course of study on the performance of radiographers when highlighting fractures on trauma radiographs: "The Red Dot System"
S J Mackay, TDCR, MSc, PhD
Salford University, Frederick Road Campus, Salford, Manchester M6 6PU, UK
 |
Abstract
|
|---|
This study was set up to determine the effect of a short course of study on radiographers' ability to identify fractures in a trauma context (Red Dot system). The performance of 133 radiographers attending Red Dot courses delivered from April 1999 to September 2003 was evaluated prospectively using a selected sample of radiographs. Assessment of fracture identification occurred before, immediately after and 6 months after the course. The same film set was used and consisted of 30 axial and appendicular cases, 18 with fractures and 12 normal or normal variants. Following a test for normality of the data, a Wilcoxon Signed-Rank Test was selected and paired tests were done between each assessment for sensitivity and specificity. There were significant differences in sensitivity between all three assessments (p
0.05), the pre-course/post-course and the post-course/6 month comparison being highly significant (p0.01). Specificity showed significant differences between the pre-course/post-course (p0.01) and the post-course/6 month follow-up scores (p0.05), but no difference was found between the 6 months/pre-course scores. The participants improved their ability to identify fractures and this appears to be as a result of the course. This improvement was not demonstrable after 6 months, although only 30% of participants took part in this follow-up. Radiographers can improve their image interpretation skills from a short course of study, but probably need continuing professional development to maintain these skills.
|
Introduction
|
|---|
The Health Professions Council standards of proficiency for radiographers [1] are designed to ensure the safe and effective practice of radiographers in the UK. They state that part of the role of the radiographer is to be able to "distinguish between normal and abnormal appearances evident on images" [1].
The Quality Assurance Agency benchmark statements [2], which also describe the role of the radiographer, but from an educational perspective, are more precise on this matter and state that radiographers should be able to "recognize and respond appropriately to abnormal, aberrant and pathological appearances on radiographic images" [2].
In an accident and emergency setting, this practice is termed a Red Dot system [3] and radiographers use these skills to alert the casualty officer to fractures.
It is now over 30 years since Swinburne [4] suggested that radiographers could be used to distinguish between normal and abnormal radiographs, and such trauma flagging systems were reported in 2000 to be in operation in over 85% of accident and emergency departments in the UK (excluding Northern Ireland) [5]. Several authors have provided evidence of the potential of radiographers to perform this role with training [5, 6] and without [7]. However, these evaluations have limited external validity. This is mainly due to methodological limitation. These studies used either small numbers (n = 7) in one department [6], a small non-random national sample (n = 22) [5], or radiographers untrained in triaging casualty radiographs [7]. Many studies have also purposefully selected senior or experienced radiographers as subjects and cannot therefore provide evidence on the performance of the range of radiographers performing Red Dot or similar systems in the UK today.
A short course of study in image interpretation to enable radiographers to highlight abnormal trauma radiographs has been delivered at the author's university since 1999 and has attracted radiographers of varying experience from throughout the UK. Their performance has been evaluated in an attempt to add to the evidence base in radiographer's performance in Red Dot systems.
One study [4] selected senior or experienced radiographers, the assumption presumably being that their experience of image interpretation in their role as a radiographer will have an effect on their film viewing performance. This study will also investigate the relationship between a radiographer's years of experience and performance in a Red Dot scheme.
|
Method
|
|---|
Overview
The performance of radiographers attending one of seven Red Dot courses delivered from April 1999 to September 2003 was evaluated prospectively using a selected sample of radiographs. They were assessed before the start of the course, immediately after the course and then after 6 months of clinical practice. The same film series was used for each evaluation.
The course
This was a 2 day course with short keynote lectures and small group tutorials delivered by radiologists and reporting radiographers. Topics covered were the identification of fractures and some non-traumatic pathologies of the axial and appendicular skeleton along with normal and abnormal variants. Both adult and paediatric patients were included so as to represent the range of clinical work that might present to an accident and emergency department.
The sample
The 133 radiographers from across the UK self-selected for the courses. Their clinical experience ranged from newly qualified radiographers to those with 36 years of experience (mean = 12.1 years, SD = 10.4). 36 (27.0%) of the sample had no experience of working within a Red Dot system before and, of the 97 (73%) who had, the maximum number of years was 20 years (mean = 3.1 years, SD = 3.4). Consent was obtained from the participants for the use of their test scores, which form part of the course feedback, in this research and it was explained that the data would be anonymized so no individuals could be identified.
The film assessment
The sample of radiographs were selected by a consultant radiologist and a reporting radiographer who aimed to make them representative of the range and quality of films that might pass through an accident and emergency department at a district general hospital. There were 30 cases of which 18 were pathological and 12 normal or normal variant. The pathological films had been reported upon by a consultant radiologist in practice and the consultant and reporting radiographer who selected the film were also agreed on the diagnosis. 10 radiographs were of the upper and nine of the lower limbs, seven spine and pelvis, and four skull and face. Three were paediatric cases.
The assessment was undertaken at the university under controlled conditions. Radiographs were viewed in one session and each radiographer was allowed to view a case for 90 s before recording one of two answers, either normal (which included normal variants) or abnormal. For the 6 month follow-up assessment, radiographers were invited back to the university to complete the assessment. Those who were unable to do this were sent a CD containing the digitized film set along with instructions regarding the conditions in which to view the radiographs, but this could not be controlled.
Analysis
True and false, and positive and negative scores were calculated to determine sensitivity, specificity, and positive and negative predictive values. Accuracy values were not used, following the advice of Maisey and Hutton [8] who describe this as a global parameter, "...which is often misleading as a measure of test performance and is rarely of any value" [8].
The sensitivity and specificity data obtained was inspected for normality using a Kolmogorov-Smirnov statistic [9]. This showed that the distributions were significantly different from the normal distribution for the pre-course and post-course (p<0.01) sensitivity data and the pre-course (p<0.05) and post-course (p<0.01) specificity data. No significant differences were found for the 6 month data for either variable. Therefore data were treated as non-parametric. Comparison was required of pairs of scores in relation to the course, e.g. pre/post-course; so the non-parametric Wilcoxon Signed-Rank Test [9] was selected for analysis. All analyses were performed using the Statistical Package for the Social Sciences (SPSS v11.5; SPSS, Chicago, IL) and a 0.05 significance level.
The number of years subjects had been qualified and the number of years they had spent working within a Red Dot scheme were correlated (Spearman's Rho, p 0.05) with sensitivity and specificity values to examine the effect of experience on image interpretation performance.
Errors
False negative errors constitute missed fractures and could be considered as the more important type of detection error, which could lead to significant morbidity and/or legal liability. Therefore, the most commonly occurring cases of false negatives were highlighted and described. This will help to inform course curricula and enable radiographers to focus their professional development.
|
Results
|
|---|
Response rate
133 (100%) radiographers completed the pre-course assessment and 132 (99.2%) the post-course assessment. 39 (29.3%) radiographers completed the 6 month follow up with 10 of these (25.6% of the responders at 6 months) doing so using the digitized film set.
Sensitivity and specificity
The results are presented in 
Tables 1 and 2, and box plots (using SPSS v11.5) are presented in Figures 1 and 2. The positive and negative predictive values were not analysed using inferential statistics and are presented for comparative purposes only.
View larger version (10K):
[in this window]
[in a new window]
|
Figure 1. Sensitivity changes over time. Box plot where the box length is the interquartile range with the median identified as a thick black line traversing the box. Outliers(ringed) are cases with values between 1.5 and 3 box lengths from the upper or lower edge of the box. From SPSS v11.5.
|
|
View larger version (11K):
[in this window]
[in a new window]
|
Figure 2. Specificity changes over time. Box plot where the box length is the interquartile range with the median identified as a thick black line traversing the box. Outliers(ringed) are cases with values between 1.5 and 3 box lengths from the upper or lower edge of the box. From SPSS v11.5.
|
|
The median scores for sensitivity show that the radiographers achieved a score of 78.9% initially. This improved to 88.2% following the course, but then fell away to less than the pre-course score after 6 months. All differences were statistically significant.
The median scores for specificity were the same, but the distributions varied and showed a highly statistically significant difference between the pre-course and post-course tests, and significant differences between the post-course and 6 month follow-up. Visual inspection of the box plot in Figure 2
demonstrate that the radiographers did worse immediately after the course, with more scores below the median. They then improved slightly at 6 months but were not significantly different from the pre-course score.
6 month follow-up
Further analysis was undertaken as less than one third of the subjects completed the 6 month follow-up. This investigated whether this group of subjects (the completers) had scored differently at other assessment times to those who did not complete the assessment. This would help to clarify whether the completers were representative of the group as a whole. A Wilcoxon test found no statistically significant differences between the completers and non-completers for number of years practicing radiography or the number of years in a Red Dot scheme. The differences in the pre- and post-test scores for these groups are displayed in Table 3.
Effect of experience
No significant correlations were found between the number of years since qualification or length of time in a Red Dot scheme and sensitivity or specificity scores pre-course, post-course or 6 months after the course.
False negative errors
The 10 most common false negative errors before and immediately after the course were ranked and collated (11 cases appear in column 1 as there were 2 cases of 10.9% tied for 10th place). These key errors are where further learning should be focused. These can be found in Table 4.
|
Discussion
|
|---|
The ability of the radiographers to identify the abnormal trauma radiographs, sensitivity, increased immediately after the course, suggesting that they had learned from the course. This improvement declined over 6 months and ended up being worse than before they started the course.
The ability of the radiographers to identify normality on trauma radiographs, specificity, decreasd after the course and then improved slightly, but was then no different from the original scores. This "post-course dip" in score was also identified by another author [5] and appears to be a phenomenon inherent in the image interpretation learning process. This may be due in part to the course, the main bulk of which shows students how to recognize fractures. The students may then have an immediate tendency to look for the abnormal rather than the normal.
A similar study [5] followed up subjects for 6–10 weeks and found a decrease in sensitivity which, although not statistically significant, would support the trend found in this work suggesting that improvements in performance were short lived. This suggests that radiographers need to continue to acquire the knowledge and skills necessary for interpreting images following any course of study. Based on these data, they should be topping up their knowledge and skills before a period of 6 months has elapsed.
In light of the impending mandatory continuing professional development (CPD) requirements of the Health Professions Council, radiographers will need to consider undertaking a range of CPD activities to consolidate and enhance their image interpretation skills.
It is acknowledged that the small sample that completed the 6 month assessment may not have been representative of the group as a whole. However, comparisons between the completers and non-completers suggest that on the measures of length of experience since qualification and working in a Red Dot scheme they were not significantly different. In addition, the differences in performance between these two groups were on the post-course sensitivity scores in which the completers did better. This would suggest that the scores from this study at 6 months may even have exaggerated the performance of the whole group of radiographers. The specificity scores between these groups were not significantly different after the course and are therefore likely to be representative.
Comparison with other results [5, 6, 10] shows a range of sensitivity and specificity levels for radiographers (Table 5). Close statistical comparison is difficult as the measures of central tendency used are different. However, looking across the values it would appear that there is a need for radiographers to improve their performance in image interpretation within Red Dot protocols. No published performance levels were found for radiographers working in a Red Dot scheme. However, post-graduate reporting radiographers are expected to achieve 90–95% sensitivity and specificity levels [11]. If we apply this standard of performance to Red Dot radiographers, they are clearly falling short of this level.
One notable trend when comparing the data in this study with two others [5, 6] is that subjects in one [5] tended to over report (high sensitivity and low specificity) and in the other [6] tended to under report (low sensitivity and high specificity). In the current study, relative to these other two, the subjects maintained a more stable performance over both measures.
The areas where radiographers missed fractures have been highlighted in this study and could be added to those from another study [6]. These data could help to inform "Red Dot" course developers and evaluators to take a more evidence based approach to radiographer education.
Experience did not appear to be a factor in the performance of this group of radiographers. Further research is needed to explore the phenomenon of "experience" in radiography and try to tease out the factors that might affect the performance of radiographers in image interpretation.
|
Conclusions
|
|---|
The radiographers did appear to learn from the short course and improved their levels of performance in image interpretation. These improvements were short lived and it is recommended that regular updating is undertaken by radiographers working within a Red Dot scheme.
There appeared to be a reduction in performance immediately following the image interpretation course, which caused an increase in false positives. Radiographers should be aware of this and attempt to counteract the effect. Programme evaluators should factor in this effect when planning any assessments of image interpretation performance after a short course of study.
The scores achieved in this evaluation fall well short of the 90–95% sensitivity and specificity levels used in radiographer reporting. Radiographers would benefit from continuing to improve their image interpretation skills throughout their working lives.
Received for publication March 8, 2005.
Revision received October 7, 2005.
Accepted for publication October 31, 2005.
|
References
|
|---|
- Health Professions Council. Standards of Proficiency: Radiographers. London, UK: HPC; 2003
- The Quality Assurance Agency for Higher Education. Benchmark Statement for Radiographers. QAA; 2001
- Berman L, de Lacey G, Twomey E, Twomey B, Welch T, Eban R. Reducing errors in the accident and emergency department :a simple method using radiographers. Br Med J 1985;290:421–2.[Medline]
- Swinburn K. Pattern recognition for radiographers. Lancet 1971;1:589–90.[Medline]
- McConnell JR, Webster AJ. Improving radiographer highlighting of trauma films in the Accident and Emergency department with a short course of study – an evaluation. Br J Radiol 2000;73:608–12.[Abstract]
- Hargreaves J, Mackay S. The accuracy of the red dot system: can it improve with training? Radiography 2003;9:283–9.[CrossRef]
- Renwick I, Butt W, Steele B. How well can radiographers triage x-ray films in accident and emergency departments? Br Med J 1991;302:568–9.[Medline]
- Maisey MN, Hutton J. Guidelines for the evaluation of radiological technologies. Report No.: 13874. London, UK: British Institute of Radiology, 1995
- Pallant J. SPSS survival manual (SPSS v10). 1st edn. Buckingham, UK: O.U. Press; 2001
- Loughran CF, Raynor J, Quine M, Mulley A. The red dot system: how good is it? In: Radiology UK; May 1996
- Salford University. MSc Advanced Practice (Clinical Radiographic Reporting) Programme Handbook. 2004
Top
Abstract
Introduction
