British Journal of Radiology (2003) 76, 57-61
© 2003 British Institute of Radiology
doi: 10.1259/bjr/68918327
Accident and Emergency and General Practitioner plain radiograph reporting by radiographers and radiologists: a quasi-randomized controlled trial
S Brealey, BSc1,
D G King, MBChB, FRCS(Eng), FRCR2,
M T I Crowe, MRCP, FRCR3,
I Crawshaw, DCR, MSc2,
L Ford, DCR, MSc2,
N G Warnock, MRCP, FRCR2,
R A J Mannion, MRCP, FRCR2 and
S Ethell, BSc4
1 Department of Health Sciences, University of York, York YO1 5DD, 2 X-ray Department, York District Hospital, Wiggington Road, York YO30 8HE, 3 X-ray Department, King's Lynn and Wisbech Hospitals NHS Trust, The Queen Elizabeth Hospital, Gayton Road, Norfolk PE30 4ET, and 4 Department of Radiography and Imaging Sciences, St Martin's College, Bowerham Road, Lancaster LA1 3JD, UK
 |
Abstract
|
|---|
Two specially trained radiographers at York District Hospital have been reporting appendicular plain radiograph X-ray examinations for Accident and Emergency (A&E) patients since February 1995. This study explores the potential for further expanding their reporting role. This was achieved by assessing the two radiographers' and a group of consultant radiologists' ability to report on a retrospectively selected random stratified sample of 400 A&E and General Practitioner (GP) plain radiograph X-ray examinations for all body areas. Using receiver operating characteristic (ROC) curve analyses there was no statistically significant difference at the 5% level between the area under the ROC curves for the radiographers and consultant radiologists when reporting A&E or GP plain radiographs. It may be feasible to expand the reporting role of suitably trained radiographers to include plain radiograph X-ray examinations for all A&E patients and for GP patients, with no detriment to the quality of reports.
|
Introduction
|
|---|
From 1968 to 1991, radiologists' total workload increased by 322%, but the number of posts increased by only 213%; consequently, radiologists in England only reported 60% of work within 2 working days [1]. The Royal College of Radiologists therefore advised radiologists to explore the potential of delegating reporting to radiographers [2]. Since then, studies have shown that during clinical practice radiographers can interpret musculoskeletal plain radiographs for Accident and Emergency A&E patients to a level of accuracy comparable to radiologists [3, 4]. Indeed, radiographers now report A&E X-ray examinations in over 30 Trusts [5], which is evidence of successful implementation of this practice on a national scale. A study to explore the potential for further expanding the reporting role of radiographers was therefore undertaken. This involved assessing the ability of radiographers and consultant radiologists to report A&E and GP plain radiographs compared with a reference standard. Furthermore, unlike previous studies [6], an assessment was made on the subsequent effect of these reports on clinicians' diagnosis and treatment plans, their diagnostic and therapeutic confidence and patient outcome. Expressing the radiographers and radiologists' radiograph reading performance using sensitivity and specificity, and presenting the results about the effect of reports to clinicians and patients, will be the subject of a further paper. This paper summarizes the results of the two groups' radiograph reading performance using receiver operating characteristic (ROC) curves, which exploit observers natural tendency for probability scaling when interpreting radiographs [7] and is thus a more valid reflection of their decision making.
|
Methods
|
|---|
In February 1995, following a period of training in plain radiograph interpretation that included all body areas and different patient types, two radiographers at York District Hospital (YDH) began to report appendicular A&E X-ray examinations judged normal by the casualty officers. They also report the few appendicular A&E X-ray examinations referred from Selby War Memorial Hospital. These radiographers are now called Clinical Specialist Radiographers (CSRs). To explore the potential for expanding their reporting role, both CSRs and consultant radiologists (n=8) at YDH who regularly report A&E and General Practitioner (GP) radiographs reported on the control and experimental sample of radiographs described in Table 1
.
A retrospective stratified random sample of radiographs was selected to ensure that the two groups of professionals reported on radiographs representative of clinical practice. Therefore, each year during the period between February 1993 and January 1997 was divided into quarters. A random sample of 3 days from each quarter was selected. The Systems and Network Services Department at YDH provided a list of all the A&E and GP examinations undertaken for the selected days. A random sample of 25 A&E and 25 GP plain radiograph examinations was selected for each quarter, i.e. 400 A&E and 400 GP radiographs. Radiograph selection was also stratified by pertinent body areas in the control and experimental groups [8]. To assess the performance of the two CSRs separately, block randomization (n=2) was used to ensure each CSR reported on a comparable sample of radiographs and an equal number for the pertinent body areas. Only plain radiograph A&E and GP examinations were eligible for inclusion, except for those not routinely provided with a written report, e.g. plaster of Paris examinations and post-fixation examinations. Examinations previously reported by the reference standard were also excluded.
Both groups reported in normal viewing conditions using the departmental codes and free text. They were provided with a proforma simulating an X-ray request form that presented information about the clinical and demographic details of the patient. Any relevant previous plain radiographs were also available. However, they did not have access to the reports of these radiographs. Furthermore, each professional was asked to interpret the radiographs independently. The reference standard was a single consultant radiologist with 11 years experience in radiology and a special interest in skeletal radiology. This radiologist was provided with exactly the same information as the professionals under evaluation and reported on all 800 radiographs independently in the same viewing conditions. The reference standard was not one of the group of radiologists and did not have access to previous reports as these are a source of potential bias [9].
The arbiter who compared reports for concordance was a single consultant radiologist at a different hospital with 10 years experience in radiology and a special interest in skeletal radiology. The arbiter was blind to who produced each report. To permit ROC curve analyses the arbiter, using the details recorded on the proforma, categorized reports as definitely, probably or possibly abnormal, and probably or definitely normal. Normal was defined as within normal limits, no bony injury, or a clinically unimportant abnormality such as healed fractures. Abnormal was defined as all clinically relevant abnormalities. Reports were only judged concordant if they agreed about the presence, location and type of abnormality [10].
|
Results
|
|---|
ROC curves plot sensitivity, or true positive rate (TPR), vs 1-specificity, or false positive rate (FPR), for each category of reports and produce the value of the area under the curve the Az value. This value is the probability of a CSR or radiologist correctly deciding whether plain radiographs are normal or abnormal. An Az of 0.5 indicates guessing and an Az of 1.0 is perfect performance [11]. The AccurROC package (Accumetric Corporation, Canada) produced the Az values and 95% confidence intervals (CIs) using the method of DeLong et al [12]. A non-parametric method derived from correlated samples was used for comparing Az values between CSRs and radiologists [12]. Because the two CSRs did not report identical radiographs but a comparable sample, it is not possible to test whether there is a statistically significant difference between their Az values. However, if the 95% CIs for their individual Az values do not overlap, this is evidence of a statistically significant difference in performance at the 5% level.
When reporting on controls, CSR A and CSR B produced similar Az values of 0.79 (95% CI 0.69–0.89) and 0.76 (95% CI 0.63–0.88), respectively. Indeed, Figure 1 shows that the ROC curve for each CSR is almost identical. For the same radiographs, the CSRs and radiologists produced mean Az scores of 0.77 (95% CI 0.69–0.85) and 0.85 (95% CI 0.78–0.91), respectively. Although Figure 2 shows the radiologists are consistently more accurate for both sensitivity and specificity, there was no statistically significant difference between the CSRs' and radiologists' Az values (p=0.09).
View larger version (12K):
[in this window]
[in a new window]
|
Figure 1. Receiver operating characteristic curve comparing clinical specialist radiographer (CSR) A and CSR B for the control group. TPR, true positive rate; FPR, false positive rate.
|
|
View larger version (11K):
[in this window]
[in a new window]
|
Figure 2. Receiver operating characteristic curve comparing clinical specialist radiographers (CSRs) and radiologists for the control group. TPR, true positive rate; FPR, false positive rate.
|
|
For experimental (E) group E1, CSR A and CSR B produced Az values of 0.70 (95% CI 0.54–0.85) and 0.74 (95% CI 0.61–0.86), respectively. Figure 3 shows the two curves cross over so that neither is uniformly better. At low sensitivity CSR A was more accurate, but at high sensitivity CSR B was more accurate. When reporting these radiographs the CSRs and radiologists achieved mean Az values of 0.72 (95% CI 0.62–0.82) and 0.70 (95% CI 0.60–0.80), respectively. Figure 4 shows that the ROC curves for CSRs and radiologists are almost identical. Their Az values were not statistically significantly different (p=0.76).
View larger version (13K):
[in this window]
[in a new window]
|
Figure 3. Receiver operating characteristic curve comparing clinical specialist radiographer (CSR) A and CSR B for group E1. TPR, true positive rate; FPR, false positive rate.
|
|
View larger version (11K):
[in this window]
[in a new window]
|
Figure 4. Receiver operating characteristic curve comparing clinical specialist radiographers (CSRs) and radiologists for group E1. TPR, true positive rate; FPR, false positive rate.
|
|
When reporting on group E2, CSR A and CSR B produced Az values of 0.64 (95% CI 0.54–0.74) and 0.74 (95% CI 0.65–0.84), respectively. Figure 5 shows that CSR B consistently reported more accurately than CSR A. For the same radiographs the CSRs and radiologists produced mean Az values of 0.70 (95% CI 0.64–0.77) and 0.74 (95% CI 0.67–0.80), respectively. Figure 6 shows the two curves cross over so that neither is uniformly better. At low sensitivity the CSRs were more accurate, but at high sensitivity the radiologists were more accurate. Again, there is no statistically significant difference between the two groups' Az scores (p=0.37).
View larger version (13K):
[in this window]
[in a new window]
|
Figure 5. Receiver operating characteristic curve comparing clinical specialist radiographer (CSR) A and CSR B for group E2. TPR, true positive rate; FPR, false positive rate.
|
|
View larger version (12K):
[in this window]
[in a new window]
|
Figure 6. Receiver operating characteristic curve comparing clinical specialist radiographers (CSRs) and radiologists for group E2. TPR, true positive rate; FPR, false positive rate.
|
|
Finally, for E3 radiographs CSR A and CSR B produced Az values of 0.71 (95% CI 0.62–0.81) and 0.77 (95% CI 0.68–0.86), respectively. Figure 7 shows that CSR B consistently reported E3 radiographs more accurately than CSR A. The CSR's and radiologist's mean Az values were 0.74 (95% CI 0.68–0.81) and 0.77 (95% CI 0.70–0.83), respectively. Although Figure 8 shows that the radiologists were more accurate or equal to the CSRs for all values, there is again no statistically significant difference (p=0.51).
View larger version (13K):
[in this window]
[in a new window]
|
Figure 7. Receiver operating characteristic curve comparing clinical specialist radiographer (CSR) A and CSR B for group E3. TPR, true positive rate; FPR, false positive rate.
|
|
View larger version (12K):
[in this window]
[in a new window]
|
Figure 8. Receiver operating characteristic curve comparing clinical specialist radiographers (CSRs) and radiologists for group E3. TPR, true positive rate; FPR, false positive rate.
|
|
|
Discussion
|
|---|
The ROC curves suggest that consultant radiologists tend to report plain radiograph X-ray examinations for A&E or GP patients more accurately than the CSR. However, for the different sub group of radiographs there was no statistically significant difference between the two groups' Az values. Furthermore, combining the CSR and radiologist Az values for all four sub groups, to test for a difference in their respective mean Az values, produced a mean difference of 0.05 (95% CI -0.04–0.13), which is not statistically significant (p=0.29). Indeed, it was only for the controls that the p-value approached statistical significance (p=0.09). Az values of lower than 0.8, however, do not appear that impressive. This means that for only 80% of the radiographs were the observers making the correct decision compared with the reference standard. This is probably explained by the strict criteria used to judge reports for concordance in terms of the presence, location and type of abnormality [10].
Various precautions were taken to ensure the results of this study were robust. These included independence in radiograph interpretation by both the professionals under evaluation and the reference standard, independence in the comparison of reports by the arbiter and application of the reference standard to each radiograph. Furthermore, a representative and valid sample of radiographs was selected and the professionals reported in normal viewing conditions. The results should therefore be generalizable to the CSRs' and consultant radiologists' A&E and GP radiograph reading performance in clinical practice. However, it is important to note that the study design did not entirely reflect clinical practice as the two professional groups interpreted radiographs independently and could not seek advice or use previous relevant reports as would normally occur.
Limitations of the design include the use of a minimum acceptable reference standard [13], which affects the validity of the study, and the sample size, which affects the precision of the results as reflected in the width of the CIs. It was also not possible to employ block randomization to the process of selecting which individual radiographs the consultant radiologists interpreted during the audit sessions when they reported these radiographs. This was because it was logistically difficult to ensure which radiologists would be available and how much time they could devote. Therefore, the researcher who organised these sessions could only select a consecutive series of between 20 and 30 radiographs for whichever radiologist was available. No explicit bias in radiograph selection was present as neither the radiologist nor the researcher selectively chose the radiographs.
Finally, this study has some features common with a "diagnostic outcome" study (characterized by a large representative selection of radiographs from clinical practice, with an experienced single consultant radiologist as the reference standard) [13]. This type of study is useful for assessing whether one professional group could substitute or complement another in clinical practice. However, a large proportion of normal and obviously abnormal radiographs may mask statistically significant differences in performance when reporting on subtle but clinically important abnormalities. These missed abnormalities might result in serious long-term disability and costly medicolegal claims. This study was comprised of subtle abnormal A&E X-ray examinations, as those judged normal, insignificantly abnormal or equivocal by the casualty officers were included in the A&E sample. Nevertheless, further research using a "diagnostic accuracy" study design would be useful to specifically assess differences in performance between the two groups for these subtle cases. This would involve selecting a smaller sample of A&E and GP plain radiographs, with an equal ratio of normal and abnormal radiographs covering a range of pathology, body areas and degrees of conspicuity, and for which a more robust double/triple blind consultant radiologist report is developed [13].
|
Conclusion
|
|---|
This study found that the CSRs' and consultant radiologists' radiograph reading performance was not statistically significantly different when reporting plain radiograph X-ray examinations for A&E and GP patients. This is evidence that if CSRs are judged competent to report on appendicular radiographs for A&E patients, there is the potential, if desirable, to further expand their reporting role. Moreover, this study demonstrates that the CSRs performance is comparable with consultant radiologists. If these findings are replicated in other studies it has major implications for the future development of radiographers' professional status and practice.
|
Acknowledgments
|
|---|
The authors are grateful to the radiologists and radiographers at York District Hospital who reported the radiographs included in the study, and those staff in the X-ray Department without whose assistance this study would not have been feasible. The authors also thank the referees' for their comments.
Received for publication February 4, 2002.
Revision received September 17, 2002.
Accepted for publication September 24, 2002.
|
References
|
|---|
- The Royal College of Radiologists. Staffing and Standards in Departments of Clinical Oncology and Clinical Radiology. London: Royal College of Radiologists, 1993.
- The Royal College of Radiologists. Medico Legal Aspects of Delegation (FCR/2/93). London: The Royal College of Radiologists, 1993.
- Robinson PJA. Plain film reporting by radiographers—a feasibility study. Br J Radiol 1996;69:1171–4.[Abstract]
- Loughran CF. Reporting of fracture radiographs by radiographers: the impact of a training programme. Br J Radiol 1994;67:945–50.[Abstract]
- Price RC, Le Masurier SB, High L, Miller LR. Changing times: a national survey of extended roles in diagnostic radiography. Br J Radiol 1999;72(S):7.
- Brealey S. Measuring the effects of image interpretation: an evaluative framework. Clin Radiol 2001;56:341–7.[CrossRef][Medline]
- Manning DJ. Evaluation of diagnostic performance in radiography. Radiography 1998;4:49–60.
- Renwick IGH, Butt WP, Steele B. How well can radiographers triage X-ray radiographs in the accident and emergency department? BMJ 1991;302:568–9.
- Brealey S, Scally A. Bias in plain film reading performance studies. Br J Radiol 2001;74:307–16.[Abstract/Free Full Text]
- Robinson PJA. The nature of image reporting. In: Paterson A, Price R, editors. Current Topics in Radiography 2. London: W B Saunders, 1996:70–82.
- Metz CE. Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol 1989;24:234–45.[Medline]
- DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988;44:837–45.[CrossRef][Medline]
- Brealey S, Scally AJ, Thomas NB. Methodological standards in radiographer plain film reading performance studies. Br J Radiol 2002;75:107–13.[Abstract/Free Full Text]
This article has been cited by other articles:
|
|
|
|
 
R. O. Bude, J. P. Fatchett, and T. A. Lechtanski
The use of additionally trained sonographers as ultrasound practitioners: our first-year experience.
J. Ultrasound Med.,
March 1, 2006;
25(3):
321 - 327.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
S Brealey, D G King, S Hahn, C Godfrey, M T I Crowe, K Bloor, S Crane, and D Longsworth
The costs and effects of introducing selectively trained radiographers to an A&E reporting service: a retrospective controlled before and after study
Br. J. Radiol.,
June 1, 2005;
78(930):
499 - 505.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
S D Brealey and P A Scuffham
The effect of introducing radiographer reporting on the availability of reports for Accident and Emergency and General Practitioner examinations: a time-series analysis
Br. J. Radiol.,
June 1, 2005;
78(930):
538 - 542.
[Abstract]
[Full Text]
[PDF]
|
|
|
Top
Abstract
Introduction
