First published online December 6, 2006
British Journal of Radiology (2007) 80, 406-413
© 2007 British Institute of Radiology
doi: 10.1259/bjr/41006673
Evidence of reference standard related bias in studies of plain radiograph reading performance: a meta-regression
S D Brealey, PhD1,
A J Scally, MSc2,
S Hahn, PhD3 and
C Godfrey, BA4
1 York Trials Unit, Department of Health Sciences, University of York, York YO10 5DD, 2 Division of Radiography, University of Bradford, Bradford BD5 0BB, UK, 3 Medical Research Collaborating Center, Seoul National University College of Medicine, Seoul 110-744 Korea, 4 Department of Health Sciences, University of York, York YO10 5DD, UK
Correspondence: Stephen Brealey, PhD, York Trials Unit, Department of Health Sciences, University of York, York YO10 5DD, UK. E-mail: sb143{at}york.ac.uk
 |
Abstract
|
|---|
The aim is to determine the effect of reference standard related bias on estimates of plain radiograph reading performance using studies conducted in clinical practice. Data were extracted on study eligibility, clinical and reference standard characteristics and reading performance. The choice of reference standards and the prevalence of bias are presented descriptively. Associations between bias and reading performance are estimated using a regression model that produces relative diagnostic odds ratios (RDOR) with 95% confidence intervals (CIs). Three of the 20 eligible studies addressed all five reference standard related biases; 15 studies addressed three or more. When the reference standard report is influenced by knowledge of an observer's opinion this is associated with a significant overestimation in reading performance (RDOR, 3.7; 95% CI, 1.6 to 8.3; p = 0.01). There is limited evidence that reading performance is inflated when the observer is aware of the reference standard report before commenting on the radiograph (RDOR, 1.7; 95% CI, 0.6 to 5.1) and deflated when a less valid reference standard is used (RDOR, 0.5; 95% CI, 0.1 to 2.5). There is no evidence that reading performance is affected by application of the reference standard depending on an observer's opinion and using different reference standards in the same study. In conclusion we found variation in the choice and application of reference standards in studies of plain radiograph reading performance, but only when reference standards report in the knowledge of an observer's opinion does this contribute to a significant overestimation in reading performance.
|
Introduction
|
|---|
An increasing demand in a health care system with scarce resources has brought about changes in policy over the last decade in the flexible and creative use of allied heath care professional skills [1–3]. Clinical practice has changed with an increasing number of experienced and postgraduate trained radiographers reporting on a diverse range of medical images [4–7]. The evidence supporting these changes is also shaping the future vision of radiographers' reading of medical images [8, 9].
With the introduction of skill mix and role extension initiatives there is a need to develop quality standards for assessing radiographers' performance when reading medical images. This is to ensure that there is a robust evidence base to underpin the transfer of responsibility from radiologists to radiographers. It is also important for addressing the increasing expectations of patients and frequency in litigation [10]. Robinson pointed out that radiology's Achilles heel is "error and variation in the interpretation of the Roentgen image" and that this represents the weakest aspect of clinical imaging [11]. Concordance between three experienced consultant radiologists when reporting on skeletal, chest and abdomen plain radiographs has been shown to be 74%, 61% and 51%, respectively [12]. Even for experts observer variation when reporting on plain radiographs is substantial. Subsequently when designing research and audit to assess the introduction and monitoring of radiographer plain radiograph reading performance, an important methodological consideration is the choice and application of the reference standard, such as the opinion of expert observers like consultant radiologists, against which radiographers' performance is assessed.
In previous articles we discussed the potential sources of bias in plain radiograph reading studies [13, 14] and described the frequency with which they are present [15, 16]. Our systematic review of radiographers' reporting on plain radiographs shows that when the reference standard is aware of the radiographer's report this has a statistically significant effect on reporting performance [8]. A limitation of the analysis is that there were only 12 studies so there might be insufficient data to detect a statistically significant finding about the effect of other bias (a Type II error) and when this did occur it might have been a chance finding as a result of multiple testing (increasing Type I error). Now we have completed a systematic review about radiographers' red dot or triage of accident and emergency (A&E) plain radiographs we combine the two reviews so that we have more data to estimate the effect of reference standard related design characteristics on both the direction and the magnitude of plain radiograph reading performance [8, 17]. Lijmer et al [18] applied a similar approach to estimate the weight that should be attached to limitations in design characteristics of studies that evaluate the accuracy of diagnostic tests.
The aim of this paper is to describe the choice of reference standards that have been applied in studies that assess plain radiograph reading performance and the prevalence with which the reference standard related biases are present. We then determine the quantitative effect of reference standard related design characteristics on estimates of plain radiograph reading performance in clinical practice. Improving awareness of these potential biases should underpin the validity of the evidence base used to guide policies, influence good practice or direct research.
|
Methods and materials
|
|---|
Data sources and data extraction
As described in our earlier reviews we searched electronic sources such as MEDLINE and EMBASE, hand searched journals and supplements, made personal contact with experts and checked reference lists of eligible articles to identify studies [8, 17]. Eligible studies assessed radiographers' plain radiograph reading performance in clinical practice compared with a reference standard and provided data to construct 2x2 contingency tables. No language or geographical restrictions were applied. Data extraction from the reviews [8, 17] meant that it was feasible for SB alone to identify the relevant information on study eligibility, clinical and methodological characteristics and reading performance.
Assessment of study validity
Various terms are used to describe the standard against which the validity of individuals' reading performance are assessed including reference, gold, criterion and index standard. In our study we use the term reference standard and classify a single consultant radiologist reference standard report (with or without validation by clinical follow up) as more valid than the other reference standards such as individual radiologists of varying seniority.
We define bias as systematic (or one-sided) deviation in study results from the truth. The focus of our study was on reference standard related biases that we adapted from an evidence-based assessment tool called QUADAS that is used in systematic reviews of diagnostic test accuracy [19]. The summary box presents the different questions that were asked about these biases and defines each bias in turn.
Each question was answered "yes", "no" or "unclear". We classified "unclear" as "no" in the analysis. Three other items from the QUADAS tool could also have been adapted. Was the observer's opinion incorporated into the reference standard report? Was there a time delay between the generation of the reference standard report and the observer's opinion? Was the execution of the reference standard described in sufficient detail? Although an observer's opinion sometimes influenced the generation of a reference standard report it was not explicitly incorporated into that report. The same plain radiographs were used against which the reference standard report and the observers' opinions were generated so time delay was not relevant and the reference standards were always executed as a formal text report. We therefore judged that these methodological considerations were not applicable to our study.
Statistical analysis
The choice of reference standard and the prevalence of the different reference standard related biases are presented descriptively. For individual studies we then calculate the observers' plain radiograph reading performance using sensitivity, specificity and the diagnostic odds ratio (DOR) which is sensitivity/(1–specificity) divided by (1–sensitivity)/specificity. The Littenberg–Moses summary receiver operating characteristic method is used to test a hypothesis of equal DOR when a reference standard related bias is absent or not [20]. This regression model comprises the logarithm of the DOR as a dependent variable and two explanatory parameters: one for the intercept or the common DOR; and one for the slope which expresses the extent of variation in the DOR across individual studies due to threshold differences. Covariates are added to this model to represent whether biases are absent or not. The antilogarithm transformation of the resulting estimated parameters are interpreted as a relative DOR (RDOR) of the corresponding covariate. They indicate the change in plain radiograph reading performance per unit increase in the covariate, i.e. the difference between the two categories of the dummy covariate that represent whether the bias is absent or not. The RDOR can range from zero to positive infinity. If the RDOR is larger than 1 then studies in which a reference standard related bias is present yield larger estimates of the DOR than studies in which a bias is absent.
Our model comprised the logarithm of the DOR as the dependent variable. When a study assessed both radiographer and radiologist performance the data for both groups were combined. This is because we are not examining different professionals' reading performance but the effect of the presence of bias on such studies. The common DOR and the threshold parameter are included in the model as continuous independent variables. As there are only 20 studies we created dummy independent variables that represented the clinical characteristics of the studies as follows: reading method (reporting = 0, other = 1), hospital setting (district = 0, other = 1), type of patient (A&E = 0, other = 1) and body area (skeleton = 0, other = 1). Whilst simultaneously controlling for these continuous and binary variables a covariate for each reference standard related design characteristic is added to the model in turn to quantify the effect on reading performance (absent = 0, present = 1). Whilst we test a hypothesis of equal DOR we expect the presence of each bias to overestimate reading performance. This assumption is based on a priori expectations and evidence from the Lijmer et al [18] study evaluating aspects of study quality on diagnostic accuracy. To account for within study variances we use linear regression analysis weighted by the inverse variance of the log of the DOR. SPSS® (version 12.0) and Meta-Disc are used for data synthesis [21].
|
Results
|
|---|
20 studies were eligible [22–41]. Table 1
presents information on study characteristics, 2x2 data and accuracy estimates with 95% CIs for each study. 12 studies were based on the reporting of plain radiographs, five studies on the triage of plain radiographs, and three studies used red dot systems. Most studies were conducted in a district hospital setting (n = 12) and included a sample of A&E plain radiographs (n = 18) of the skeleton (n = 15).
Table 2 describes the type of reference standard that was chosen for each study and whether a reference standard related bias was present or not. 18 of the 20 studies used a radiologist report as the reference standard: 11 studies used a single consultant radiologist, six studies used different radiologists of varying seniority, and one study used a consultant radiologist whose report was validated by clinical follow up. In the remaining studies the choice of reference standard was a single radiographer with postgraduate training in image interpretation and another study used a variety of different health care professionals. Three of the 20 studies eliminated all five potential biases; 15 studies did this for three or more biases. Reference standard review bias was present in nine studies, reference standard bias was present in eight studies, partial verification bias and differential verification bias in seven studies, and observer review bias in two studies.
Tables 1
and 2 show that reference standard bias and differential verification bias are more frequent in the red dot or triage studies because the reference standard often comprised radiologists of varying seniority. In addition these studies were susceptible to reference standard review bias as the application of red dots to the radiographs meant that the reference standard was aware of the observer's opinion before making their report. In contrast partial verification bias was often present in reporting studies. This is because the process of assessing whether the observer report was correct affected the application of the reference standard report.
The results of the regression analyses for the reference standard biases are presented in Table 3. When holding all other explanatory variables constant the most salient finding is that the reference standard review variable contributes to a significant overestimation in the DOR when this bias is present (RDOR, 3.66; 95% CI, 1.61 to 8.32; p = 0.01). The R2 was 0.81 also demonstrating that this bias is more strongly associated with the observers' reading performance than the other biases although some unexplained residual heterogeneity remains. There is limited evidence to suggest that reading performance is inflated when the observer is aware of the reference standard report before commenting on the radiograph (RDOR, 1.7; 95% CI, 0.6 to 5.1; p = 0.33) and deflated when a less valid reference standard is used (RDOR, 0.5; 95% CI, 0.1 to 2.5; p = 0.33). There is no evidence that reading performance is affected by the application of the reference standard depending on an observer's opinion (partial verification bias) or using different reference standards in the same study (differential verification bias).
|
Discussion
|
|---|
We found variation in the choice and application of reference standards in studies that assess plain radiograph reading performance in clinical practice. It is arguable that this variation in methodology might affect the validity of study findings and hence the estimates of reading performance. Our analyses show that of the five biases only reference standard review bias is associated with a statistically significant effect in overestimating reading performance. There is only limited evidence to suggest how the observer's knowledge of the reference standard report before commenting on the radiograph and the use of a less valid reference standard might affect reading performance. These findings could be a consequence of how we defined whether biases are absent or not and the lack of statistical power because of the small sample of studies. Furthermore the associations between these study level covariates and estimates of reading performance are between study associations so do not necessarily have a causal interpretation. Great care should be taken when interpreting these findings as discussed below.
Our findings show that when reference standard bias is present or a less valid reference standard is used there is limited evidence that this deflates reading performance. This finding was not statistically significant so is likely to be a chance effect. A possible explanation for this finding, however, might be that the observers under evaluation are more competent than the less valid reference standards. This could result in their reports being wrongly assessed as incorrect which inappropriately deflates their performance thus emphasising the need for valid reference standards. We also expected that partial verification bias would inflate estimates of reading performance. This is because omission of the reference standard denies the potential for identifying other errors and could be exacerbated if the reference standard is aware that they are reporting radiographs when there is discordance between observers' reports [14]. There is no evidence, however, that this bias affected estimates of reading performance. Similarly there is no evidence that differential verification bias affects reading performance. The last two biases refer to independence (or blinding) in the reading of plain radiographs. The results of the analyses for the presence of observer bias provide limited evidence that this might inflate estimates of reading performance but did not reach conventional statistical significance so might be explained by chance alone.
The largest effect on the estimation of plain radiograph reading performance was the presence of reference standard review bias. Not only is this bias associated with almost a fourfold overestimation in the DOR but it was also a significant finding (p = 0.01). This evidence complements the results from our earlier review about radiographer plain radiograph reporting that this is the only methodological characteristic to significantly affect estimates of performance [8]. This bias is particularly prevalent in red dot studies because of the addition of an adhesive label to the radiograph. It should not be too difficult to devise a scheme so that the reference standard is blind to the observers' interpretation of the radiographs. It is also arguable that the performance of observers in reporting studies may be influenced by the use of a red dot system, as well as the reference standard. Both red dot and reporting systems are used in conjunction in many centres. In our opinion it is unlikely that this introduces another bias as in a reporting study both the observers and the reference standard know the outcome of the red dot system so they both have access to the same information.
Only three of the 20 studies addressed all the reference standard related design characteristics [23, 24, 31]. These studies assessed radiographers' performance when reporting on A&E plain radiographs of the skeleton and avoided bias by using a single consultant radiologist as the reference standard who reported all the radiographs in the sample and both the radiographers and the reference standard reported independently of each other. Time and thought should be given to avoid or minimize reference standard related design biases, but possibly owing to resource constraints and study objectives there might be trade-offs between methodological purity and what is practical. We found that whilst there is variation in the choice and application of reference standards there is no or little evidence that this affects estimates of plain radiograph reading performance. This implies that for the purpose of clinical audit the more feasible methods of choosing and applying reference standards presented here are acceptable. For example, it may not always be feasible to apply a consultant radiologist as the reference standard to all reports. The evidence suggests that differential verfication bias does not have a significant effect on the estimate of accuracy so it might be acceptable to use radiologists of varying seniority as the reference standard. Precautions should still be taken though to explore whether bias has been introduced. The analyses could be stratified by seniority of radiologist to test whether estimates of accuracy vary with seniority. If variable estimates of accuracy were found then this should be reported as a possible limitation of the study. One bias, however, that should always be avoided is reference standard review bias. Despite the need for blinding in medical research being well documented this bias is the most prevalent bias present in the included studies. When the reference standards report in the knowledge of an observer's opinion this significantly inflates reading performance.
The reference standard related biases discussed here are only one type of many important design features such as the selection of the sample of radiographs to be reported on, the choice and application of the arbiter who judges whether reports are concordant, and the non-random withdrawal of radiographs to be reported on [14, 15]. When few studies meet most of the methodological criteria clinicians and policy makers are left with difficult decisions as are reviewers when weighing the quality of the evidence [18]. Strict application of methodological criteria implies that only a small minority of available data can be used to inform clinical practice. Alternatively inclusion of a wide range of imperfect studies requires weighting of evidence according to the relative importance of criteria. This study only provides limited evidence to help inform these difficult decisions. In the absence of clear empirical guidance it is all the more important that authors reliably and completely describe the design and conduct of their studies; assessment of quality is only feasible when there is complete clarity on the methodology. This should be facilitated with the development of tools such as QUADAS that is used in systematic reviews of diagnostic test accuracy [19] and the STARD initiative for improving the writing up of diagnostic accuracy studies [42]. Greater editorial vigilance has been recommended to raise authors' awareness of current methodological standards [18].
Finally there is growing evidence that postgraduate trained radiographers can report A&E plain radiographs in a comparable manner to radiologists of varying seniority [4, 8, 43]. Observer variation between consultant radiologists when reporting on plain radiographs is also substantial [12, 44]. It is questionable whether we should try to generate the elusive gold standard report that underpins the assessment of error or accuracy in medical image reading studies. A more appropriate and feasible design to consider in future evaluative studies and audit, particularly for the reading of more complex images, is to examine the intra- and interobserver agreement between the same and different groups of professionals compared with the report of an index radiologist who is judged to be the local standard. Such a design could still reflect whether or not the observers are "undercalling" or "overcalling" as shown by estimates of sensitivity and specificity in accuracy studies and the findings presented as simple proportions or using the Kappa statistic [45].
|
Conclusion
|
|---|
Ideally all conceivable bias should be eliminated but resource constraints and study objectives might affect what is practical. We found that whilst there is variation in the choice and application of reference standards there is no or little evidence that this affects estimates of plain radiograph reading performance so the more feasible methods of choosing and applying reference standards presented here might be considered acceptable. Great care should always be taken though to exclude reference standard review bias as, when the reference standards report in the knowledge of an observer's opinion, this significantly inflates reading performance. The challenge to those designing and executing future studies is to achieve the right balance between what can be critically judged as acceptable and what is feasible.
|
Appendix 1. Definitions of reference standard related biases
|
|---|
Reference standard bias
Is the reference standard report likely to correctly classify the target condition? The veracity of a reference standard affects the validity of an observer's performance so this bias occurs when a less valid reference standard is used. This bias may inflate reading performance when observers' opinions converge more with a less valid reference standard report, i.e. the errors in the observers' opinions are positively correlated with errors made by the less valid reference standard.
Partial verification bias
Were all the observers' opinions or a random selection of the observers' opinions verified against a reference standard report? This bias occurs when rationale other than verification against the reference standard report determines whether observers' opinions are correct or not. For example, only applying the reference standard when two groups of observers' reports disagree can introduce this bias. Alternatively a radiographer's performance might be assessed against that of a radiologist. When both reports agree the radiographer's report is correct but when the radiographer report disagrees with the radiologist's a second radiologist is asked to report on the radiograph. The first report that is concordant with the second radiologist's report is judged to be correct. Whilst implicitly the reference standard is the second radiologist the process of appraising the report obfuscates the process of verification.
Differential verification bias
Was the same reference standard report applied regardless of the observer's opinion? When different reference standards of varying validity are used within the same study this will have a variable effect on estimates of reading performace and again is likely to inflate these estimates when a less valid reference standard is applied.
Observer review bias
Was the observer's opinion made without knowledge of the reference standard report? When an observer comments on a radiograph in the knowledge of the reference standard report this might lead the observer to change their opinion to agree with that report and thus inflate estimates of reading performance.
Reference standard review bias
Was the reference standard report made without knowledge of the observer's opinion? When a reference standard reports on a radiograph and is aware of the observers' opinion this might lead the reference standard to consider something they otherwise would not have done and is likely to inflate estimates of reading performance.
|
Acknowledgments
|
|---|
We are indebted to the authors of the included studies and also thank the referees for their comments.
Received for publication March 15, 2006.
Revision received August 23, 2006.
Accepted for publication September 5, 2006.
|
References
|
|---|
- College of Radiographers. Reporting by radiographers: A vision paper. London: COR, 1997
- Department of Health. The NHS plan: A plan for investment. A plan for reform. London: Stationary Office, 2000
- Department of Health. Meeting the challenge: A strategy for the allied health professions. London: Stationary Office, 2000
- Brealey S, King DG, Hahn S, Godfrey C, Bloor K, Crowe MTI, et al. The costs and effects of introducing selectively trained radiographers to an A&E reporting service: A retrospective controlled before and after study. Br J Radiol 2005;78:499–505.[Abstract/Free Full Text]
- Robinson PJA, Culpan G, Wiggins M. Interpretation of selected accident and emergency radiographer examinations by radiographers: a review of 11 000 cases. Br J Radiol 1999;72:1–6.[Medline]
- Moller JM, Larsen L, Bovin J, Lausten GS, Hasselqvist M, Jensen CM, et al. MRI diagnosis of fracture of the scaphoid bone: impact of a new practice where the images are read by radiographers. Acad Radiol 2004;11:724–8.[Medline]
- Wivell G, Denton ERE, Eve CB, Inglis JC, Harvey I. Can radiographers read screening mammograms? Clin Radiol 2003;58:63–7.[CrossRef][Medline]
- Brealey S, Scally AJ, Hahn S, Thomas N, Godfrey C, Coomarasamy A. Radiographer plain radiograph reporting performance during clinical practice: a meta-analysis. Clin Radiol 2005;60:232–41.[CrossRef][Medline]
- College of Radiographers. Medical image interpretation & clinical reporting by non-radiologists: The role of the radiographer. London: COR, 2005
- Berlin L, Berlin JW. Malpractice and radiologists in Cook County, IL: trends in 20 years of litigation. AJR Am J Roentgenol 1995;165:781–8.[Abstract/Free Full Text]
- Robinson PJA. Radiology's Achilles heel: error and variation in the interpretation of the Roentgen image. Br J Radiol 1997;70:1085–98.[Abstract]
- Robinson PJA, Wilson D, Coral A, Murphy A, Verow P. Variation between experienced observers in the interpretation of accident and emergency radiographs. Br J Radiol 1999;72:323–30.[Abstract]
- Brealey S. Measuring the effects of image interpretation: An evaluative framework. Clin Radiol 2001;56:341–7.[CrossRef][Medline]
- Brealey S, Scally AJ. Bias in plain film reading performance studies. Br J Radiol 2001;74:307–16.[Abstract/Free Full Text]
- Brealey S, Scally AJ, Thomas NB. Methodological standards in radiographer plain film reading performance studies. Br J Radiol 2002;75:107–13.[Abstract/Free Full Text]
- Brealey S, Scally AJ, Thomas NB. Presence of bias in radiographer plain film reading performance studies. Radiography 2002;8:203–10.[CrossRef]
- Brealey S, Scally A, Hahn S, Thomas N, Godfrey C, Crane S. Accuracy of radiographers red dot or triage of accident and emergency radiographs in clinical practice: a systematic review. Clin Radiol 2006;61:604–15.[CrossRef][Medline]
- Lijmer JG, Mol BW, Heisterkamp S, Bonsel GJ, Prins MH, van der Meulen JH, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA 1999;282:1061–6.[Abstract/Free Full Text]
- Whiting P, Rutjes AWS, Reitsma JB, Bossuyt PM, Kleijnen J. The development and validation of methods for assessing the quality of diagnostic accuracy studies. Health Technol Assess 2004;8:(25)
- Deeks J. Systematic reviews of evaluations of diagnostic and screening tests. In: Egger M, Smith GD, Altman G, editors. Systematic reviews in health care: meta-analysis in context. London: BMJ Publishing Group, 2001: 248–82
- Zamora J, Muriel A, Abraira V. Meta-DiSc for Windows: A software package for the meta-analysis of diagnostic tests. XI Cochrane Colloquium. Barcelona, 2003. Available at http://www.hrc.es/investigacion/metadisc_en.htm
- Piper KJ, Paterson AM, Ryan CM. The implementation of a Radiographic Reporting Service for trauma examinations: final analyses of accuracy. Br J Radiol 2000;73:(S)91[Abstract]
- Pitchers M. Radiographic interpretation by emergency nurse practitioners and radiographers in a Minor Injuries Unit (thesis). Hatfield: University of Hertfordshire, 2002
- Carter S, Manning D. Performance monitoring during postgraduate radiography training in reporting – a case study. Radiography 1999;5:71–8.[CrossRef]
- Manning D. Results from audit process of students on a postgraduate course (personal communication). Lancaster: University of Lancaster, 1999
- Robinson PJA. Plain film reporting by radiographers – a feasibility study. Br J Radiol 1996;69:1171–4.[Abstract/Free Full Text]
- Balcam S, Hood A. Are radiographers with suitable training competent to "cold" report plain films from the accident and emergency department? (personal communication). Hull: Hull Royal Infirmary, 1998
- Eyres RD, Williams P. Results of a postgraduate training programme (personal communication). Salford: University of Salford, 1999
- Eyres RD, Williams P. Monitoring the effectiveness of a clinical reporting and educational programme: are radiographers getting it right? Br J Radiol 1997;70:(S)123
- Wolfe G. How well do radiographers identify and describe abnormalities on axial and appendicular radiographs in A&E X-ray (BSc thesis). Glasgow: Glasgow Caledonian University, 2002
- Ford L, Crawshaw I. Results of audits (personal communication). York: York District Hospital, 1999
- Loughran CF. Reporting of fracture radiographs by radiographers: the impact of a training programme. Br J Radiol 1994;67:945–50.[Abstract/Free Full Text]
- Snaith BA. Has radiography outgrown the red dot? Br J Radiol 2000;73:(S)32
- Morrison ID, Hendry R, Fell C, Stothard K. An audit of radiographers' accuracy in recording abnormalities on casualty radiographs using the red dot protocol. Clin Radiol 1999;54:862–3.[CrossRef]
- Bowman S. Introducing an abnormality detection system by radiographers into an accident and emergency department: An investigation into radiographers' concerns about the introduction of such a system. Res Radiogr 1991;1:2–20.
- Hargreaves J, Mackay S. The accuracy of the red dot system: can it improve with training? Radiography 2003;9:283–9.[CrossRef]
- Rolph H. Red dot radiographers: assessment of their ability to detect abnormalities and write a short quality description (BSc Thesis). Glasgow: Glasgow Caledonian University, 2003
- Renwick IGH, Butt WP, Steele B. How well can radiographers triage X-ray films in the accident and emergency department? BMJ 1991;302:568–9.[Abstract/Free Full Text]
- Khiroya T, Taylor S, Remedios D, de Lacey G. Trauma radiology: the accuracy of radiographers' assessment of appendicular trauma. Br J Radiol 1997;70:(S)123
- Ross C. The effectiveness of a short course of study on the ability of radiographers to highlight and describe abnormalities on skeletal accident and emergency films (MSc thesis). London: South Bank University, 2002
- Remedios D, Ridley N, Taylor S, de Lacey G. Trauma radiology: extending the red dot system. Br J Radiol 1998;71:(S)60
- Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, et al. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Fam Pract 2004;21:4–10.[Abstract/Free Full Text]
- Brealey S, King DG, Hahn S, Crowe MTI, Williams P, Rutter P, et al. Radiographers and radiologists reporting plain radiographs requested from accident and emergency and general practice. Clin Radiol 2005;60:710–17.[CrossRef][Medline]
- Tudor GR, Finlay D, Taub N. An assessment of inter-observer agreement and accuracy when reporting plain radiographs. Clin Radiol 1997;52:235–8.[CrossRef][Medline]
- Altman DG. Practical statistics for medical research. London: Chapman and Hall, 1991:403–9
This article has been cited by other articles:
|
|
|
|
 
S BREALEY, C HEWITT, A SCALLY, S HAHN, C GODFREY, and N THOMAS
Bivariate meta-analysis of sensitivity and specificity of radiographers' plain radiograph reporting in clinical practice
Br. J. Radiol.,
July 1, 2009;
82(979):
600 - 604.
[Abstract]
[Full Text]
[PDF]
|
|
|
Top
Abstract
Introduction
