This Article Abstract Full Text (PDF) Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Brealey, S Articles by Scally, A J Search for Related Content PubMed PubMed Citation Articles by Brealey, S Articles by Scally, A J

Bias in plain film reading performance studies

¹ Department of Health Sciences & Clinical Evaluation, University of York, York YO1 5DD ² Division of Radiography, University of Bradford, Bradford BD5 0BB, UK

	Abstract

Top Abstract Introduction Biases due to patient... Biases due to observer... Biases associated with the... Biases due to measurement... Independence of interpretation... Further biases in film... Conclusion References

 
Radiographers and other healthcare professionals are becoming increasingly involved in radiological reporting, for instance plain radiographs, mammography and ultrasound. Systematic reviews of research evidence can help to assimilate a knowledge base by ordering and evaluating the available evidence on the reporting accuracy of different professional groups. This article reviews the biases that can undermine the results of plain film reading performance studies. These biases are subdivided into three categories. The first category refers to the selection of subjects, including both films and professionals, and covers the validity of generalizing results beyond the study population. The other two categories are concerned with study design and the interpretation both of films and of reports and the effect on study validity. An understanding of these biases is essential when designing such studies and when interpreting the results of existing studies.

	Introduction

Top Abstract Introduction Biases due to patient... Biases due to observer... Biases associated with the... Biases due to measurement... Independence of interpretation... Further biases in film... Conclusion References

 
The NHS and Community Care Act (1990) produced a climate in which the traditional methods of delivering healthcare were challenged, resulting in the blurring of professional boundaries [1]. A recent national survey found that radiographers were involved in reporting Accident and Emergency (A&E) radiographs at 37 Trusts [2], whereas a similar survey performed just 4 years earlier identified only four Trusts [3]. There is also evidence of emergency nurse practitioners interpreting A&E radiographs [4, 5], and of radiographers and non-radiological medical practioners reporting images other than plain films, such as mammograms [6, 7], abdominal ultrasound [8] and CT head examinations [9].

Conducting a systematic review involves locating, appraising and synthesizing evidence from scientific studies to provide empirical answers to well defined research questions [10]. This requires adherence to strict scientific design to ensure the review is both comprehensive and minimizes bias, thus providing reliable results from which decisions about the delivery of healthcare are made [11, 12]. It is most straightforward to synthesize results from well planned and well executed randomized controlled trials (RCTs), as this study design is least subject to bias and there are statistical models available for pooling estimates of effect [13]. In some areas of medicine and healthcare there are very few, if any, RCTs despite an extensive literature of research data [14]. A large scale RCT that compares the film reading performance of two different professional groups would protect against some of the biases. However, such studies are expensive and may not be amenable to a rapidly evolving political climate. It is therefore important to be aware of the weaknesses and strengths of alternative study designs.

There is also a conceptual hurdle to overcome—when evaluating film reading performance how does one relate patient outcome to the reports made by different professionals when other factors such as therapy are involved? This difficulty was resolved in the context of imaging technologies by applying a hierarchical framework first suggested by Fineberg et al [15] and subsequently extended by the Institute of Medicine [16]. The categories in the framework are: technical capability; diagnostic accuracy; diagnostic impact; therapeutic impact; and impact on health [17]. Film reading performance studies are comparable with the "diagnostic accuracy" category. Studies could also measure changes in referring clinicians' diagnosis (diagnostic impact), management plans (therapeutic impact) or patient health status (impact on health) following reports made by different professionals. However, the subject of this article is those studies that assess only the plain film reading performance of healthcare professionals (i.e. observers). Such studies involve selecting a sample of observers from the same or different professions to interpret a sample of films. Then, a healthcare professional (i.e. arbiter) judges whether the reports made by the observers are concordant with a reference standard (consultant radiologist). This allows the investigator to assess how accurately the observers can interpret the films. Table 1 illustrates the classification of such studies, including clinical examples [18].

	Biases due to patient (or film) selection

Top Abstract Introduction Biases due to patient... Biases due to observer... Biases associated with the... Biases due to measurement... Independence of interpretation... Further biases in film... Conclusion References

Film cohort bias
Following referral, the criteria used to establish which films are eligible for inclusion will also affect the characteristics of the sample selected. Depending on whether the purpose of the study is to assess the potential for radiographers to report in clinical practice or is more pragmatic in design determines whether spectrum or population bias arises.

Spectrum bias
If the focus is to assess the potential of radiographers to report, for instance students on an "image interpretation" course reporting a validated bank of radiographic examinations [20], a limited range of disease type, severity, duration or clinical demographics in the film sample can considerably bias performance [21–23]. Prevention of this bias is more concerned with internal validity, as the aim is to assess radiographers' performance when reporting an unrepresentative but more difficult batch of films. Therefore, stratifying the sample in favour of a higher prevalence and variety of pathology is desirable.

Population bias
In contrast, the emphasis may be on external validity if the aim is to assess radiographers' A&E film reading performance in clinical practice [24, 25]. A representative case mix is required if the results are to be generalized to how well radiographers can interpret all A&E films. Ideally, wewish to describe radiographers' ability to diagnose the disease status of the patient as positive or negative. However, in the absence ofanincontrovertible standard, radiographer performance is often compared with a reference standard such as the diagnosis of a consultant radiologist who would initially have a greater clinical knowledge and experience. We are therefore evaluating radiographers' ability to predict the radiologist's diagnosis, which may be wrong, rather than the patient's true disease status. Subsequently, when sensitivity and specificity are calculated, they are related to the prevalence of the abnormality [26]. As the reference standard is not always correct, it is prudent, if possible, to evaluate radiographers' performance using a random sample of films from clinical practice to reflect the same prevalence of disease [27].

Film selection bias
This bias occurs if the radiographers do not interpret all the films eligible for inclusion in the study and/or they have the opportunity to choose which eligible films they want to interpret [28, 29]. Thus, if radiographers' A&E film reading performance was assessed and they were asked to report films when time permitted, they may only interpret films they are confident to report. By not reporting the more difficult cases, this could inflate their indices of performance.

	Biases due to observer selection

Top Abstract Introduction Biases due to patient... Biases due to observer... Biases associated with the... Biases due to measurement... Independence of interpretation... Further biases in film... Conclusion References

 
Observer selection bias
Careful thought should be given to defining which films are eligible for inclusion, as whether performance data are collected from all such films or from a sample, conclusions will only be drawn about the films fulfilling the inclusion criteria. Similarly, if only selectively trained radiographers within a single department are being assessed, conclusions should not be made concerning how well all radiographers report. The only inferences drawn are how well the specific group of radiographers report on the selected films. It is again important that criteria are explicitly recorded so an assessment of their relevance can be made. Two examples of observer selection bias are described below.

Observer cohort bias
This would occur if an inappropriate selection of radiographers was included in a study, being very dependent on the research question being asked. To estimate the influence on external validity, it is important to know the number of radiographers and the level of their experience or training.

Observer cohort comparator bias
This bias can occur when two or more groups of observers are compared without the appropriate use of matching; again, this is dependent on the research question. For example, a study might apply the principles of a controlled trial to demonstrate the effectiveness of a training programme (intervention) by comparing a study group (radiographers who have received training) and a control group (radiographers without training) [20]. The two groups of radiographers should be matched for certain characteristics such as number of years experience in the profession and/or in a relevant specialty. It is important to ensure comparability between the two groups so that differences in performance can be attributed to the training programme rather than differences between radiographers.

	Biases associated with the application of the reference standard

Top Abstract Introduction Biases due to patient... Biases due to observer... Biases associated with the... Biases due to measurement... Independence of interpretation... Further biases in film... Conclusion References

 
Verification bias
This bias can have dramatic consequences in terms of sensitivity and specificity [26]. This would occur if not all films interpreted by the radiographers were interpreted by the same reference standard. Economic limitations may prevent a reference standard report being produced for each film in the sample. The reference standard may not be applied if the clinical signs and symptoms suggest that the film is normal.

Work-up bias
There are contradicting opinions as to whether there is any difference between verification bias and work-up bias [22, 26]. Work-up bias is defined here as a specific type of verification bias. It occurs when not all films receive confirmation with the reference standard owing to the report of the observer under evaluation.

Work-up bias would occur if an investigator compared the A&E film reading performance of radiographers and casualty officers when reporting the same films and assumed that if the pair of reports agree there was no need to apply the reference standard [24, 25]. If the standard was applied, it may be discordant with the pairs of reports. The statistical consequence is an overestimation of the two professions' film reading performance, as omission of the standard denies the potential for identifying other false negatives and/or false positives. A further example would be if only radiographers' A&E film reading performance was evaluated and the investigator assumed that if a report was normal it was unnecessary to apply the reference standard. If only abnormal reports receive verification with a standard, this will artificially inflate sensitivity by underestimating the number of false negatives [30]. This bias is exacerbated if the reference standard knows that the reason they are reporting the films is because there was discordance between a radiographer and casualty officer, or in the case of radiographers alone that it was reported as being abnormal.

The problem of verification and work-up bias may be avoided if the observers' reports are not known before the reference standard is applied. It is also possible in certain circumstances to correct the results obtained if data are available on a stratified random sample of negative films as well as the clinical details of each patient [30–32].

Incorporation bias
This occurs if the observer under evaluation is incorporated into the process of generating the reference standard or is used as the reference standard. For example, a study may assess the film reading performance of a group of radiographers vs radiologists of varying seniority. Incorporation bias exists if a radiologist's report within a cohort was used to generate the reference standard, for example a double blind radiologist's report. This will artificially inflate the performance of the radiologists, as there will be confounding of the radiologist's report within the cohort and the reference standard report.

	Biases due to measurement of results

Top Abstract Introduction Biases due to patient... Biases due to observer... Biases associated with the... Biases due to measurement... Independence of interpretation... Further biases in film... Conclusion References

 
Disease progression bias
This occurs if there is a long delay between the radiographer's report and the use of patient re-attendance as a proxy for patient outcome and the reference standard. To avoid this bias, it is appropriate to use consensus review to identify why the patient re-attended, perhaps for another examination or to seek further treatment. When assessing radiographic reporting of A&E films it would be important to identify whether a fracture was missed on the initial examination or an occult fracture was only visible on examinations following patient re-attendance. This bias is not applicable if the reference standard involves only film interpretation.

Withdrawal bias
Any non-random exclusion of films that have been judged eligible for inclusion in a study will bias the results. Furthermore, if films are excluded prior to receiving the reference standard it will introduce work-up bias or verification bias depending on the reason for exclusion. The following describes two kinds of withdrawal bias.

Indeterminate results
Failure to include indeterminate (equivocal) film interpretations in the analysis may result in a possibly biased assessment of radiographers' performance [26]. Their inclusion is valuable for economic assessment, for example, if radiographers ask for repeat examinations but radiologists interpret films correctly without the need for repeats this will save direct healthcare costs, and for generalization of results. If indeterminate results are included but regarded as negative, specificity is artificially increased and sensitivity is decreased, and the reverse is true if they are classified as positive [23]. For these reasons, it is important that studies record the frequency of equivocal reports and the way these results are used in the calculation of radiographers' performance.

Loss to follow-up
The films reported by radiographers may be lost and the reference standard cannot therefore be applied. If this is systematic, it may distort the performance results of radiographers.

Observer variability
Reliability is a major consideration in studies involving judgement [33], not only in image interpretation but also when comparing the report under evaluation with the standard. The ability of observers to produce reliable reports is also reflected in their ability to accurately interpret films. For example, a low level of reproducibility is incompatible with a high level of accuracy [34]. Observer variation in plain film reporting is pervasive. A recent study examined the interobserver variation between three experienced radiologists with the three major types of plain film examination; abdominal, chest and skeletal. Concordance between all three was found in only 51%, 61% and 74% of radiographs, respectively [35]. Interobserver variation is usually greater than intraobserver variation and is measured using the Kappa statistic [36]. The following variabilities can be estimated.

• Interobserver variability: observers within different groups should independently report the same or similar sample of films. This would allow an assessment of consistency or variability in film interpretation between, for example, radiographers and casualty officers.
  
• Intraobserver variability: this is concerned with the reproducibility of film interpretation by a single observer at different points in time. Radiographers and casualty officers could re-interpret a subsample of A&E radiographs at a suitable later date to ascertain which cohort performed more consistently.
  

Arbiter variability
Even when explicit and objective decision-making criteria are available for comparing observers' reports with the reference standard, it is important to assess whether it can be applied consistently by different people (arbiters) on the same occasion or by the same person on different occasions. Variation in the decisions made by an arbiter can affect the indices of performance calculated. The application and interpretation of the criteria used to measure radiographers' performance therefore influence the reliability of study results. The following variabilities can also be estimated using the Kappa statistic.

• Interarbiter variability: this is when two independent arbiters (e.g. consultant radiologists) compare a subsample of radiographers' reports with the reference standard to assess whether the criteria are applied consistently by different people.
  
• Intraarbiter variability: this is when the same arbiter compares a subsample of the radiographers' reports with the reference standard at a suitable later date to assess whether the criteria are applied consistently.
  

	Independence of interpretation biases

Top Abstract Introduction Biases due to patient... Biases due to observer... Biases associated with the... Biases due to measurement... Independence of interpretation... Further biases in film... Conclusion References

 
Whether a study is susceptible to these biases depends on the purpose of the evaluation. If the aim is to measure radiographers' "true" film reading performance, they should report independently without conferring and be blind to all other reports, as prior knowledge can dramatically influence results [37]. Conversely, if the objective is to assess radiographers' performance during clinical practice it may be permissible for them to have access to reports of previous examinations and to discuss individual cases with colleagues.

Assessments that involve clinical judgement are also susceptible to bias owing to prior expectation [38]. The arbiter should therefore be blind to who is responsible for the reports because preconceived ideas can affect their decision as to whether two reports are concordant. The following terms have been coined specifically for plain film reading performance studies.

Observer review bias
This occurs if the radiographers are aware of the reference standard report when interpreting films and can be avoided by blinding the radiographers to this report. If the reference standard used is clinical follow-up, as long as the study is prospective the results of the definitive diagnosis must be unknown at the time of interpretation by the radiographers. This source of bias can lead to falsely elevated indices of performance.

Reference standard review bias
This is the opposite of observer review bias. It occurs if radiographers' reports are known when the films are interpreted by the reference standard [24]. If the reference standard was a consultant radiologist, they must be blind to the radiographers' reports. This source of bias could falsely elevate or even deflate radiographers' indices of performance, depending on how this knowledge affects the reference standard report.

Observer bias
This bias is present if individual radiographers in a study do not report films independent of each other. They should therefore be blind to other reports. If the study is performed during clinical practice and it is normal for radiographers to communicate with colleagues, then this bias is not applicable.

Observer comparator bias
If an explicit attempt is made to compare the performance of individual radiographers, each should report on the same films or the films should be randomly allocated so the radiographers report on a comparable sample. Differences between radiographers can then be attributed to differences in individual performances rather than the case mix of films.

Co-image bias
This occurs if additional images are available to a cohort of observers other than the images they are being asked to interpret. If radiographers' plain film reading performance is being assessed, they should not have access to images from other modalities that could assist their interpretation of the plain films. However, the availability of previous plain films would be permissible if the aim was to simulate clinical practice, particularly as there is evidence that this improves accuracy [39].

Arbiter review bias
The severity of this bias depends on whether (a) the arbiter is also an observer under evaluation or (b) the arbiter was the reference standard, with the former having a greater potential affect on study results. If radiographers' performance was being evaluated, the same radiographers should not be involved in the process of deciding whether their reports concord with the reference standard. They may be too critical about their own reports or alternatively they may not be critical enough. Neither should the reference standard be the arbiter, as they are responsible for one of the reports. This could bias their decision as to whether pairs of reports agree.

Arbiter bias
This occurs if the arbiter was aware of whether the report was made by a radiographer or the reference standard. The arbiter does not need to know which report is the reference standard or the radiographer's and therefore should be blind to which report was made by whom.

Film access bias
This bias is present if the arbiter has access to the sample of films whilst judging whether the reports being compared are concordant. The arbiter's interpretation of the films can incorrectly influence the decision as to whether the reports agree. Furthermore, the arbiter's judgement could be affected by an incorrect report when viewing the films.

Clinical review bias
Several authors have demonstrated improved performance when clinical data are available [40–42]. Others have found clinical details to be unhelpful in lesion detection [43, 44]. In routine clinical practice, knowledge of patients' age, sex and symptoms is required to ensure the most appropriate procedure is carried out and to avoid time and effort searching for findings that would be irrelevant in the clinical context. These requirements heavily outweigh any potential advantage of eliminating bias by withholding relevant clinical data [34]. However, if an unblinded study is undertaken, it is prudent to account for the possible influence of other factors and covariates [14].

Cohort comparator bias
This bias is present when a study assesses the performance of two groups who do not interpret films independently. If radiographers' A&E film reading performance was compared with casualty officers, both groups should be blind to each other's reports. Furthermore, the radiographers and casualty officers should report on the same or a comparable batch of films so that any difference in performance is attributed to differences between cohorts rather than differences in the case mix of films.

Co-image comparator bias
This bias would occur if the plain film reading performance of radiographers is compared with radiologists and the latter have access to images from other modalities such as CT.

Arbiter comparator bias
This bias is present if two or more groups are compared and the arbiter is aware of which group made the reports. If the arbiter has a preconceived conception that radiologists should perform better than radiographers, it may systematically influence their judgement and subsequently distort the indices of performance.

	Further biases in film reading studies

Top Abstract Introduction Biases due to patient... Biases due to observer... Biases associated with the... Biases due to measurement... Independence of interpretation... Further biases in film... Conclusion References

 
This paper focuses on studies of plain film reading performance. For more complex procedures, there are additional potential biases that require consideration.

If an assessment of interpreting, for example, CT head examinations was performed, then additional biases, possibly in the selection of images, may be pertinent. A neurological centre can refer rare or problem cases with a different prevalence and type of disease to that of a district hospital (centripetal bias). Experts may preferentially include and keep track of challenging or interesting cases (popularity bias) and differences in financial and geographical access to CT may affect the study group (diagnostic access bias) [45].

When interpreting barium enema examinations, observers' performance is influenced largely by what is seen fluoroscopically and reports would be biased by the images presented for reporting. In a study where the influence of two factors are being controlled, i.e. who performs the examination and who interprets the images, a factorial design would be appropriate. Random allocation, stratified perhaps by certain clinical details, would ensure that a comparable case mix of patients was included in each arm of the trial.

	Conclusion

Top Abstract Introduction Biases due to patient... Biases due to observer... Biases associated with the... Biases due to measurement... Independence of interpretation... Further biases in film... Conclusion References

 
We have identified numerous potential sources of bias to address when designing plain film reading performance studies. The biases may arise in the selection of films and observers, the application of the standard and the measurement and interpretation of films and reports. We have also indicated examples of further biases that may occur when evaluating observers' interpretation of images from more complex procedures.

It is also important to note that even if all biases are eliminated, inadequate attention to other methodological factors will limit the value of a study. If too small a sample of films is selected, this will produce imprecise estimates of indices of performance. Therefore, a RCT designed to compare radiographers' and radiologists' plain film reading performance should calculate an appropriate sample size to detect as statistically significant a real difference of a given magnitude between the two groups [27]. Confidence intervals should also be constructed to illustrate the range of values that one can beconfident includes the true film reading performanceof radiographers and radiologists [27]. A detailed analysis of the specific methodological factors that influence the quality of a plain film reading performance study will be the subject of a further paper.

Observer variation is substantial and image interpretation, whether of plain films or not, is considered the weakest area of clinical imaging [34]. It is therefore imperative that these biases are understood and consideration is given to avoiding or minimizing them when designing studies to assess different observers' competence to interpret plain films or any other image. It is also important to be aware of these biases when systematically appraising such studies, as their presence will compromise the quality of research. Improving awareness of these biases should underpin the validity of the evidence base used to guide policies, to influence good practice or to direct research.

	Acknowledgments

 
The authors are grateful for the advice of Steven Kelly at the University of Leeds and of the two referees.

Received for publication August 2, 2000. Revision received November 29, 2000. Accepted for publication January 24, 2001.

	References

Top Abstract Introduction Biases due to patient... Biases due to observer... Biases associated with the... Biases due to measurement... Independence of interpretation... Further biases in film... Conclusion References

 

1. The College of Radiographers. Role development inradiography. London, UK: College of Radiographers, 1996.
2. Price RC, Le Masurier SB, High L, Miller LR. Changing times: a national survey of extended roles in diagnostic radiography. Br J Radiol 1999;72(Suppl.):7.
3. Paterson AM. Role development—towards 2000: a survey of role developments in radiography. London, UK: College of Radiographers, 1995.
4. Meek S, Kendall J, Porter J, Freij R. Can accident and emergency nurse practitioners interpret radiographs? A multicentre study. J Accid Emerg Med 1998;15:105–7.[Abstract/Free Full Text]
5. Remedios D, Ridley N, Taylor S, de Lacey G. Trauma radiology: extending the red dot system. Br J Radiol 1998;71(Suppl.):60.
6. Pauli R, Hammond S, Cooke J, Ansell J. Radiographers as film observers in screening mammography: an assessment of competence under test and screening conditions. Br J Radiol 1996;69:10–4.[Abstract/Free Full Text]
7. Haiart DC, Henderson J. A comparison of interpretation of screening mammograms by a radiographer, a doctor and a radiologist: results and implications. Br J Clin Pract 1991;45:43–5.[Medline]
8. Bates JA, Conlon RM, Irving HC. An audit of the role of the sonographer in non-obstetric ultrasound. Clin Radiol 1994;49:617–20.[Medline]
9. Craven CM, Blanshard KS. Computed tomography (CT) head scans reported by an experienced CT radiographer. Radiography 1997;2:105–11.
10. NHS Centre for Reviews and Dissemination. Preface. In: Undertaking systematic reviews of research on effectiveness, CRD Report 4. York, UK: The University of York, 1996:i–ii.
11. Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. A comparison of results of meta-analyses of randomised controlled trials and recommendations of clinical experts. Treatments for myocardial infarction. JAMA 1992;268:240–8.[Abstract/Free Full Text]
12. Oxman AD, Guyatt GH. The science of reviewing research. Ann N Y Acad Sci 1993;703:125–33.[Medline]
13. Russell I, Di Blasi Z, Lambert M, Russell D. Systematic reviews and meta-analyses: opportunities and threats. In: Templeton AA, O'Brien PMS, editors. Evidence-based fertility treatment. London, UK: RCOG Press, 1998:15–64.
14. Kelly S, Berry E, Roderick P, Harris KM, Cullingworth J, Gathercole L, et al. The identification of bias in studies of the diagnostic performance of imaging modalities. Br J Radiol 1997;70:1028–35.[Abstract]
15. Fineberg HV, Bauman R, Sosman M. Computerized cranial tomography: effect on diagnostic and therapeutic plans. JAMA 1977;238:224–7.[Abstract/Free Full Text]
16. Institute of Medicine. Policy statement: Computed tomographic scanning. Washington DC: National Academy of Sciences, 1977.
17. Mackenzie R, Dixon AK. Measuring the effects of imaging: an evaluative framework. Clin Radiol 1995;50:513–8.[Medline]
18. Brealey S, Glenny AM. A framework for radiographers planning to undertake a systematic review. Radiography 1999;5:131–46.
19. Feinstein AR. Clinical biostatistics XXXI: on thesensitivity, specificity and discrimination of diagnostic tests. Clin Pharmacol Ther 1975;17:104–16.[Medline]
20. Boynes S, Scally AJ, Webster AJ, Kay K. Radiographic reporting of the axial and appendicular skeleton by radiographers and nurse practitioners. Br J Radiol 1977;70(Suppl.):123.
21. van der Schouw YT, Verbeek AL, Ruijs SH. Guidelines for the assessment of new diagnostic tests. Invest Radiol 1995;30:334–40.[Medline]
22. Ransohoff DF, Feinstein AR. Problems of spectrum bias in evaluating the efficacy of diagnostic tests. N Engl J Med 1978;299:926–30.[Abstract]
23. Carrington RM, Lachs MS, Feinstein AR. Use of methodological standards in diagnostic test research: getting better but still not good. JAMA 1995;274:645–51.[Abstract/Free Full Text]
24. Loughran CF. Reporting of accident and emergency radiographs by radiographers: a study to determine the effectiveness of a training programme. Br J Radiol 1994;67(Suppl.):93.
25. Robinson PJ. Plain film reporting by radiographers—a feasibility study. Br J Radiol 1996;69:1171–4.[Abstract/Free Full Text]
26. Begg CB. Biases in the assessment of diagnostic tests. Stat Med 1987;6:411–23.[Medline]
27. Altman DG. Practical statistics for medical research. London, UK: Chapman & Hall, 1991.
28. Renwick IGH, Butt WP, Steele B. How well can radiographers triage X-ray films in the accident and emergency department? BMJ 1991;302:568–9.
29. Berman L, de Lacey G, Twomey E, Twomey B, Welch T, Eban R. Reducing errors in accident department; a simple method using radiographers. BMJ 1985;290:421–2.
30. Choi BCK. Sensitivity and specificity of a single diagnostic test in the presence of work-up bias. J Clin Epidemiol 1992;45:581–6.[Medline]
31. Begg CB, Greenes RA. Assessment of diagnostic tests when disease verification is subject to selection bias. Biometrics 1983;39:207–15.[Medline]
32. Greenes RA, Begg CB. Assessment of diagnostic technologies: methodology of unbiased estimation from samples of selectively verified patients. Invest Radiol 1985;20:751–6.[Medline]
33. Donabedian A. Evaluating the quality of medical care. Millbank Memorial Fund Quarterly 1966:166–206.
34. Robinson PJA. Radiology's Achilles' heel: error and variation in the interpretation of the Röntgen image. Br J Radiol 1997;70:1085–98.[Abstract]
35. Robinson PJA, Wilson D, Coral A, Murphy A, Verow P. Variation between experienced observers in the interpretation of accident and emergency radiographs. Br J Radiol 1999;72:323–30.[Abstract]
36. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960;20:37–46.
37. Aideyan UO, Berbaum K, Smith WL. Influence of prior radiological information on the interpretation of radiographic examinations. Acad Radiol 1995;2:205–8.[Medline]
38. Sackett DL, Haynes RB, Guyatt GH, Tugwell P. Clinical epidemiology: a basic science for clinical medicine (2nd edn). London, UK: Little, Brown and Company, 1991:45.
39. White K, Berbaum K, Smith WL. The role of previous radiographs and reports in the interpretation of current radiographs. Invest Radiol 1994;29:263–5.[Medline]
40. Doubilet P, Herman PG. Interpretation of radiographs: effects of clinical history. AJR 1981;137:1055–8.[Abstract/Free Full Text]
41. Schreiber MH. The clinical history as a factor in roentgenogram interpretation. JAMA 1963;185:399–401.
42. Rickett AB, Finaly DB, Jagger C. The importance of clinical details when reporting accident and emergency radiographs. Injury 1992;23:458–60.[Medline]
43. Good BC, Cooperstein LA, DeMarino GB, Miketic LM, Gennari RC, Rockette HE, et al. Does knowledge of the clinical history affect the accuracy of chest radiograph interpretation? AJR 1990;154:709–12.[Abstract/Free Full Text]
44. Eldevik OP, Dugstad G, Orrison WW, Haughton VM. The effect of clinical bias on the interpretation of myelography and spinal computed tomography. Radiology 1982;145:85–9.[Abstract/Free Full Text]
45. McMaster University Health Sciences Centre. How to read clinical journals: III: to learn the clinical course and prognosis of disease. Can Med Assoc J 1981;124:869–72.[Medline]

This article has been cited by other articles:

				S BREALEY, C HEWITT, A SCALLY, S HAHN, C GODFREY, and N THOMAS Bivariate meta-analysis of sensitivity and specificity of radiographers' plain radiograph reporting in clinical practice Br. J. Radiol., July 1, 2009; 82(979): 600 - 604. [Abstract] [Full Text] [PDF]

				S Brealey and M Westwood Are you reading what we are reading? The effect of who interprets medical images on estimates of diagnostic test accuracy in systematic reviews Br. J. Radiol., August 1, 2007; 80(956): 674 - 677. [Abstract] [Full Text] [PDF]

				S D Brealey, A J Scally, S Hahn, and C Godfrey Evidence of reference standard related bias in studies of plain radiograph reading performance: a meta-regression Br. J. Radiol., June 1, 2007; 80(954): 406 - 413. [Abstract] [Full Text] [PDF]

				G. T. Sica Bias in Research Studies Radiology, March 1, 2006; 238(3): 780 - 789. [Abstract] [Full Text] [PDF]

				Y Balabanova, R Coker, I Fedorin, S Zakharova, S Plavinskij, N Krukov, R Atun, and F Drobniewski Variability in interpretation of chest radiographs among Russian clinicians and implications for screening programmes: observational study BMJ, August 13, 2005; 331(7513): 379 - 382. [Abstract] [Full Text] [PDF]

				S Brealey, D G King, S Hahn, C Godfrey, M T I Crowe, K Bloor, S Crane, and D Longsworth The costs and effects of introducing selectively trained radiographers to an A&E reporting service: a retrospective controlled before and after study Br. J. Radiol., June 1, 2005; 78(930): 499 - 505. [Abstract] [Full Text] [PDF]

				P. M. Bossuyt, J. B. Reitsma, D. E. Bruns, C. A. Gatsonis, P. P. Glasziou, L. M. Irwig, D. Moher, D. Rennie, H. C.W. de Vet, and J. G. Lijmer The STARD Statement for Reporting Studies of Diagnostic Accuracy: Explanation and Elaboration Ann Intern Med, January 7, 2003; 138(1): W1 - W12. [Abstract] [Full Text] [PDF]

				P. M. Bossuyt, J. B. Reitsma, D. E. Bruns, C. A. Gatsonis, P. P. Glasziou, L. M. Irwig, D. Moher, D. Rennie, H. C.W. de Vet, and J. G. Lijmer The STARD Statement for Reporting Studies of Diagnostic Accuracy: Explanation and Elaboration Clin. Chem., January 1, 2003; 49(1): 7 - 18. [Abstract] [Full Text] [PDF]

				S Brealey, D G King, M T I Crowe, I Crawshaw, L Ford, N G Warnock, R A J Mannion, and S Ethell Accident and Emergency and General Practitioner plain radiograph reporting by radiographers and radiologists: a quasi-randomized controlled trial Br. J. Radiol., January 1, 2003; 76(901): 57 - 61. [Abstract] [Full Text] [PDF]

				S Brealey, A J Scally, and N B Thomas Methodological standards in radiographer plain film reading performance studies Br. J. Radiol., February 1, 2002; 75(890): 107 - 113. [Abstract] [Full Text] [PDF]

This Article Abstract Full Text (PDF) Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Brealey, S Articles by Scally, A J Search for Related Content PubMed PubMed Citation Articles by Brealey, S Articles by Scally, A J