British Journal of Radiology 74 (2001),621-628 © 2001 The British Institute of Radiology
ROC curve analysis of lesion detectability on phantoms: comparison of digital spot mammography with conventional spot mammography
W M Yip, BSc,
S Y Pang, BSc,
W S Yim, BSc and
C S Kwok, PhD
Department of Optometry and Radiography, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong
Correspondence: Dr Cheuk Sang Kwok
 |
Abstract
|
|---|
Although conventional screen–film mammography has excellent spatial resolution and is commonly used as a screening tool, certain inherent limitations prevent its further improvement. New digital mammography techniques, despite lower spatial resolution than screen–film mammography, may overcome these limitations. This study compared lesion detectability between charge coupled device-based digital spot mammography and conventional spot mammography. A total of 100 sets of images of specially designed breast phantoms was acquired, with variable background achieved by overlapping several layers of grapefruit fibre on a 4 cm thick lucite slab, using both modalities. 75 sets were "normal" images and 25 sets were images with simulated lesions. Four radiologists assessed the images according to a five-point confidence scale. The results were used to construct receiver operating characteristic curves. No statistical difference was observed between the two sets of curves for individual radiologists as well as pooled data. The lower spatial resolution of digital mammography was compensated for by its higher contrast sensitivity relative to conventional spot mammography.
|
Introduction
|
|---|
Despite the technically and clinically demanding nature of breast imaging, mammography still remains one of the few essentially film-based radiological imaging techniques in a modern medical imaging department [1]. However, some inherent limitations do exist in conventional mammography and restrict further technical improvement. For example, a compromised situation occurs when the film must act as an image acquisition detector and also a storage and display device [2].
Digital mammography is considered to be the technique that has "the greatest potential impact on the management of breast cancer" [3]. In digital mammography, image acquisition, storage and display are performed independently, allowing optimization of each stage. Unlike conventional imaging, which requires low X-ray energy to ensure high contrast of the film, digital detectors permit wider variation in the X-ray energy. Once the image is stored, it can be displayed with contrast independent of the detector properties.
The spatial resolution of conventional mammography is up to 20 line pairs per millimetre (lp mm-1). To achieve this, digital mammography must have pixels spaced no further apart than 0.025 mm which is currently achievable only in small field digital detectors. Spatial resolution is one of the strengths of conventional mammography. However, there is some evidence that digital mammography with a resolution less than 10 lp mm-1 may have better lesion detectability than screen–film mammography owing to improved contrast detail [2]. Therefore, research comparing screen–film mammography and digital mammography with high spatial resolution is much needed. Previous research studies have used digitized screen–film mammograms for comparison, but the quality of the digital image remained limited by the quality of the original information on the film. During the digitization process there was also a possibility of information loss [1, 4]. These studies reported no significant difference in lesion detection between the digitized film and analogue images [4–6].
Some studies compared storage phosphors with conventional screen–film mammography. It was found that computed radiography approaches the performance of screen–film mammography for survey views, but may be superior to screen–film mammography in the detection of microcalcifications in a spot view [7–9]. However, studies comparing charge coupled device (CCD) imaging detectors for digital mammography and conventional mammography are limited. One such study found that the contrast detail detectability of the digital system was significantly superior to the conventional system [10]. The phantom used by Liu et al had constant uniform background, in contrast to the variable background of clinical mammograms. The present study was performed to compare the lesion detectability between CCD-based digital spot mammography and conventional spot mammography under a variable background. The CCD detector used is one of the highest resolution detectors currently available in the digital mammography industry.
|
Materials and method
|
|---|
Breast phantoms
100 breast phantoms were made by superimposing three to five layers of grapefruit fibre, each about 0.6 mm thick, on a slab of 4 cm thick lucite in such a way that the outline of a compressed breast was simulated and the veins of the fibre converged on a point on the midline of the slab. This point simulated the nipple of a patient. The lucite was used to simulate the 4.4 cm mean thickness of compressed breast tissues in terms of attenuation [11]. To the best of our knowledge, a breast phantom of this design has not been previously reported in the literature. Validation of the phantoms was achieved by comparing images of the phantoms with mammograms.
Simulated lesions
Fragments of eggshell [8] and chalk powder [12] were used to simulate high and moderate contrast microcalcifications, respectively. The size of each microcalcification [13] was measured by means of an optical microscope and a calibrated graticule. The simulated microcalcifications ranged from 0.5 mm to 0.14 mm in diameter and were grouped 3, 4, 5 or 10 grains per cluster within a 1 cm2 area. Circular pieces of X-ray film (thickness 0.19 mm) and aluminum foil (thickness 0.02 mm) with diameters of 6 mm were used to simulate low contrast masses. Two such lesions were comprised only of a piece of aluminum foil, while one lesion consisted of a piece of aluminum foil and a piece of X-ray film. Each of these simulated lesions could be randomly positioned on top of a breast phantom to yield abnormal phantom images.
Phantom images
In total, 100 cases for each mode of imaging were acquired using a Siemens Mammomat 3000 (Siemens, Erlangen, Germany) mammography unit equipped with the Opdima digital detection system. 75 cases were "normal" with no lesions and 25 cases were "abnormal" with simulated clusters of microcalcifications or masses. This ratio aimed at bringing our study close to the clinical situation [7]. Among the 25 sets of abnormal phantom images, three sets had simulated low contrast masses, three sets had clusters of microcalcifications simulated by chalk powder and 19 sets had clusters of microcalcifications simulated by tiny bits of eggshell. More high contrast eggshell lesions than moderate and low contrast lesions were imaged in this study because in a pilot study there was no significant difference in the detectability of eggshell lesions using digital and conventional techniques.
A normal screen–film spot mammographic image was acquired by placing a breast phantom on a magnification table of the Siemens mammographic unit. The exposure was taken at 28 kVp. A Kodak Min-R 2 cassette (Kodak, Rochester, NY) mounted with a single Min-R 2000 screen, loaded with a Kodak Min-R Diagnostic Film 2000, was placed inside the cassette holder. The image was coned by a magnification spot collimator and exposed using a fine focal spot and an automatic exposure detector device. A digital image of the same phantom was acquired by positioning the phantom on the cassette holder with a Siemens Opdima CCD-based detector plate inside. The CCD was 49 mm x 85 mm. The same exposure of 28 kVp was used and was terminated using the same automatic exposure detector device. The standard resolution mode was selected for digital images using a large focal spot. The matrix size was 1024 x 1792 with a corresponding pixel size of 48 µm x 48 µm. These procedures were repeated on 75 different normal breast phantoms and 25 sets of breast phantoms with simulated lesions as explained above. Irradiated films were developed in a Kodak X-Omat 2000 processor, maintained at 33 °C. The digital images were stored and viewed on a Sun Ultra computer with a 1 K x 1 K colour monitor. Certain image processing programs, such as contrast enhancement and image filtering, were also supplied by the Opdima system.
Interpretation
The analogue and digital images were read by four radiologists, each with at least 4 years of clinical experience in mammography. To avoid reading order effect, the digital and conventional images were interpreted on different days. The sequences of image presentation were also different for each set of images [5]. Each observer was allowed a brief introduction and practice time on the usage of the image processing program before they interpreted the digital images [12, 14]. In viewing the analogue and digital mammographic images, the observers were free to adjust their optimal viewing conditions. They had to score the images according to a five-point confidence scale: 1, abnormality definitely not present; 2, abnormality probably not present; 3, indeterminate; 4, abnormality probably present; 5, abnormality definitely present [7]. If an abnormality on a digital image was suspected, the observer also had to record the co-ordinates of the lesion on the scoring sheet.
Data analysis
The interpretation data were collected to generate receiver operating characteristic (ROC) curves. The gold standard in our study was our record of phantom settings. ROC analysis is extensively used in radiology to investigate the effect of individual parameters on an imaging system or to compare the performance of one system with another [13, 15]. It is a plot of all the sensitivity and specificity pairs from continuously varying the decision threshold [16].
The total number of scores for each confidence level assigned by each observer was counted and input into a ROCKIT program supplied by Dr Charles E Metz of the University of Chicago. The corresponding ROC curve was plotted using his PlotROC software. A pooled ROC curve for each imaging modality was plotted by summing up the performance scores assigned by all the observers for the same modality.
The bivariate 
2 goodness-of-fit test was used to determine any statistical difference between the pair of ROC curves of individual observers [15]. The number of correct interpretations made by each observer for each class of simulated lesions was also recorded for further analysis. Kendall's coefficient of concordance was used to test agreement between individual observers.
|
Results
|
|---|
Representative images of the breast phantoms acquired by the Opdima digital system are given in Figure 1
. The top panel is an image of a breast phantom without any simulated lesion. The middle panel corresponds to the image of a phantom with a simulated mass lesion. The bottom panel shows the image of a phantom with a simulated cluster of microcalcifications. According to the four radiologists recruited for this study, the phantom images closely resemble mammograms.
View larger version (122K):
[in this window]
[in a new window]
|
Figure 1. Representative digital images of breast phantoms. Top, without any simulated lesion; middle, with a simulated mass lesion; bottom, with a simulated cluster of microcalcifications.
|
|
Summaries of scores assigned by the four radiologists to the normal and abnormal images acquired by the conventional or digital techniques are given in Table 1 and Table 2, respectively. The correlation between individual radiologists' scores, according to Kendall's coefficient of concordance, W, is given in Table 3. The radiologistsshowed agreement in scoring all the various types of images except the conventional phantom images with chalk powder. However, this constitutes a borderline case, having a probability of equal scoring slightly smaller than 0.05.
In considering individual observers, Dr A (Figure 2), Dr C (Figure 3) and Dr D (Figure 4) appeared to perform better when assessing the digital phantom images than when assessing the conventional images. However, this was not true for Dr B, who appeared to have a higher diagnostic accuracy when assessing the conventional phantom images (Figure 5). The 2 test did not support any significant difference between the two sets of images for any of the observers. Individual observers' interpretations were therefore pooled together. The ROC curves of the pooled data are shown in Figure 6.
Table 4 gives the properties of the ROC curves of individual observers coded as A, B, C or D as well as the ROC curves for the pooled data. a is the vertical intercept, b is the slope of the ROC curve and Az is the area index under the curve. The area under the pooled ROC curve for the conventional images was 0.74±0.03 SE and the area under the pooled digital ROC curve was 0.78±0.03 SE. The difference of 0.04 in area was not statistically significant (the bivariate 2 test=1.094; p=0.5787). Table 4 also gives the properties of the ROC curves generated by taking the mean values of a and b of the ROC curves of the four individuals.
View this table:
[in this window]
[in a new window]
|
Table 4. Properties of receiver operating characteristic (ROC) curves of individual observers, pooled data and averaged ROC data. Mean±SE values are presented
|
|
Table 5 shows the detection accuracy of conventional spot mammography and digital spot mammography for the three different groups of simulated lesions. The digital technique could detect 25±8% of chalk powder simulated clusters of microcalcifications and 67±8% of simulated masses, while the conventional technique could detect only 8±8% and 17±8% of simulated lesions, respectively. The errors quoted are the minimum uncertainty representing the percentage of one out of the total number of interpretations for each group of lesions. However, the conventional technique appears to be marginally superior to the digital technique in the detection of eggshell simulated clusters of microcalcifications. Their detection ratios are 88±1.3% and 83±1.3%, respectively. The differences between the two imaging techniques for each of the three groups of simulated lesions, according to the unpaired t-test, are statistically significant at p<0.05 level.
View this table:
[in this window]
[in a new window]
|
Table 5. Detection accuracies of conventional spot mammography and digital spot mammography for three different groups of simulated lesions. Errors quoted are the minimum uncertainty
|
|
|
Discussion
|
|---|
The presence of the background structure of the breast itself distinguishes mammograms from most phantom images. The latter usually only involve assessment of the detectability of small objects against a uniform background. Radiologists have to search mammograms for specific anatomical abnormalities against a complex background. Mammograms contain noise from film granularity and quantum fluctuations in addition to structural variation [17, 18]. For this reason, we constructed breast phantoms with variable backgrounds. Such phantoms yielded radiographic images similar to those of patients. 100 breast phantoms were made to provide each set of phantom images with a unique background. This prevented the observers from becoming accustomed to particular features or orientations of the phantoms [19].
ROC analysis has been widely used in medical imaging studies over the past 2 decades. A ROC test is considered optimal with regard to sensitivity ifthe area under a ROC curve is between 0.75 and 0.80 [19]. In the present study, the areas under the ROC curves were within 0.67–0.89. Therefore, our ROC curves are close to the optimal.
Based on three of the four sets of the plotted ROC curves, digital spot mammography appeared to be superior to conventional spot mammography in lesion detection but the difference was not statistically significant. Even though the spatial resolution of 11 lp mm-1 of the spot digital mammography system used in this study [20] was lower than the 16 lp mm-1 resolution measured for our conventional spot mammography system, the superior contrast resolution of digital mammography could compensate for this loss. Research work conducted by Liu et al [10] also confirmed that CCD detector-based digital mammography had significantly superior contrast detail detectability than conventional mammography. In our work, the digital technique resulted in significantly higher detection accuracies than the conventional technique for low to medium contrast lesions (Table 5
). However, the detection accuracy of the conventional spot mammography for small, high contrast lesions, represented by eggshell bits in this study, may be better than that of digital mammography (Table 5). This could be because the Min-R 2000 screen–film system has a better modulation transfer function than that of the standard resolution mode of the CCD-based detector (Figure 7) [20] at all frequencies. A larger scale study must be carried out to confirm that our findings are not a consequence of the numbers of films in the study featuring the respective details. The Opdima system also allows acquisition of spot digital images in a high resolution mode with pixel size half that of the normal mode. A spatial resolution of at least 16 lp mm-1 can be achieved by the high resolution mode with a concomitant increase in the mean glandular dose of only about 20%. It will be of interest to compare the high resolution digital spot imaging technique with the conventional spot magnification technique in lesion detectability in a separate study. Spatial resolution is undoubtedly the crucial issue in the further development of digital mammography [21,22].
Kendall's coefficient of concordance suggested that correlation between individual radiologists' scoring was high and this agrees with our ROC statistical analyses. After scoring 100 sets of phantom images, the radiologists involved in the present study agreed that the digital spot technique was better than the conventional spot technique in image manipulation. They found it easier to locate lesions on the digital phantom images, which had higher contrast resolution. Extensive usage of brightness and contrast control, as well as enlargement of regions of interest, in the digital images was reported. Brightness and contrast control were especially useful for detecting low and medium contrast lesions. Small lesions were easier to visualize after magnification. The system also provided several image filters including sharpen, smooth, noise and edge enhancement but the radiologists rarely applied them.
When two or more observers read images under identical conditions, two approaches are available to generate a single ROC curve to describe the overall performance. One is the pooled data approach. Another approach is to average the a and b values across observers to produce an averaged ROC curve [23]. According to Metz, pooled data suffer from downward bias while the averaged ROC does not [23]. This bias underestimates the sensitivity and specificity of an imaging modality. The Az values of the pooled conventional spot mammography and digital spot mammography data were 0.74±0.03 SE and 0.78±0.03 SE, respectively (Figure 6 and Table 4). The Az values of the averaged ROC curves for the two modalities were 0.80±0.06 SE and 0.80±0.07 SE, respectively (Figure 8). It is obvious that the areas under the curves of the pooled data were smaller than those under the averaged curves.
This study has several limitations. Firstly, reading order effect is a bias that favours the image read last [23]. In our study, since each radiologist interpreted the conventional phantom images first, the study was potentially biased in favour of the digital phantom images. To reduce this effect, the two sets of images were assessed in different orders and on different days. Secondly, the number of observers was only four. When the number of observers cannot be increased, Nab et al [5] suggested that the greater the number of phantom images used, the higher the accuracy of the study will be. Thirdly, craniocaudal and mediolateral views are taken at the same time for comparison in a clinical situation. The design of our phantoms only permitted one view to be taken for each phantom. This presented some difficulty to the radiologists when identifying the space-occupying lesions in the phantom images.
Direct small field-of-view digital mammography has been commercially available for guided breast biopsy and for spot compression mammography for several years. Contrast detail phantom studies [10] and the current study demonstrate that CCD-based digital spot mammography is capable of at least a comparable detectability of lesions as the conventional spot mammography. This finding may be extended to currently available full field digital mammography devices if their spatial resolution is at least 11 lp mm-1. Further efficacy evaluations of conventional and digital (either spot or full field) mammography in patients are warranted.
|
Conclusion
|
|---|
Digital spot mammography with a spatial resolution of about 11 lp mm-1 was slightly superior in overall lesion detection to conventional spot mammography with a spatial resolution of about 16 lp mm-1, although the difference was not statistically significant. The enhanced contrast sensitivity of digital mammography could compensate for its lower spatial resolution. Therefore, to achieve diagnostic accuracy equal to that of film, it is not necessary to match the film in all respects.
Boyle et al [24] stated that, over the next decade, direct acquisition digital mammography islikely to supplant conventional screen–film mammography in many medical centres. The potential advantages of digital mammography favour further developments, while studies to date have also shown encouraging results.
|
Acknowledgments
|
|---|
We thank Dr John Lo from the United Christian Hospital, Dr L F Chiu from the Queen Elizabeth Hospital, Dr K W Cheung from the Tuen Mun Hospital and Dr W G Yang from the Prince of Wales Hospital for acting as interpreters of the phantom images. We also thank Mr C Wong, Mr W K Yan and Mr KY Yuen for their technical assistance. This work was supported by a large equipment grant from the Research Grants Council of the Hong Kong Polytechnic University (Project No. 9025) and research funding of the Department of Optometry and Radiography, PolyU.
Received for publication August 18, 2000.
Revision received December 12, 2000.
Accepted for publication April 4, 2001.
|
References
|
|---|
-
Cowen AR, Parkin GJS, Hawkridge P. Direct digital mammography image acquisition. Eur Radiol 1997;7:918–30.[Medline]
-
Feig SA, Yaffe M. Current status of digital mammography. Semin Ultrasound CT MR 1996;17:424–43.[Medline]
-
Cheung L, Bird R, Chitkara A, Rego A, Rodriguez C, Yuen J. Initial operating and clinical results of a full field mammography system. In: Karssemeiger N, Thijssen M, Hendriks J, Erning LV, editors. Digital mammography. London: Kluwer Academic Publishers, 1998:11–26.
-
Adler DD, Wahl RL. New methods for imaging the breast: techniques, findings, and potential. AJR 1995;164:19–30.[Abstract/Free Full Text]
-
Nab HW, Karssemeijer N, Van Erning LJ, Hendriks JH. Comparison of digital and conventional mammography: a ROC study of 270 mammograms. Med Inform (London) 1992;17:125–31.
-
Karssemeijer N, Frieling JT, Hendriks JH. Spatial resolution in digital mammography. Invest Radiol 1993;28:413–9.[Medline]
-
Brettle D, Ward SC, Parkin GJ, Cowen AR, Sumsion HJ. A clinical comparison between conventional and digital mammography utilizing computed radiography. Br J Radiol 1994;67:464–8.[Abstract/Free Full Text]
-
Funke M, Breiter N, Hermann KP, Oestmann JW, Grabbe E. Storage phosphor direct magnification mammography in comparison with conventional screen–film mammography—a phantom study. Br J Radiol 1998;71:528–34.[Abstract]
-
Gaspard-Bakhach S, Dilhuydy MH, Bonichon F, Barreau B, Henriques C, Maugey-Laulom B. ROC analysis comparing screen film mammography and digital mammography. J Radiol 2000;81:133–9.[Medline]
-
Liu H, Fajardo LL, Barrett JR, Baxter RA. Contrast-detail detectability analysis: comparison of a digital spot mammography system and an analog screen–film mammography system. Acad Radiol 1997;4:197–203.[Medline]
-
Helvie MA, Chan HP, Adler DD, Boyd PG. Breast thickness in routine mammograms: effect on image quality and radiation dose. AJR 1994;163:1371–4.[Abstract/Free Full Text]
-
Lado MJ, Tahoces PG, Souto M, Mendez AJ, Vidal JJ. Real and simulated clustered microcalcifications in digital mammograms. ROC study of observer performance. Med Phys 1997;24:1385–94.[Medline]
-
Kallergi M, Carney GM, Gaviria J. Evaluating the performance of detection algorithms in digital mammography. Med Phys 1999;19:1015–23.
-
Chan HP, Vyborny CJ, MacMahon H, Metz CE, Doi K, Sickle EA. Digital mammography ROC studies of the effects of pixel size and unsharp-mask filtering on the detection of subtle microcalcifications. Invest Radiol 1987;22:581–9.[Medline]
-
Goddard CC, Gilbert FJ, Needham G, Deans HE. Routine receiver operating characteristic analysis in mammography as a measure of radiologists' performance. Br J Radiol 1998;71:1012–7.[Abstract]
-
Zweig M, Campbell G. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem 1993;39:561–76.[Abstract/Free Full Text]
-
Caldwell CB, Yaffe MJ. Development of an anthropomorphic breast phantom. Med Phys 1990;17:173–280.
-
Kotre CJ. The effect of background structure on the detection of low contrast objects in mammography. Br J Radiol 1998;71:1162–7.[Abstract]
-
Olsen JB, Skretting A, Widmark A. Assessment of image quality and total performance in Norwegian mammography laboratories. Acta Radiol 1998;39:507–13.[Medline]
-
Thunberg S, Sklebitz H, Ekdahl B, Batz L, Lundin A, Moller H, et al. Opdima: Large-area CCD-based X-ray image sensor for spot imaging and biopsy control in mammography. Proc Soc Photo-Optical Instrum Engin 1999;3659:150–8.
-
Kimme-Smith CK, Haus AG, DeBruhl N, Bassett LW. Effects of ambient light and view box luminance on the detection of calcifications in mammography. AJR 1997;168:775–8.[Abstract/Free Full Text]
-
Kimme-Smith CK, Lewis C, Beifuss M, Williams MB, Bassett LW. Establishing minimum performance standards, calibration intervals, and optimal exposure values for a whole breast digital mammography unit. Med Phys 1998;25:2410–6.[Medline]
-
Metz CE. Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol 1989;24:234–45.[Medline]
-
Boyle ER, Pak D, Williams JB. Motion artifact seen on slot-scanning direct digital mammography. AJR 1999;172:697–701.[Abstract/Free Full Text]
This article has been cited by other articles:
|
|
|
|
 
K. B. Krug, H. Stutzer, R. Schroder, J. Boecker, J. Poggenborg, and K. Lackner
Image Quality of Digital Direct Flat-Panel Mammography Versus an Analog Screen-Film Technique Using a Low-Contrast Phantom
Am. J. Roentgenol.,
September 1, 2008;
191(3):
W80 - W88.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
K. B. Krug, H. Stutzer, R. Girnus, M. Zahringer, A. Gossmann, G. Winnekendonk, and K. Lackner
Image Quality of Digital Direct Flat-Panel Mammography Versus an Analog Screen-Film Technique Using a Phantom Model
Am. J. Roentgenol.,
February 1, 2007;
188(2):
399 - 407.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
H. H. Kim, E. D. Pisano, E. B. Cole, M. R. Jiroutek, K. E. Muller, Y. Zheng, C. M. Kuzmiak, and M. A. Koomen
Comparison of calcification specificity in digital mammography using soft-copy display versus screen-film mammography.
Am. J. Roentgenol.,
July 1, 2006;
187(1):
47 - 50.
[Abstract]
[Full Text]
[PDF]
|
|
|
Top
Abstract
Introduction
