This Article Abstract Full Text (PDF) Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Mello-Thoms, C Search for Related Content PubMed PubMed Citation Articles by Mello-Thoms, C

The problem of image interpretation in mammography: effects of lesion conspicuity on the visual search strategy of radiologists

University of Pittsburgh, Department of Radiology, 300 Halket Street, Suite 4200, Pittsburgh, PA 15228 – USA

Correspondence: Claudia Mello-Thoms, University of Pittsburgh, Department of Radiology, 300 Halket Street, Suite 4200, Pittsburgh, PA 15228, USA. E-mail: cmellothoms{at}magee.edu

	Abstract

Top Abstract Introduction Materials and methods Results Discussion References

 
Radiologists make the decision to report or dismiss a possible cancer based not only on the finding itself, but also in the comparison with selected areas of the background. We examined the effects of fixating, for the first time, the location where the radiologist either reported the presence of a malignant mass or visually inspected the mass but did not report it, and the effects of pairing radiologists to read the same cases. Four experienced mammographers participated in this experiment. They read a set of 20 cases twice. Eye-position tracking was used to monitor the visual search behaviour of the observers. Spatial frequency analysis was used to determine the characteristics of the areas of the background fixated by the observers. Radiologists had more fixations in the cases where they agreed how to manage the lesion than when they disagreed. Correlation between the areas of the background sampled by the radiologists and an "average" representation of the background increased after the observers fixated for the first time a malignant mass that they reported. Fixating, for the first time, a location where the radiologist reports a malignant mass or a location containing a cancer that the radiologist visually inspects but decides not to report, has a significant effect on any further sampling of the background. Furthermore, care should be taken when pairing radiologists, because some observers showed such a similar visual search behaviour that not much would be gained by having them read the same cases.

	Introduction

Top Abstract Introduction Materials and methods Results Discussion References

 
Breast cancer is the second most deadly form of cancer for women in the USA, and it will claim an estimate of more than 40 000 lives in 2006 [1]. Furthermore, the nature of this disease is such that, particularly for younger women, early detection can effectively determine the patient's 5 year survival [2–5]. Due to its effectiveness, widespread availability and low cost, mammography is still the most commonly used screening tool for this disease. In 2004, nearly 40 million mammograms were performed in the USA [6], with cancer being detected about five times for each 1000 cases screened [7]. The combination of a high volume of images and low prevalence of the disease makes the radiologists' task very difficult. Moreover, several characteristics intrinsic to breast cancer detection compound this difficulty; for example, the wide range of appearance of the breast parenchyma [8] and the visual subtlety of the initial signs of cancer [9]. The level of expertise of the radiologist has also been shown to significantly impact their detection rates [10]. Models of medical image perception advocate that the ability to properly detect breast cancer involves not only the acquisition of an internal database with samples of lesion-containing and lesion-free tissue, but also the development of an efficient visual search strategy, to be used to properly sample the informative areas of the parenchyma [11]. Upon image onset, the observer forms a "global impression" of what is been seen, often without the need for eye movements [12]. In this phase several areas of possible perturbation are flagged; these areas either contain suspicious features or do not conform to the observer's a priori expectations of the image. To each of these areas the observer guides the high-resolution beam of the visual system, the fovea, in order to solve the discrepancy in perception. This is done by comparing each perceived finding with selected areas of the background parenchyma, in order to determine the finding's uniqueness. Following this process of foveation of all flagged areas, the observer engages in a "discovery scanning" search [13], which aims to pick up conspicuous features that have not yet been subject to detailed foveal scrutiny. Because of these two processes, it has been shown that about 70% of unreported cancers receive significant amounts of visual dwell [14], thus suggesting that faulty visual search is not the main reason why these lesions were not reported.

In this paper, we determine how the perception of a malignant mass affects the visual sampling strategy of experienced radiologists by comparing the background areas looked at by them before they first fixate the location of a reported finding, and the background areas looked after they first fixate this location. We also compare the visual search strategy of radiologists when they agree on the management of a visually inspected lesion (being the agreement to either report the lesion or to dismiss it) vs when they do not agree on how to manage the inspected cancer. Finally, we compare the visual search behaviour of pairs of radiologists, in order to determine whether random pairing (such as in double-reading scenarios) always yields comparable improvements in cancer detection rates.

	Materials and methods

Top Abstract Introduction Materials and methods Results Discussion References

 
Reader selection
Four Mammography Quality Standards Assurance-qualified, experienced mammographers from the Breast Imaging Division, Department of Radiology, University of Pittsburgh Medical Center, participated in this experiment. The mammographers had between 7 years and 20 years experience reading mammograms. More than 40 000 screening mammograms are performed every year by the combination of our main hospital and its five satellite clinics.

Case selection
We used a case set containing 2 two-view (craniocaudal, CC, and mediolateral oblique, MLO) mammogram cases. 15 cases contained a biopsy-proven malignant mass, which was visible in both views, and 5 cases were lesion-free ("normal") and had been stable for 2 years. The cancer cases contained two sets of images: (1) the mammograms in which the mass was discovered at mammography screening, called the "latest" mammograms; and (2) the first prior mammograms, in which the lesion was retrospectively deemed to be visible: it was not reported when these mammograms were originally read at mammography screening, called the "prior" mammograms. In order to keep the comparison cases as stable as possible, we only used one set of images for the lesion-free cases. Masses were approximated as an ellipsoid, and the mean longest diameter was 1.096 cm (range 0.37–3.09 cm) in the "latest" and 0.808 cm (range 0.222–2.340 cm) in the priors [15].

Image digitization and display
The mammograms were cropped to remove most of the background from the image. They were then digitized using a Lumiscan Model 85 digitizer (Lumisys, Sunnyvale, CA), with a pixel size of 50 µmx50 µm and a grey level resolution of 12 bits. Two ORWIN displays, model DS5100P (Clinton Electronics, Rockford, IL), each with a screen size of 2048x2560 pixels and a nominal luminance setting of 80 foot-Lambert (ftL), were placed side-by-side and showed the two-view mammograms. The displays were calibrated using the DICOM standard [9], and the grey scale range of each mammogram was automatically adjusted using a window/level function. Once these levels were set, the observers were not permitted to change them during the experiment.

Eye-position tracking
We used ASL's model 501 (Applied Science Laboratories, Bedford, MA) eye-tracking system, to monitor the observers' gaze. This system has an infrared beam that it uses to monitor the position of the pupil and the first corneal reflection. It also has a magnetic head tracker, which allows for the monitoring of the observers' head. It has an accuracy of less than 1° of visual angle.

The observers on average sat about 35 cm away from the centre line that divided the displays. The eye tracker was initially calibrated by having the observers look, at the experimenter's request, to each point of a 9-point, 3x3 grid shown in each display. This was done with the observers' head being held steady. After calibration the observers were free to move their head from side to side, as well as inward as close as 20 cm away from the displays.

Experimental protocol
The observers read the case set in two trials of 20 cases each. On any given trial a balanced combination of "latest" and "prior" mammograms was used, with the restriction that only one version of each abnormal case would appear on a given trial. All lesion-free cases were seen in both trials. The average interval between the trials was 2 months (range 1–3 months).

The observers were instructed to search for malignant masses only. In the instructions we told them that they would see both cancer and normal cases, but not the proportions of each type of case. Furthermore, they were not told that each cancer case contained only one mass, visible in both views, and they were allowed to report as many malignant masses per case as they deemed necessary. The experiment was divided in two phases: (i) visual search; and (ii) report and localization. During the first phase, the observers visually examined the breast parenchyma, while their eye-positions were monitored until they determined whether they thought that the case was "normal" (no lesions to report) or "abnormal" (there would be malignant masses to report). Furthermore, if they thought that the case was "abnormal", we asked them not to terminate visual search until they decided on all locations that they would report in the second phase of the experiment. Once these two decisions were made, the observer used a mouse-controlled cursor to click a button in the display indicating the observer's impression of the case, namely, "normal" or "abnormal". At that point, eye-position monitoring was terminated and the observer used the mouse-controlled cursor to indicate the (x,y) coordinates of all malignant masses that they wished to report. On finishing that (or in the cases deemed to be "normal") the observer used the mouse-controlled cursor to indicate readiness to move on to the next case.

Response scoring
All areas marked by the observers were scored according to whether they corresponded to a true malignant mass (a list of all malignant masses present in the case set was contained in a truth table), thus yielding "true positive" (TP) decisions, or not, which yielded "false positive" (FP) responses. We used as criterion for this scoring the following: if the observer's mark was within 2.5° of visual angle in any direction from the (x,y) location listed in the truth table as containing a malignant mass, it was scored as a TP; if it was beyond that, it was scored as a FP. Moreover, if no mark was placed within 2.5° of visual angle in all directions of a true malignant mass, it was scored as a "false negative" (FN) decision. Note that we used a distance criterion of 2.5° of visual angle because this corresponds to the radius of the "useful visual field" [16, 17]. Finally, lesion-free areas that attracted prolonged visual attention (>330 ms) [18] and that did not yield a mark by the observer (and thus were presumably correctly interpreted as being lesion-free) were scored as "true negative" (TN) responses.

Spatial frequency representation of the responses
Representations in the spatial frequency domain for the four types of decision outcomes made by the observers were generated as follows. Each area that yielded a TP, FP, FN, or TN was segmented from the mammogram. Each segmented area measured 256x256 pixels (this roughly corresponded to an approximation of a circle with diameter 5° of visual angle). Each of these areas was processed using a filter bank in which each filter was sensitive to different orientations and different spatial frequency ranges, in a process known as the "Wavelet Packet Transform" (WPT) [19]. In this decomposition the traditional downsampling was suppressed, which generated an over-complete representation, but at the same time, better preserved high-frequency information. The WPT decomposition was carried out to two-levels deep, which generated 20 spatial frequency bands (SFBs). Each of these contained image elements that favoured a certain orientation (horizontal, vertical, diagonal) and scale (going from coarse to finer detail). We used Daubechies filters because of their compact support and smooth decay [20]. In order to reduce the dimensionality of the dataset, we calculated the energy of the WPT coefficients at the output of each filter. For normalization purposes, we calculated the logarithm of the energy at the output of each filter. Energy was chosen as a representative feature because an energy model has been used to successfully explain threshold vision (that is, the difference between "seeing" and "not seeing") [21]. In addition, it has been shown that the retina performs a logarithmic operation before further processing of the visual signal [22].

Spatial frequency representation of the background
The background as sampled by the observers
Each lesion-free area of the background where the observers had prolonged visual dwell (that is, cumulative fixations totalling 330 ms or longer), but where the observers did not report the presence of a finding (previously defined as TN decisions), were used as samples of the areas of the background that had been used in the observers' cognitive processing of the image.

The "average" background
In order to obtain an "average" representation of the background, we delimited the skin line and filled out the parenchyma (both the CC and the MLO views) with as many non-overlapping 256x256 squares as possible, without crossing the skin line. We then processed each square using the WPT method previously described, and, in order to generate an "average" representation of the background, we calculated the mean, per spatial frequency band, of all of the squares, for each the CC and the MLO view, per case. Thus, each case had two "average" representations of the background, one that represented the CC view and one that represented the MLO view. When the observers' decision was made in the CC view, we used the "average" CC representation of the background, and when it was made in the MLO view, we used the "average" MLO representation of the background for comparisons.

Data analysis
We run two types of analysis on the data. In the first analysis, the observers were paired, and the analyses were run for each pair of observers. This analysis determined both the characteristics of the visual search strategy used by the observers when they agreed on lesion management (i.e. both observers in the pair agreed to report the lesion or both observers agreed not to report it) and when they disagreed on lesion management (that is, when one observer in the pair agreed to report the lesion and the other one did not). We also compared changes in visual search strategy per observer pair "before" and "after" the observers' eyes first hit the location of a response. In the second analysis, all observers were used so as to determine how their background sampling strategy changed as a function of fixating a true lesion that they did report (a TP decision), or a lesion-free area that they reported as containing a lesion (a FP decision), or a true malignant mass that they visually inspected but decided not to report (a FN decision).

	Results

Top Abstract Introduction Materials and methods Results Discussion References

 
Paired observer analyses
Agreement in background sampling
For the cancer cases (both the "latest" and the "priors"), we paired the observers and sought to determine whether each pair visually sampled the background similarly as a result of agreeing to report the lesion (a TP decision), visually inspecting but agreeing not to report the lesion (a FN decision), or disagreeing whether to report the lesion (i.e. one observer in the pair reports the lesion but the other does not). Table 1 shows the mean number of fixations in the cases agreed upon by the observers and in the cases disagreed upon by the observers. For the cases agreed upon by the observers, we further determined whether the agreed decision was to report the lesion (a TP decision) or to dismiss it (a FN decision). In addition, Table 1 also shows the mean number of fixations agreed upon by the observers in the cases where they agreed and where they disagreed to report the lesion. As can be seen from Table 1, the observer pairs have a greater number of fixations in the cases where they agreed how to handle the lesion (mean 12, range 7–16 when they do report the lesion; mean 16, range 10–22 when they do not report the lesion) than in the cases where they disagree about lesion management (mean 11, range 0–23). When the observers agree on how to manage the lesion, they agree on most of the fixations they make (58% when they report the lesion vs 63% when they do not) than when they disagree how to manage the lesion (55% of the fixations).

View this table: [in this window] [in a new window]   Table 3. Statistically significant changes(according to Fisher's r to z transform, p<0.05) in the number of spatial frequency bands significantly correlated between the areas of the background sampled by the observers "before" and "after" their eyes first hit the location of a true positive (TP), false positive (FP), or false negative (FN) decision and both the "average" representation of the background and the representation of the area of the response, for both the "latest" and the "prior" mammograms. In the table, the " " indicates the number of spatial frequency bands (SFBs) whose correlation significantly changed, between "before" and "after" fixating the location of the response

	Discussion

Top Abstract Introduction Materials and methods Results Discussion References

In this study we looked at the changes in background sampling strategy as the observers first fixated the location of a response from two viewpoints: from the viewpoint of all observers and from the viewpoint of individual pairs of observers. The first approach allows us to draw conclusions about an "average" observer, whereas the second approach allows us to try to understand how each observer behaves when subject to the same case set. While the first approach is certainly more useful in terms of making generalizations, the second approach allows us to understand what happens in situations such as double-reading, in which two sometimes randomly paired observers have to read the same cases. Our second approach allows us to question whether any two pairs are the same; namely, does the choice of who goes in the pair affect the final result?

Our data seem to suggest that the choice of who forms the pair does indeed matter. Accordingly, some pairs had many more fixations in common in the cases where they agreed on lesion management (whether that was to report or to incorrectly dismiss the lesion), as well as in the cases where they disagreed in lesion management (that is, when one observer reported the lesion but the other dismissed it). Furthermore, agreement in background sampling was pretty tight in some pairs, with no differences being observed before or after the lesion was fixated, whereas for some other pairs differences that existed before the lesion was fixated were eliminated afterwards. These results seem to suggest that double-reading of mammograms by radiologists may be more effective if they are paired in accordance with how they each individually read a case. In other words, if agreement in their reading is so strong that no differences can be observed in background sampling before or after they fixate the lesion, then probably not much can be gained from that particular pairing, because very likely the two radiologists will read the cases very similarly. If, however, differences can be observed in how they sample the background, then perhaps more could be gained, because one radiologist could bring a perspective to the visual interpretation of the image that would differ from the perspective of the other radiologist.

Regarding the analysis done with all observers, our data suggest that for the "average" observer there are changes after fixating for the first time the location of a true malignant mass that the observer reports. In this case, correlation with the "average" background and with the area containing the response increase after the location is initially fixated, thus indicating a shift in visual search strategy, whereby the observer is trying to find similar areas in the image (i.e. the radiologist is trying to determine whether the lesion occurs in several locations or in a single location). Interestingly, a similar effect is mostly observed after the radiologists first fixate the lesion-free areas where they report the presence of a malignant mass (a FP decision). Albeit in this case the "lesion" itself is not a true lesion, it is real enough for the observer that the same changes in visual search strategy are observed. For the lesions that the observers visually inspect but dismiss, we noticed that after fixating the lesion for the first time the correlation of the background samples with the "average" background was reduced, whereas the correlation with the location of the lesion was significantly increased. This seems to suggest that the observer is actively trying to conciliate the perception of the lesion with the perception of the image, and is thus trying to place the percept in the context of the background parenchyma.

In summary, we can say that our data suggest that initially fixating a location where the observers make a response (this response being correct, a TP, or not, either a FP or a FN) does influence the observers' visual sampling of the rest of the image. Furthermore, our data also suggest that criteria should be established before pairing observers for double-reading cases, because the visual search behaviour of certain observers is virtually indistinguishable, and in this case there is very little to be gained by joining these observers together to read a case.

This work was partially funded by the following grants: NIH/NCI CA100107 and NIH/NIBIB EB002120

	References

Top Abstract Introduction Materials and methods Results Discussion References

 

1. Cancer Facts and Figures 2006. American Cancer Society, 2006
2. Warwick J, Tabar L, Vitak B, Duffy SW. Time-dependent effects on survival in breast carcinoma: results from 20 years of follow-up from the Swedish two-county study. Cancer 2004;100:1331–6.[CrossRef][Medline]
3. Ganry O, Peng J, Dubreuil A. Influence of abnormal screens on delays and prognostic indicators of screen-detected breast carcinoma. J Med Screening 2004;11:28–31.[CrossRef][Medline]
4. Buseman S, Mouchawar J, Calonge N, Byers T. Mammography screening matters for young women with breast carcinoma: evidence of downstaging among 42-49-year-old women with a history of previous mammography screening. Cancer 2003;97:352–8.[CrossRef][Medline]
5. Yankaskas BC, Taplin SH, Ichikawa L, Geller BM, Rosenberg RD, Carney PA, et al. Association between mammography timing and measures of screening performance in the United States. Radiology 2005;234:363–73.[Abstract/Free Full Text]
6. Brem RF. The never ending controversies of screening mammography: what is the appropriate callback rate for women undergoing screening mammographic examination? Cancer 2004;100:1549–52.[CrossRef][Medline]
7. Elmore JG, Barton MB, Moceri VM, Polk S, Arena PJ, Fletcher SW. Ten-year risk of false positive screening mammogram and clinical breast examinations. New Engl J Med 1998;338:1089–96.[Abstract/Free Full Text]
8. Kundel HL, Nodine CF, Thickman D, Carmody D, Toto L. Nodule detection with and without a chest image. Investigative Radiol 1985;20:94–9.
9. Burhenne LJW, Wood SA, D'Orsi CJ, Feig SA, Kopans DB, O'Shaughnessy KF, et al. Potential contribution of computer-aided detection to the sensitivity of screening mammography. Radiology 2000;215:554–62.[Abstract/Free Full Text]
10. Nodine CF, Kundel HL, Mello-Thoms C, Weinstein SP, Orel SG, Sullivan DC, et al. How experience and training influence mammography expertise. Acad Radiol 1999;6:575–85.[CrossRef][Medline]
11. Nodine CF, Mello-Thoms C. The nature of expertise in Radiology. In: Beutel J, Kundel HL, Van Metter RL, editors. Handbook of medical imaging, volume 1: progress in medical physics and psychophysics. SPIE Press, 2000:859–94
12. Kundel HL, Nodine CF. A visual concept shapes image perception. Radiology 1983;146:363–8.[Abstract/Free Full Text]
13. Kundel HL, Nodine CF. Modeling visual search during mammogram viewing. Proc SPIE Image Perception, Observer Performance and Technology Assessment 2004;5372:110–5
14. Nodine CF, Kundel HL, Lauver SC, Toto LC. Nature of expertise in searching mammograms for breast masses. Acad Radiol 1996;3:1000–6.[CrossRef][Medline]
15. Mello-Thoms C. How does the perception of a lesion influence visual search strategy in mammogram reading? Acad Radiol 2006;13:275–88.[CrossRef][Medline]
16. Kundel HL, LaFollette PS. Visual search patterns and experience with radiological images. Radiology 1972;103:523–8.[Medline]
17. Mackworth NH. Stimulus density limits the useful field of view. In: Monty RA, Senders JW, editors. Eye movements and psychological processes. Hillsdale, NJ: Lawrence Erlbaum Publishers, 1976: 307–21
18. Henderson JM, Hollingworth A. Eye movements, visual memory, and scene representation. In: Peterson MA, Rhodes G, editors. Perception of faces, objects, and scenes – analytic and holistic processes. Oxford University Press, 2003:356–83
19. Mallat S. A wavelet tour of signal processing. New York, NY: Academic Press, 1998
20. Popivanov I, Muller RJ. Similarity search over time-series data using wavelets. Proc. of the 18th International Conference on Data Engineering 2002;1063:6382/02
21. Manahilov V, Simpson W. Energy model for contrast detection: spatiotemporal characteristics of threshold vision. Biological Cybernetics 1999;81:61–71.[CrossRef][Medline]
22. Deutsch S, Deutsch A. Understanding the nervous system – an engineering perspective. Wiley-IEEE Press, 1993
23. Nodine CF, Mello-Thoms C, Weinstein SP, Kundel HL, Conant EF, Heller-Savoy RE, et al. Blinded review of retrospectively visible unreported breast cancers: an eye position analysis. Radiology 2001;221:122–9.[Abstract/Free Full Text]
24. Mello-Thoms C, Hardesty L, Sumkin J, Ganott M, Hakim C, Britton C, et al. Effects of lesion conspicuity on visual search in mammogram reading. Acad Radiol 2005;12:830–40.[CrossRef][Medline]

This Article Abstract Full Text (PDF) Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Mello-Thoms, C Search for Related Content PubMed PubMed Citation Articles by Mello-Thoms, C