This Article Abstract Figures Only Full Text (PDF) Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Alberdi, E Articles by Slack, R Search for Related Content PubMed PubMed Citation Articles by Alberdi, E Articles by Slack, R

Use of computer-aided detection (CAD) tools in screening mammography: a multidisciplinary investigation

¹ Centre for Software Reliability and ² Psychology Department, City University, Northampton Square, London EC1V 0HB and ³ Institute for Communicating and Collaborative Systems School of Informatics, The University of Edinburgh, 1 Buccleuch Place, Edinburgh EH8 9LW, UK

	Abstract

Top Abstract Introduction 1. The HTA trial 2. Supplementary analyses of... 3. Follow-up studies 4. Ethnographic studies of... 5. Discussion 6. Concluding remarks References

 
We summarise a set of analyses and studies conducted to assess the effects of the use of a computer-aided detection (CAD) tool in breast screening. We have used an interdisciplinary approach that combines: (a) statistical analyses inspired by reliability modelling in engineering; (b) experimental studies of decisions of mammography experts using the tool, interpreted in the light of human factors psychology; and (c) ethnographic observations of the use of the tool both in trial conditions and in everyday screening practice. Our investigations have shown patterns of human behaviour and effects of computer-based advice that would not have been revealed by a standard clinical trial approach. For example, we found that the negligible measured effect of CAD could be explained by a range of effects on experts' decisions, beneficial in some cases and detrimental in others. There is some evidence of the latter effects being due to the experts using the computer tool differently from the intentions of the developers. We integrate insights from the different pieces of evidence and highlight their implications for the design, evaluation and deployment of this sort of computer tool.

	Introduction

Top Abstract Introduction 1. The HTA trial 2. Supplementary analyses of... 3. Follow-up studies 4. Ethnographic studies of... 5. Discussion 6. Concluding remarks References

 
In this paper we summarise the results of a variety of studies and analyses conducted to investigate the use of a computer-aided detection (CAD) tool in mammography. This work was done as part of the Dependability Interdisciplinary Research collaboration (DIRC; http://www.dirc.org.uk), a collaborative, multidisciplinary project dealing with the dependability of computer-based systems: systems encompassing computer software and hardware together with their human users and the social context of computer use. The use of computer aids in breast screening is a good example of this sort of "human–computer system".

As is common in the radiological literature, we will use the abbreviation "CAD" to mean the computer tool, whenever the context does not create ambiguities with its literal meaning "detection activity aided by a computer".

CAD is used to alert (prompt) a human expert (typically a radiologist) to areas of a mammogram (a radiography image of the breast) where computerised image analysis suggests that abnormalities may be found. Typically, the CAD tool processes a digitised version of a mammogram and marks it with "prompts" to highlight mammographic features that the reader should examine. The design goal for CAD is to aid human experts (hereafter referred to as "readers") to notice features in a mammogram that might indicate cancer but that they may otherwise miss.

Our interest in the human–CAD system originated from a clinical trial funded by the UK Health Technology Assessment (HTA) programme [1]. The goal was to assess the impact of a particular CAD tool, R2 ImageChecker^® M1000 (R2 Technology, Inc., Los Altos, CA) [2], in breast screening. The trial was designed to test whether CAD increases readers' sensitivity (the proportion of cancers recalled out of all cancers) without adversely affecting their specificity (the proportion of normal cases not recalled out of all normal cases). In Section 1 of this paper we outline aspects of the HTA trial (reported in detail in [1]) that are relevant to our research.

Our approach is multidisciplinary and combines insights from various disciplines: reliability engineering, computing, psychology, human factors and sociology. We were granted access to the HTA trial data and conducted supplementary analyses to investigate in more detail the effects of CAD on readers' decisions. Our analyses were influenced by reliability modelling in engineering [3, 4] and focused on how, inter alia, the effects of CAD vary between cases or depend on whether the tool prompts mammograms correctly or not. Section 2 summarises some results of these analyses, which have been partly reported in an earlier publication [5]. We also conducted two follow-up studies, outlined in Section 3, to investigate how readers react to incorrect CAD output [6]. The HTA trial and the follow-up studies were complemented by ethnographic studies of CAD use (see Section 4), which have been partly reported by Hartswood et al [7].

The discussion in Section 5 attempts to integrate the different findings and highlights their potential implications for the design, evaluation and use of CAD. This demands that we consider how we may extrapolate from data obtained in trial conditions to the real-world use of CAD in breast screening; we reflect on the potential benefits and as yet unresolved questions of using a multidisciplinary approach to address this important issue.

	1. The HTA trial

Top Abstract Introduction 1. The HTA trial 2. Supplementary analyses of... 3. Follow-up studies 4. Ethnographic studies of... 5. Discussion 6. Concluding remarks References

 
The goal of the HTA trial was to assess the impact of a CAD tool (R2 ImageChecker^® M1000 [2]) in breast screening. This CAD tool had been shown to have a high sensitivity, that is it prompts a high proportion of breast cancers. Castellino et al [8] report an overall sensitivity of up to 90%. However, high sensitivity comes at the expense of low specificity, that is it generates many false prompts, with an average of 2.06 prompts per case in one study.

The trial was run with 50 readers experienced in breast screening, and used 180 cases (a mixture of 60 cancers and 120 normal cases) distributed in three sets of 60. All participating readers saw all the cases in two different experimental conditions: (a) "unprompted condition", i.e. without CAD; and (b) "prompted condition", i.e. with CAD. The order of conditions was randomised across participants. In both conditions, the participants saw two versions of each case: (1) the mammograms positioned on a standard viewing roller; and (2) a digitised version of the mammograms printed on paper. In the prompted condition, the printouts contained the prompts generated by CAD. Participants were also asked to make their decisions regarding whether a case should be recalled for further tests as if they were viewing the mammograms as single readers in the screening programme. More details of the procedures can be found in Taylor et al [1].

Analysis of the results showed no statistically significant impact of CAD (no improvement and no reduction) on the readers' sensitivity and specificity [1].

	2. Supplementary analyses of the data from the HTA trial

Top Abstract Introduction 1. The HTA trial 2. Supplementary analyses of... 3. Follow-up studies 4. Ethnographic studies of... 5. Discussion 6. Concluding remarks References

 
The analyses we describe here were inspired by studies of the dependability of software systems with diverse redundancy [3]. These studies have shown that variation and co-variation of difficulty of input cases for different system components substantially affect the dependability of the overall system. They also indicate that focusing on the average probabilities of the failures of the components, or assuming statistical independence among their failures, can be misleading. We explored the possibility that conclusions from those studies could also apply to the system formed by CAD and the human reader. We focused our analyses on the role of the correctness of CAD output and evaluating its interaction with "case difficulty", which we define as the probability of a reader, randomly selected from the given set, reaching a wrong decision about a given case.

Throughout this paper, we will be talking about "errors" of human (reader) or machine (CAD). We clarify here how we define these errors and present the terminology we will be using.

• "Error by a human" (or "incorrect decision") means that s/he recalls a normal case or fails to recall a cancer. "Correct decision" means that s/he fails to recall a normal case or recalls a cancer.
  
• We talk of "error by CAD", or "incorrect output": (i) for a cancer, if CAD "misses" the cancer (i.e. it gives a "false negative"), which it can do in two ways: (a) by failing to place any prompt on the mammogram ("unmarked" cancer); or (b) by placing prompts in areas other than the actual location of the cancer ("incorrectly marked" cancer: all prompts are "false"). We say that CAD provides "correct output" for a cancer (i.e. it processes it correctly; true positive) if it prompts the area on the mammogram where the cancer is located ("correctly marked" cancer), even if CAD may also have prompted other areas of the mammogram; and (ii) for a normal case, if CAD places any prompt on the mammogram ("incorrectly marked" normal case; false positive). We say that CAD provides "correct output" for a normal case if it places no prompts ("unmarked normal"; true negative).
  

2.1. Potential of CAD for improving readers' sensitivity
It is interesting to estimate how much CAD could potentially improve readers' sensitivity for the test set in the trial.

In the HTA trial, readers in the unprompted condition (without CAD), made 2994 decisions about cancers (50 readers x 60 cancers–6 missed data points), 741 of which were incorrect. Thus, the estimated probability of human error for cancers in the unprompted condition is 741/2994=0.247 (95% confidence interval 0.232–0.263). Of these 741 errors, 314 occurred on cases correctly prompted by CAD.

If we assume (optimistically) that readers in the prompted condition will make the best possible use of CAD, i.e. (1) will recognise and recall all the cancers correctly prompted by CAD and (2) will still recall all cancers that they recalled in the unprompted conditions, even with incorrect output from CAD, then the expected number of errors for cancers in the prompted condition is 741–314=427 (of all 2994 decisions), i.e. the probability of failure in the prompted condition is estimated to be 427/2994=0.143 (95% confidence interval 0.130–0.156). Considering only the mean estimates, the estimated potential for improving readers' sensitivity equals 314/2994=0.105 (10.5%). Using a similar approach in another study, Warren Burhenne et al [9] reported "...CAD prompting could have potentially helped reduce..." the initial rate of false negative errors (0.21) by 77%, i.e. they estimated the potential for improving readers' sensitivity as 0.21 x 77%=16.2% (0.162).

Analysis of the HTA trial data [1] shows that the estimated potential improvement in sensitivity of 10.5% does not seem to be realistic, at least for this experimental setting, and that the assumptions for the above calculation were not verified. In the prompted condition, readers missed correctly prompted cancers; they also made errors that they did not make in the unprompted condition, as shown in Table 1 for different categories of cancers.

We ignore those outcomes in which the two decisions were either both correct or both wrong, and consider only the following outcomes:

• the decision by a reader for a case was incorrect in the unprompted condition and correct in the prompted condition ("prompted decision better"); or
  
• the reader's decision was correct in the unprompted condition and incorrect in the prompted condition ("prompted decision worse").
  

If there was statistical independence between the outcomes "prompted decision better/worse" and "CAD output correct/incorrect", then we could conclude that the correctness of CAD output does not affect readers' decisions. We tested for independence using Fisher's Exact Test [11] for the contingency tables given in Table 2. Independence would mean that the probability of the prompted decision being better rather than worse would be the same for cases correctly processed by CAD as for cases incorrectly processed by CAD.

The results indicate that correct CAD output is likely to help in reaching a correct decision and that incorrect CAD output makes it more difficult.

2.3. Error rates of readers and CAD for different populations of cases in different conditions
To investigate the effect of correctness of CAD output, we estimated the probabilities of human error in both the prompted and the unprompted conditions, for cases categorised according to case type (normal or cancer) and the output of CAD (correct or incorrect).

These estimates are shown in Table 3. Both for cancers and for normal cases, all the estimated probabilities of error do not differ significantly between the prompted and unprompted conditions, except for those normal cases that CAD did not prompt. For this category of cases, the readers' rate of false recalls in the prompted condition was smaller than in the unprompted condition by 0.06.

We tested the correlation between errors made by readers and CAD for significance with Fisher's Exact Test [11] applied to the contingency tables for correct and wrong decisions by readers, on cancers and on normal cases in both conditions (see Table 4). The test indicated significant correlation for all four contingency tables (p<0.05).

We define "non-obvious" as a case on which at least one reader made an incorrect decision, either in the prompted or unprompted condition.

Analysis of variance (ANOVA) indicated that the value d–d_p is significantly different for groups of cases processed by CAD correctly and incorrectly:

• all cancers: F(1,58)=4.1899, p=0.0452;
  
• non-obvious cancers: F(1,44)=4.0676, p=0.0498;
  
• normal cases: F(1,118)=9.3384, p=0.00278;
  
• non-obvious normal cases: F(1,114)=9.0746, p=0.00319.
  

In an earlier study [5], we applied logistic regression to highlight general patterns in the effect of CAD. It appears that CAD tends to make cancers that are relatively easy (i.e. with d<0.6) less difficult (i.e. d_p<d) and cases that are relatively difficult (i.e. with d>0.6) even more difficult (i.e. d_p>d).

The plot in Figure 1 illustrates this effect. The horizontal axis represents the unprompted difficulty d. The vertical axis shows the difference d_p–d. So, a point below the horizontal line indicates a cancer for which CAD appears to reduce the rate of reader errors for cancers. Points marked w and c indicate the observed values of d and d_p–d for non-obvious cancers, divided into those with correct CAD output (c) and those with wrong CAD output (w). The curves show the regression estimate for the mean value of d_p–d for cases with different difficulty d. The dashed curve corresponds to incorrectly prompted cancers, the dotted–dashed curve to the correctly prompted cancers, and the solid curve to all cancers together. For more details of our regression analyses, see Povyakalo et al [5].

View larger version (15K): [in this window] [in a new window]   Figure 1. Effect of correctness of computer-aided detection (CAD) output on case difficulty (rate of false negative errors). dp, prompted difficulty; d, unprompted difficulty; w, cancer that was wrongly processed by CAD; c, cancer that was correctly processed by CAD. ---, logistic regression curve for wrongly processed cancers; -·-·, logistic regression curve for correctly processed cancers; —, logistic regression curve for all cancers.

	3. Follow-up studies

Top Abstract Introduction 1. The HTA trial 2. Supplementary analyses of... 3. Follow-up studies 4. Ethnographic studies of... 5. Discussion 6. Concluding remarks References

The test sets in the HTA trial did not contain enough examples of cancers incorrectly processed by CAD (in particular, "unmarked" cancers; see definitions in Section 2) to allow us to draw statistically significant estimates of their effects on the readers' decisions. We ran a follow-up experiment (Study 1) with a new test set (60 cases) containing a larger proportion of cancers missed by CAD (20 of the 30 cancers in the set). Nine of the false negatives in our test set were "unmarked cancers". We kept all other characteristics of the test set as similar as possible to the sets used in the HTA trial, as we wanted the readers to perceive this study as a natural extension to the original trial and to behave in a comparable way.

The participants in Study 1 were 20 readers who had participated in the original trial. We used essentially the same procedures as used in the HTA trial, except that readers in Study 1 saw all the case, only once, always with the benefit of CAD ("prompted condition").

At that stage, we were not interested in comparing readers' performance with and without CAD; our goal was to estimate the probability of reader error when the output from CAD was incorrect. However, the results turned out to be highly unexpected: the average reader sensitivity was surprisingly low (52%) and this decrease was particularly strong for the "unmarked" cancers. This led us to suspect that CAD errors may have had a significant negative impact on readers' decisions. On the other hand, we could not exclude the alternative explanation that the cases in our study had characteristics that made them particularly difficult (perhaps mammographically undetectable) for both human readers and the CAD tool.

As a "control" for Study 1, we ran Study 2, where readers saw the same test set without CAD ("unprompted condition"). We used 19 readers from three different UK screening centres, none of whom had participated in Study 1 but who were equivalent to the group used in Study 1 in terms of years of experience and professional background. We used the same procedures as in Study 1 except that readers did not see the CAD output.

Additionally, we conducted a new test with six of the more experienced participants in Study 2 to get a better understanding of the "difficulty" of the cancers in our test set. These participants were presented again with the 30 mammograms containing cancer and were asked to rank them according to various criteria of case "difficulty". The responses of this subset of readers, as well as the performance of all participants in Study 2, strongly indicated that six of the cases in our test set were probably "occult" cancers, undetectable via mammography, and so we eliminated these cases from our analyses (see more details in Alberdi et al [6]).

Readers' average sensitivity for the remaining 24 cancers was 61% for those who saw the cases with CAD (Study 1) and 73% for those who saw them without CAD (Study 2). The difference in average sensitivity between the two sets of readers was statistically significant. In contrast, the average specificity in Study 2 was lower than in Study 1 (86% vs 90%), but the difference was not statistically significant.

Table 5 shows the proportions of incorrect human decisions in Study 1 and Study 2 for the 54 analysed cases, categorised according to case type (normal or cancer) and output of CAD ("unmarked", "correctly marked", or "incorrectly marked"). ANOVA showed statistically significant differences between Study 1 and Study 2 for the "unmarked cancers" (p<0.001), the "incorrectly marked" cancers (p<0.05) and the "unmarked" normal cases (p<0.05).

	4. Ethnographic studies of CAD use

Top Abstract Introduction 1. The HTA trial 2. Supplementary analyses of... 3. Follow-up studies 4. Ethnographic studies of... 5. Discussion 6. Concluding remarks References

 
The aim of the ethnographic studies was to examine how readers used the CAD tool and the ways in which they came to understand and explain its prompts and general behaviour. Readers' behaviour was observed both in trial conditions [7] and in vivo in everyday breast screening practice. Observations took place over a total period of approximately 1 month and were videoed and reviewed so as to examine in fine-grained detail the work of reading.

Using a "think aloud" protocol in which participants vocalised their thought processes to the observer as they read cases, we attempted to explore the sense readers made of prompts in the context of the mammograms on which they occurred. Subsequently, we discussed with readers cases that they identified as problematic (especially cases on which they had spent a substantial amount of time) to clarify how they dealt with these "difficult" cases.

A significant finding was the importance readers attached to ascertaining what a prompt "meant": how it could be explained in the context of the mammogram. In some cases, the accounts were of the order "I don't know why it's prompted that", in others readers saw the features prompted as, for example, composite shadows, and gave an account of why they thought the CAD tool had prompted the feature together with what they saw it as actually being (e.g. benign, because it could be "picked apart").

Although readers were advised to use CAD as an attention cue, and to use their own judgement to decide whether a prompted feature required recall, we observed that they sometimes used prompts to inform their decisions. For example, one reader commented: "This is a case where without the prompt I'd probably let it go ... but seeing the prompt I'll probably recall ... it doesn't look like a mass but she's got quite difficult dense breasts ... I'd probably recall." In other instances, we observed readers using the absence of a prompt as evidence for "no recall".

The explanation for this may be that the uses to which prompts are put are contingent on the specific problems posed by individual cases. For example, reading dense, feature-rich breasts poses demands very different from those of lucent or uncomplicated breasts, and the reader's comment above demonstrates how she marshals the "evidence" of the prompt in making a decision under these specific circumstances.

Over time, readers acquired a "biography" for the CAD tool: they came to believe they knew what features it would and would not prompt and they read with this putative biography as a factor in their work. For example, readers would often remark that they had anticipated that the system would prompt for a particular feature within the breast, sometimes then dismissing the prompt as they already had judged the feature to be benign. This does not mean that they would ignore the prompt, or that they would not pay serious attention to it, but that they thought the prompt could be expected given their emerging understanding of what the CAD tool could and did do.

Post-trial discussions with readers indicated some of the strengths they attributed to the CAD tool:

• an ability to draw their attention to subtle signs that they might have missed otherwise; in some instances, readers reported that a prompt had led them to recall a case that they would have "let go" had the prompt not been there;
  
• its perceived consistency; prompts were seen as useful by compensating in some consistent way for individual human weaknesses.
  

On the other hand, readers noticed the following weaknesses of the tool:

• the many false prompts: the tool was seen to prompt the wrong mammographic features; in many instances, readers explicitly dismissed prompts that they interpreted as artefacts, noise or obvious benign appearances;
  
• the distracting effect of too many prompts;
  
• the fact that the tool missed obvious features that readers thought should have been prompted.
  

	5. Discussion

Top Abstract Introduction 1. The HTA trial 2. Supplementary analyses of... 3. Follow-up studies 4. Ethnographic studies of... 5. Discussion 6. Concluding remarks References

 
5.1. Impact of CAD on readers' sensitivity
CAD in mammography has the potential to improve readers' ability to detect cancers. As we saw in Section 2.1., if readers were always to react appropriately to the prompts placed correctly by CAD, their sensitivity in the HTA trial could have increased by up to 10%. However, the trial showed no statistically significant effects of CAD on readers' average sensitivity. This result is actually consistent with most experimental measurements of the impact of the R2 ImageChecker^® and other similar CAD tools on mammogram reading (e.g. [12–16]). To improve the effectiveness of computer aids, it is desirable to explain why CAD apparently had no effect in these studies.

A simple conjecture would be that readers tend to ignore CAD outputs, possibly because the high number of false prompts creates excessive load. However, our statistical analyses and ethnographic studies do suggest systematic effects of the use of CAD, which are positive or negative depending on aspects of the cases and on CAD output. Readers' reports suggested that the presence of CAD prompts had at times alerted them to relevant mammographic features that they would have missed otherwise as well as affecting their recall decision for features that they had already noticed. These reports are corroborated by our supplementary analyses of the HTA trial data, which indicate that for a subset of cases CAD did have beneficial effects on readers' decisions (see Table 2 and Figure 1). At the same time, readers often dismissed explicitly many of the prompts as they considered them false. Analysis of the data indicates that some of these were, in fact, correct prompts: readers sometimes had difficulties distinguishing between correct and incorrect prompts.

Our statistical analyses also show that CAD output could have detrimental effects on readers' sensitivity for a subset of cases ("difficult" cancers), especially when the output of CAD was incorrect.

We argue that CAD, rather than having too little effect on readers' decisions to produce a measurable impact, had both beneficial and detrimental effects on readers' performance, but in the trial these effects compensated for each other, resulting in no significant impact on average sensitivity.

5.2. Effects of the absence of prompts
By choosing in our follow-up studies a test set with a large proportion of cancers missed by CAD, we managed to isolate (somewhat serendipitously) the potential detrimental effects of CAD on reader sensitivity. Participants who read our test set with the benefit of CAD (Study 1) showed a significantly lower sensitivity than those who saw the same cases without CAD (Study 2).

This effect was particularly marked for those cancers that CAD did not prompt and led us to conjecture that the absence of prompts may have a much bigger impact on readers' decisions than anticipated. Our analyses of the data from both the HTA trial and our follow-up studies strongly suggest that readers may have used the absence of prompts on a mammogram as a sort of reassurance for their "no recall" decisions for normal cases. It appears that, based on their experience with the tool, readers tended to (correctly) assume that the absence of prompting was a strong indication that a case was normal. The participants (both in the HTA trial and in follow-up Study 1) were very unlikely to recall cases for which CAD had issued no prompts. Again, these findings are broadly corroborated by our ethnographic observations.

One could argue that this is a rational approach. Readers perceived many of the prompts as distracting. As most mammograms contained prompts, the absence of prompts was more informative than their presence (especially if detailed analysis of every prompt was too demanding and thus practically infeasible). This can be beneficial when dealing with equivocal normal cases (see Table 2). However, as the results from our follow-up studies indicate, this can have damaging effects on readers' decisions for difficult-to-detect cancers that CAD does not prompt.

To our knowledge, such detrimental effects have not been reported before in the radiological/CAD literature. Earlier human factors studies of the effects of computer failures on human behaviour have shown that the failure of a computer aid to detect and warn of a target event could make users less likely to make the right decision for the event [17, 18]. However, the participants in such studies are typically students working in artificial laboratory settings, while our participants were experts working in relatively realistic settings relevant to their area of expertise.

One plausible mechanism to explain these effects is that the absence of prompts made readers revise their decisions for ambiguous abnormalities that they had already detected. In other words, they may have used the absence of prompts as a reassurance for a "no recall" decision when dealing with features they found difficult to interpret. It is possible that readers were using whatever evidence was available to resolve uncertainty. The implication is that CAD was being used not only as a detection aid but also as a classification or diagnostic aid, which is not what the tool is designed for. This is consistent with our observations from our ethnographic studies (Section 4) and also with earlier studies of CAD tools [19]. But we cannot exclude alternative mechanisms. For example, the absence of prompts may have caused readers to pay less attention to a case and as a result, they may have failed to detect signs that they would not have missed otherwise (as proposed by studies of "automation bias" or "over-reliance" on computer advice [17]). Although this is a plausible scenario, we have not found evidence to support its occurrence in the HTA trial or our follow-up studies.

5.3. Difficulties of extrapolating results to real-world practice
Most of the findings reported in this paper are based on studies and analyses of readers' behaviour in trial conditions. Although there was an attempt to make the experimental settings reasonably realistic, many artificialities and simplifications were unavoidable. One must be careful, therefore, when extrapolating from the behaviours observed to effects in the field. We highlight here some important differences between the trial(s) and everyday practice.

(a) A common criticism is that clinical trials are conducted with test sets containing unrealistically high proportions of pathological cases (so as to achieve sufficient statistical power with manageable numbers of cases). Evidence that radiologists do behave differently when faced with case samples containing different prevalence of disease has been reported [20, 21]. These effects have not received sufficient attention to date and are worth exploring further.

(b) In everyday breast screening, readers have access to many other sources of information in addition to CAD (e.g. earlier mammograms, medical records, etc.); these sources of information may make readers interpret CAD output in different ways from how they did in the trials.

(c) In the NHS Breast Screening Programme, reading is essentially a collaborative activity [22]; double reading is common practice and the final decision on a case is often the result of group discussion. In contrast, trial participants acted as though they were reading alone, which may have influenced how they interpreted the prompts and made their decisions.

(d) In the trial, we saw how readers attempted to make sense of the tool's behaviour. In everyday practice, readers would have a better opportunity to gain a progressive understanding of how the tool works and to adjust their interpretation of its behaviour accordingly.

Despite these difficulties, many of the considerations derived from the clinical trial are relevant and useful. The statistical analyses, in conjunction with observations of humans in this and similar tasks, indicate plausible mechanisms that would cause the effects we observed, but do not allow us to decide which of these mechanisms will be active to a perceptible extent in a given activity. However, finding evidence of them in practice, even in partially artificial environments, is prima facie evidence of them being likely to occur. Designers of tasks and the computer tools to support them should consider how these behaviours may arise, and how to adapt tools and procedures to reduce those that are considered negative; assessors should consider them as factors that may change between the clinical trial and clinical use.

5.4. Methodological implications for the evaluation of computer aids
The use of a standard clinical trial regimen for evaluating new healthcare technologies has been subject to much criticism in recent years [23–26]. One problem that has been highlighted is that trial designs inevitably ignore the contextual nature of the work being supported, raising doubts about extrapolating trial results to real settings of use. Our decision to employ a multidisciplinary approach to CAD evaluation was partly intended as an exploration of how criticisms of clinical trials might be addressed.

Our experience of using ethnographic methods to complement statistical analysis has shown some promise. Ethnography aims at understanding the context of work practices, i.e. how different are they actually performed in the workplace. The value of ethnographic methods has already become quite widely recognised as a way of informing requirements so that information technology (IT) systems are designed appropriately for their actual circumstances of use [27]. In our evaluation of CAD, we also found that ethnographic methods can be valuable in addressing the "ecological validity" of clinical trials by helping the interpretation of trial results to take into account differences between the context of the trial and a more realistic context of use.

First, it is only through ethnographic studies that we have been able to gain an understanding in detail of the character of everyday screening work and the context in which a CAD tool would be used. This has helped us to identify possible mismatches between the tool as designed and readers' requirements [7]. As an example, we observed readers' perceived need to explain the behaviour of the CAD tool in order to use it properly, and how they struggled to do this with the tool as designed. Second, ethnographic studies helped to reveal aspects of how readers actually used the CAD tool in the trial. We found evidence of readers not adhering to the trial protocol of using the tool as an attention cue and instead, using it at times as a decision aid. Ethnography and statistical analysis thus corroborated each other's conclusions. Third, ethnographic findings also influenced the choice of probabilistic models [4] by avoiding unrealistic assumptions that would invalidate results.

From a statistical viewpoint, a first consideration is simply that averages may hide substantial variations between subpopulations. Our statistical analyses, motivated by the "diversity modelling" approach (reported in references [3 and 4]) and its emphasis on how performance varies across classes of cases, have proven very useful here.

We found both beneficial and detrimental systematic effects of the use of CAD that just happen to cancel out in the trial (although the detrimental effects appear acutely in our follow-up studies). If these effects were to be present in practical clinical use, with different mixes of cases and readers from those in the trial, the net overall effect might be positive, negative or null, in addition to some possible transfer of risk between categories of patients. It might still be possible to estimate the net effect in future clinical use from the results in the trial [4] if further studies showed that the {case, reader} pairs can be classified by variables that can be estimated in both situations before introducing CAD and that are sufficiently predictive.

5.5. Implications for CAD design and deployment
We consider here the implications of four of our main sets of findings.

(a) Limited diversity between human and machine errors. In the HTA trial, CAD errors were heavily correlated with the "difficulty" of cases (see Section 2.3.; Table 3). To increase the "potential" advantage that we estimated in Section 2.1., CAD should prompt correctly those cases where unaided readers would tend to fail: its error pattern should be as "diverse" as possible from the readers" (cf. the mathematical models in references [3 and 4]). Some improvement in CAD effectiveness could be sought by increasing this diversity, even without improving the average sensitivity or specificity of CAD. The tool could be tuned to be more sensitive for classes of cases on which readers tend to be less effective, as these cases are natural candidates for CAD to make a difference. Perhaps CAD thus tuned to be "more diverse" from its users may improve the latter's performance more than CAD simply tuned to be very good (in terms of sensitivity, specificity or any weighted combination of the two). We believe that this possibility is worth exploring, although how much gain (if any) it would produce depends on the details of the specific CAD algorithms and the degrees of freedom in tuning them. It might even be desirable to adapt CAD tuning differently for each individual reader, automatically or manually.

(b) Evidence of systematic positive and negative effects of CAD. Specific human behaviours that are considered undesirable may be targeted with methods for either avoiding them or correcting their negative effects ("fault tolerance"). We could think of many solutions, whose acceptability in the specific working environment would need to be checked before they are adopted. For example, the cognitive load on readers could be reduced by not repeating prompts on features that readers have already noticed and marked. Similarly, the self-calibration of readers, and their reliance at times on the CAD outputs as a decision aid instead of an attention cue, could be corrected by including in their normal workload fictitious cases with incorrect CAD outputs, rarely but yet frequently enough to refresh their memory of types of possible errors by CAD. Changes could also be made to the CAD tool so that it makes readers' violations of prescribed protocols of use impractical or harder to justify.

(c) Readers' evident concern to explain the presence or absence of prompts. One approach would be to explore how the CAD tool could provide explanations of its behaviour on demand. The challenge would be to produce the sorts of accounts that would be useful to a reader, which calls for an understanding of the sorts of explanations where confusion may arise. The question also exists whether such exacting specifications could be implemented reliably enough not to make the explanation facility a source of further problems.

(d) Readers seemed to be overinfluenced by the absence of prompts. When using CAD, readers are required to pay attention to prompts and not to their absence, but, in reality, they appeared to be doing the opposite. Design changes might be implemented to combat this problem (e.g. by introducing mechanisms that make readers aware of the risks of this behaviour). However, one could argue that the requirement that readers not be influenced at all by the absence of prompts is possibly psychologically impossible to satisfy as well as normatively incorrect, if the absence of prompts has indeed informative value; in this case, there could be merit in seeking to provide readers with a heuristic procedure to follow that would give absence of prompts approximately the right weight in decisions, rather than attempting to have them ignore it altogether.

	6. Concluding remarks

Top Abstract Introduction 1. The HTA trial 2. Supplementary analyses of... 3. Follow-up studies 4. Ethnographic studies of... 5. Discussion 6. Concluding remarks References

 
The overall aim of our work is to examine more closely the effects of decision aids on human judgement and the factors that affect the dependable performance of human–computer systems. We have used CAD as a case study for exploring some of the subtler interactions between human decisions and computer-generated evidence. Our multidisciplinary approach combining quantitative and qualitative methods has helped to detail and understand some of the possible factors underlying the apparent lack of effects of CAD in some trials, and to identify areas for further development. Statistical analyses identified some possibly detrimental effects of CAD on reader performance. These findings were corroborated by the ethnographic studies that provided important insights into the manner in which readers came to use the tool as a part of their work, as well as by follow-up experiments. In this way, the various methods used helped to orient each other and to improve confidence in our findings.

Our multidisciplinary approach has helped to address some of the recognised limitations of the clinical trial as an evaluation methodology for IT-based healthcare interventions. The use of ethnographic observation in particular has helped to put the clinical trial results into a real-world context of use and thereby gain a better understanding of their implications for the everyday work of reading mammograms. We would note also that the introduction of a new technology such as CAD may change work practices and thus, in an iterative process, make new demands on the technology. Evaluation methodologies that do not take into account the influence of adaptation and learning may fail to deliver meaningful results for users and designers alike and fail to contribute to improving the technology. Again, our multidisciplinary approach has enabled us to gain some perspectives on how readers learn and adapt to the behaviour of the CAD tool and has suggested some promising lines of enquiry for how the tool and the procedures for using it may themselves be adapted.

In conclusion, our studies have provided some insight about the effects of CAD on readers' performance, as well as some methodological indications. Both results may be of interest for a larger class of tasks and computer aids than that of CAD for mammography.

	Acknowledgments

 
The work described in this paper has been partly funded by the UK Engineering and Physical Sciences Research Council (EPSRC), through the Dependability Interdisciplinary Research Collaboration (DIRC), a project investigating the dependability of computer-based systems. We would like to thank: DIRC collaborator Mark Rouncefield for his substantial contributions to the ethnographic work summarised here; R2 Technologies Inc. (and very especially Gek Lim, Jimmy Roehrig and Julian Marshall) for their support in obtaining the data samples for our follow-up studies; Paul Taylor and Jo Champness (from UCL) for granting us access to their data, facilitating the follow-up studies and helping run them; all the readers who volunteered to take part in the follow-up studies; and last but by no means least, David Wright for his invaluable feedback on earlier drafts of this paper.

	Footnotes

 
The work described in this paper has been partly funded by the UK Engineering and Physical Sciences Research Council (EPSRC) through the Dependability Interdisciplinary Research Collaboration (DIRC), a project investigating the dependability of computer-based systems.

	References

Top Abstract Introduction 1. The HTA trial 2. Supplementary analyses of... 3. Follow-up studies 4. Ethnographic studies of... 5. Discussion 6. Concluding remarks References

 

1. Taylor PM, Champness J, Given-Wilson RM, Potts HWW, Johnston K. An evaluation of the impact of computer-based prompts on screen readers' interpretation of mammograms. Br J Radiol 2004;77:21–7.[Abstract/Free Full Text]
2. US Food and Drug Administration. Pre-market approval decision. Application P970058. 26 June 1998. http://www.fda.gov/cdrh/pdf/p970058.pdf [accessed 10 November 2004].
3. Littlewood B, Popov P, Strigini L. Modelling software design diversity – a review. ACM Computing Surveys 2001;33:177–208.
4. Strigini L, Povyakalo AA, Alberdi E. Human–machine diversity in the use of computerised advisory systems: a case study. In: Proceedings of DSN 2003, International Conference on Dependable Systems and Networks; 2003 June 22–25; San Francisco, CA. Los Alamitor, CA: Institute of Electrical and Electronic Engineers, 2003:249–58.
5. Povyakalo AA, Alberdi E, Strigini L, Ayton P. Evaluating "Human + Advisory computer" system: a case study. In: Proceedings of the 18th British HCI Group Annual Conference; 2004 September 6–10; Leeds, UK. 2004;2:93–6.
6. Alberdi E, Povyakalo AA, Strigini L, Ayton P. Effects of incorrect computer-aided detection (CAD) output on human decision-making in mammography. Acad Radiol 2004;11:909–18.[Medline]
7. Hartswood M, Procter R, Rouncefield M, Slack R, Voss A. "Repairing" the machine: a case study of the evaluation of computer aided detection tools in breast screening. In: Dourish P, Fitzpatrick G, editors. Proceedings of the 8th European Conference on Computer-Supported Cooperative Work; 2003 September 14–17; Helsinki, Finland. Dordrecht: Kluwer Academic Publishers, 2003.
8. Castellino R, Roehrig J, Zhang W. Improved computer aided detection (CAD) algorithms for screening mammography. Radiology 2000;217(P):400.
9. Warren Burhenne LJ, Wood SA, D'Orsi CJ, et al. Potential contribution of computer-aided detection to the sensitivity of screening mammography. Radiology 2000;215:554–62.[Abstract/Free Full Text]
10. Hartswood M, Procter R, Williams L. Prompting in mammography: computer-aided detection or computer-aided diagnosis? In: Berry E, Hogg DC, Mardia KV, Smith MA, editors. Proceedings of Medical Image Understanding and Analysis, MIUA 98; 1998 July 6–7; Leeds, UK. University of Leeds, 1998:101–4. http://www.cogsci.ed.ac.uk/~rnp/papers/miua-98/final.pdf
11. Agresti A. Categorical data analysis. New York, NY: Wiley, 1990:59–66.
12. Thurfjell E, Thurfjell G, Egge E, Bjurstam N. Sensitivity and specificity of computer-assisted breast cancer detection in mammography screening. Acta Radiol 1998;39:384–8.[Medline]
13. Moberg K, Bjurstam N, Wilczek B, Rostgard L, Egge E, Muren C. Computed assisted detection of interval breast cancers. Eur J Radiol 2001;39:104–10.[CrossRef][Medline]
14. Marx C, Malich A, Grebenstein U, et al. Are unnecessary follow-up procedures induced by computer-aided diagnosis (CAD) in mammography? Comparison of mammographic diagnosis with and without use of CAD. Eur J Radiol 2004;51:66–72[Medline]
15. Brem RF, Schoonjans JM. Radiologist detection of microcalcifications with and without computer-aided detection: a comparative study. Clin Radiol 2001;56:150–4.[CrossRef][Medline]
16. Ciatto S, Del Turco MR, Risso G, et al. Comparison of standard reading and computer aided detection (CAD) on a national proficiency test of screening mammography. Eur J Radiol 2003;45:135–8.[CrossRef][Medline]
17. Skitka LJ. Accountability and automation bias. Int J Human-Computer Studies 2000;52:701–17.[CrossRef]
18. Meyer J. Effects of warning validity and proximity on responses to warnings. Hum Factors 2001;43:563–72.[Medline]
19. Hartswood M, Procter R. Computer-aided mammography: a case study of error management in a skilled decision-making task. Top Health Inf Manage 2000;20:38–54.[Medline]
20. Egglin TK, Feinstein AR. Context bias. A problem in diagnostic radiology. JAMA 1996;76:1752–5.
21. Rutter CM, Taplin S. Assessing mammographers' accuracy: a comparison of clinical and test performance. J Clin Epidemiol 2000;53:443–50.[CrossRef][Medline]
22. Hartswood M, Procter R, Rouncefield M, Slack R. Performance management in breast screening: a case study of professional vision and ecologies of practice. In: Johnson C, editor. Human error and medical work. Journal of Cognition, Technology and Work 2002;4:91–100.[CrossRef]
23. Heathfield H, Wyatt J. Philosophies for the design and development of clinical decision support systems. Methods Inf Med 1993;32:1–8.[Medline]
24. Heathfield H, Buchan I. Current evaluations of information technology in health care are often inadequate. BMJ 1996;313:1008.[Free Full Text]
25. Heathfield H, Pitty D, Hanka R. Evaluating information technology in health care: barriers and challenges. BMJ 1998;316:1959–61.[Free Full Text]
26. Sullivan FM, Pagliari HC, Mitchell ED. Health informatics. London, UK: Royal College of General Practitioners, 2002.
27. Hughes J, King V, Rodden T, Anderson R. Moving out from the control room: ethnography and systems design. In: Proceedings of the ACM Conference on Computer-Supported Cooperative Work. 22–26 October 1994; Chapel Hill, NC. New York, NY: ACM Press, 1994:429–39.

This Article Abstract Figures Only Full Text (PDF) Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Alberdi, E Articles by Slack, R Search for Related Content PubMed PubMed Citation Articles by Alberdi, E Articles by Slack, R