BJR
HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
This Article
Right arrow Figures Only
Right arrow Full Text (PDF)
Services
Right arrow Related articles in BJR
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Halligan, S
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Halligan, S
British Journal of Radiology 75 (2002),193-194 © 2002 The British Institute of Radiology

Correspondence

Reproducibility, repeatability, correlation and measurement error

The Editor—Sir,

Gillard and colleagues claim to have determined the reproducibility of CT measurements of cerebral perfusion, stating there is "little variability in absolute values for cerebral blood flow in individuals studied 24 h apart" [1]. This is based upon finding positive correlation between repeated measurements on the same individuals. The title of the article cites reproducibility, which is defined as the value below which the difference between two single test results, obtained under different conditions, may be expected to lie with a specified probability [2]. More simply defined, reproducibility is the strength of agreement between the replicates [3]. Repeatability refers to the strength of agreement between repeated measurements obtained under similar circumstances [2]. Unfortunately, the Pearson correlation coefficient does not measure agreement, but simply determines the strength of linear association between two variables, which is quite different.

Perfect correlation occurs when a scatter plot of the first variable against the second results in a straight line [4] (see Figure 1Go). When the first and second variables are replicate measurements using the same technique, as in this study, there will evidently be perfect correlation if the second measurement equals the first, as would happen with perfect reproducibility or repeatability (Figure 1Go, dataset A). However, what happens when the second measurement is exactly twice the size of the first? Again there would be perfect correlation, implying perfect reproducibility using the authors' logic, but this time with a measurement error of 100% (Figure 1Go, dataset B)! The same comment applies if the second measurement is one-tenth or ten-times the first (Figure 1Go, dataset C, D) and so on as long as a linear relationship between the variables is maintained. Furthermore, samples of subjects who differ widely will tend to produce higher correlation coefficients than those who do not [3]. Clearly, simple correlation is a poor tool with which to assess reproducibility [4].



View larger version (10K):
[in this window]
[in a new window]
  		Figure 1. Scatter graph of four datasets of repeated measurements. The first measurement is 10, 20, 30, 40, 50 in all cases. In dataset A ({blacksquare}) the second measurement equals the first, in dataset B ({blacktriangleup}) the second measurement is exactly twice the first, in dataset C ({blacktriangledown}) the second measurement is exactly one-tenth the first and in dataset D ({blacklozenge}) the second measurement is exactly ten-times the first. In all cases, Pearson's correlation coefficient (r) is 1.

 
A more appropriate approach would be to determine the level of measurement error between repeated values. Measurement error is defined as the variation between measurements of the same quantity on the same individual [5], which is clearly the focus of this article. Obviously, repeated measurements on the same subject will tend to differ because of variation in the subject (in this study subjects were examined 24 h apart) or variation in the technique (the focus of the authors' investigation). Repeated measurements will vary from the "true" measurement (which we do not know—if we did, the statistical problem would be one of calibration) because of measurement error. Therefore, the standard deviation of repeated measurements provides an estimate of measurement error; the within-subject standard deviation [3, 5]. This is simplified when there are only two measurements per subject, as there were in Gillard et al's study, since their variance is half the square of their difference. 95 per cent of the time, the difference between a measurement and the true value will be less than 1.96 times the measurement error. Alternatively, the measurement error can be presented as the "repeatability", defined as 2.77 times the measurement error [1, 5].

Inappropriate use of simple correlation to assess agreement remains common in medicine, despite being well recognized [6]. It seems especially important to subject the data to more vigorous analysis when subjects have twice been exposed to irradiation and iv contrast medium.

Yours etc.,

S Halligan

Intestinal Imaging Centre, St. Mark's Hospital, London, UK

Received for publication September 11, 2001. Accepted for publication September 17, 2001.

References

  1. Gillard JH, Antoun NM, Burnet NG, Pickard JD. Reproducibility of quantitative CT perfusion imaging. Br J Radiol 2001;74:552–5.[Abstract/Free Full Text]
  2. British Standards Institution. Precision of test methods, part 1: guide for the determination of repeatability and reproducibility for a standard test method. BS 5497, Part 1. London: BSI, 1979.
  3. Bland JM, Altman DG. Measurement error and correlation coefficients. BMJ 1996;313:41–2.[Free Full Text]
  4. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986;i:307–10.
  5. Bland JM, Altman DG. Measurement error. BMJ 1996;313:744–53.[Free Full Text]
  6. Murray GD, Miller R. Statistical comparison of two methods of clinical measurement. Br J Surg 1990;77:385–7.[Medline]

Related articles in BJR:

Authors' reply
J H Gillard, N M Antoun, N G Burnet, and J D Pickard
BJR 2002 75: 194-195. [Full Text]  



This article has been cited by other articles:


Home page
 JNMHome page
L. M. Velasquez, R. Boellaard, G. Kollia, W. Hayes, O. S. Hoekstra, A. A. Lammertsma, and S. M. Galbraith
Repeatability of 18F-FDG PET in a Multicenter Phase I Study of Patients with Advanced Gastrointestinal Malignancies
J. Nucl. Med., October 1, 2009; 50(10): 1646 - 1654.
[Abstract] [Full Text] [PDF]

Home page
 Am. J. Neuroradiol.Home page
S. Bisdas, K. Spicer, and Z. Rumboldt
Whole-Tumor Perfusion CT Parameters and Glucose Metabolism Measurements in Head and Neck Squamous Cell Carcinomas: A Pilot Study Using Combined Positron-Emission Tomography/CT Imaging
AJNR Am. J. Neuroradiol., August 1, 2008; 29(7): 1376 - 1381.
[Abstract] [Full Text] [PDF]

Home page
 RadiologyHome page
V. Goh, S. Halligan, and C. I. Bartram
Quantitative Tumor Perfusion Assessment with Multidetector CT: Are Measurements from Two Commercial Software Packages Interchangeable?
Radiology, March 1, 2007; 242(3): 777 - 782.
[Abstract] [Full Text] [PDF]

Home page
 RadiologyHome page
M.-P. Revel, A. Bissery, M. Bienvenu, L. Aycard, C. Lefort, and G. Frija
Are Two-dimensional CT Measurements of Small Noncalcified Pulmonary Nodules Reliable?
Radiology, May 1, 2004; 231(2): 453 - 458.
[Abstract] [Full Text] [PDF]

Home page
 RadiologyHome page
M.-P. Revel, C. Lefort, A. Bissery, M. Bienvenu, L. Aycard, G. Chatellier, and G. Frija
Pulmonary Nodules: Preliminary Experience with Three-dimensional Evaluation
Radiology, May 1, 2004; 231(2): 459 - 466.
[Abstract] [Full Text] [PDF]

This Article
Right arrow Figures Only
Right arrow Full Text (PDF)
Services
Right arrow Related articles in BJR
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Halligan, S
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Halligan, S

HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
BJR DMFR IMAGING  ALL BIR JOURNALS