British Journal of Radiology (2004) 77, S126-S132
© 2004 British Institute of Radiology
doi: 10.1259/bjr/17464219
Image filtering techniques for medical image post-processing: an overview
C P Behrenbruch, PhD1,2,
S Petroudi, MSc1,
S Bond, MA1,
J D Declerck, PhD2,
F J Leong, MD, PhD, ARPS2,3 and
J M Brady, PhD, FRS, FREng1
1 Medical Vision Laboratory, Engineering Science, Oxford University, Parks Road, Oxford OX1 3PJ, 2 Mirada Solutions Ltd., 23–38 Hyth Bridge Street, Oxford OX1 2EP, UK and 3 Department of Medical and Molecular Pharmacology, University of California Los Angeles, Los Angeles CA, USA
 |
Introduction
|
|---|
Images from an ordinary consumer digital camera convey information at a wide range of spatial (and temporal) scales and enable the viewer to decompose the image into regions that are uniform in some way (colour, texture, ...), recognize familiar objects, determine spatial relationships between objects, and detect abnormalities (e.g. textural markings on a region expected to be plain). Though modern digital cameras are equipped with low noise electronics and excellent lenses that minimize pin-cushion (and similar) distortions, images also contain noise and artefacts such as red-eye in flash images. Widely distributed software packages such as Photoshop provide a set of "filtering" operations which enable the user to improve the image in some way: from image smoothing (typically local averaging) that removes noise and high frequencies, sharpening that increases high frequency content, contrast stretching, through to specialized algorithms, for example for red-eye reduction. Such image filtering is designed to improve the appearance of an image, relying on the human visual system to disregard any unwanted change of content of the image.
Medical image analysis poses a far tougher challenge. First, there is an even greater need for image filtering, because medical images have a poorer noise-to-signal ratio than scenes taken with a digital camera, the spatial resolution is often frustratingly low, the contrast between anatomically distinct structures is often too low to be computed reliably using a standard image processing technique, and artefacts are common (e.g. motion and bias field in MRI). Second, changes to image content must be done in a highly controlled and reliable way that does not compromise clinical decision-making. For example, whereas it is generally acceptable to filter out local bright patches of noise, care must be taken in the case of mammography not to remove microcalcifications.
This paper briefly explores some of the key areas of development in the area of filtering in Medical Imaging and how these techniques impact generally available software packages in routine use in a diagnostic setting. It is interesting to note that a great deal of image filtering takes place at what is usually regarded as a "pre-processing" stage in the formation of a medical image and is relatively invisible to a radiologist. However, there is an increasing awareness of the impact of post-processing algorithms – particularly filtering – in diagnostic software applications and an awareness of these types of techniques is useful.
|
Noise equalization
|
|---|
All imaging modalities, but especially those that are relevant for medical imaging, generate image noise, whether due to stability of a low-flip angle MRI acquisition [1], ultrasound speckle [2], quantum noise in an X-ray [3] or out of field counts in a PET scan [4]. Virtually all imaging systems also perform filtering on the image acquisition data both at an electronic level prior to reconstruction as well as during the image reconstruction phase. Indeed, much recent advancement in reconstruction techniques for 3D imaging focus on including noise removal as part of the reconstruction optimization process [5].
Most initial attempts at removing image noise focus on "smoothing" the pixel or voxel data by performing some sort of local averaging function. For example, Gaussian smoothing is an easily implemented smoothing algorithm; however it is clearly not desirable to locally smooth a data set in all cases (effectively removing high-frequency and highly spatially localized image components). For example, as we noted above, a mammographic X-ray is only diagnostically valuable if the resolution and spatial accuracy is sufficient to capture attenuation due to microcalcifications. Therefore increasingly "smart filters" based on techniques such as anisotropic diffusion [6], which smoothes the image to different extents in the direction of the intensity gradient (across a boundary) and along the boundary, or wavelets [7] (a standard but highly mathematical reference) or [8] (an easily understood introduction to the Matlab wavelet toolkit) are very useful because they can remove noise from an image while recognizing that certain noise-like components need to be preserved. In this way the entire fields of image processing and computer vision open up to yield interesting techniques for embedding knowledge of anatomy, tissue characteristics and the physics of imaging into the filtering process [9].
By way of example, the images in Figure 1
show a small segment of an X-ray mammogram that has been digitized at 50 µm and which contains two small clusters of calcifications and a vessel. It is evident that the image is extremely noisy: the visual impact of the noise being accentuated by visualizing the image as a surface (height equates to brightness). By simply performing an iterative local averaging or smoothing process [10], the overall structure of the image fragment becomes clearer. However, the precise locations of the features in the image are poorly preserved, essentially because the filter cannot discriminate between what is high-frequency noise and what is a highly spatially localized (and therefore also noise-like) "spike" of feature which is a calcification. A "smart" filter, based on a suitable wavelet, and which is matched to the expected shape of a microcalcification has the effect of removing the noise, by some local averaging; but, when a calcification is encountered, the image structure is better preserved [11].
View larger version (27K):
[in this window]
[in a new window]
|
Figure 1. (a) A small section of a 50 µm mammogram with microcalcifications and a vessel visible. (b) The unprocessed image displayed as a surface map. (c) The filtering of the image segment using diffusion (smoothing) techniques [10] with (d) showing the benefit using a more selective filtering approach such as a wavelet [7] which has better structure preservation.
|
|
As radiology becomes overwhelmingly digital for all modalities, including those which have traditionally been film-based, clinicians will need to have a deeper understanding of the relationship between the imaging process and the display of the image. A lot of filtering and post-processing is performed either to enhance visual characteristics of images or to make an image more quantitative. Understanding these techniques and their impact on the image characteristics is important for good decision-making.
|
Bias field correction
|
|---|
In the previous example of filtering for "de-noising" an image, the high-frequency ("spiky") parts of the image are removed. This type of artefact removal is the one that is most commonly thought of as being an image filtering step, but it is important to recognize that "filtering" has much broader applicability.
A good example of a filtering technique that is, in a certain sense, at the opposite extreme from noise filtering is MRI bias field correction. Small variations in the magnetic field introduced by the radiofrequency (RF) system (the B1 field introduce slowly undulating (low frequency) inhomogeneities in the image which can be visually distracting), can impact the textural significance of an area, and because they can substantially reduce the contrast in different image regions, is a barrier to using any kind of segmentation or region delineation tool which is based on thresholding (assigning all of those voxels above a fixed intensity to a particular tissue class). In this case, filtering aims to remove a low-frequency component to the image, rather than predominantly high frequency noise as in the previous section, as Figures 2 and 3.
View larger version (86K):
[in this window]
[in a new window]
|
Figure 2. An example of bias field correction using "smart filters" that can detect inhomogeneities in the image [12, 13]. The left image shows an MRI slice of the colon with clear bias field artefacts. The right image has been significantly improved, both for visualization and the application of quantitative and computer-aided techniques.
|
|
View larger version (43K):
[in this window]
[in a new window]
|
Figure 3. Bias field corrected images (refer to Figure 2 ) can have automated algorithms successfully applied to reconstruct the shape of the colon and provide an important starting point for computer-aided detection algorithms.
|
|
In this method, a filtering approach is used to estimate an intensity correction distribution, which is applied to the image [12, 13]. This is also an example of a "smart filter" because in order to work robustly, the filter needs to include knowledge that the magnetic susceptibility of tissue varies, depending on the tissue type. Therefore, in order for the filtering algorithm to work properly, the different tissue components in the image need to be identified and a series of separate filters for each tissue class created.
Bias field correction is an example of a fundamental image processing step which is critical to building computer-aided tools for aiding diagnosis – such as computer-aided detection (CAD) algorithms [14] and three-dimensional (3D) visualization tools which can selectively segment anatomy for improved display and interpretation. A good example is the automated segmentation of the colon for building tools to help detect the presence of polyps. In this example, an automated contouring algorithm looks for the intensity transition of the colon so that cleverer feature detection algorithms have a more focused search region in which to look for textural inconsistencies in the colonic volume. Equally importantly, focusing the search ensures that any features detected will be in precisely the part of the image that is of interest to the clinician, avoiding the sharp reduction in the clinician's confidence were a feature to be reported in an image region that is not anatomically relevant. Just the same way as bias fields make diagnosis more difficult for a radiologist, algorithms (which are a lot less intelligent than radiologists...) are also confused, so this correction is critical.
|
Visualization-driven filtering
|
|---|
Visual filters are likely to be the filters that are most familiar to the reader. "Low Pass", "High Pass", "Edge Enhancement", BNR (noise reduction) are commonly available operator options in most radiology software packages. Adjusting image sharpness and contrast can be very valuable for differentially discerning abnormalities in an image. The challenge for software designers is to make these filtering actions fast and as interactive as possible.
Perhaps less obvious to the user of a radiological software package who may not be expert in image analysis is the relationship between image interpolation and filtering. The reader may be familiar with terms such as a "nearest neighbour" or "bi-cubic" interpolation. These techniques are used to make low resolution data or highly zoomed images appear smother and more consistent. These interpolation techniques are effectively filtering operations which generally provide a better visual effect but which also dramatically change the spatial characteristics of the image data. Indeed, the true voxel accuracy of an image can be deceptive when highly interpolated. This is illustrated in Figure 4 for a neurological PET image.
View larger version (34K):
[in this window]
[in a new window]
|
Figure 4. Most radiologists do not consider different interpolation techniques as a filtering process, but it clearly is. The magnified image segments illustrate (from left to right) "nearest neighbour", linear and cubic spline interpolation, respectively. These images importantly illustrate how many basic visualization techniques can dramatically impact the presentation – and misrepresentation – of the true image data.
|
|
When performing filtering on top of an interpolated image, one is effectively performing a sequence of two or more filtering operations. Users of imaging software need to be aware that it may be better to perform filtering on uninterpolated data than on the results of highly interpolated data – especially if the data is low resolution. In the example below, a low resolution therapy planning CT is interpolated using cubic spline interpolation (Figure 5). The resolution and the intensity transitions of the skull cause (Gibbs "ringing") artefacts that are visually disruptive.
View larger version (49K):
[in this window]
[in a new window]
|
Figure 5. (A) A low quality therapy planning CT with typically thick slab profile and hence poor through-plane resolution. (B) and (C) illustrate the same volume with linear and spline interpolation, respectively. In this example, the spline interpolation (usually considered to be the best method) actually generates a poorer quality re-sampled image with "ringing" artefacts clearly visible as shown in the magnified image D. Although a technical example, illustrated for clarity, caution is relevant to many filtering and re-sampling functions.
|
|
|
Physics-based filtering
|
|---|
Often when deciding upon a filtering strategy to improve image quality it is important to consider the fundamental physics of the acquisition process. For example, nuclear medicine and PET scanners utilize scatter and out-of-plane count simulation to build models that can sharpen picture quality. Although the model is driven by the physics and the engineering of the imaging system, the computation of the image enhancement is essentially a smart filter. In fact, physics-based filtering is a really key aspect of building a complete imaging system because sometimes it may be more effective to model an image degrading process and correct it in software, than to change hardware or acquisition protocol.
A good example of this is scatter correction in digital mammography. An antiscatter grid is used to provide a degree of collimation so that image quality is sharpened (scatter has the effect of blurring an X-ray image). The trade off is that the grid attenuates the X-ray beam, so to compensate the amount of radiation exposure to the patient is increased. The challenge of building a directly digital system which balances image sharpness with radiation dose is one that is moving strongly into the domain of software, where scatter correction can be partially performed in both hardware and software to optimize both aspects. An example of image sharpening through software-based scatter correction for X-ray mammography is illustrated in Figure 6 [9].
View larger version (85K):
[in this window]
[in a new window]
|
Figure 6. A very nice example of physics-based filtering where a model of the image formation process of an X-ray mammography image is used to construct a special filter which can improve the noise and spatial representation of microcalcification clusters [9]. The top insert image shows how scatter and the spectral characteristics of the X-ray source have blurred the image, almost completely masking the calcifications (indicative of ductal carcinoma). The bottom insert image shows a vast improvement in the quality of the appearance of the microcalcifications.
|
|
|
Texture analysis
|
|---|
In normal English usage, the word "texture" refers to a pattern of colour, shading, or material changes on a surface. Some textures are geometrically regular, for example a check pattern. Some textures are primarily "statistical" in nature, for example a carpet weave of slightly different colours, in the sense that there is no obvious geometrical regularity to the pattern. Most often, textures have both a geometrically regular component and a statistical component: for example, Harris Tweed or a raffia pattern. In image analysis, the word texture is generalized to refer to regions of images that are the projections of spatially distributed "entities", for example a lawn, a pebbled path, or the foliage of a tree.
The human visual system is remarkable in many ways; but not least in its ability to discern regions of images that are homogeneously textured. We can segment images in to textured regions that we do not recognize as being other than "textures", and we can recognize effortlessly regions that correspond to textures (grass, leaves, pebbles, ...) that we know well. Unfortunately, it has proven almost impossible to date to develop mathematical and computational models that approach human competence in texture recognition and segmentation. Various models have been developed that attempt to capture the idea of "repetition", at least in a statistical sense, based essentially on estimating the local image autocorrelation. Other models have been developed in the frequency domain – again attempting to capture the notion of regular repetition. Since a texture is localized both in the spatial domain and in the frequency domain, it is natural to turn to recently developed mathematical tools such as wavelets [7, 8]. One particular mathematical model of a texture is based on the concept of a fractal – a shape (or surface) which is self similar at all scales (or nearly so) and whose dimension is fractional [15]. Despite significant effort over the past 30 years, we still lack a mathematical or computational model of textures that works well in general: rather, we have a stock of techniques that work well for classes of textures and/or applications.
Most recently, attempts have been made [16–18] to learn texture models from a set of training examples. A computer system is presented with dozens of instances of a texture, all of which are examples of the texture of interest, but under different illumination, orientation, and other typical variations, applies a bank of filters and from the responses, builds a "texton" model that enables subsequent recognition of texture instances. Initial results have been encouraging (Figure 7).
View larger version (45K):
[in this window]
[in a new window]
|
Figure 7. Classification of a mammogram into Wolfe Patterns using textons. (a) A mammogram belonging to Wolfe class N1. (b) The texton "labelled" image of the mammogram. The texton histogram corresponding to this image is used to classify the image into the pattern Wolfe N1. This type of classification forms the basis of "next generation" computer-aided detection (CADe) and diagnosis (CADi) algorithms for digital mammography.
|
|
Filtering a medical image to facilitate texture classification or segmentation has broad application. For example, new form CJD tends to be accompanied by a characteristic texture on the surface of the cortex. In general, abnormal patterns, such as architectural distortions, are often the first sign that pathology may be present. One instance of this is again in mammography, an idea first promulgated by Wolfe and more recently refined in the BIRADS classification of dense breast tissue in a mammogram (or the six category classification). Petroudi et al [18] have shown that the recently developed idea of learning texture classifications can be applied to mammography and have trained a system to discriminate between the Wolfe classes with very encouraging results.
|
Feature detection and tracking
|
|---|
The brief discussion about texture analysis in the previous section introduced the idea that a collection of filters can be created to look for different types of image characteristics and classify an image accordingly. This has obvious applications in computer-aided detection as well as a being an important algorithmic component in a lot of imaging applications, which are designed to optimize the workflow around measuring things (such as accurately delineating blood vessels in an angiography study).
The use of "libraries" of filters to track features over time is particularly important in the case of dynamic imaging or longitudinal studies. Most often, such feature tracking is intended to help automate the measurement of a change in a feature – such as a tumour volume in response to chemotherapy. Filtering for feature tracking also has application in studies where specific types of feature motion or change is indicative of pathology. A good example is cardiac ultrasound where endocardial and epicardial wall motion can be used to highlight ischaemic regions of the heart [19, 20]. In order to build such wall motion analysis tools, the image features corresponding to the myocardium must be filtered from ultrasonic speckle noise over successive imaging frames (Figure 8) [1].
View larger version (86K):
[in this window]
[in a new window]
|
Figure 8. This figure illustrates how feature detection across a series of image frames can be used to estimate the motion of the heart. In the set of images (a), the shape and location of the endocardium and epicardium is estimated from image features over successive frames and then used to drive a deformation model [19, 20]. (b) The wall motion kinetics and thickening measurements between the sets of contours can be used to create a map of the "activity" level of the myocardium which correlates well with other modalities [20].
|
|
As a general note, ultrasound is a modality that is heavily dependent on filtering for optimal clinical use, both as a visual post-processing step as well as for building quantitative applications. A tremendous amount of filtering is used to control the visualization of harmonics and even techniques such as "colour flow" or Doppler imaging are dependent on the good separation of temporal and spatial echo returns – itself a filtering operation.
|
Conclusions
|
|---|
The field of medical image analysis is extremely broad and filtering is a fundamental aspect of virtually every implementation of an image analysis and visualization solution. Filtering is a system component of all medical imaging modalities and as illustrated in this brief paper, a basic part of many diagnostic applications.
This overview paper has tried to address some of the misconceptions about medical image post-processing and the idea that filtering is somehow about reducing the diagnostic value of an image. It is true that filtering can be inappropriately used and can have a negative impact on the diagnostic content of an image (a few simple examples were shown). However, "smart" filters can significantly aid diagnosis by highlighting regions and accentuating image characteristics, which may be lost in the enormous complexity of a medical image.
Finally, an important consideration is the fact that although imaging algorithms such as filters are generally considered to be part of the post-processing domain, a tremendous amount of filtering takes place before the image is even presented to a radiologist. To this end, a greater awareness of how an image is processed prior to display is increasingly important as radiology becomes digital. By extension, an awareness of the capabilities of image analysis techniques such a filtering, may drive a greater acceptance of the role of computer-aided radiology.
|
References
|
|---|
- Hashemi RH, Bradley WG. MRI the basics. Lippincott Williams and Wilkins, 2002.
- Wagner RF. Statistics of speckle in ultrasound B-scans. IEEE Trans Sonics Ultrasonics 1983;30:156–63.
- Highnam RP, Brady M, English R. Detecting film-screen shot noise in mammography using a model-based approach. IEEE Trans Med Imaging 1999;18:1016–24.[Medline]
- Cherry SR, Sorenson JA, Phelps ME. Physics in nuclear medicine. W B Saunders, 2003.
- Qi J, Leahy R. Resolution and noise properties of MAP reconstructions in fully 3D PET. IEEE Trans Med Imaging 2000;19:493–506.[Medline]
- Weickert J. Anisotropic diffusion in image processing, ECMI Series. Stuttgart, Germany: Teubner-Verlag1998.
- Mallat S. A wavelet tour of signal processing. Academic Press, 1998.
- Wavelet Toolbox User's Guide (Matlab), The Mathworks, 1997–2000.
- Brady M, Highnam R. Mammographic image analysis. Kluwer Series on Medical Image Understanding. Kluwer, 1999.
- Forsyth DA, Ponce J. Computer vision: a modern approach. Prentice Hall, 2003.
- Behrenbruch CP. Image fusion for the management of breast cancer. Doctoral Thesis, Department of Engineering Science, University of Oxford, 2002.
- Guillemaud R, Brady M. Estimating the bias field of MR images. IEEE Trans Med Imaging 1997;16:238–51.[Medline]
- Styner M. Parametric estimate of intensity inhomogeneities applied to MRI. IEEE Trans Med Imaging 2000;19:153–65.[CrossRef][Medline]
- Hayton P, Brady M, Smith S, Moore N. A non-rigid registration algorithm for dynamic breast MR images. Artificial Intelligence 1999;114:125–56.
- Dubuisson MP, Dubes RC. Efficacy of fractal features in segmentation images of natural textures. Pattern Recognition Letters 1994;15:419–31.[CrossRef]
- Leung T, Malik J. Representing and recognizing the visual appearance of materials seeing three-dimensional textons. Int J Computer Vision 2001;43:29–44.[CrossRef]
- Varma M, Zisserman A. Classifying images of materials: achieving viewpoint and illumination independence. Proceedings of the European Conference on Computer Vision, Copenhagen, Denmark. 2002;255:271.
- Petroudi S, Kadir T, Brady M. Automatic classification of mammographic parenchymal patterns: A statistical approach. In: Proceedings of EMBC, International Conference on Engineering in Medicine and Biology. IEEE, 2003:798–801.
- Jacob G, Noble JA, Behrenbruch C, Kelion AD, Banning AP. A shape-space-based approach to tracking myocardial borders and quantifying regional left-ventricular function applied in echocardiography. IEEE Trans Med Imaging 2002;21:226–38.
- Boukerroui D, Noble JA, Robini MC, Brady M. Enhancement of contrast regions in suboptimal ultrasound images with application to echocardiography. Ultrasound Med Biol 2001;27:1583–94.
Top
Introduction
Noise equalization
