British Journal of Radiology (2004) 77, S108-S113
© 2004 British Institute of Radiology
doi: 10.1259/bjr/45222871
Navigating and visualizing three-dimensional data sets
N W John, PhD1 and
R F McCloy, MD, FRCS2
1 School of Informatics, University of Wales, Bangor LL57 1UT and 2 University of Manchester, Oxford Road, Manchester M13 9PL, UK
 |
Abstract
|
|---|
This paper provides an overview of the main techniques being used for three-dimensional (3D) visualization of medical data sets and highlights some of the clinical benefits that can be obtained. One of the major advantages of using a 3D representation is that all of the slice data produced by the latest multislice CT and high gradient MR scanners can be utilized, and then presented to the clinician in an intuitive format. Continued advances in technology mean that high resolution 3D representations of patient specific anatomy can now be routinely obtained and so provide valuable input to diagnosis, planning and navigation tasks. Examples from these areas are presented and illustrated below. Future developments and possibilities are also discussed.
|
Introduction
|
|---|
The latest generation of medical scanners, such as multislice/multidetector-row CT or high gradient MR scanners, are capable of producing large numbers of high resolution images in fast acquisition times. Series of 512 images or more is becoming the norm. Soon multislice scanners will be delivering thousands of data slices routinely, leading to problems for data servers and viewing the data sets. The basic light box display environment is no longer an efficient tool for the display and analysis of this many images as typically only 60 can be displayed at one time. Although a skilled radiologist will make an accurate diagnosis using a light box environment, it is preferable to make all the data available at the same time. This goal can be achieved by converting the image set into a three-dimensional (3D) representation. The use of 3D can also be a more intuitive format for the non-radiological specialist such as surgeons, and also produce more accurate measurements of, for example, the change in size of a tumour due to response to therapy. Experts in the field are already predicting efficiency gains and benefits to patients through the use of 3D techniques [1].
Work has been ongoing for over 20 years to provide high quality 3D visualizations of medical data. The scanner manufacturers all provide workstations today that support 3D and such workstations are becoming faster and the software more sophisticated. The two common techniques used for the 3D visualization of medical data are surface extraction and volume rendering, and these are described in the next section. Navigation of 3D data sets is then discussed, and the use of virtual environments for diagnosis and planning is introduced. The paper ends with example clinical examples that highlight the benefits that can be obtained through the uses of 3D.
|
Visualization techniques
|
|---|
The aim of volume visualization is to realistically represent and analyse volume data (often referred to as voxel data). There are two basic classes of volume visualization algorithms in use today:- Surface extraction algorithms
- Direct volume rendering algorithms
These techniques allow the user to see the internal structure and topology of the volume data. In fact, it was the medical field that was the first to exploit volume visualization and the earliest clinical applications were reported in the late 1970s on the visualization of bone from CT in craniofacial surgery and orthopaedics.
Surface extraction
The key idea of surface-based rendering methods is to extract an intermediate surface description of the relevant objects from the volume data. Only this information is then used for rendering. One of the first techniques developed was contour tracking. Here the user is required to trace the contour representing the region of interest on each 2D slice. A triangular mesh is then constructed between the stack of contours so formed. This technique is prone to inaccuracies, however, particularly if the resulting 3D surface contains branching structures.
The most popular surface extraction technique in use today is the Marching Cubes algorithm [2, 3], or one of its derivations. The algorithm produces an isosurface representing the locations of a certain intensity value in the data. Depending on whether a voxel is inside the object (i.e. above a threshold value), a surface representation of up to four triangles is placed within the cube. The exact location of the triangles is found by linear interpolation of the intensities at the voxel vertices. The result is a highly detailed surface representation. The surface orientations needed for shading are calculated from grey level gradients. Figure 1
shows an example of such a surface. It has been generated by endovascular surgical planning software developed at the Manchester Visualization Centre (MVC) to meet the specific needs of interventional neuroradiologists evaluating the suitability of intracranial aneurysms for endovascular coiling [4]. The choice of threshold value is extremely important for extracting the required surface. In this case, low values are required to obtain as much vascular structure as possible but noise in the original scanner data can cause the creation of a large number of artefacts that may obscure the aneurysm. Filtering algorithms can be used to reduce noise – in this case a 3D tangential smoothing filter is effective. We have also used a simpler method for removing unwanted isosurface objects based on their size (number of triangles); a recursive search through the arrays of vertices and triangles is made, building up a list of topologically separate objects. The clinician then selects which of the isosurface objects are needed to be displayed; these will be the largest objects at the head of the list.
View larger version (110K):
[in this window]
[in a new window]
|
Figure 1. Surface extraction generated from time-of-flight MR angiography data set of a patient suffering from a brain haemorrhage. The threshold value is selected using the histogram tool.
|
|
An extensive overview of surface rendering is given by Schreyer [5].
Direct volume rendering
With volume rendering, images are created directly from the volume data, and no intermediate geometry is extracted. All of the grey level information originally acquired during the scanning process is maintained. This makes it an ideal technique for interactive data exploration. Threshold values and other parameters that are not clear from the beginning can be changed interactively. Furthermore, volume-based rendering allows a combined display of different aspects such as opaque and semitransparent surfaces, cuts and maximum intensity projections (MIPs).
A voxel data set is often treated as a set of different coloured transparent gels and each voxel is assigned a colour value and a transparency value. A segmentation step is needed to classify and shade the voxels according to some opacity transfer function or look up table, for example, Drebin [6] developed a probabilistic (fuzzy) scheme to classify air, fat, tissue, and bone types in CT data. This method relies on the fact that not more than two tissue type distributions overlap. A voxel therefore contains one of seven possibilities: air; air and fat; fat; fat and tissue; tissue; tissue and bone; bone. Each voxel is assigned a material percentage, either directly or by using probabilistic classification, e.g. the maximum-likelihood classifier. Given any material property and the material percentage models, a composite model for that property is calculated by multiplying the percentage of each material by the property assigned to each material. One of two basic scanning strategies for traversing the volume will then be used:
- Feed backward projection or image order traversal. The pixels in the image plane are traversed and imaginary rays are cast through each pixel into the volume. The path of the ray determines the pixel value, which is the composite of the intensity values from back to front for all voxels along each ray. Levoy [7] proposed the original ray casting algorithm. There have been many improvements and variations since.
- Feed forward projection or object order traversal. The data volume is traversed and each voxel in the volume is projected onto the image plane. The splatting algorithm [8] is a good example of such a technique.
Figure 2 is another example from MVC's endovascular surgical planning software, but here ray casting has been used to provide a 3D representation of the aneurysm and surrounding blood vessels. Note that as this example is using time-of-flight MR angiography (MRA) data, the transfer function can be chosen so that flow speed can be visualized by the volume rendering. By reducing the opacity of the slow flowing blood the observer can identify the faster flowing regions which likely act as the inflow jet to the aneurysm.
View larger version (88K):
[in this window]
[in a new window]
|
Figure 2. Volume rendering generated from time-of-flight MR angiography data set of a patient suffering from a brain haemorrhage. The transfer function is described using the colour-map editor. Red is used to indicate regions of faster blood flow.
|
|
There are numerous algorithms in use today to efficiently apply volume rendering techniques. Voxar (Edinburgh, UK) 3D is a good example of a fast software solution that is highly optimized for a PC. Dedicated hardware such as the VolumePro PCI card from TeraRecon, Inc. (San Mateo, USA) can also be used to provide increased performance [9]. Another hardware approach is volume slicing, which uses texture mapping hardware [10]. Originally available only on SGI workstations, this technique is now supported by commodity PC graphics cards.
A more complete overview of volume rendering with details of the algorithms used can be found in [11].
|
Navigation techniques
|
|---|
A virtual environment (VE) is a collection of technologies that allow people to interact efficiently with 3D computerized databases in real time using their natural senses and skills. The potential to use a VE in the medical field is significant [12, 13]. The ability to efficiently interact with and navigate 3D volume visualizations are key requirements of a successful medical VE, and use of stereoscopic viewing and/or haptic interfaces can also enhance the immersive experience.
A straightforward but nevertheless effective system has been developed at the University of Manchester for allowing a surgeon to interact with 3D volume renderings of patient specific data whilst in the operating theatre [14]. The system is called Op3D, and makes use of texture mapping hardware on a high performance computer to deliver the volume rendered data to a laptop client at interactive rates (Figure 3). It is important that the software is easy to use in an operating theatre environment and does not provide any distraction. Therefore, no menus are used and all interaction can be performed simply with a button click on the joystick. We studied the way in which the surgeon would interact with the 3D data and found that it invariably followed a regular pattern: he would rotate the volume; change transparency settings; zoom into an area of interest; and then manipulate a clip plane to look inside the volume. Clip plane manipulation involves rotation of the clip plane, translation of the clip plane and rotation of the volume currently being displayed. An "interaction profile" of the surgeon emerged and was encoded and stored in a text file. The profile is automatically loaded into Op3D at start up and used to control the state changes within the software. More advanced navigation techniques are being developed as part of ongoing work.
View larger version (125K):
[in this window]
[in a new window]
|
Figure 3. The University of Manchester's Op3D system. An inexpensive "computer games" joystick is being used to interact with the 3D data. The joystick is easy to use and can be made sterile by simply enclosing it within a sterile bag.
|
|
Virtual endoscopy is an excellent example of an application now offered by all of the major scanner manufactures that requires both 3D visualization and sophisticated navigation techniques. Clinically, it is proving to be valuable for screening and is already being applied to colonoscopy, ventriculoscopy, angioscopy and bronchoscopy [15]. The patient does not have to go through the uncomfortable optical endoscopy procedure as the 3D anatomy is reconstructed from a medical scan. Not only is a virtual endoscopy cheaper than the optical procedure, but it is not invasive and avoids potential complications and even mortality, and has the added advantage that the clinician can inspect areas behind any entity of interest. Of course, it is not possible to take a physical biopsy for histopathology but "virtual" biopsies can interpret the X-ray characteristics within a suspicious lesion.
Both surface extraction and volume rendering techniques can be used to reconstruct the anatomy (e.g. a colon) being examined – see Bartz [16] for a good technical overview. Figure 4 is using a volume rendering technique and a large polyp can be clearly seen within the reconstructed colon. Note that the virtual camera will often be inside the organ of interest and so a perspective projection must then be used to create the 3D view as this prevents both distortion and objects from being obscured. In addition, there must be enough tissue contrast to allow for shading of the 3D reconstruction. For CT, the ideal image source would have approximately a 200 Hu attenuation contrast. Figure 4 is an example image from a virtual colonoscopy.
View larger version (84K):
[in this window]
[in a new window]
|
Figure 4. Voxar Colonscreen 3D colour volume rendering showing a large polyp identified by the endoluminal orthographic view. Data courtesy of Hope Hospital, Manchester, UK.
|
|
Navigation through the inside of the 3D anatomy is often achieved by using a planned or automatic system, whereby the clinician specifies a camera path and an animated fly-through is generated offline. The clinician can then play through the animation. Interaction is limited, however, as if it transpires that the structure of interest is not well covered, the only options are to generate a completely new animation sequence, or to make sure that the animation path visits all structures (even irrelevant ones) [17]. Free navigation is the alternative approach, but this too has problems. The anatomical structures are frequently complex and even a trained clinician can find it difficult to navigate to the target. Computer resources also increase, particularly if collision avoidance is required. These technical problems are being solved with better algorithms and faster, cheaper hardware becoming available all of the time.
Another vital area requiring both 3D and advanced navigation tools is surgical navigation [18]. Surgical instruments are tracked and their positions are registered within the frame of reference of the 3D reconstruction. The exact position of internal probes whose tips are not directly visible can then be established. The majority of surgical navigation applications to date address neurosurgery where it is possible to keep the patient's head in a fixed position and there is little internal movement of the anatomy. Other surgical specialities requiring minimally invasive procedures will benefit. The major problem is to dynamically create 3D reconstructions of anatomy that is moving during the procedure. Use of fast intraoperative imaging techniques such as ultrasound will be needed. This aspect of surgical navigation is an active research area.
|
Clinical benefits
|
|---|
With fast software and/or hardware now available for 3D visualization, clinical use is set to increase dramatically. This section highlights just some of the benefits of 3D.
Figure 5 is an example of a 3D image generated from 1000 contrast enhanced CT images of the upper abdomen. Viewing in axial mode would have been unwieldy and time consuming. The 3D volume renderings clearly show the horseshoe kidney with triple arterial supply, and a large tumour arising from the collecting system or right hemikidney associated with partial hydronephrosis. Such representations of the data are invaluable for the pre-operative evaluation of this renal mass.
View larger version (128K):
[in this window]
[in a new window]
|
Figure 5. Voxar 3D image showing segmented 3D colour volume-rendered image demonstrating the arterial supply of a tumour in a horseshow kidney. Image data is courtesy of Christoph Wald, MD, PhD, Lahey Clinic Medical Center.
|
|
Figure 6 is generated from a time-of-flight MRA scan (with gadolinium as the contrast agent) of the brain from a patient suffering from blurred vision. A meningioma is revealed in the right posterior parietal–occipital parasagittal region. Note that the pericallosal artery is markedly enlarged. The normal practice would be to use MIP images to analyse the data. The 3D images, however, provide a far better representation of which vessels lie along the medial, lateral, superior and inferior margins of the mass.
View larger version (118K):
[in this window]
[in a new window]
|
Figure 6. Voxar 3D image showing right posterior parietal–occipital parasagittal meningioma with primary supply by the enlarged pericallosal artery. Image data is courtesy of John J Warner, MD, Marshfield Clinic, USA.
|
|
Interaction with the 3D data sets allows for pre-operative surgical planning. In our own experience with Op3D we have found that about one in four patients planned for hepatobiliary-pancreatic tumour surgery have been inoperable when the 3D data sets have been interrogated by the surgeon, compared with the radiological diagnosis provided from inspection of 2D films. Figures 7 and 8 show a 3D data set reconstructed from 203 slices of 512 x 512 data, giving a voxel size of 0.75 mm x 0.75 mm x 1.25 mm. The data shows a very large calcified endocrine tumour of the body/tail of pancreas closely applied to the splenic vein and a large metastasis in the left lobe of liver extending from the hilum to the hepatic veins.
View larger version (132K):
[in this window]
[in a new window]
|
Figure 7. Transverse clip plan of data set demonstrating endocrine tumour of pancreas with liver metastasis.
|
|
View larger version (146K):
[in this window]
[in a new window]
|
Figure 8. By tilting the clip plane vertically the relationship between the confluence of the superior mesenteric, splenic and portal veins with the tumour is demonstrated.
|
|
Figures 9 and 10 illustrate how small cystic lesions of the pancreas can be interrogated by Op3D in a far more surgeon friendly way than traditional X-ray films. There is the added advantage that this interrogation can be performed live by the surgeon by viewing the images on the wall of theatre during the operation (Figure 3
) to improve the accuracy of the surgery [14].
View larger version (175K):
[in this window]
[in a new window]
|
Figure 9. A vertical clip plane view of a 385 slice reconstruction, giving a voxel size of 0.78 mm x 0.78 mm x 1.0 mm, illustrating the body and tail of the pancreas.
|
|
View larger version (173K):
[in this window]
[in a new window]
|
Figure 10. Moving the clip plane through the data set vertically allows visualization of two small cystic lesions in the tail of the pancreas and their relationship to the splenic vessels.
|
|
|
Conclusion and future expectations
|
|---|
Volume visualization of medical images can bring real added value to a clinician. Applications range from diagnostic use, to pre-operative planning on patient-specific data. The ever-improving algorithms, and increasing availability of hardware solutions, will dramatically increase the use and acceptance of volume rendering as a valuable clinical tool.
In the future, use of 3D visualization and medical virtual environments will certainly expand and also have key roles in emerging areas such as microsurgery and nanotechnology applications. Microrobots and even nanobots designed for use within the human body will need supervision by skilled operators equipped with advanced visualization equipment. There already exist "microsubmarines" powered by an induction motor; at 4 mm long and 650 µm in diameter, it is small enough to pass down a hypodermic needle and has the potential for various diagnostic or therapeutic applications [19]. For example, Figure 11 is a simulation of a microsubmarine navigating through the human cochlear. We envisage that the device will be tracked within the human body and referenced to a patient-specific data set to allow the supervisor to navigate using virtual images.
View larger version (77K):
[in this window]
[in a new window]
|
Figure 11. Microsubmarine navigating inside a semitransparent 3D model of the human cochlea (not to scale). The cochlea is from a surface extraction generated using data courtesy of Prof. Alan Jackson, University of Manchester, UK.
|
|
There can be no doubt as to the potential role of 3D visualization and navigating techniques with medical data sets. To truly benefit from this potential, however, a change in medical opinion will be required and a significant learning curve will have to be overcome if the advances already achieved, never mind those to come, are to be translated into realities in healthcare.
|
Acknowledgments
|
|---|
The authors would like to acknowledge all of the staff at the Manchester Visualization Centre who worked on the projects referenced in this paper, James Perrin, Simone Herrman, Mary McDerby, and many more; and Michelle Dimmock of Voxar who provided access to those Voxar3D images that have been used as illustrations in this paper.
|
References
|
|---|
- Megibow AJ. Three-D offers workflow gains, new diagnostic options. Diagnostic Imaging. November 2002:83–93.
- Lorenson WE, Cline HE. Marching cubes: a high resolution 3D surface reconstruction algorithm. Computer Graphics 1987;21:163–9.
- Wyvill G, McPheeters C, Wyvill B. Data structure for soft objects. The Visual Computer 1986;2:227–34.
- Perrin JS, Lacey A, Laitt R, Jackson A, John NW. A visualization system for the clinical evaluation of cerebral aneurysms from MRA data. Eurographics Short Presentations Proceedings. 1017-4656. 2001;177–83.
- Schreyer AG, Warfield SK. Surface rendering. In: Caramella D, Bartolozzi C, editors. Medical radiology - diagnostic imaging, 3D image processing. Technique and clinical applications. Berlin, Germany: Springer-Verlag GmbH & Co. KG, 2002:31–4.
- Drebin RA, Carpenter L, Hanrahan P. Volume rendering. Computer Graphics 1988;22:65–74.
- Levoy M. Efficient ray tracing of volume data. ACM Trans. Graphics 1990;9:245–61.
- Westover L. Footprint evaluation for volume rendering. Computer Graphics 1991;24:367–76.
- Pfister H, Hardenbergh J, Knittel J, Lauer H, Seiler L. The VolumePro Real-Time Ray-Casting System. Proc of ACM SIGGRAPH, 1999:251–60.
- Cabral B, Cam N, Foran J. Accelerated volume rendering and tomographic reconstruction using texture mapping hardware. Proc. ACM/IEEE Symposium on Volume Visualization. 1995:91–8.
- John NW. Volume rendering. In: Caramella D, Bartolozzi C, editors. Medical radiology - diagnostic imaging, 3D image processing. Technique and clinical applications. Berlin, Germany: Springer-Verlag GmbH & Co. KG, 2002:35–41.
- McCloy RF, Stone R. Virtual reality in surgery. BMJ 2001;323:912–5.[Free Full Text]
- John NW. Basis and principles of virtual reality in medical imaging. In: Caramella D, Bartolozzi C, editors. Medical radiology - diagnostic imaging, 3D image processing. Technique and clinical applications. Berlin, Germany: Springer-Verlag GmbH & Co. KG, 2002:35–41.
- McCloy RF, John NW. Remote visualization of patient data in the operating theatre during hepato-pancreatic surgery. Computer Assisted Radiology and Surgery (CARS). Elsevier, 2003:53–8.
- Neri E, Vagli P, Spinelli C. Virtual endoscopy. In: Caramella D, Bartolozzi C, editors. Medical radiology - diagnostic imaging, 3D image processing. Technique and clinical applications. Berlin, Germany: Springer-Verlag GmbH & Co. KG, 2002:43–53.
- Bartz D. Advanced virtual medicine: techniques and applications for virtual endoscopy (course #52). ACM SIGGRAPH 2002.
- He T, Hong L, Chen D, Liang Z. Reliable path for virtual endoscopy: ensuring complete examination of human organs. IEEE Trans on Visualization and Computer Graphics 2001;7:333–42.[CrossRef]
- Peters TM. Image-guided surgery: from X-rays to virtual reality. Comput Methods Biomech Biomed Engin 2000;4:27–57.[Medline]
- microTEC Online. http://www.microtec-d.com/html/a_homepage/e_start.html (accessed 9 May 2003).
Top
Abstract
Introduction
