Three Dimensional Reconstruction of Human Heart Surface from Single Image-View Under Different Illumination Conditions

The 3D reconstruction from a single-view image is a longstanding issue in computer vision literature, especially in the medical field. Traditional medical imaging techniques that provide information about the heart and which are used to reconstruct the heart model, include Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) images. However, in some cases, they are not available and the applications that use these techniques to model the human heart only produce acceptable results after a long process, which involves acquiring the input data, as well as the segmentation process, the matching process, effort and cost. Therefore, it would be useful to be able to use a 2D single image to reconstruct the 3D heart surface model. We introduce an image-based human heart surface reconstruction from a single image as input. To model the surface of the heart, the proposed method, first, detects and corrects the specular reflection from the heart’s surface, which causes deformation of the surface in the R3. Second, it extrudes the three axes for each image pixel (e.g., x, y and z axes) from the input image, in which the z-axis is calculated using the intensity value. Finally, a 3D reconstruction of the heart surface is created to help the novice cardiac surgeon to reduce the period of time in learning cardiac surgery and to enhance their perception of the operating theatre. The experimental results for images of the heart show the efficiency of the proposed method compared to the existing methods


INTRODUCTION
Reconstruction of the 3D human heart surface is becoming very useful for many clinical procedures, especially in education and training for novice surgeons (Jacobs et al., 2008;Guo et al., 2005;Knox et al., 2005). While 2D images can be acquired from medical scan devices-X-ray, MRI, CT, Ultrasound, they require experts, are time consuming to process and costly. Furthermore, not all hospitals have such devices. In addition, segmentation, calibration and matching, until one obtains a 3D reconstructed model is time consuming and suffers from reliability and consistency issues, which result in an unrealistic model of the human heart.
In this study, we propose a method to build a 3D model of the surface of the human heart to help teach novice cardiac surgeons how to conduct real cardiac operations. A pre-processing step is proposed to correct the artefacts on the surface of the heart that affect the reconstruction process, such as specular reflections, which cause a deformed heart surface model. Removing the specular reflection from the surface of the heart is based on the threshold value and inpainting algorithm. A simple method for the Science Publications AJAS detection and correction of the reflections in the region of interest was proposed by (Schirnibeck et al., 2004), using a sequence of greyscale images to detect specularities. The method detects the coronary artery on the epicardial surface and also identifies the importance of removing the glossiness from the surface of the heart before segmentation of the coronary arteries. This could be time consuming since it needs a dilation operation of the threshold areas and requires time for a fill-in method based on chain coding (Freeman, 1961;Kuhl and Giardina, 1982) for the reconstruction of the underlying image colours.
Reconstruction of the underlying structure by detection of specularities was proposed by (Groger et al., 2001), based on structural tensor interpolation and a diffusion scheme. The method uses a greyscale value as a threshold value to detect the specular reflection and the filling-in information from the neighbouring regions. This method might result in the loss of the underlying image data in the threshold area and can only be applied in small specular regions. A light field method to cover large specular regions was proposed by (Vogt et al., 2002), to remove specularities from endoscopic image sequences. This method uses a sequence of images from different directions to remove the specularities; however, the method consumes much time and effort.
The separation of specular reflection and diffuse components is one of the methods for specular reflection removal using a single image based on colour analysis, as proposed by (Tan and Ikeuchi, 2005). It introduced a specular free image to avoid reflection, which does not require colour segmentation. While this method runs based on two neighbouring pixels to minimize the error of the outcome, it causes the processing time to increase and requires previous information concerning the illumination. It also iteratively compares the intensity logarithmic differentiation with the specular free image for which the chromaticity of the light source must be known. Further, the specular reflection components cannot be separated from the low-saturation colours, such as grey. When applied to images of the heart, this method changes a significant part of the surface colour of the heart.
In terms of robotic laparoscopic images, (Stoyanov and Zhong, 2005) proposed a method for the removal of specular reflection and the temporal characteristics of such images. This method detects the surface of the heart during robotic heart surgery and, at the same time, preserves the underlying image structure. The specular components and diffuse components are separated after recovering the chromaticity from the spatiotemporal volume. Although this method is computationally efficient, the surface shape and image geometry tend to disregard the important information needed to identify the specularities.
A significant amount of research was undertaken into specular detection and correction by linear transformation of the RGB colour space to another colour space (Stehle, 2006) using a YUV (luminance Y, chrominance UV components) colour space with low pass filters. The work presented by (Oha et al., 2007) transforms RGB to the HSV (Hue, Saturation and Value) colour space in order to detect the specularities. It also proposes an efficient method for informative frame classification based on a Discrete Fourier Transform. However, the reflections require the computational cost of the detection methods and the choice of colour space. These studies have the disadvantage of a lack of practicality and applicability.
In the filling method, an iterative process has been used to replace the specular pixel with the information from the average of the non-specular neighbouring pixels (Greenspan et al., 2009). This method also produces a good result for small specular regions. However, for the surface of the heart, the specular regions are large. Furthermore, as the reflections are sometimes found in two different regions, the replaced pixels are blended together in these two regions and the result is not clear.
More recently, (Arnold et al., 2010) proposed an efficient method, which directly uses the RGB colour space for specular reflection segmentation based on median filtering and correction of the specularities based on an inpainting algorithm, which has given visually good results. This method proposes two modules to detect the different characteristics of specular reflections. In the first step, a very high intensity pixel with a greyscale value as the threshold on the green and blue colour channel are determined. The second step detects the small intense parts of the specular reflection in the image using another threshold from the first step as a lower threshold and a modified median filter; subsequently, it uses a convolution operation in the post-processing step in the region close to the camera. The inpainting for the reflection pixels is done by manually setting a specific distance from the centre of the specular region and Science Publications AJAS replacing the specular pixel by the colour of the centred pixel for small and uniform regions. However, the results appear to be strongly blurred when applied to large regions. Furthermore, it involves many computations, requires assigning several parameter values manually to detect the specular reflection and needs a large amount of effort to find the optimal threshold values.
These specularities affect the 3D reconstruction process. Recent medical 3D image reconstruction methods require a set of 2D slices to create a 3D model, which can be recorded using image acquisition techniques, such as X-ray, MRI, CT and ultrasound. The reconstruction process of the heart from multiple image sequence slices obtained from such devices needs considerable effort and is time consuming. Several works have reconstructed cardiac organs. Peifer et al. (1990) reported a method, which superimposes 3D arteries from X-ray angiograms on an artificially modelled ventricle. Halmann et al. (1989) used X-ray angiograms for reconstruction of the coronary artery and MRI volume images for that of the ventricle, while others used MRI. Nam et al. (2013) proposed a method based on the parallelized implementation of an iterative compressed sensing reconstruction method for 3D radial acquisitions using a commercial graphics processing unit using B-spline or bi-cubic Hermite surface patches and piecewise polynomial surfaces for surface reconstruction.These methods investigate surface reconstruction from multiple cross sections (Kuwahara and Eiho, 1988) or projections (Young et al., 1989). However, the user manually sets the landmarks for each slice and the geometry of the shape must be known for surface reconstruction.
Several methods have reported the feasibility and usefulness of rapid prototyping technology to understand the anatomy and pathology of congenital heart anomalies (Jacobs et al., 2008;Knox et al., 2005). However, most of the reconstructed models have been made of solid and rigid resin, using a mould, which requires an experienced mould maker and is a timeconsuming and expensive process. In addition, the result is not suitable to understand surgical simulation. Creswell et al. (1992) reconstructed 3D finite element models from the boundary representations of the endocardial and epicardial surfaces, which manually set the contour points of the boundaries.
In other 3D shape reconstruction, Deguchi et al. (1996) proposed the normalization process of the factorization method, which produces rough sketches of global 3D shapes of human internal organs derived from endoscope image sequences. Recently, using Shape From Shading (SFS) with a single image-view, Kimmel et al. (1995) applied SFS to the conventional case of a distant light source and orthographic projection. To overcome the distant light problem, Oliensis and Dupuis (1993) proposed a method based on the Markov Chain approximation and assumed differentiable depth to obtain the depth information with no prior information about the surface, however, it cannot deal with non-smooth surfaces. Furthermore, Okatani and Deguchi (1997) proposed a method to reconstruct a shape from its shaded image in the case where a point light source is near and at the projection centre; however, they do not tackle topological ambiguities and no fitting can be applied to the final result due to all surface points producing the same angle. In this study we develop a learning system environment via augmented reality, which will allow novice surgeons to reduce the period of time required to learn cardiac surgery and to enhance their perception of the operating field, as shown in Fig. 1. This study deals with image acquisition and specular reflection detection and correction to get a clear 3D reconstructed human heart surface model. The detection and correction of specular reflections is considered to be a significant part of these methods, which cause a distortion of the depth reconstruction of the heart surface. A method for detection and correction of the specularities in the human heart surface is proposed in this article. To illustrate the problem, Fig. 2 shows an example of an image of a real human heart with specularities taken by a digital camera during a cardiac surgery.
In this article, a novel and automated method is proposed to reconstruct the human heart 3D surface model, which detect and correct the specular reflection from the surface of a human heart from a single image view. The detection is based on the difference between the standard deviation of the RGB colour values and the blue colour value at each pixel position and the correction of the specular is done by the average of the L-shape inverse (Г) neighbouring pixels.
This article presents the detection of the specular region of interest, then outlines the process how to correct the specularities, followed by extractions of the 3D axes values and the 3D reconstruction process. The Experimental results and evaluation are discussed also.

Human Heart Images
Several images were taken during a cardiac surgery from the operating theatre in Universiti Kebangsaan Malaysia Medical Center (UKMMC). The images are acquired by digital camera Sony cyber-shot DSC-T30 digital camera, with focal length between 6.33-19.0mm, resolution 7.2 megapixels and 3x optical zoom with aperture range f3.5-10f.

Detection of Specular Reflection
Operating theatre light sources emit a huge amount of light rays that reflect from the surface of the heart as it is a moist surface. This results in large and bright regions across different parts of the heart surface.
Most of the recent methods depend on at least two or three threshold values to detect the specular reflection from the moist surfaces. These threshold values are difficult to determine and are manually chosen in most methods. The proposed method provides an efficient way to detect the specularities based on the greyscale value as in (Arnold et al., 2010). Using a single threshold value, which is found after a preliminary testing of the method using a different RGB colour channel parameters on the human heart images from different patients, a threshold value has been set. The following condition will be satisfied if a pixel is within a specular region Equation 1: where, gs is the greyscale value of the pixel p, maxB is the maximum value of the blue colour channel and SD is the standard deviation of the RGB colour channel values. We carried out several experiments on the image RGB channel parameters with a large number of heart images taken by digital camera during cardiac surgery. As a result, we found that the threshold value depends on the blue colour channel and standard deviation values to identify pixels if they are within the specular reflection region or otherwise. In addition, the intensity of the red and green colour channels is relatively larger than that of the blue colour. Especially in specular regions, the red channel has a very high intensity and the green channel also has a high intensity, while the blue channel has a very low intensity. The result of specular reflection detection of the heart is shown in Fig. 3. Several experiments were conducted under different illumination conditions, such as images captured when the operating theatre room light, the ceiling lamps and the surgeon's headlight all hit the surface of the heart at the same time. As we can see in Fig. 4a adjacent to the surface of the heart is a small circle of light from the surgeon's headlight and a larger area of light from the operating theatre lights.
The proposed method is robust even in a region of large specular reflection. Figure 5 illustrates the results of the detection of specularities from the worst case reflections appearing on the surface of the heart. Figure 6 shows that the different lighting sources in the operation theatre emit a huge amount of lighting rays that are reflected from the surface of the heart, which causes a loss of the original visual information in the specular regions.

Correction of Specular Reflections
Currently, the method used to correct specular reflection is based on the nearest pixels neighbouring the specular pixel, such as an iterative process to replace the specularities by the average of the nonspecular surrounding pixels (Greenspan et al., 2009), or, more recently, correction by the centroid colour of the pixels within a specific distance using filters (Arnold et al., 2010). However, these methods have the disadvantage of being computationally costly and lack practicality and applicability.
Extensive experiments have been undertaken using different window sizes for the neighbouring pixels, such as 3×3, 5×5, 7×7 and 9×9. The results from these experiments are the replacement of the specular reflection with blurred regions. In addition, the time consumption for computation increases as the size of the window increases. The results of the different mean filter windows are shown in Fig. 7.
Extensive experiments were carried out to select the best surrounding pixels for the correction process, such as one pixel from the nearest pixels to the specular pixel to correct the reflection. This is a not an efficient method when it comes to deciding which pixel to choose. Even to select two or more pixels from the surrounding area of the specular pixel is a tedious process to find the best pixels to correct the reflection. In this matter, we propose a novel correction method, which is called the L-inverse shape (Г) method using only the nearest four neighbouring pixels to the specular reflection pixel, as shown in Fig. 8. The method starts by examining pixel by pixel from the top-left corner pixel 1 of the neighbour pixels and moves vertically down the left-hand column through pixel 2 to reach the bottom pixel 3, after which the algorithm then considers the neighbouring pixel 4 above the pixel under examination. This repeats until the full image of the heart has been scanned. If a specular pixel is found, it will be corrected by the average of the Г neighbour pixels, since these Г pixels have been previously visited and are not specular pixels. The row processing is the same as the column processing, provided that the pixels have been previously visited and are not specular pixels, the only difference is that the four surrounding pixels that we proposed Г shape will look like as rotated L-shape (Г). The proposed method has the advantage of simplicity and reduces the time required because the processes are carried out with only four pixels and it is a non-iterative process.

AJAS
In addition, it requires no prior information on the condition of the illumination. Furthermore, it improves the efficiency of the correction process and the computational cost. The results of the Г method are shown in Fig. 9.

Extraction of Three Dimensional Axes
The 3D surface models are usually built by projection of all the pixel coordinates x, y and z axes into the R 3 domain. Much of the extraction process for these axes in the 3D reconstruction of the medical images is done using image sequences/slices, in which several works extracted these axes from multiple image sequences, such as Chiang et al. (2012). Other researchers, such as Amini et al. (2001), reduced the amount of input data with B-spline surface representation. Single image reconstruction from a medical image was proposed by Okatani and Deguchi (1997) based on the shape from shading, however, it only works with endoscopic images and only for small regions of the image. Recently, 3D reconstruction from a single image based on the intensity value of the pixels was proposed by Al-Surmi (2008) which we apply in this study using human heart surface images.
The extraction process for the axes that represent the x, y and z values, which are needed to visualize the 3D surface model, is done by extracting the positions of the image pixels as x, y axes and intensity pixel value as the z-axis. The values of these three axes are saved to a text file and the results show the heart surface using C++ with OpenGL library, as shown in Fig. 10, in which we can resize the result and rotate it through every degree using freeform rotation.

3D Heart Surface Reconstruction
To our knowledge, single image reconstruction to build a 3D human heart surface from a real heart image has not been previously reported. A surface model of the heart of interest can be easily viewed from any angle. However, specular reflection affects the 3D reconstruction process, which causes deformation of the reconstructed heart surface. It is obvious that the occurrence of specular reflections causes protrusions in the final result, which are removed by using the proposed method. A result of the 3D reconstructed surface from different viewpoints is shown in Fig. 11. . 11. Different views of the 3D heart surface reconstruction (a) Original heart images with specular reflection (b) Proposed method (c) Inpainting algorithm (Arnold et al., 2010)

RESULTS AND DISCUSSION
The proposed method has already been implemented and tested in more than 100 data sets. In fact, the detection and the correction of the specular reflection have been continuously tested on more than 70 data sets for qualitative evaluations to monitor improvements in the method. In addition to these evaluations, quantitative methods are also applied to determine the common performance metrics, such as the specificity, accuracy, precision and sensitivity (Han et al., 2011) of the method. We currently have ground truth for 30 data sets. The quantitative test results from these data sets are given in The evaluation Table 1, started by manually creating a set of ground truths by labelling and correcting the specular pixels in a set of images of a human heart from different patients. Then the performance of the proposed method was compared with the inpainting algorithm proposed by (Arnold et al., 2010) and the eight neighbour pixels method by (Muthukumar et al., 2010). For each image, we optimized the parameters of the (Arnold et al., 2010) method and the eight neighbour pixels method (Muthukumar et al., 2010), while the proposed method was automatically optimized.
From the results reported in Table 2, it can be seen that the accuracy of the proposed method has improved on the other methods presented in (Arnold et al., 2010;Muthukumar et al., 2010). Moreover, the cost reduction of the False Positives (FP) and False Negatives (FN) for the proposed method compared to (Arnold et al., 2010) and (Muthukumar et al., 2010) was 76.26% and 76.16%, respectively. Furthermore, the proposed method can process the image faster due to the need for only four neighbouring pixels in the correction process. The results are visually shown in Fig. 12, which compares the results of the three methods in the form of the original image, the results from the proposed Г method, the results from the inpainting method (Arnold et al., 2010) and the results from the eight neighbour pixel method (Muthukumar et al., 2010), respectively. It can be seen that the results of the proposed method are the best compared to the other methods, while the inpainting method has a better result in the dark regions. However, the large specular reflection region was not accurately corrected. Furthermore, if the specular region appeared in two different colour regions, the specular pixels were replaced with a combination of a different colour. Figure 13 shows mask maps before and after specular reflection correction. The original images are shown in column (a). Column (b) shows the intensity masks of the original images while column (c) shows the corrected image. The masks in the corrected imag are in column (d). As seen in Fig. 13, most of the masks that arise from specular reflection are actually removed by the correction process.   (Arnold et al., 2010) and to the eight neighbour pixels method in (Muthukumar et al., 2010). The proposed method achieved a cost reduction 76.16% compared to the method in (Muthukumar et al., 2010

CONCLUSION
In this article, we have presented a new automatic detection method for specularities. The method of the heart surface reconstruction is sensitive to specular reflections. To overcome this problem, we developed a method based on a blue colour channel combined with standard deviation and a new specular reflection correction method. Using a single image of a human heart, the reflection correction process replaces the lost information by four local Г surrounding pixels without any iteration process. The results obtained from the images tested show a significant improvement and robustness in the detection and correction process, which has a better performance and a higher accuracy. Moreover, we have shown the effect of the specular reflection on 3D reconstructions. The method can also process high-resolution images.

ACKNOWLEDGEMENT
We would like to thank the Universiti Kebangsaan, Malaysia Medical Centre, for the data and support of the work. We would especially like to thank Mirko Arnold, School of Computer Science and Statistics, Trinity College Dublin, Ireland, for help in the evaluation of the algorithm performance.